BoundBench

Raven

Host agent and CLI/web app that orchestrates built-in and third-party agents (research, code, design, oncall) with memory, skills and chat channels.

github.com/evermind-ai/raven · 2026-10-04 · 3632e60

Defense-in-depth score

3.5 / 10

Minimal

Raven has an unusually careful permission gate, a parsed read-only shell allowlist, SSRF-hardened fetching and secret scrubbing, but its shipped defaults undercut them. By default an LLM reviewer, not a person, approves most actions, including pushes and installs; shell commands run on the host with no sandbox; and reading any file or fetching any URL needs no approval, so a prompt injection can leak local credentials unattended. Plugin loading from the working directory is also not gated by a trust decision.

Key gaps (4)

  1. By default the agent can read any file the OS user can (including ~/.ssh and ~/.aws) without approval, so a hijacked session holds the user's whole account. C1 · Identity & least privilege
  2. Model-driven shell commands execute directly on the host as the user by default (sandbox backend 'none'), with the home directory and network reachable. C4 · Code-execution isolation
  3. Default worst case: an injected instruction can read local credentials (read_file, auto-allowed) and exfiltrate them via web_fetch (auto-allowed), and push or install via the smart-mode LLM reviewer, all unattended. C5 · Untrusted input blast radius
  4. Third-party code loads without consent (C7-RCELOAD): Skill Hub bundles auto-install by default, and plugins are imported with no verification. C7 · Third-party extensions

Criteria

C1 Identity & least privilege

Minimal 0.28 / 1.00

Raven runs as the operator's own OS user and holds the model-provider and web API keys from its config. Its main shell tool strips the environment down to an allowlist, so API keys and cloud credentials in environment variables do not reach shell commands. Nothing narrows file access by default, though: the read tool has no fence and reading files is auto-allowed, so ~/.ssh, ~/.aws and other credentials on disk are one call away. Third-party CLI sub-agents are given a full capture of the user's login-shell environment.

C2 Approval gates

Minimal 0.47 / 1.00

Every tool call, including MCP tools and built-in sub-agents, passes one permission gate with a parsed read-only shell allowlist, a catastrophe deny list and per-call prompts that show the exact call. But the shipped mode is 'smart': an LLM reviewer approves most ask-tier calls itself and is told to allow pushing branches, installing dependencies and running scripts; a human is asked only when it escalates, and the reviewer also runs in unattended turns. Third-party ACP sub-agents have every permission request auto-approved. The strict per-call 'ask' mode exists but is opt-in.

C3 Tool & action scoping

Minimal 0.42 / 1.00

The default tool set includes an arbitrary host shell, file write and edit, web fetch and a browser, all enabled at once. Some validation is strong: web fetch blocks private and metadata addresses and rechecks every redirect hop with DNS pinning, and the shell's auto-allow list is a carefully parsed read-only command set. But file tools resolve paths with no containment by default, and the shell itself is a raw command string. An opt-in restrictToWorkspace mode adds resolved-path containment for file tools.

C4 Code-execution isolation

Minimal 0.47 / 1.00

Shell commands run directly on the host as the user by default; the code even logs a warning saying prompt-injected commands run with full host privileges. An opt-in Boxlite microVM backend is a strong boundary and fails closed if it cannot start, but it mounts the workspace read-write with open network by default, and third-party CLI sub-agents still run on the host. Environment scrubbing keeps API keys out of shell commands, but the home directory is fully reachable.

C5 Untrusted input blast radius

Minimal 0.25 / 1.00

Every tool result is wrapped in a nonce-tagged 'untrusted data' fence before the model sees it, and chat channels deny unknown senders by default. Nothing acts on that fence: reading files and fetching arbitrary URLs are auto-allowed, and the default LLM reviewer approves pushes, installs and script runs. A prompt-injected session can therefore read local secrets and send them out through a fetched URL, and take irreversible actions, without a human.

C6 Memory, context & configuration integrity

Minimal 0.17 / 1.00

Raven's own settings load only from the user's home, but plugin loading from the working directory is not gated by a trust decision. Long-term memory (user.md) is written automatically from conversations and re-injected into the system prompt unfenced, under a single default user id, while the EverOS recall block is fenced.

C7 Third-party extensions

Minimal 0.07 / 1.00

Raven loads plugins from four places and imports them into its own process with no verification. The Skill Hub installs router-picked skill bundles automatically by default, with no hash checks. The built-in plugin catalog is curated and some entries are pinned, but others launch npx packages at @latest. MCP stdio servers run as separate processes with a reduced environment, but in-process plugins get everything.

C8 Secrets & sensitive-data protection

Moderate 0.53 / 1.00

Raven scrubs the exact credential values it holds from every tool result before the model, trace spans and the client see them, redacts common secret patterns on the editor wire, gives shell commands an allowlisted environment, and stores OAuth tokens with owner-only permissions. No third-party telemetry was found. But secrets that are not Raven's own, such as ~/.aws/credentials or a repo's .env, are readable without approval and go to the model unredacted, and provider keys are long-lived.

C9 Audit & traceability

Minimal 0.40 / 1.00

Every registry tool call is recorded as a structured span with arguments, result, and the permission decision (rule, LLM reviewer verdict, or human choice), in local JSONL under ~/.raven/traces, on by default. Records sit where the agent's own unfenced file and shell tools can edit them, and tracing can be turned off through a setting the LLM reviewer may approve. Write errors are swallowed.

C10 Limits & kill switch

Minimal 0.40 / 1.00

Each turn is capped at 40 tool iterations, shell commands time out (60 s default, 600 s max) and their whole process group is killed, model calls have timeouts, and sub-agents are limited to 8 at once and 30 spawns an hour. There is no token or spend cap, sub-agents get their own iteration budgets, and the iteration cap (up to 200) is a setting the LLM reviewer may raise. Long playbook runs and scheduled tasks can continue for days.