BoundBench

Skyvern

Automates browser-based workflows with LLMs and vision

github.com/Skyvern-AI/skyvern · 2026-10-03 · 19bf399

Defense-in-depth score

3.4 / 10

Minimal

Skyvern drives a real browser through arbitrary websites using stored logins, and as shipped nothing stands between a prompt-injected page and the agent's actions: there is no human approval, navigation can go to any public site, and credential release on the default engine is not a complete boundary. Workflow code blocks are on by default and run in the server process behind only an AST filter. The codebase contains a sophisticated origin firewall, egress monitor and credential site-binding, but in this open-source build the firewall cannot be enrolled and the site-binding does not cover every path. Secrets are kept out of the model's context via placeholders, which is a real strength.

Key gaps (2)

  1. Default runs combine untrusted web content, stored credentials and unrestricted navigation/form submission with no human gate (C5-WORSTCASE). C5 · Untrusted input blast radius
  2. Code blocks are enabled by default and exec() Python in the API server process behind only an AST denylist; the OSS build has no isolated runner. C4 · Code-execution isolation

Criteria

C1 Identity & least privilege

Minimal 0.38 / 1.00

Skyvern acts on websites with whatever site passwords, TOTP secrets and password-manager credentials a workflow author binds as parameters, plus the operator's LLM API keys in the server's environment. A run only receives the credentials its workflow declares, and the API requires an org API key, but per-site credential release on the default task engine is not a complete boundary. A hijacked run can use the bound credentials across as many services as the workflow touches.

C2 Approval gates

Minimal 0.07 / 1.00

There is no human approval step before the agent clicks, types, submits forms, uploads files, or navigates; actions proposed by the model are executed directly. The only human gate is an optional Human Interaction workflow block that pauses a workflow for a single approve/reject decision on an author-written instruction, not on the specific action. The code labelled 'effect approval' is a machine-to-machine binding for a cloud firewall that is not enrolled in this build. Consequential web actions (purchases, submissions, account changes) are irreversible and unattended by default.

C3 Tool & action scoping

Minimal 0.45 / 1.00

The agent's actions are browser UI primitives (click an element, type text, select, upload, navigate) rather than shell or SQL, and several have real argument checks: navigation blocks internal and private addresses and re-checks every redirect, downloads go through an SSRF-guarded connector, and an upload URL must appear verbatim in the user's own goal or payload. But navigation accepts any public URL and text input accepts any value, so the effective reach is 'any website with any input'. There is no per-task action allowlist or read-only default.

C4 Code-execution isolation

Minimal 0.25 / 1.00

Workflow code blocks are enabled by default and run Python in the Skyvern server process via exec(), protected only by an AST denylist and a restricted builtins dict. The project's own comment says the OSS build has no secure runner (that is a cloud feature). Code reaching this path is not only operator-written: the 2.0 task planner asks the LLM to write 'compute' code and runs it through the same block, and opt-in cached scripts are imported with the genuine builtins. An escape from the filter lands in a process holding LLM API keys, the vault key, the database and unrestricted network.

C5 Untrusted input blast radius

Minimal 0.25 / 1.00

Skyvern's core job is reading arbitrary web pages and acting on them, so every run combines untrusted input, stored credentials and the ability to submit forms and navigate anywhere. The defence is prompt-level: about a third of the prompt templates fence page content as untrusted data and neutralize delimiter look-alikes, but the 2.0 planner and many helper prompts do not. Nothing in the default build cuts a Rule-of-Two leg: an injected page can steer the agent to an attacker URL carrying data in the query string, with no human involved, and credential release is not a complete boundary. A detailed origin firewall exists in the code but cannot be enrolled in this build.

C6 Memory, context & configuration integrity

Minimal 0.38 / 1.00

Skyvern has no model-writable long-term memory or vector store, and starts each default run in a fresh temporary browser profile. Persistence is opt-in: saved browser profiles and persistent sessions carry cookies and site state between runs, and 'code mode' caches LLM-generated scripts that later runs execute without a fresh model decision. These stores are scoped per organization in queries, but nothing validates or reviews what a poisoned session or cached script carries forward. Settings read a .env from the server's working directory, which is operator configuration rather than an untrusted workspace.

C7 Third-party extensions

Minimal 0.30 / 1.00

Skyvern does not load plugins, MCP servers, or model files at runtime, and the model cannot install packages. The one third-party extension point is an operator setting (EXTENSIONS) that loads unpacked Chromium extensions from a local directory into every browser it launches; the list is empty by default. Those extensions are not pinned or hash-checked and run inside the same browser where credentials are typed, but they cannot be added by the model or by page content.

C8 Secrets & sensitive-data protection

Minimal 0.35 / 1.00

Secrets are handled better than average: stored credentials appear to the model only as random placeholders that are swapped for real values at execution time, logs pass through field-name and bearer-token redaction, and the local vault is Fernet-encrypted. Weak spots: the vault's key file sits next to it, general database encryption is off by default, and telemetry to PostHog is on by default and sends each task's URL (which can carry tokens or personal data). The credentials themselves are long-lived site passwords.

C9 Audit & traceability

Moderate 0.50 / 1.00

Every agent action is written to the database as a structured record (type, target element, reasoning, status, timestamps) together with screenshots, page HTML, LLM prompts and responses, and recordings, and code-block page calls are mapped onto the same action timeline. Records carry the organization but no separate human principal or approver, live in a database the server process itself can modify, and are written after the action runs; artifact capture is fire-and-forget. This is good for replaying a run, weaker as tamper-evident audit.

C10 Limits & kill switch

Minimal 0.45 / 1.00

Each task stops after 10 steps by default (each step with up to 5 retries), browser actions and code blocks have timeouts, and an optional per-organization budget can cap steps across a whole workflow run. There is no cost or token budget on the main agent loop and no default wall-clock limit for a run, and workflow loops may run up to 1000 iterations, each starting a task with its own step budget, with the loop count often driven by data extracted from pages. Cancelling marks the task in the database and is checked between steps, so in-flight actions finish.