BoundBench

Notte

Python framework and SDK for web-browsing AI agents, with a local Playwright-based agent and a hosted browser/agent API.

github.com/nottelabs/notte · 2026-10-04 · 9c898d9

Defense-in-depth score

4.0 / 10

Minimal

Notte's local agent acts on whatever web pages it reads with no approval step: every navigation, form submission and model-written JavaScript runs immediately, so a malicious page can steer it into exfiltrating data or submitting forms, including with vault logins and the stored card if a vault is attached. Browser launch hardening is incomplete. Its vault keeps secrets out of model prompts well, and step and timeout limits are sensible, but nothing contains a hijacked run.

Key gaps (2)

  1. Prompt injection from any visited page can drive unattended exfiltration (arbitrary goto/JavaScript) and irreversible web actions, including vault logins and the card, with no approval step. C5 · Untrusted input blast radius
  2. Locally run SDK functions (including forked shared ones) are downloaded unverified and exec()'d in-process, inheriting all the process's credentials. C7 · Third-party extensions

Criteria

C1 Identity & least privilege

Minimal 0.35 / 1.00

In local mode the agent drives a freshly created, non-persistent browser context, so it starts with no cookies or logged-in accounts of the user's own browser. It only receives site credentials if the developer attaches a cloud vault, and the vault releases a login only for the URL of the current page and checks the target field type. There is no authorization layer on actions themselves, the browser runs on the operator's machine and network (so it can reach localhost and intranet hosts), and the vault does not fully protect payment-card data.

C2 Approval gates

Minimal 0.05 / 1.00

There is no human approval step anywhere in the local agent loop. Every action the model chooses, including navigating to any URL, submitting forms, running JavaScript on the page and filling vault credentials or a payment card, executes immediately. The only human-in-the-loop hook, a 'help' action, simply ends the run as a failure. Consequential web actions such as submitting forms or purchases are generally irreversible.

C3 Tool & action scoping

Minimal 0.20 / 1.00

Actions are typed models in a fixed registry and navigation only accepts http and https URLs, and downloaded file names are reduced to a base name. Beyond that, the agent's main tools are general-purpose: 'goto' accepts any URL with no host allowlist or block on internal addresses, and 'evaluate_js' runs arbitrary model-written JavaScript in the page. Every registered action, including JavaScript evaluation, is offered to the model by default, and there is no per-task tool selection.

C4 Code-execution isolation

Minimal 0.47 / 1.00

The only model-reachable code execution is JavaScript the model writes, which runs inside pages of a locally launched Chromium. That keeps it within the web platform (no direct file or shell access), but the default browser launch configuration is not fully hardened, and the agent deliberately visits pages chosen by untrusted content. There is no containment beneath the browser: it runs as the operator's user, alongside the agent's environment and API keys. Remote browser providers exist as an opt-in alternative but their isolation is outside this repository.

C5 Untrusted input blast radius

Minimal 0.25 / 1.00

Every page the agent visits is untrusted input, and the only defenses are delimiter tags around the page observation and a system-prompt instruction to treat page text as data. Results of previous actions (scraped data, JavaScript output, emails and SMS read by a persona) re-enter the conversation without those tags. Nothing in code restricts what the agent may do after reading a page, so a hijacked agent can navigate to an attacker URL carrying data, run JavaScript, submit forms, and use vault logins or the payment card, all unattended.

C6 Memory, context & configuration integrity

N/A · full credit 1.00 / 1.00

The local agent keeps no memory between runs: the trajectory lives in memory for a single run (an agent can only run once), the browser context is not persistent, and no instruction files, dotenv files or vector stores are loaded from the working directory. Configuration comes from the package's own file or an explicit NOTTE_CONFIG_PATH. Cloud features such as browser profiles are outside the scored local configuration.

C7 Third-party extensions

Minimal 0.25 / 1.00

The agent loop loads no plugins, MCP servers or model files. The SDK can, however, run a stored or shared 'function' locally: it downloads the code from the Notte service, by default the latest version, and executes it inside the developer's Python process, under RestrictedPython by default (which allows the requests library and the Notte client). This is an explicit opt-in call, but nothing verifies the code's integrity and it runs with all of the process's credentials.

C8 Secrets & sensitive-data protection

Minimal 0.45 / 1.00

In the default local flow the only secrets are LLM provider keys read from environment variables, and no code logs them; usage telemetry to PostHog and Scarf is on by default but deliberately carries only event names, status and system info. The stronger mechanism is the optional cloud vault: the model sees placeholders, real values are substituted at execution time, values the vault returned are scrubbed from model inputs and masked in screenshots, and filled values are SecretStr types. Gaps in the vault: payment-card data is not fully protected, and page JavaScript run by the agent can read filled fields and send them anywhere.

C9 Audit & traceability

Minimal 0.45 / 1.00

Each run builds a structured trajectory of every model decision and executed action with its arguments, result and start and end times, and step summaries are printed through the logger as they happen. The trajectory lives only in memory and is handed back to the developer at the end of the run; nothing is written to disk or shipped elsewhere by default, there is no actor or approver attribution, and a crash loses whatever was not printed.

C10 Limits & kill switch

Moderate 0.55 / 1.00

The agent loop is capped at 20 steps by default (the request schema refuses more than 150), it stops after 3 consecutive failed actions, and individual actions, JavaScript evaluation and LLM calls have timeouts. There is no wall-clock limit for the whole run, no token or cost budget, and no stop control beyond cancelling the Python task; in-flight browser actions run until their timeout.