Browser automation & QA (Browserbase)

Nova agents can drive a real remote browser to fill forms, click through flows, read rendered pages, and screenshot results — then have a vision model judge what actually happened. This is the same engine behind #nova-directories submissions, but the capability is generic: any agent (for example a future nova-qa that you @nova to "go verify X on the live site") can use it by being granted the browser_* tools.

The browser runs on Browserbase (clean residential-ish IPs, stealth, session replay, in-session captcha solving), driven through Stagehand (an LLM observe → act loop over Playwright). Implementation: apps/admin/src/lib/directories/submission/stagehand-client.ts; the agent-facing tools live under the agent_runtime provider (integrations/providers/agent-runtime/actions/browser.ts).

The browser_* tools

A run that has these in its tool list can drive a browser end-to-end:

Sessions are keep-alive with a 30-minute timeout because a real task is a multi-call agentic loop (navigate → observe → act ×N → screenshot) with LLM latency between each stateless tool call.

Captcha solving needs a residential proxy (the main lesson)

Browserbase auto-solves reCAPTCHA v2 / hCaptcha / Turnstile in the background when the session is created with solveCaptchas: true. It signals progress with two page console events — browserbase-solving-started and browserbase-solving-finished (roughly 5 to 30 seconds), which we log so you can confirm it fired.

But solveCaptchas alone is not enough. reCAPTCHA v2 is IP-reputation- sensitive: from a raw Browserbase datacenter IP the solver gets no solvable challenge (zero solving-started events) and any token validates server-side as "invalid". The fix is to pair it with a residential proxy:

browserSettings: {
  viewport: { width: 1288, height: 900 },
  solveCaptchas: true,
  proxies: true,   // ← Browserbase built-in US residential proxy (required for the solver)
}

Rules for the agent (already encoded in the directories skill — reuse for any browser agent that hits a captcha):

• Never click / scroll / browser_act on the captcha widget or its iframe — reCAPTCHA iframes are cross-origin (you'll get "unable to obtain a content frame") and interacting races Browserbase's solver.
• Never reload or re-navigate to "get a fresh captcha" — that resets the solve.
• Wait for browserbase-solving-finished (or re-observe/screenshot a moment later); typically the checkbox clears in 15 to 30 seconds, then proceed to submit.

Cheaper alternative (future): a BrightData external proxy — proxies: [{ type: "external", server, username, password }]. See the infra cost benchmark and the 2026-06-14-directories-browser-infra-benchmark.md design note for the GCE + BrightData + 2Captcha path that trades clean IPs for lower per-submission cost. Geolocation knob if a site needs a region: proxies: [{ type: "browserbase", geolocation: { city: "NEW_YORK", state: "NY", country: "US" } }].

Live view — watch the browser in real time

Pass live_view_channel + live_view_thread_ts to browser_session_start and the session auto-posts a 🖥️ live-view link to the thread (the Browserbase devtools inspector URL) so a human can watch the mouse/clicks as the agent works. This is channel-agnostic — it routes through postThreadReply, so it works whether the run was triggered from Slack or from the web (/<threadId>) or elsewhere.

Session reuse across an approval pause (HITL)

If a flow pauses for human approval mid-task (for example a "Submit this? (yes/no)" gate), the resumed turn is a fresh conversation turn: the prior tool-call results are NOT in context, so the agent doesn't "remember" the sessionId. The pattern:

1. Right after browser_session_start, persist the sessionId somewhere durable (a record row, run metadata).
2. On resume, read it back and reuse that session — it's still alive (keep-alive, 30-min) with the page state intact (form still filled).
3. Only start a fresh session if the saved one is missing or errors as expired/closed — and then re-do the page setup before continuing.

Starting a new session on resume lands you on a blank page and loses all state. This is the most common mistake here.

Verifying with vision (don't trust page text alone)

Page text can lie (a "success" banner that's actually a stale toast, or an error hidden below the fold). Corroborate with a screenshot + a vision model via the connected Vertex integration:

vertex_generate_content({
  model: "gemini-2.5-flash",
  images: [screenshotUrl],
  prompt: "Judging ONLY by what's visible, did <action> actually complete? " +
          "Reply with one of: success | pending | needs-verification | failed | unclear — then one sentence of evidence.",
})

If the vision verdict contradicts the page text, trust the screenshot and report the safer outcome. (Browserbase records rrweb, not MP4 — so the analyzable artifact is the screenshot, not a video.)

Pointing this at QA / verification (nova-qa)

To let an agent QA the live product (@nova verify the signup flow still works, @nova check the pricing page renders on mobile), give that agent the browser capability. Per the agent-tools rule, a new capability on a flow @mention agent must be granted in three places (updating only one silently fails):

1. agent.tools — add the browser_* tools (+ vertex_generate_content for the vision check) so they resolve at all.
2. The flow node's data.allowedTools — node allowlists only narrow agent.tools, never add; include the browser tools there too.
3. The node's data.template — the per-node prompt must permit the QA/browse behavior (a tightly-scoped persona will otherwise refuse an off-script "go browse" request).

Then the QA logic itself belongs in a skill (agents read skills) — a browser-qa skill describing the verification playbook (what to navigate, what "healthy" looks like, when to screenshot, how to phrase the Gemini check), kept thin in the prompt (use_skill('browser-qa')). This mirrors how the directories submit agent works: a fat skill + the generic browser tools.

Cost & escalation

Browserbase is the reliable default (clean IPs, stealth, replay, built-in captcha solving) but bills per session-minute + proxy GB, which adds up at scale. The tracked optimization is to move "easy" form-only targets to a cheaper GCE + headless Chromium + BrightData + 2Captcha path and keep Browserbase as the Tier-2 escalation for anything that path fails (Cloudflare / Turnstile / IP block). See the infra benchmark note for the decision rule.

Related

• Sandboxes — the broader remote-compute model.
• Agents — the three-place tool-grant rule.
• Integrations — Browserbase, Vertex, 2Captcha.