feat: add Yutori Navigator n1.5 as a computer-use (CUA) agent provider

[lawrencechen98]

Draft — opening early for visibility/feedback. Happy to adjust scope or split (see "Notes for reviewers").

Summary

Adds Yutori Navigator n1.5 as a computer-use agent provider, alongside the existing OpenAI / Anthropic / Google / Microsoft CUA clients. Navigator is a computer-use model (screenshot in, coordinate-based tool_calls out in a normalized 1000×1000 space) served via an OpenAI-compatible Chat Completions API at https://api.yutori.com/v1. Because it's OpenAI-compatible, this reuses Stagehand's existing openai dependency and the provider-agnostic V3CuaAgentHandler — no new dependencies, no handler-shape changes for other providers.

const agent = stagehand.agent({ mode: "cua", model: "yutori/n1.5-latest" });
await agent.execute({ instruction: "...", maxSteps: 30 });

Auth via YUTORI_API_KEY (or clientOptions.apiKey / baseURL). Ships the core tool set (browser_tools_core-20260403).

What's included

• YutoriCUAClient — screenshot-per-turn loop; tool_call → AgentAction with 1000×1000 → viewport coordinate denormalization; role:"tool" results with a Current URL: suffix; request payload trimming (keep recent screenshots under ~9.5 MB); completion when no tool_calls; stop-and-summarize on max steps. Mirrors the Yutori Python SDK reference loop.
• Provider registration — AgentProvider, AVAILABLE_CUA_MODELS, AgentType, and providerEnvVarMap (yutori → YUTORI_API_KEY); Navigator-specific ClientOptions (toolSet, disableTools, jsonSchema, userTimezone, userLocation) + the cloud API / OpenAPI schema.
• Keyboard modifiers implemented generically: a new optional modifiers option on the understudy page.click() / page.scroll() that sets the CDP mouse-event modifier bitmask (reusable by any provider). Plus hold_key and refresh (via page.reload, with a faithful agent-replay step).
• Evals — the local bench harness no longer builds an AI-SDK text client for CUA-only models (getAISDKLanguageModel has no provider for them and initV3 ignores it in CUA mode). General fix; also unblocks local evals for microsoft/fara-7b.
• Tests — unit coverage for the client (action mapping, message/trajectory shape, structured output, error recovery, stop-and-summarize), the helpers (coordinate denorm/validation, key mapping, payload trimming), the handler (modifiers/hold/refresh + URL freshness), and API serialization; plus a usage example.

Testing

• pnpm build (typecheck) + the new unit suites pass; prettier/eslint clean.
• Verified live end-to-end against the real Navigator API (local headless Chrome): multi-step click/type/keypress tasks complete with correct DOM end-state.

Notes for reviewers

• Core tool set only. The expanded/DOM tool set (extract_elements, find, set_element_value, execute_js) is intentionally a follow-up.
• The cloud-API/OpenAPI additions expose Navigator config through the hosted API schema for consistency; happy to drop or split these (and/or the evals harness fix) into separate PRs if you'd prefer a smaller first PR.
• mouse_down/mouse_up are disabled by default (no equivalent in the shared action handler; drag covers press-move-release).

Maintained by the Yutori team.

---

Summary by cubic

Adds the Yutori Navigator n1.5 computer-use model as a new provider via an OpenAI-compatible Chat Completions API, now defaulting to the expanded DOM tool set for richer page interaction.

• New Features

  • New yutori/n1.5-latest CUA model with YUTORI_API_KEY auth and options (toolSet, disableTools, jsonSchema, userTimezone, userLocation, temperature).
  • Expanded tools (default): extract_elements, find, set_element_value, execute_js built on the a11y snapshot + deepLocator; coordinate tools can target a ref (resolved to on-screen center with scroll-into-view) and recover on stale refs.
  • YutoriCUAClient: screenshot-per-turn loop, 1000×1000 coordinate mapping, payload trimming, per-tool results with current URL, stop-and-summarize on max steps (fully flow-logged), and structured parsed_json on AgentResult.output.
  • Generic click/scroll modifiers via CDP bitmask (captured and replayed), hold_key delay, and refresh with a recorded replay step; API/OpenAPI exposes provider yutori and Navigator options; .env.example includes YUTORI_API_KEY; local eval harness skips AI-SDK text clients for CUA-only models; example and tests included.

• Migration

  • Set YUTORI_API_KEY (and optional baseURL), then use: stagehand.agent({ mode: "cua", model: "yutori/n1.5-latest" }).
  • Optional model options: toolSet, disableTools, jsonSchema, userTimezone, userLocation, temperature (use toolSet: "browser_tools_core-20260403" for coordinate-only).

^{Written for commit fcfbdb1. Summary will update on new commits.}

[changeset-bot]

🦋 Changeset detected

Latest commit: fcfbdb1

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@browserbasehq/stagehand	Patch
@browserbasehq/stagehand-evals	Patch
@browserbasehq/stagehand-server-v3	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

This PR is from an external contributor and must be approved by a stagehand team member with write access before CI can run.
Approving the latest commit mirrors it into an internal PR owned by the approver.
If new commits are pushed later, the internal PR stays open but is marked stale until someone approves the latest external commit and refreshes it.

[cubic-dev-ai]

2 issues found across 23 files

Confidence score: 3/5

• There is some real merge risk: stopAndSummarize() in packages/core/lib/v3/agent/YutoriCUAClient.ts bypasses expected flowLogger instrumentation, which can reduce traceability and make agent behavior/debugging less reliable.
• packages/core/lib/v3/handlers/v3CuaAgentHandler.ts has a concrete replay correctness concern—modifiers used during CUA action execution are not persisted, so replay can diverge for modifier-dependent actions.
• Given a high-confidence medium/high-severity instrumentation gap plus a replay-divergence bug, this looks mergeable only with caution rather than a low-risk merge.
• Pay close attention to packages/core/lib/v3/agent/YutoriCUAClient.ts and packages/core/lib/v3/handlers/v3CuaAgentHandler.ts - missing flow logging and non-persisted modifiers can cause observability and replay consistency issues.

Architecture diagram

sequenceDiagram
    participant User as User Code
    participant Stagehand as Stagehand Instance
    participant Agent as Agent Provider
    participant Client as YutoriCUAClient
    participant Handler as V3CuaAgentHandler
    participant Page as Understudy Page
    participant CDP as Chrome DevTools Protocol
    participant Navigator as Yutori Navigator API

    Note over User,Navigator: NEW: Yutori Navigator n1.5 CUA Agent Provider

    User->>Stagehand: stagehand.agent({ mode: "cua", model: "yutori/n1.5-latest" })
    Stagehand->>Agent: create provider (modelToAgentProviderMap)
    Agent->>Client: new YutoriCUAClient(type, model, instructions, clientOptions)
    alt API key missing
        Client-->>Agent: throw Error
    end
    Client->>Client: configure toolSet, disableTools, jsonSchema, userTimezone, userLocation
    Stagehand-->>User: agent instance

    User->>Stagehand: agent.execute({ instruction, maxSteps })
    Stagehand->>Handler: new V3CuaAgentHandler(..., client=YutoriCUAClient)

    Handler->>Client: setScreenshotProvider()
    Handler->>Client: setActionHandler()
    Handler->>Handler: capture screenshot (page.screenshot)
    Handler->>Client: setCurrentUrl(page.url())
    Client->>Client: build message history (system prompt + user instruction with location/timezone context)

    loop Step Loop (maxSteps)
        Client->>Client: clone messages for request
        Client->>Client: trimImagesToFit (drop old screenshots under ~9.5 MB, keep latest)
        Client->>Navigator: POST /v1/chat/completions (OpenAI-compatible)
        Note over Client,Navigator: Extra params: tool_set, disable_tools, json_schema
        Navigator-->>Client: response with tool_calls (1000x1000 normalized coordinates)
        Client->>Client: parse tool_calls from assistant message

        alt No tool_calls
            Client-->>Handler: return final result (completed)
        else Has tool_calls
            par For each tool_call
                Client->>Client: denormalizeCoordinates (1000→viewport pixels)
                Client->>Client: mapNavigatorKeyToPlaywright (Navigator keys→Playwright keys)
                Client->>Handler: actionHandler(action)
                alt Action type: click (with possible modifier)
                    Handler->>Page: click(x, y, { modifiers })
                    Page->>CDP: dispatchMouseEvent(..., modifiers bitmask)
                else Action type: scroll (with possible modifier)
                    Handler->>Page: scroll(x, y, deltaX, deltaY, { modifiers })
                    Page->>CDP: dispatchMouseEvent(..., modifiers bitmask)
                else Action type: keypress (with optional holdMs delay)
                    Handler->>Page: keyPress(key, { delay })
                else Action type: refresh
                    Handler->>Page: reload({ waitUntil: "load" })
                    Page->>CDP: Page.reload
                else Action type: type, goto, back, forward, wait, drag
                    Handler->>Page: execute action
                end
                alt Action succeeded
                    Page-->>Handler: success
                else Action threw error
                    Page-->>Handler: error
                    Handler->>Page: still update client URL (page.url())
                    Handler-->>Client: action result with [ERROR]
                end
            end
            Client->>Client: append tool result (role:"tool" + "Current URL:" suffix)
            Client->>Client: captureScreenshot() for next turn
        end

        alt Payload size > max bytes
            Client->>Client: trimImagesToFit (strip old screenshots, keep recent)
        end
    end

    alt Max steps reached (no completion)
        Client->>Client: formatStopAndSummarize(task)
        Client->>Navigator: final request with summarize prompt (no json_schema)
        Navigator-->>Client: summary text
        Client-->>Handler: return result (completed=false, summary message)
    end

    Handler-->>Stagehand: AgentResult (output may include parsed_json)
    Stagehand-->>User: execution result

Loading

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>