Voice Agents

Enable governed speech-to-speech for an autonomous chat agent, choose the right SDK settings, and tune the conversation with evidence.

Voice agents add a managed realtime speech model to an autonomous chat agent. The realtime model listens and speaks, while the agent's separate loop.model reasons and chooses from the signed tool surface. Every tool call still passes through policy, DLP, trust, approvals, metering, and audit.

Enable voice on a chat agent

Speech-to-speech requires all three of these declarations:

Create the agent with defineAutonomousAgent.
Add an enabled chat trigger with speechToSpeech settings.
Declare the only tools the call may use in the signed loop.tools surface.

Example

import { defineAutonomousAgent } from "@forge/sdk";

export default defineAutonomousAgent({
  identity: {
    name: "Claims Voice Guide",
    scope: "dept/claims",
    authority: "Explain claim status and collect follow-up details.",
  },
  trust: { initialTier: "medium" },
  triggers: [
    {
      id: "voice-chat",
      type: "chat",
      enabled: true,
      speechToSpeech: {
        providerId: "openai",
        model: "gpt-realtime-2.1",
        voice: "marin",
        requestedMode: "strict",
        reasoningEffort: "low",
        recording: "disabled",
      },
    },
  ],
  goal: [
    "Help the caller understand their claim status.",
    "Use short spoken sentences and confirm important identifiers.",
    "Ask before calling a tool that changes data.",
  ].join(" "),
  loop: {
    providerId: "anthropic",
    model: "claude-sonnet-4-6",
    tools: [
      {
        name: "get_claim_status",
        ref: { source: "mcp", connectorId: "claims", toolId: "get_status" },
      },
    ],
    maxTurns: 12,
    tokenBudget: 80_000,
  },
});

The control plane validates realtime capability, voice eligibility, residency, and route policy at deployment and again when a session starts. A prompt cannot add a tool that was not declared and approved in loop.tools.

Speech-to-speech settings

Setting	Choices	Guidance
`providerId`	Managed realtime provider ID	Use `openai` for the current managed realtime path.
`model`	Realtime-capable speech model	Use `gpt-realtime-2.1` for the current managed speech model. This is separate from `loop.model`.
`voice`	`alloy`, `ash`, `ballad`, `cedar`, `coral`, `echo`, `marin`, `sage`, `shimmer`, `verse`	Test with your real scripts and audience. Defaults to `marin`.
`requestedMode`	`strict` or `native`	Start with `strict`. It holds audio and transcript for applicable DLP checks. `native` reduces latency but may release audio before final-response DLP finishes.
`reasoningEffort`	`minimal`, `low`, `medium`, `high`, `xhigh`	Start at `low`. Raise it only when evaluations show the realtime turn needs more reasoning and the latency tradeoff is acceptable.
`recording`	`disabled` or `optional`	Leave disabled unless the product needs recording. `optional` still requires tenant and user permission for each session.

voice, requestedMode, reasoningEffort, and recording may be omitted; they normalize to marin, strict, low, and disabled before the manifest is signed.

Use native only as a deliberate latency tradeoff. A caller can hear audio before the final response has completed DLP evaluation, even though tool actions remain governed.

Tune the conversation

Treat model personality as a starting hypothesis, not a provider guarantee. Across conversational evaluations, Anthropic loop models often produce warmer, more expansive replies. OpenAI loop models often produce tighter, more precise replies. The live speech path currently uses OpenAI GPT Realtime 2.1 in either case, so the agent goal, realtime voice, signed tools, and your own evaluations have more influence on the spoken experience than the provider label alone.

For a personable voice agent:

describe the relationship and tone in the goal instead of asking for a vague "friendly" voice,
ask for short spoken sentences, one question at a time, and explicit confirmation of critical details,
give the loop model examples of a good greeting, interruption recovery, tool wait, and handoff,
test names, numbers, acronyms, silence, interruptions, and noisy microphones,
measure task completion, correction rate, tool-call accuracy, approval waits, and call latency.

For a precise voice agent, state the answer format and decision boundary directly. Require it to separate known facts from assumptions and to ask for missing information instead of filling gaps.

Optimize safely

Change one variable at a time: first the goal and examples, then the loop model, then voice and reasoning effort. Re-run the same evaluation set after each change so a warmer conversation does not quietly reduce factual accuracy or tool discipline.

Keep destructive or egress-capable tools behind approval. Start with strict mode and recording disabled, then consider lower latency or recording only after the privacy owner understands the tradeoff. Review Autonomous agents and loops for tool governance, caps, and prompt-injection guidance.

Enable voice on a chat agent

Speech-to-speech settings

Tune the conversation

Optimize safely

On this page