Gemini 3.8 Live: how does a realtime voice Agent turn one sentence into a JSON tool call?

The user says “turn on the living-room light.” The model hears an audio stream. When it reaches, it still emits name, id, and an args object. The Live API does not execute for you, and it does not auto-return. If the client drops the id, or sends a misheard room name into the executor, the audio will keep saying “okay” after the side effect has already happened. Same as a hosted text Agent: the loop can be fast; the contract must be able to refuse. How the text side closes: After the Agents API, why JSON matters more.

What 3.8 Live actually shipped

As of 2026-09-22, the stable model string is gemini-3.8-live. Migrating from gemini-3.1-flash-live-preview, the docs require a model-name change and dropping thinking_level / thinking_config. Input can be text, image, audio, or video; output is text and audio. Context is 131,072 / 65,536 tokens.

Pin the capability edges that matter for this piece:

• Function calling: supported. Default behavior: NON_BLOCKING. BLOCKING still works for older clients.
• Structured outputs: not supported. Do not configure a Live session as generateContent JSON-reply mode.
• Search grounding: supported. It can sit in the same tools array as function declarations.
• Code execution / File search / Maps / URL context: not supported.
• Response shape: audio. If you need visible text, turn on output transcription separately. Proactive audio is permanently on; turning it off errors. The affective-dialogue switch was removed from the API.

There is also gemini-3.8-live-extended-thinking: heavier background reasoning, Function Calling async only — setting BLOCKING is a hard error. Use 3.8 Live for the default low-latency path. Do not mix the two behaviors.

From voice to Tool: five hops

The official Live tools page is blunt about client duty: declare functions in the session config; after a toolCall, send a set of FunctionResponse with session.send_tool_response. Unlike generateContent, there is no automatic tool response.

1. Open the session. client.aio.live.connect(model="gemini-3.8-live", config=...). Put response_modalities: ["AUDIO"] and tools in the config.
2. Send content. Live audio, or send_client_content. 3.8 lets you update client content throughout the session; turn_complete=true interrupts the reply being generated.
3. Receive response.tool_call. Walk function_calls; read name, id, args. One prompt can emit parallel or chained calls.
4. You execute locally. Validate args against the declared Schema first, then run. On failure, return an error object. Do not “try to execute anyway.”
5. send_tool_response. Each item carries the same id / name; response is a JSON object. The session keeps emitting 24 kHz audio.

This is the same shape as a text Agent’s required_actions → tool_result, with a Live-event shell. Compare: Agents API vs MCP.

No Structured Output — why Schema matters more

The model card is explicit: Live does not support Structured outputs. The final “the light is on” is audio, not an object you can pin with text.format. Developers then assume “Live has no JSON contract.” The wrong layer:

• Structured Output constrains the final reply.
• Function Calling constrains the tool args.

Voice makes the second layer more dangerous. A slip, an accent, a room name heard as another enum value — the model can still emit args that look legal. There is no reply Schema to catch it. Parameter validation before execution is the only hard gate. Definition: What is Structured Output — that side owns the reply; this side owns the reach.

The transcription API has its own WordInfo JSON. Do not mix it with toolCall.args. Boundary: 3.5 Transcribe.

Async by default: NON_BLOCKING and scheduling

Function Calling on 3.1 Flash Live Preview was sync only: no return, the model does not keep talking. 3.8 Live made async the default. With behavior: NON_BLOCKING, the tool can run while the user keeps talking. If you need “look it up first, then speak,” add BLOCKING to that declaration.

On an async return, use scheduling to tell the model how to use the result:

• INTERRUPT: cut the current speech and speak the result now.
• WHEN_IDLE: wait until the current stretch finishes.
• SILENT: note it, use it later.

The docs sample puts scheduling inside FunctionResponse.response, next to result. That is a scheduling hint, not a business field. Close the business object yourself. Do not let scheduling leak into your executor.

How to declare functions in the session

Declarations live in session config.tools. The official minimal sample has only name. In production, at least add description and parameters. Gemini type fields are often uppercase OBJECT / STRING:

{
  "functionDeclarations": [
    {
      "name": "set_light",
      "description": "Set one allowlisted room light. Use when the user asks to turn a light on or off.",
      "behavior": "NON_BLOCKING",
      "parameters": {
        "type": "OBJECT",
        "properties": {
          "room": {
            "type": "STRING",
            "enum": ["living_room", "kitchen", "bedroom"],
            "description": "Allowlisted room id. Do not invent rooms."
          },
          "on": { "type": "BOOLEAN" }
        },
        "required": ["room", "on"]
      }
    }
  ]
}

The same config can also add {"google_search": {}}. Search is platform-run; set_light is still yours to execute. Do not write one over-wide tool for “any room, any command.” The permission shape matches a text Agent; see limit tool permissions with Schema.

A toolCall you can accept locally

Below folds the docs’ light control into “pending call + return.” Treat it as the core object first. Do not wrap an SDK on day one.

{
  "toolCall": {
    "functionCalls": [
      {
        "id": "call_light_01",
        "name": "set_light",
        "args": {
          "room": "living_room",
          "on": true
        }
      }
    ]
  },
  "functionResponse": {
    "id": "call_light_01",
    "name": "set_light",
    "response": {
      "result": {
        "room": "living_room",
        "on": true,
        "applied": true
      },
      "scheduling": "INTERRUPT"
    }
  }
}

The args contract closes rooms to an enum. If speech says “hall” or “dining” and it does not match, refuse and return a speakable error — do not guess:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "additionalProperties": false,
  "required": ["room", "on"],
  "properties": {
    "room": {
      "type": "string",
      "enum": ["living_room", "kitchen", "bedroom"]
    },
    "on": { "type": "boolean" }
  }
}

Compare the shapes that should fail: room as garage; an extra extra_cmd; on as "on". Those objects should fail validation in the Workbench. Growing a Schema from a sample: How to generate a Schema. How to pull args out of the event is a step-by-step tutorial for the next queue item.

How to accept it in JSON Toolbox

1. Paste args and response.result into JSON Format and confirm they parse.
2. Use JSON → Schema to generate the core contract. Add enum, required, and additionalProperties.
3. Run the same fields through Function Calling; if another Host will reuse them, wrap MCP Tool.
4. Prepare objects that should fail, and confirm Schema validation refuses them. When you need a reply-shape contrast, use Structured Output — remember the Live session itself does not consume that reply contract.

All data is processed in the browser and never uploaded. Have an object that can refuse bad args first. Then hang the function on the Live session.

FAQ

Does Gemini 3.8 Live support Structured Output?

The model card says no. The final reply is audio. What you can pin is the function parameter Schema, plus a check of args before execution.

Does the Live API execute tools for you?

No. You must handle toolCall yourself, then send_tool_response. Copy id and name on every return.

How do I choose NON_BLOCKING vs BLOCKING?

Async is the default; the call can continue. If you must wait for the result before speaking, use BLOCKING. Returns can use INTERRUPT / WHEN_IDLE / SILENT. The extended-thinking variant allows async only.

After voice enters the session, are tool args still JSON?

Yes. The model hears audio. When it reaches, it still emits an object. Voice is less stable, so validate before you execute.

Is this the same as Gemini 3.5 Transcribe?

No. Transcribe emits WordInfo; Live emits bidirectional audio and toolCall. Do not mash the two JSON shapes into one contract.

Summary

Gemini 3.8 Live made the low-latency voice loop the default model: audio in, audio out, tools async by default. It did not take execution away, and it did not give the final reply a Structured Output. What you still hold is functionDeclarations in the session config, and a JSON Schema on toolCall.args before they enter the executor. Voice changes the trigger, not the contract. The next step is not more room tools. It is a light-control object that can refuse garage and extra_cmd.

Related tools