What is Gemini 3.5 Transcribe? JSON output and timestamps

"JSON output" in the title makes people jump to writing responseSchema. The dedicated transcription model first returns ASR annotation. The meeting JSON you want is usually a local reshape or a second Structured Output call.

What is Gemini 3.5 Transcribe?

Gemini 3.5 Transcribe is Google's dedicated speech-to-text model. It entered public preview on 2026-08-26. It is not "drop any audio on Gemini Flash." Official positioning: turn a recording or a microphone stream into readable text, and on the pre-recorded path, add speaker labels and word-level timestamps.

Google's own numbers (Artificial Analysis): streaming average WER 4.0%, non-streaming 2.6%; about 70% faster to a final draft than Chirp 3. On FLEURS, a main-language set is 5.50% streaming and 5.04% non-streaming. These are launch figures, not an acceptance line for your production audio.

• Smart transcription: handles "Tuesday—no, Wednesday," strips um / ah, auto-adds punctuation and number formatting.
• Custom vocabulary: brand names, SKUs, drug names, internal codes.
• Auto-detects 85+ languages; supports switching between Chinese and English in one clip.
• Already on consumer surfaces: Android Rambler, the macOS Gemini app. Developer entry is Gemini API / AI Studio; enterprise entry is Gemini Enterprise Agent Platform.

Two APIs, two model IDs

Pick the path first, then talk JSON. Model ID suffixes in the official announcement and Cloud docs are not identical. When you integrate, copy the IDs from your platform. Don't mix them.

The live path is for "show it first, then commit." The pre-recorded path is the entry for caption alignment, call QA, and speaker-sliced clips.

Native JSON: WordInfo is not responseSchema

In the Interactions API, after you turn on timestamp_granularities: ["word"], each word is one WordInfo:

• type: always word_info
• text: the word itself
• start_offset / end_offset: time from the start of the audio (only with word-level timestamps on)
• speaker: e.g. spk_1 (only with diarization_mode: speaker)

The Agent Platform sync API is more direct: audio_transcription.words[] has word, start_offset, end_offset; the speaker is speaker_label on the part.

So "JSON output" is two layers:

1. ASR JSON: WordInfo / words arrays. Fields are set by Google.
2. Business JSON: your meeting notes, caption track, QA form. Use Structured Output to infer a Schema from a sample, then make a second call or convert locally.

How to turn timestamps on, and what you lose

1. Pre-recorded: word-level start_offset / end_offset

Gemini API uses timestamp_granularities: ["word"]; Agent Platform uses word_timestamp=True. Google is explicit: word-level timestamps lower transcription accuracy. If you want karaoke-style captions, accept "better alignment, possibly worse WER," then spot-check numbers and proper nouns on a second quiet recording.

A minimal Interactions config (structure only — not a complete authenticated request):

{
  "model": "gemini-3.5-transcribe",
  "input": [{
    "role": "user",
    "content": [{
      "type": "audio",
      "uri": "FILE_URI",
      "mime_type": "audio/mp3"
    }]
  }],
  "generation_config": {
    "transcription_config": {
      "language_hints": ["zh-CN", "en-US"],
      "diarization_mode": "speaker",
      "timestamp_granularities": ["word"]
    }
  }
}

2. Live: utterance-level only, no word-level

Live API pushes two event types: interim_input_transcription (a hypothesis that can change) and input_transcription (a final you can commit). Cloud docs: live supports utterance-level timestamps and does not support word-level timestamps or speaker diarization. A continuous session lasts about 10 minutes; longer sessions you slice and stitch yourself.

Show interim on screen. Persist only final. If you write hypotheses into the database, the timeline will not line up.

Speaker diarization: how many, and when it is experimental

The launch post stresses "up to 3 speakers" on the pre-recorded path; 3+ is marked experimental. The Agent Platform table goes up to 8, and still marks 3+ as Experimental. Ship as "usable within 3 people; more speakers are a sampling item."

The live path has no speakers. For "who said the order number at 01:12," record first, then use the pre-recorded model.

Don't mix Smart and verbatim

verbatim keeps fillers, slips, and self-corrections — good for evidence, QA, and caption sources. Smart cleans slips, drops filler, and formats numbers and lists. Current docs: Smart is incompatible with speakers / word-level timestamps.

Run the same recording twice: verbatim + timestamps as source data; Smart as the human-readable notes. Don't let Smart overwrite the source.

A transcript sample that is enough to generate a Schema

Below is a common shape after you fold WordInfo into the business layer. Use seconds as numbers, not "00:12" strings — comparing, sorting, and drawing a timeline all get easier.

{
  "source": "gemini-3.5-transcribe",
  "mode": "verbatim",
  "language": "zh-CN",
  "segments": [
    {
      "speaker": "spk_1",
      "start": 1.24,
      "end": 4.08,
      "text": "把订单 ORD-1024 推到预发。"
    },
    {
      "speaker": "spk_2",
      "start": 4.21,
      "end": 6.9,
      "text": "好，我先对一下 SKU。"
    }
  ]
}

With this site's inference rules, this sample yields:

Once the sample is ready, open the Structured Output tool to generate each platform wrapper, then review required and additionalProperties by hand.

Duration, languages, and preview status

• Still Preview. Quotas, regions, and model ID suffixes can change. Run your own meeting recordings before you scale.
• Duration. Cloud: live about 10 minutes; sync files about 15 minutes when diarization / timestamps are on. The announcement does not write "one hour" as a hard limit. Follow your platform's current docs; slice long audio yourself.
• Languages. 85+ auto-detect. If you know the language, pass BCP-47, e.g. cmn-Hans-CN, en-US. Vocabulary is at most 1000 entries; official guidance is to start under 100 proper nouns.
• Custom vocabulary. Brands, names, SKUs only. Don't stuff everyday words.

How to accept it in JSON Toolbox

1. Paste a real WordInfo or a cleaned segments payload into JSON Format and confirm it parses.
2. Generate a first draft with JSON → Schema or Structured Output.
3. Validate a second recording with Schema validation. The first sample is always optimistic.
4. When the timeline is off, use JSON Diff to see whether start drifted or the speaker cuts changed.
5. Convert to Zod when the frontend needs checks, and to OpenAPI 3.1 when you need API docs.

Everything stays in the browser and is never uploaded.

FAQ

What is Gemini 3.5 Transcribe?

Google's dedicated speech-to-text model, released 2026-08-26. Pre-recorded audio uses gemini-3.5-transcribe; live uses gemini-3.5-transcribe-live.

Can it return timestamped JSON directly?

Yes. WordInfo includes text, start_offset, and end_offset; with diarization on, it also includes speaker. This is ASR annotation, not responseSchema.

Can Smart mode give speakers and word-level timestamps at the same time?

Per current docs, don't mix them. For labels or timestamps, use verbatim first.

Does the live API support speaker diarization?

No. Live only has interim / final text and utterance-level timing.

Are the dedicated transcription model and Structured Output the same thing?

No. A custom summary + segments shape needs a second call with a general model, or you fold WordInfo into your Schema locally.

Summary

Gemini 3.5 Transcribe solves "hear it, label who, align time." JSON landing is two layers: take WordInfo first, then fold it into a business Schema. Word-level timestamps and speakers are pre-recorded only. Don't turn Smart on with those annotations. Keep one core Schema. Validate, Zod, and OpenAPI all grow from it.

Related tools