Nemotron 3.5 Transcribe is NVIDIA’s open-weights speech LLM built for transcription across 40 languages and recordings up to 1 hour. The model supports a 65,536-token context and reaches a 4.71% mean WER on 9 public English sets, leading open-weights peers. A text instruction controls output style: verbatim, written, speaker-attributed, contextual, or translated.
What is Nemotron 3.5 Transcribe?
It’s a speech-augmented language model (SALM) that turns audio plus text instructions into text. An audio encoder feeds embeddings into an LLM decoder, making transcription promptable. NVIDIA positions Nemotron 3.5 Transcribe as a transcription specialist rather than a general-purpose assistant.
The core idea: there is no single correct transcript. Numbers, fillers, and domain terms should adapt to the reader’s needs. You select the output style in the prompt: verbatim, written form, speaker-attributed, contextual, or translated.
One open model replaces separate ASR, diarization, and alignment tools. For teams, that means a simpler pipeline and fewer moving parts to maintain.
How to run Nemotron 3.5 Transcribe?
It’s designed for offline runs. In BF16 it operates on 1× H100 80 GB. FP8 (~35 GB) and NVFP4 (~21 GB) checkpoints are available; the NVFP4 variant runs on RTX 5090. GGUF files target NeMo-Speech.cpp. A single checkpoint handles up to 1 hour of audio with a 65,536-token context.
The flow is straightforward: choose a checkpoint format (BF16/FP8/NVFP4), load the model with its audio encoder, and specify an instruction to steer the output style. The model ingests audio plus a text prompt, returns a transcript with speaker labels, and can emit timestamps and translation.
Nemotron 3.5 Transcribe is offline only. That suits environments without cloud calls or where local data control is required. You’ll need sufficient GPU memory depending on the chosen checkpoint.
How does the architecture work?
The design has three parts: a Parallel Expert Encoder, an LLM backbone, and add-ons for faster decoding and alignment. This split makes transcription promptable without complicating the main decoder.
Parallel Expert Encoder. A trainable 0.6B Transformer ASR encoder runs beside a frozen Nemotron 3 Diarization model. The diarizer tracks up to 8 speakers, and its activity is injected into ASR features every 80 ms. The LLM thus sees who spoke when without architecture changes.
LLM backbone. Nemotron 3.5 Lightning 30B-A3B, a hybrid Mamba-Transformer MoE with 52 layers: 23 Mamba-2, 23 MoE, and only 6 attention layers. Each MoE layer routes to 6 of 128 experts plus 1 shared expert. Fewer attention layers keep the KV cache small for hour-long audio.
Add-ons. A shared multi-token prediction (MTP) layer drafts tokens for speculative decoding. A 50M-parameter CTC adapter aligns the final transcript to audio and yields word timestamps.
The name says 30B-A4B, yet the checkpoint holds about 34B parameters due to the 1.3B-parameter MTP module and audio components. About 4.3B are active (~4B). The context window is 65,536 tokens for hour-long recordings.
Training. One supervised fine-tuning stage on 256 H100 GPUs for 28,500 steps, followed by a short RL stage: 400 GSPO updates with a reward of 1 minus WER. The model saw roughly 1.72M unique hours of audio, or 6.5M hours with repeats.
How does it score on benchmarks?
According to NVIDIA’s technical report, Nemotron 3.5 Transcribe is the best open-weights model on the English and multilingual Open ASR Leaderboards. It delivers a 4.71% mean WER across 9 public English ASR sets. The next open model posts 4.94%. Still, “top commercial APIs still lead.”
The weak spot is AliMeeting Mandarin far-field meetings with 34.95% cpCER. This suggests more work is needed for that specific scenario or that a different stack may be preferable.
Strengths and weaknesses of Nemotron 3.5 Transcribe
Strengths: a single open checkpoint covers transcripts, speaker labels, timestamps, and translation across 40 languages. Prompting allows you to tailor style from verbatim to written or contextual. The 65,536-token context supports up to 1 hour per recording.
Weaknesses: it’s offline only and shows weak accuracy on Mandarin far-field meetings (34.95% cpCER). If cloud-based calls or that exact scenario are critical, factor in the risk.
| Parameter | Nemotron 3.5 Transcribe | Google Speech-to-Text | Microsoft Azure Speech | IBM Watson Speech to Text |
|---|---|---|---|---|
| Languages | 40 | — | — | — |
| Max audio length | 1 hour | — | — | — |
| Context window | 65,536 tokens | — | — | — |
| Mean WER (EN) | 4.71% | — | — | — |
| Worst case (cpCER) | 34.95% (AliMeeting) | — | — | — |
| Weights | open-weights | — | — | — |
Часті питання
What is it and how does it work?
Nemotron 3.5 Transcribe is a SALM that ingests audio and a text instruction, then outputs a transcript in the requested style (verbatim, written, speaker-attributed, contextual, translated). A Parallel Expert Encoder combines a trainable 0.6B ASR encoder with a frozen diarizer (up to 8 speakers, updates every 80 ms). The LLM backbone is a hybrid Mamba-Transformer MoE with 52 layers and a 65,536-token context.
Nemotron 3.5 Transcribe pricing?
The source provides no pricing or licensing details. It states the model is open-weights and runs offline. Plan for inference resources: 1× H100 80 GB for BF16; FP8 and NVFP4 reduce memory to about 35 GB and 21 GB, with NVFP4 running on RTX 5090. This lets you choose on-prem hardware without cloud dependence.
Nemotron 3.5 Transcribe vs Google Speech-to-Text comparison?
No direct Google numbers are given in the source. NVIDIA reports Nemotron 3.5 Transcribe as the best open-weights model on Open ASR Leaderboards with 4.71% WER (English), while the next open model has 4.94%. It also notes that “top commercial APIs still lead.” Without Google’s figures, any head-to-head remains limited.
Nemotron 3.5 Transcribe for multilingual teams?
The model covers 40 languages, includes speaker labels, timestamps, and translation. It supports up to 1 hour per recording and a 65,536-token context to keep long meetings in one pass. Open-weights and offline-only operation help where local control is needed. Note the 34.95% cpCER on AliMeeting (Mandarin, far-field) as a caveat.
How are alignment and diarization handled?
A frozen Nemotron 3 Diarization tracks up to 8 speakers, injecting activity every 80 ms into ASR features. A 50M-parameter CTC adapter aligns the final transcript to audio for word-level timestamps. Speculative decoding is enabled via an MTP module (~1.3B parameters) that drafts multiple tokens ahead.
Based on MarkTechPost.
