Embedding Gemma Transcribe: Turning an Embedding Model into a 288x Speech-to-Text Engine on Apple Neural Engine

In modern deep learning, models are typically treated as narrow, single-purpose appliances. Automatic Speech Recognition (ASR) models (like OpenAI Whisper or NVIDIA Parakeet) are trained specifically to generate text tokens from mel-spectrograms. Embedding models (like Google’s EmbeddingGemma 2) are trained strictly to output dense numerical vectors for search and clustering.

Recently, independent AI researcher Haku unveiled an audacious architectural experiment at eg2.haku.sh that flips this assumption on its head:

“Google’s EmbeddingGemma 2 is an embedding model. We turned it into a speech-to-text model: an hour of audio in about 13 seconds on a base MacBook Air, entirely on device.”

By exploiting the geometric properties of a shared multimodal vector space, Embedding Gemma Transcribe achieves an astounding 288x real-time transcription speed, outrunning dedicated speech models running on heavy GPUs.

Here is an architectural dissection of how this vector inversion works under the hood, why it executes with zero battery penalty on Apple’s Neural Engine, and what it teaches us about the geometry of multimodal foundation models.

1. The Core Insight: Vector Space Nearest-Neighbor Transcription

To understand how an embedding model can transcribe speech without ever being trained as an ASR decoder, you have to look at how EmbeddingGemma 2 structures reality:

┌────────────────────────────────────────────────────────────────────────┐
│             Google's EmbeddingGemma 2 Shared 768d Sphere               │
│                                                                        │
│               "apple" (Text Token)                                     │
│                     ▲                                                  │
│                     │ Cosine Proximity (~0.88)                         │
│                     ▼                                                  │
│             [ Audio of spoken word "apple" ]                           │
│                                                                        │
└──────────────────────────────────┬─────────────────────────────────────┘
                                   │
                                   ▼
┌────────────────────────────────────────────────────────────────────────┐
│                        The Inversion Mechanism                         │
│  1. Freeze the base EmbeddingGemma 2 Audio Encoder entirely.           │
│  2. Slice incoming audio into short temporal frames.                   │
│  3. Map audio embeddings directly into the text vocabulary space.      │
│  4. The closest word vector out of ~229,000 candidates wins!           │
└────────────────────────────────────────────────────────────────────────┘

When Google DeepMind trained EmbeddingGemma 2, they aligned audio, text, images, and video into a unified 768-dimensional contrastive space.

When a human speaks the word “architecture”, the audio encoder projects that acoustic wave to almost the exact same coordinate cluster as the text string "architecture".
The Hypothesis: If spoken words already land directly on top of their written counterparts in vector space, why bother training a massive 1B+ autoregressive decoder? Just read the closest word out of the space via nearest-neighbor search!

2. Architectural Pipeline: How the Inversion Operates

The complete transcription pipeline comprises four lightweight stages:

[ Raw Audio Waveform (.m4a / .wav) ]
                  │
                  ▼
┌─────────────────────────────────────────────────────────────┐
│          1. Frozen EmbeddingGemma Audio Encoder             │
│  (Conformer-based acoustic front-end; zero modifications)    │
└──────────────────────────────┬──────────────────────────────┘
                               │ High-dimensional acoustic features
                               ▼
┌─────────────────────────────────────────────────────────────┐
│          2. Lightweight Projection Reader Module            │
│  (Trained in < 2 hours on 8×H100 on open English speech)    │
└──────────────────────────────┬──────────────────────────────┘
                               │ Slices mapped to text vector coordinates
                               ▼
┌─────────────────────────────────────────────────────────────┐
│          3. 229,000-Word Vector Nearest Neighbor Search     │
│  (Finds closest vocabulary token with cosine similarity)    │
└──────────────────────────────┬──────────────────────────────┘
                               │ Unpunctuated token sequence
                               ▼
┌─────────────────────────────────────────────────────────────┐
│          4. Lightweight Post-Processing Punctuation Head    │
│  (Adds capitalization, periods, commas, and paragraph breaks│
└─────────────────────────────────────────────────────────────┘
                               │
                               ▼
[ Final Clean Transcript (1 Hour Audio in 12.6s) ]

The Remarkable Training Footprint:

The base EmbeddingGemma 2 model was never fine-tuned or modified. Its weights remain entirely frozen.
Only the tiny projection reader and punctuation module were trained.
Total compute required for training: Under 2 hours on 8× NVIDIA H100 GPUs using openly licensed English speech datasets.

3. Benchmark Reality: 288x Real-Time on an 8GB MacBook Air

By eliminating the autoregressive decoding loop (where models generate one token at a time sequentially), Embedding Gemma Transcribe processes entire audio streams in bulk parallel batches.

When benchmarked against leading open-source speech models:

┌────────────────────────────────────────────────────────────────────────┐
│                Speech-to-Text Throughput & Hardware Benchmark          │
├──────────────────────────┬──────────────┬──────────────┬───────────────┤
│ Model                    │ Compute Unit │ Hardware     │ Speed Ratio   │
├──────────────────────────┼──────────────┼──────────────┼───────────────┤
│ EmbeddingGemma Transcribe│ Neural Engine│ M3 Air (8GB) │ 288× Real-Time│
│ IBM Granite Speech 5.0   │ GPU (MLX)    │ M3 Air (8GB) │ 150× Real-Time│
│ NVIDIA Parakeet TDT 0.6B │ Neural Engine│ M4 Pro       │ 156× Real-Time│
│ NVIDIA Parakeet TDT 0.6B │ Neural Engine│ M5 Pro       │ 335× Real-Time│
│ Phonon-2 (FluidAudio)    │ Neural Engine│ M5 Pro       │ 480× Real-Time│
│ OpenAI Whisper Large-v3  │ Metal / GPU  │ M3 Air (8GB) │ ~15× Real-Time│
└──────────────────────────┴──────────────┴──────────────┴───────────────┘

Key Quantitative Takeaways:

288× Processing Speed: A 60-minute recorded interview or podcast is transcribed end-to-end in 12.6 seconds.
Zero Fan Noise / Cool Thermals: Because the workload runs compiled against CoreML targeting the Apple Neural Engine (ANE), it consumes less than 3 Watts of power, leaving the main CPU and GPU idle.
Word Error Rate (WER): Currently stands at approximately ~5% WER. While slightly behind bleeding-edge specialized ASR models (like Whisper Large-v3 or Parakeet at 2–3% WER), a 5% error rate is more than sufficient for search indexing, meeting recaps, and personal dictation.

4. Systems Implications: The Death of Cloud Dictation APIs

For privacy-conscious developers and enterprise systems architects, this project delivers three profound lessons:

1. Zero Cloud Latency and Total Audio Privacy

Services like Otter.ai, Rev, or cloud Whisper endpoints require streaming sensitive board meetings, medical consultations, and legal discussions to remote third-party servers. Embedding Gemma Transcribe runs 100% air-gapped on device. Holding down the Right Option key allows instantaneous, zero-latency dictation into any desktop application.

Historically, researchers treated embeddings as a “one-way hash” of human language. Embedding Gemma Transcribe demonstrates that high-dimensional multimodal contrastive spaces are so tightly organized that feature extraction and generation are simply two sides of the same geometric coin.

3. Edge Silicon Dominance

Running on a basic $999 consumer laptop with 8GB of unified memory and matching the performance of dedicated server-grade GPU clusters underscores the immense untapped potential of dedicated Neural Processing Units (NPUs).

5. Architectural Summary

Dimension	Embedding Gemma Transcribe Specification
Foundation Model	Google DeepMind EmbeddingGemma 2 (Frozen Audio Backbone)
Decoding Methodology	Vector space nearest-neighbor projection into 229k text tokens
Throughput Speed	288× real-time (1 hour transcribed in 12.6 seconds)
Execution Hardware	Apple Neural Engine (ANE) via CoreML on base M3 MacBook Air
Word Error Rate (WER)	~5% on standard English speech benchmarks
Privacy Footprint	100% on-device local execution; zero telemetry or network calls

The Engineering Verdict:

Embedding Gemma Transcribe is a masterclass in architectural elegance: identifying latent geometric capabilities in an existing foundation model, training a lightweight adapter in under two hours, and achieving state-of-the-art edge performance.

It proves that the future of speech intelligence will not be dominated by larger cloud models, but by ingenious mathematical shortcuts deployed directly on consumer edge silicon.