Spirit LM Explained: How Speech and Text Interleave

How Spirit LM Connects Speech and Text

Spirit LM is a research language model that places speech units and text tokens in one autoregressive sequence. That lets the model continue from speech to text, text to speech, speech to speech, or text to text without forcing every spoken input through a plain transcript first.

The goal was not to create a general personal assistant. It was to study whether one model could retain the semantic strengths of a text language model while representing information in speech, including some expressive features that ordinary text drops.

flowchart LR
    A["Written words"] --> B["BPE text tokens"]
    C["Speech audio"] --> D["Phonetic speech units"]
    C --> E["Pitch and style units<br/>Expressive model"]
    B --> F["One interleaved token sequence"]
    D --> F
    E --> F
    F --> G["Autoregressive continuation"]
    G --> H["Text tokens"]
    G --> I["Speech units reconstructed as audio"]

Why a transcript-first pipeline loses information

A common speech workflow uses automatic speech recognition to turn audio into text, sends that text to a language model, then uses text-to-speech to create audio.

That cascade is practical and modular. It can also discard timing, pitch, emphasis, speaking style, hesitation, and other information when speech becomes text. A later speech synthesizer has to invent expressive details that the transcript never preserved.

Spirit LM investigates a different representation. Speech and text remain distinct token types, but the language model learns to predict both in one stream.

What Spirit LM Base represents

The Spirit LM paper starts from a 7 billion parameter pretrained text language model based on Llama 2. The researchers continue training it on text and speech units.

Written language uses subword BPE tokens. Speech uses discrete phonetic units produced with HuBERT. These units are not raw audio samples and are not ordinary written words. They compress aspects of speech into tokens the model can predict.

Spirit LM Base focuses on linguistic content. The model can receive speech units, text tokens, or a mixture, then continue in either modality.

What Spirit LM Expressive adds

Speech contains more than phonetic content. Pitch and speaking style can change how a sentence feels even when the words remain the same.

Spirit LM Expressive adds pitch and style units alongside the phonetic speech units. The purpose is to preserve more expressive information during generation.

More representation is not automatically better on every measure. The paper reports tradeoffs. Expressive units add complexity and can weaken some lexical or semantic modeling compared with the Base version. The right question is which information matters for the intended task.

How speech and text become interleaved

The researchers used an automatically curated aligned speech and text corpus. At word boundaries, the training sequence could switch between speech units and text tokens. That teaches the model that the two modalities can continue the same linguistic sequence.

The paper reports 300 billion text tokens, about 460,000 hours of speech-only data, and about 110,000 hours of aligned speech and text. The authors also acknowledge that automatic alignment can contain errors.

Those details are important because interleaving is not magic. The quality of alignment, tokenization, speech reconstruction, and data coverage shapes what the model can learn.

What the evaluation did and did not show

The paper evaluates generation across speech and text. It also includes few-shot experiments such as automatic speech recognition, text-to-speech, intent classification, and expressive speech tasks.

Spirit LM did not replace strong cascaded systems across the board. The paper reports that an automatic speech recognition, Llama 2, and text-to-speech cascade substantially outperformed Spirit LM on several clean benchmark comparisons.

That result does not erase the research contribution. It clarifies it. Spirit LM explored a unified sequence model and expressive transfer, while mature task-specific components remained stronger in important settings.

Audio examples are useful for hearing selected outputs. They do not establish consistency across accents, languages, disabilities, microphones, noise, emotion, or long conversations. They also do not prove factual reliability or safe use with real voices.

Access does not make it a general assistant

The official Spirit LM repository contains model weights, inference code, evaluation scripts, and a model card. That gives researchers material to inspect and run.

The controlling FAIR Noncommercial Research License limits covered materials and outputs to noncommercial research. It includes prohibited uses and does not promise support.

The repository is therefore a research artifact. It is not a commercial voice platform, a private local assistant, or a drop-in replacement for a notebook or podcast tool.

What a real voice workflow must add

A team evaluating speech technology needs representative speakers, languages, accents, recording conditions, and accessibility cases. It needs permission to process voices, clear retention rules, security controls, disclosure, and a process for impersonation or harmful output.

Evaluation should measure intelligibility, semantic accuracy, expressive fidelity, latency, speaker similarity, failure patterns, and human-review time. If the system generates a recognizable voice, consent and likeness rights need explicit ownership.

[[What Open Source Means for an AI Release]] explains why public weights under a research license should not be collapsed into open source. [[How to Evaluate an AI Research Release]] provides the field-test structure.

The lasting value of Spirit LM is the research question. How much spoken meaning is lost when a system treats text as the only bridge between listening and speaking? The paper gives one technically concrete attempt to answer it.

Editorial note

This research explainer was developed with AI assistance from the recovered E044 captions and the linked Spirit LM paper, repository, announcement, model card, and license. Dalton Anderson remains the author. Technical, source, license, voice-rights, privacy, safety, accessibility, and founder review are mandatory before publication or use. Publication is not authorized.