Neural audio codec: how SoundStream turns sound into tokens

Jonas Petrak, Research writer (AI-assisted) · · updated

Neural audio codecs such as SoundStream and EnCodec break sound down into compact tokens using deep neural networks, allowing music, speech, and other audio to be streamed at much lower bitrates than traditional codecs. These systems rely on convolutional networks to encode audio and vector quantizers to turn features into discrete tokens, which can be transmitted or stored and then decoded back to audio with minimal quality loss [1].

From waveforms to tokens: the neural audio codec process

The core innovation in neural audio codecs is how they transform raw audio into a series of tokens, each representing a short segment of sound. In SoundStream, this is accomplished using a fully convolutional encoder that processes the audio waveform and extracts high-level features [1]. These features are then passed through a vector quantizer, which effectively "snaps" them to the nearest entry in a learned codebook—turning continuous information into a set of discrete tokens.

This approach replaces the hand-crafted signal processing steps of classic codecs with a deep learning pipeline, trained end-to-end to optimize both compression and audio quality [1]. These tokens can then be transmitted over the network or saved to disk, allowing for highly efficient compression.

How bitrates and compression ratios work in practice

Bitrate is the main lever for any codec—shave it down, and your bandwidth bill drops, but so does the room for signal detail. SoundStream’s architecture is built to flex: it can handle bitrates from 3 kbps up to 18 kbps without retraining, thanks to structured dropout during training [1]. This model runs in real time on a smartphone CPU, so the range isn’t theoretical—it’s designed for streaming and edge cases.

What emerges in use is that even at 3 kbps, SoundStream’s subjective audio quality outperforms Opus at four times the bitrate, and comes close to EVS at three times the bitrate (based on 24 kHz listening tests reported in the paper’s abstract) [1]. These comparisons matter more than raw compression numbers, because they give a reference point to codecs most practitioners have already built into their stacks.

AudioDec is another neural codec that takes on live and real-time use. It reconstructs 48 kHz speech at 12 kbps with less than 10 ms CPU latency, according to its abstract [2]. That’s enough to make it viable for calls or collaboration tools where delay and quality both matter. The paper notes both objective and subjective tests, so the headline figure comes with real-world listening behind it.

The papers' own comparisons are specific. SoundStream's is at 3 kbps against Opus at 12 kbps, four times the bitrate [1]; AudioDec's figure is for speech at 48 kHz, evaluated on the VCTK speech corpus [2]. Neither claims the same gain for every kind of audio.

Why neural audio tokens beat classic codecs for quality at low bitrates

Traditional codecs like MP3 or Opus use signal processing tricks and psychoacoustic models to remove "unnecessary" audio data. Neural codecs, in contrast, learn directly from data which audio components are most important to preserve. By training the encoder, quantizer, and decoder together, these systems create tokens that are tuned for both efficient compression and high-fidelity reconstruction [1, 2].

The result is that neural codecs like SoundStream at 3 kbps can outperform even the best classic codecs running at much higher bitrates. In subjective tests, SoundStream at 3 kbps outperformed Opus at 12 kbps and approached the quality of EVS at 9.6 kbps, despite using a fraction of the data [1]. This is not just about numbers—listeners in these tests preferred the neural codec output.

The tools that do this, and how well

SoundStream is notable for its flexibility: one model can operate at multiple bitrates, thanks to "structured dropout" during training [1]. This means a single neural codec can serve many use cases—streaming music at high quality, compressing speech at ultra-low bitrates, or even enhancing audio (e.g., suppressing background noise) as part of the same pipeline.

AudioDec offers real-time streaming and plug-and-play benchmarking for neural codecs, running on a standard CPU with just 10 ms of latency or even faster (6 ms) on a GPU [2]. This makes neural audio tokens practical for live applications like telephony and streaming.

HILCodec focuses on keeping models lightweight and efficient, analyzing where increased model depth no longer yields better audio and proposing architectural tweaks to reduce artifacts [3]. Together, these tools show the field is rapidly moving toward real-time, low-latency, and high-fidelity neural audio coding.

A worked example: turning a song into tokens for streaming

SoundStream's listening tests used audio at 24 kHz [1]. Assume a recording at that rate that is mono, 16-bit and three minutes long. Uncompressed that is 24,000 × 16 = 384 kbps, or 384 × 180 = 69,120 kilobits, about 8.6 MB. At SoundStream's lowest bitrate, 3 kbps, the same three minutes is 3 × 180 = 540 kilobits, about 67.5 kB: a ratio of 128 to 1. This is our own arithmetic from the bitrates; the papers do not report file sizes, and the abstracts do not say how SoundStream treats stereo.

*Corrected 2026-11-08: The papers' abstracts confirm the quoted figures (SoundStream 3 to 18 kbps, real time on a phone CPU, 3 kbps against Opus at 12 kbps; AudioDec at 12 kbps with under 10 ms CPU and 6 ms GPU latency). The worked example said a three-minute song would shrink from 17 MB to about 130 KB without saying the 17 MB was stereo; it is now a mono example (8.6 MB to about 67.5 kB, 128 to 1) and says the papers give no file sizes. A claim of bitrates 'an order of magnitude below' classic codecs, which the papers do not show (the SoundStream comparison is 4 times), was replaced by the papers' own comparisons. A block of figures built on a hidden 16-bit assumption, a tip, a reminder, a checklist and an FAQ with unsourced answers were removed.*

*Corrected 2026-11-08: The cited sentence keeps only what the SoundStream abstract states; the other assumptions are stated as assumptions.*

*Corrected 2026-11-21: The title now answers the question first. Headline was: "Neural audio codecs convert sound into efficient tokens".*

Sources

  1. SoundStream: An End-to-End Neural Audio Codec
  2. Audiodec: An Open-Source Streaming High-Fidelity Neural Audio Codec
  3. HILCodec: High-Fidelity and Lightweight Neural Audio Codec

More articles