A 5-second clip becomes a 201 by 498 spectrogram at 16 kHz

Jonas Petrak, Research writer (AI-assisted) · · updated

A spectrogram is a way to see the changing frequencies present in an audio signal, displayed as an image. Audio AI models use them because they reveal the structure in speech, music, and noise far more clearly than raw waveforms do. This article explains how a spectrogram is formed, what information it gives, and why models that "see" audio as pictures often outperform those that work directly with the raw sound.

A spectrogram shows how sound changes across time and frequency

A spectrogram maps time (horizontal axis), frequency (vertical axis), and amplitude (shown by color or brightness) to make the structure of a sound visible [1]. For speech and music, this visualization exposes which harmonics or notes are present at each instant and how they shift over time.

Building a spectrogram means slicing the audio signal into very short, overlapping frames—commonly 25 milliseconds each—then running the short-time Fourier transform (STFT) on each segment to reveal its frequency makeup [2]. The result: each vertical slice in the image shows the frequency content during that window, and stacking these slices reconstructs the whole evolution of the sound.

The settings you pick for frame and hop length control the kind of detail you see. Short frames make it possible to spot rapid changes, like the burst in a plosive consonant, but trade off frequency precision. Long frames sharpen the frequency resolution—which is essential for tracking stable pitches in music—but blur short-lived sound events [1]. The tradeoff is fundamental: you cannot maximize time and frequency resolution simultaneously [1].

Audio models use spectrograms because patterns are clearer in pictures

Many audio classifiers, among them the Audio Spectrogram Transformer (AST) and its multiscale successor MAST, take a spectrogram as input rather than the raw waveform [3]. Harmonics, onsets, and other features form visible structures in spectrograms: vertical lines for transients, horizontal striations for harmonics [1]. The structure is explicit in the picture, so a model built for images can use it directly; other models learn from the raw waveform instead, and the sources here do not compare the two. For speech and animal sound analysis, these structures are used directly to identify phonemes and call types [1].

Both the Audio Spectrogram Transformer (AST) and the Multiscale Audio Spectrogram Transformer (MAST) operate on spectrogram input. MAST introduces hierarchical pooling along both axes, letting the model capture features at different time and frequency scales. On VGGSound, MAST achieves a 4.7% higher top-1 accuracy than AST, with both models using the same spectrogram representations [3]. This difference comes not from the spectrogram itself, but from the model’s ability to use its structure—pooling, in this case, lets MAST exploit patterns that a flat transformer misses.

No source in this set reports a direct accuracy comparison between spectrogram-based and waveform-based models. What the sources do show is that, when models are built to read spectrograms, they can use the visible structure of audio patterns—especially when the network architecture aligns with the shapes and scales present in the spectrogram.

How a spectrogram is generated, step by step

To see where audio models get their input, start with the steps that turn a waveform into a spectrogram, using the short-time Fourier transform [2]:

  1. The audio signal is divided into short, overlapping frames—typically 25 milliseconds long, shifted by 10 milliseconds at a time (the "hop").
  2. Each frame is multiplied by a window function (often a Hann window) that tapers the ends to reduce edge artifacts [2].
  3. A Fourier transform is applied to each windowed frame, decomposing it into frequency components [2].
  4. For each frequency bin, the magnitude is measured and the phase is ignored [1,2].
  5. These magnitudes are stacked into a matrix, with time on one axis, frequency on the other, and color or brightness representing strength [1].

Most audio spectrograms use a logarithmic scale for amplitude (decibels) so both quiet and loud sounds are visible, and sometimes a logarithmic scale for frequency to highlight musical relationships [1]. While 3D "waterfall" plots exist, the 2D heat map is standard in audio AI, as it matches what models expect in their input [1].

The tools that do this, and how well

Many audio classification models start with a spectrogram. The Multiscale Audio Spectrogram Transformer (MAST) is one example; it aggregates features at multiple time and frequency scales, which helps it recognize both short, sharp events and longer, evolving ones [3]. Compared to the earlier AST model, MAST improved top-1 accuracy by 4.7% on the VGGSound dataset, used 5× fewer multiply-accumulate operations, and had 42% fewer parameters [3]. For real-time or embedded applications where memory and compute are limited, this reduction in parameters and operations is a practical gain—the difference between a model that fits on a low-power device and one that does not.

No source gives a direct, consistent comparison between models trained on spectrograms and those trained directly on waveforms for classification tasks. What the MAST paper [3] does show is that spectrogram-based models such as AST and MAST can be made both more accurate and more efficient by tuning how they process the spectrogram, and that models adapted from vision can transfer their advances to audio.

The spectrogram exposes patterns in a form that these models, especially those built on vision transformer backbones, can use. MAST achieves its efficiency by working at multiple time and frequency scales, compressing the input where it matters least, and preserving detail where it matters most [3].

What spectrograms miss, and why it matters

A spectrogram is not a perfect copy of the original sound. It discards the phase information, which tells you exactly where in its cycle a frequency is at each moment [1]. While the amplitude (energy) at each frequency is captured, the exact timing details that can be crucial for reconstructing the sound are lost [1]. This means you cannot turn a spectrogram back into the original audio perfectly; at best, you get a close approximation, which is often enough for classification but not always for high-fidelity generation [1].

There is also a tradeoff in how you set up the spectrogram. A long analysis window gives you better frequency detail but smears quick changes in time; a short window catches rapid events but blurs the frequency [1]. This is rooted in the Heisenberg uncertainty principle as it applies to signal processing [1]. In practice, 25 ms frames and 10 ms hops are a common compromise for speech and music [2].

A worked example

Suppose you have a 5-second audio clip at 16,000 Hz. You use 25 ms frames and 10 ms hops to generate the spectrogram. Our own computation of these sizes, done two ways (by the formula and by stepping the frame through the clip), is on the site's data page [4]: 498 frames, 201 bins, 100,098 cells for this clip; 98 frames for 1 second, 998 for 10 seconds, 2,998 for 30 seconds; and 552 bins at 44,100 Hz [4].

  • Total frames: 1 + floor((5 - 0.025) / 0.01) = 498.
  • Each frame covers 201 frequency bins (up to 8,000 Hz).
  • The resulting spectrogram is a 201 (frequency) × 498 (time) matrix.
  • This image is then used as input for an audio model, which can spot patterns such as harmonics, onsets, or frequency sweeps.

AST and MAST read a mel spectrogram, whose frequency axis is spaced by pitch rather than the linear bins used here, so their grids have a different shape from this one.

Questions readers ask

What does a spectrogram reveal that a waveform hides?

A waveform shows only amplitude over time but does not separate frequencies. A spectrogram reveals which frequencies are present and how they change, letting you see harmonics, overtones, and noise in ways a waveform cannot [1].

Can you turn a spectrogram back into audio?

Not perfectly—spectrograms discard phase information, which is needed to reconstruct the exact waveform. Some tools approximate the original sound, but the result will not match the source exactly [1].

Why not just use the raw audio for models?

Raw audio contains all the information, phase included, and some models learn from it directly. A spectrogram rearranges the audio so that harmonics and onsets are visible as shapes, which is what image-style models such as AST and MAST are built to read [3]. The sources here do not compare the two inputs on the same task, so this article makes no claim about which performs better.

*Corrected 2026-11-07: The sentence on the settings AST and MAST use stated figures (10 ms hop, 128 mel bins) that the cited paper page does not print; it now says only what is checkable.*

Sources

  1. Spectrogram - Wikipedia
  2. Short-time Fourier transform - Wikipedia
  3. Multiscale Audio Spectrogram Transformer for Efficient Audio Classification
  4. Our own computation (2026-11-07): how big a spectrogram is, frames and bins for standard settings, two ways

More articles