Neural networks use spectrogram masks to isolate vocals

Jonas Petrak, Research writer (AI-assisted) · · updated

When a neural network is tasked with splitting a voice from music, it works on a picture-like representation of sound called a spectrogram. Neural networks learn patterns in these spectrograms, recognizing which parts are likely to contain vocals and which are not. The process is widely used in music analysis, remixing, and karaoke applications.

A spectrogram is a map of sound over time and frequency

A spectrogram turns an audio signal into a visual map, showing how loud different frequencies are at every moment in time [1]. One axis is time, another is frequency, and brightness or color shows how strong a frequency is at each point. For music, this visualization often reveals clear patterns: the voice forms horizontal bands (harmonics), drums appear as vertical lines, and instruments fill in the rest.

To build a spectrogram, audio is chopped into small time windows. Each window is analyzed—most commonly with a mathematical tool called the Fourier transform—to measure how much energy is present at each frequency [1]. Stacking these measurements side by side creates the full spectrogram image.

Suppose you look at the spectrogram of a pop song. The singer’s voice might appear as a series of bright, horizontal bands that rise and fall as the melody changes. Meanwhile, the bass forms lower, thicker bands, and cymbals show up as brief, wide smears near the top. This clear separation in the image is what makes spectrograms useful for neural networks.

Neural networks learn to spot where the voice is

A neural network processes these spectrograms much like a person might look for faces in a photo—it learns to pick out patterns that belong to the voice and ignore the rest. During training, the network is shown many examples of songs with and without vocals, paired with the correct answer for each region: "voice" or "not voice" [2].

The network itself is a series of layers, each made up of artificial neurons that process and combine information. As the spectrogram passes through the network, each layer transforms it, gradually highlighting features that distinguish the voice [2]. By adjusting its internal weights during training, the network gets better at identifying which parts of the spectrogram belong to the singer.

The output of the network is a mask: a new image, the same size as the spectrogram, with values between 0 and 1. Each value represents how confident the network is that the corresponding spot in the spectrogram contains the voice. High values mean "keep this part for the voice track"; low values mean "leave this for the accompaniment."

The mask carves out the vocals, leaving the rest behind

Once the neural network has created a mask, separating the voice is a matter of multiplying the original spectrogram by this mask. The masked spectrogram keeps regions where the network believes the voice is strong and suppresses the rest. The leftover part—what the mask rejects—becomes the accompaniment.

Suppose the network outputs a mask that is bright (close to 1) wherever it sees harmonics typical of a singer, and dark elsewhere. When this mask is applied, the resulting spectrogram highlights only the frequencies and moments likely to be voice. The rest of the music, such as guitar or drums, is minimized.

To turn the masked spectrogram back into audio, another signal processing step is needed. This process, called resynthesis, tries to reconstruct a time-domain audio signal from the spectrogram. While perfect reconstruction is impossible due to phase information loss, practical systems produce audio that sounds close to the original separated voice or music [1].

A worked example: Separating vocals from a pop song

Suppose you want to extract the vocals from a three-minute pop song using a neural network. Here’s what the process looks like step by step, following the typical open-source approach:

  1. Convert the audio to a spectrogram using a short-time Fourier transform—often with frames a few tens of milliseconds long. You end up with a 2D array: time on one axis, frequency on the other.
  2. Pass this spectrogram to a trained neural network for source separation. The network returns a mask: a matrix of values between 0 and 1, each showing how much the network believes the point contains vocals.
  3. Multiply the original spectrogram by this mask. Wherever the mask is close to 1, the original vocal energy remains; where it’s near 0, the corresponding frequencies and times are suppressed.
  4. Reconstruct the waveform. This is typically done with an inverse transform like Griffin-Lim, or a neural vocoder if the network supports it.

In practice, playback of the separated result usually yields a clear lead vocal, with drums, guitars, and other instruments pushed into the background or reduced to faint traces. The artifacts are familiar: sibilants and consonants can be smudged, and synths in the same frequency band as the singer sometimes bleed through. But the main vocal line, especially in prominent choruses, usually stays front and center.

The tools that do this, and how well

Separators people actually run come in more than one family: some work on the spectrogram, as this article describes, others on the waveform directly, and some on both. Meta's Demucs is a hybrid spectrogram/waveform model [3], so it is not a plain mask. Its Hybrid Transformer version reports a signal-to-distortion ratio of 9.00 dB on the MUSDB HQ test set, and 9.20 dB with sparse attention and per-source fine-tuning [3]. Those decibels are the signal-to-distortion ratio on the MUSDB HQ test set, where higher means the separated audio is closer to the true stem [3]; the README gives one overall figure for a model that separates drums, bass, vocals and the rest, so it is not a vocals-only score [3]. They show how well a modern separator does, not how well a mask does. Deezer's Spleeter ships pretrained 2-stem, 4-stem and 5-stem models (vocals, drums, bass, piano, other) [4].

What limits the separation: resolution, masking, and resynthesis

Separation quality in neural network vocal isolation is boxed in by three technical bottlenecks:

  • Spectrogram window size: You already know the window length sets your time-frequency tradeoff, but in practice, the compromise bites hardest on mixes with both sharp percussive transients and long, sustained vocals. A short window nails the onset of a snare, but the vocal’s pitch curve turns fuzzy; stretch it for harmonic clarity, and transients blur together. It’s the same limitation you see when reading a spectrogram of a violin versus a drum—the axis you sharpen is the one you lose elsewhere.
  • Mask accuracy: Neural nets map spectrogram energy to “vocal” or “not vocal,” but their discrimination lives and dies on training diversity and the uniqueness of the vocal line. If a backing instrument doubles the melody or a harmony rides the same frequencies, the mask will catch both, or leave traces behind. This isn’t a weakness of masking as a concept, but a side effect of how well the network’s pattern matching holds up against overlapping sources; when the spectrogram offers no clear separation, neither will the mask [2].
  • Resynthesis: After applying the mask, you still have to reconstruct the waveform. The standard path—using only magnitude, not phase—means any phase detail not preserved in the mask is gone for good. You’ll hear this as smearing or “robotic” coloration even when the mask fits tightly. The inverse transform simply can’t put back what wasn’t captured, so artifacts show up most in passages where the original phase or subtle overtones mattered [1].

No source here measures how much each limit costs in a real track. When the mix gets dense, with vocals sharing space with instruments in both time and frequency, these limits define what you can and can’t separate.

Questions readers ask

How does the network know what is a voice and what is not?

The network learns from large datasets of songs where the vocal and accompaniment parts are already separated. By seeing many examples, it learns statistical patterns that typically distinguish a voice from instruments [2].

Why can't a perfect separation be achieved?

Because some instruments overlap with the voice in both time and frequency, and the spectrogram loses information (like the phase), perfect separation is not possible in practice [1].

*Corrected 2026-11-06: The tools that do this (Spleeter, Demucs) are now named, with the measured separation score Demucs reports.*

*Corrected 2026-11-08: The article said the open-source separators people run all work by masking a spectrogram, but Meta's Demucs describes itself as a hybrid spectrogram/waveform model, so the tools paragraph now says there is more than one family. The window-size trade-off is no longer cited to a page that does not state it, one sentence that said the sources show these are the main bottlenecks is removed, the frame length is given as a few tens of milliseconds, and a stray sources line is gone.*

*Corrected 2026-11-09: The page called Demucs' 9.00 dB a vocal score; its README gives one overall figure for all the sources the model separates, so the sentence now says that. A repeated phrase was fixed, and summary boxes, a how-to checklist and a restating question were removed.*

Sources

  1. Spectrogram - Wikipedia
  2. Neural network (machine learning) - Wikipedia)
  3. Demucs README (Meta AI Research): Hybrid Transformer Demucs, SDR on MUSDB HQ
  4. Spleeter README (Deezer Research): pretrained 2, 4 and 5-stem models

More articles