MFCC, CQT and STFT: Audio feature Extraction Compared

Jonas Petrak, Research writer (AI-assisted) ·

MFCCs, constant-Q transform (CQT), and short-time Fourier transform (STFT) are the most widely used advanced techniques for extracting features from audio. Each has its own strengths, trade-offs, and application areas. Knowing when and how to use each can give you more relevant and robust features for your audio models and experiments.

MFCCs match human hearing for speech and device identification

Mel-frequency cepstral coefficients (MFCCs) transform short segments of audio into a series of coefficients that capture how the human ear perceives sound frequencies. They do this by:

  • Taking the Fourier transform of each windowed segment of the signal
  • Mapping the spectrum onto the mel scale (which is linear below 1,000 Hz and logarithmic above)
  • Applying a logarithm to the powers at each mel frequency, then a discrete cosine transform to these log powers

This series of steps produces coefficients that are especially effective for speech recognition, speaker identification, and even detecting the device that recorded the audio [1]. Unlike a simple spectrum, MFCCs capture both the vocal tract characteristics and device-specific coloration, because the convolutional distortions from a phone or microphone are preserved in the resulting features.

Example: In a speaker authentication system, MFCCs are used to compare a user's voice sample to stored references. If the same person speaks into two different phones, the MFCCs will reflect both the unique properties of the speaker and the subtle differences in each phone's microphone and circuitry, allowing the system to identify not just the person but, in some cases, the recording device as well [1].

Constant-Q transform is tuned for musical signals

The constant-Q transform (CQT) provides a time-frequency representation with frequency bins spaced logarithmically, matching how musical pitch works [2]. Each bin has a constant ratio of center frequency to bandwidth (the Q), so lower frequencies get wider windows (better frequency resolution) and higher frequencies get narrower ones (better time resolution). This is especially valuable for music analysis, where notes are spaced by octaves and musical intervals.

CQT's time-frequency trade-off means it can distinguish closely spaced low notes in a piano recording better than STFT, but with less precise timing for fast attacks at low frequencies. It is also invertible with the right implementation, allowing for signal reconstruction after processing [2].

Example: In a music genre classification task, CQT features can capture the harmonic content and chord changes over time, providing a more musically meaningful feature space than linear-frequency spectrograms. This is particularly useful for tasks like cover song detection or instrument recognition, where pitch relationships matter [2].

STFT gives a flexible window into time-varying spectra

The short-time Fourier transform (STFT) is the backbone of most audio analysis pipelines and the basis for spectrograms [3]. It divides the audio into overlapping frames, applies a window function (like Hann or Gaussian), and computes the Fourier transform for each frame. The result is a 2D time-frequency map that shows how energy at each frequency evolves over time.

STFT is flexible: you can tune the window length for narrowband (good frequency resolution, poor timing) or wideband (good time resolution, poorer frequency) analysis. However, it uses fixed window sizes for all frequencies, so it cannot adapt its resolution to both low and high frequencies at once. This limitation led to the development of CQT and wavelet transforms [3].

Example: For environmental sound classification (like identifying sirens, alarms, or footsteps), an STFT spectrogram captures both the spectral shape and timing of events, making it a strong baseline feature for many machine learning models [3].

The tools that do this, and how well

  • MFCCs are a standard in speech-recognition libraries and are part of ETSI standards for mobile phones [1]. They are fast, well-tested, and robust to moderate noise, but can discard phase information and may be less effective for music than for speech.
  • CQT is implemented in libraries like LibROSA (Python) and MATLAB, using efficient fast Fourier transform-based methods for speed [2]. It excels with musical audio, but may be less intuitive for speech or environmental sounds.
  • STFT is available in nearly every audio processing toolkit, with fast FFT-based calculation and support for custom windows and overlap settings [3]. It remains the go-to for general-purpose audio analysis and visualization.

A practical workflow often starts with STFT or MFCCs for speech tasks, and CQT for music tasks. For hybrid cases—like singing voice detection or audio event classification—combining features from multiple transforms may yield the best results.

Questions readers ask

How do I choose between MFCC, CQT, and STFT for my audio project?

If your task involves speech or speaker recognition, MFCCs are usually the best starting point. For music, especially where pitch or harmony is important, use CQT. For general audio analysis, exploratory work, or when you need a 2D time-frequency representation, start with STFT and adjust window sizes to your needs.

Can I combine features from more than one technique?

Yes. Many advanced audio systems combine MFCCs, CQT, and STFT-derived features to capture complementary information. This is common in music information retrieval, environmental sound analysis, and even some modern neural models.

Are there cases where these methods fail?

Each method has blind spots. MFCCs can lose critical information for non-speech sounds; CQT may not be ideal for tasks requiring precise timing at low frequencies; STFT’s fixed resolution can miss details that vary in both time and frequency. Understanding your signal and task lets you pick or combine the right tools.

Sources

  1. Mel-frequency cepstrum - Wikipedia
  2. Constant-Q transform - Wikipedia
  3. Short-time Fourier transform - Wikipedia

More articles