What to check when an audio AI model fails on your recording

Jonas Petrak, Research writer (AI-assisted) ·

Audio AI models can give poor results on your files for several technical reasons. The most common are sample rate or channel mismatches, missing decoders, distortion or loudness problems, loss of frequency content during conversion, or using audio that doesn't match the model's training data. This article works through each cause and how to check and fix it, with calculated numbers for a 44.1 kHz stereo file converted for a model such as Whisper [1].

Sample rate and channels: the most frequent mismatch

Most audio AI models expect a fixed sample rate and number of channels. For example, Whisper processes 16,000 Hz mono audio, not the 44,100 Hz stereo format typical of music recordings [1,2]. Demucs works at 44.1 kHz stereo, but both models will resample or downmix input if needed [1].

If your file has the wrong sample rate or channel count, the model will either resample it (sometimes with quality loss) or fail to run. Whisper's code and README document using ffmpeg to convert any input to 16,000 Hz mono before processing [1,2,5].

You can check your file's format with a tool like ffprobe or any audio editor. If you need to convert manually, use a command like:

``` ffmpeg -i input.wav -ar 16000 -ac 1 output.wav

```

But most models will do this conversion for you automatically if ffmpeg is installed [1].

Worked example: what does resampling actually do?

Suppose you have a 60-second stereo recording at 44,100 Hz, 16-bit PCM—standard for a music file [1]. Whisper expects 16,000 Hz mono, so the file is downmixed and resampled.

Here is what happens in numbers, from our own arithmetic [1]:

Quantity44.1 kHz stereo 16-bit16 kHz mono 16-bit
Samples per channel (60 s)2,646,000960,000
Channels21
Total samples5,292,000960,000
Bytes per sample22
File size (bytes)10,584,0001,920,000
Highest frequency kept22,050 Hz8,000 Hz

The conversion keeps only 960,000 of the original 5,292,000 samples—about 18.1 percent [1]. Everything above 8,000 Hz (the new Nyquist limit) is lost. This means cymbals, "air," and other high-frequency detail are gone before the model sees your file.

Missing or broken decoder: ffmpeg is essential

Both Whisper and Demucs rely on ffmpeg to decode audio files [1,5]. If ffmpeg is not installed or is broken, the models cannot read your files.

To check, run:

``` ffmpeg -version

```

If this fails, install ffmpeg using your system's package manager. On Windows, Demucs specifically falls back on ffmpeg due to limited torchaudio support [1].

If you cannot install ffmpeg, convert your file to uncompressed wav format using any standard tool.

Clipping and loudness: when the file is too hot

If your recording is too loud (so-called "hot") or distorted (clipped), models may give poor or strange results. Demucs tries to prevent output clipping by rescaling stems by default, but this can change the relative loudness of the instruments. Alternatively, it can apply hard clipping with `--clip-mode clamp`, which preserves levels but distorts peaks [1].

Check your waveform for flat tops at full scale (which signal clipping), and listen for crackle or distortion. If these are present, lower your input file's volume before running separation, or choose your model's clipping mode carefully [1].

Bandwidth loss: when downsampling discards content

Downsampling from 44,100 Hz to 16,000 Hz, as Whisper does, discards all content above 8,000 Hz—the new Nyquist frequency [1]. For music or audio where high frequencies matter, this can strip away important detail and harm results.

To check, compare the spectrum of your original file and the converted one in an audio editor.

If you need to preserve high frequencies, use a model that works at the original rate.

Training data mismatch: your audio vs. what the model knows

Every model is trained on certain data. Whisper was trained on 680,000 hours of multilingual audio from the internet, with wide-ranging sources but not every possible recording condition [1,3,4]. Demucs is benchmarked on MUSDB18, a set of 150 stereo music tracks [1].

If your recording is very different—for example, a noisy field recording or a language, instrument, or mic setup not in the training data—the model may not perform well.

Check whether your audio resembles the model's training data in language, style, and quality. If not, try a model trained closer to your domain, clean or trim your audio, or fine-tune if the model supports it [1].

How benchmarks are produced and why your score may differ

Benchmarks like the 9.00 dB SDR for Hybrid Transformer Demucs are measured on specific datasets (MUSDB HQ) and report the median score across all tracks and sources [1]. They do not guarantee the same result on your own recordings, especially if those differ from the benchmark data in language, channel count, or mixing style.

Questions readers ask

What if my audio is longer than 30 seconds?

Whisper processes audio in 30-second windows, so long files are split and processed chunk by chunk. Errors may appear at window edges if context is lost [1].

Can I trust the model's benchmark scores for my own files?

No. Benchmark scores are measured on specific datasets and may not match your results, especially if your audio differs in content or quality [1].

My file is mono at 16,000 Hz but still gets bad results—what next?

Check for clipping, distortion, or a mismatch in the type of audio (e.g., language or recording style). Also, ensure ffmpeg is installed and working [1].

How the numbers work

Computed from the numbered sources and the assumptions named below; change an assumption and the results change with it.

FigureValueWhere it comes from
Your recording's sample rate44,100 Hzassumption: the reader's own recording; typical value from source [1]
Your recording's channel count2assumption: the reader's own recording; typical value from source [1]
Your recording's duration60 sassumption: the reader's own recording; typical value from source [1]
Model's required sample rate (Whisper)16,000 Hzsource [1]
Model's required channel count (Whisper)1source [1]
Bit depth (PCM encoding)16 bitsassumption: typical uncompressed PCM from source [1]
Samples per channel in your recording2,646,000computed: Your recording's sample rate × Your recording's duration
Total samples in your recording5,292,000computed: Samples per channel in your recording × Your recording's channel count
Bytes per sample (PCM)2 bytescomputed: Bit depth (PCM encoding) ÷ 8
Uncompressed file size of your recording10,584,000 bytescomputed: Total samples in your recording × Bytes per sample (PCM)
Highest frequency kept in your recording (Nyquist limit)22,050 Hzcomputed: Your recording's sample rate ÷ 2
Samples per channel after resampling to model rate960,000computed: Model's required sample rate (Whisper) × Your recording's duration
Total samples after resampling and downmixing960,000computed: Samples per channel after resampling to model rate × Model's required channel count (Whisper)
Highest frequency kept after resampling to model rate8,000 Hzcomputed: Model's required sample rate (Whisper) ÷ 2
Work it out with your own numbers
  • Your recording's sample rate: 44100 Hz
  • Your recording's channel count: 2
  • Your recording's duration: 60 s
  • Model's required sample rate (Whisper): 16000 Hz
  • Model's required channel count (Whisper): 1
  • Bit depth (PCM encoding): 16 bits

Works out: Samples per channel in your recording; Total samples in your recording; Bytes per sample (PCM); Uncompressed file size of your recording; Highest frequency kept in your recording (Nyquist limit); Samples per channel after resampling to model rate; Total samples after resampling and downmixing; Highest frequency kept after resampling to model rate

Sources

  1. Our own data, 15 Nov 2026: Why an audio model fails on your recording: a sourced checklist
  2. raw.githubusercontent.com
  3. (2212.04356) Robust Speech Recognition via Large-Scale Weak Supervision
  4. (2212.04356) Robust Speech Recognition via Large-Scale Weak Supervision
  5. GitHub - openai/whisper: Robust Speech Recognition via Large-Scale Weak Supervision · GitHub

More articles