What to check when an audio AI model fails on your recording
Jonas Petrak, Research writer (AI-assisted) ·
Audio AI models can give poor results on your files for several technical reasons. The most common are sample rate or channel mismatches, missing decoders, distortion or loudness problems, loss of frequency content during conversion, or using audio that doesn't match the model's training data. This article works through each cause and how to check and fix it, with calculated numbers for a 44.1 kHz stereo file converted for a model such as Whisper [1].
Sample rate and channels: the most frequent mismatch
Most audio AI models expect a fixed sample rate and number of channels. For example, Whisper processes 16,000 Hz mono audio, not the 44,100 Hz stereo format typical of music recordings [1,2]. Demucs works at 44.1 kHz stereo, but both models will resample or downmix input if needed [1].
If your file has the wrong sample rate or channel count, the model will either resample it (sometimes with quality loss) or fail to run. Whisper's code and README document using ffmpeg to convert any input to 16,000 Hz mono before processing [1,2,5].
You can check your file's format with a tool like ffprobe or any audio editor. If you need to convert manually, use a command like:
``` ffmpeg -i input.wav -ar 16000 -ac 1 output.wav
```
But most models will do this conversion for you automatically if ffmpeg is installed [1].
Worked example: what does resampling actually do?
Suppose you have a 60-second stereo recording at 44,100 Hz, 16-bit PCM—standard for a music file [1]. Whisper expects 16,000 Hz mono, so the file is downmixed and resampled.
Here is what happens in numbers, from our own arithmetic [1]:
| Quantity | 44.1 kHz stereo 16-bit | 16 kHz mono 16-bit |
|---|---|---|
| Samples per channel (60 s) | 2,646,000 | 960,000 |
| Channels | 2 | 1 |
| Total samples | 5,292,000 | 960,000 |
| Bytes per sample | 2 | 2 |
| File size (bytes) | 10,584,000 | 1,920,000 |
| Highest frequency kept | 22,050 Hz | 8,000 Hz |
The conversion keeps only 960,000 of the original 5,292,000 samples—about 18.1 percent [1]. Everything above 8,000 Hz (the new Nyquist limit) is lost. This means cymbals, "air," and other high-frequency detail are gone before the model sees your file.
Missing or broken decoder: ffmpeg is essential
Both Whisper and Demucs rely on ffmpeg to decode audio files [1,5]. If ffmpeg is not installed or is broken, the models cannot read your files.
To check, run:
``` ffmpeg -version
```
If this fails, install ffmpeg using your system's package manager. On Windows, Demucs specifically falls back on ffmpeg due to limited torchaudio support [1].
If you cannot install ffmpeg, convert your file to uncompressed wav format using any standard tool.
Clipping and loudness: when the file is too hot
If your recording is too loud (so-called "hot") or distorted (clipped), models may give poor or strange results. Demucs tries to prevent output clipping by rescaling stems by default, but this can change the relative loudness of the instruments. Alternatively, it can apply hard clipping with `--clip-mode clamp`, which preserves levels but distorts peaks [1].
Check your waveform for flat tops at full scale (which signal clipping), and listen for crackle or distortion. If these are present, lower your input file's volume before running separation, or choose your model's clipping mode carefully [1].
Bandwidth loss: when downsampling discards content
Downsampling from 44,100 Hz to 16,000 Hz, as Whisper does, discards all content above 8,000 Hz—the new Nyquist frequency [1]. For music or audio where high frequencies matter, this can strip away important detail and harm results.
To check, compare the spectrum of your original file and the converted one in an audio editor.
If you need to preserve high frequencies, use a model that works at the original rate.
Training data mismatch: your audio vs. what the model knows
Every model is trained on certain data. Whisper was trained on 680,000 hours of multilingual audio from the internet, with wide-ranging sources but not every possible recording condition [1,3,4]. Demucs is benchmarked on MUSDB18, a set of 150 stereo music tracks [1].
If your recording is very different—for example, a noisy field recording or a language, instrument, or mic setup not in the training data—the model may not perform well.
Check whether your audio resembles the model's training data in language, style, and quality. If not, try a model trained closer to your domain, clean or trim your audio, or fine-tune if the model supports it [1].
How benchmarks are produced and why your score may differ
Benchmarks like the 9.00 dB SDR for Hybrid Transformer Demucs are measured on specific datasets (MUSDB HQ) and report the median score across all tracks and sources [1]. They do not guarantee the same result on your own recordings, especially if those differ from the benchmark data in language, channel count, or mixing style.
Questions readers ask
What if my audio is longer than 30 seconds?
Whisper processes audio in 30-second windows, so long files are split and processed chunk by chunk. Errors may appear at window edges if context is lost [1].
Can I trust the model's benchmark scores for my own files?
No. Benchmark scores are measured on specific datasets and may not match your results, especially if your audio differs in content or quality [1].
My file is mono at 16,000 Hz but still gets bad results—what next?
Check for clipping, distortion, or a mismatch in the type of audio (e.g., language or recording style). Also, ensure ffmpeg is installed and working [1].
How the numbers work
Computed from the numbered sources and the assumptions named below; change an assumption and the results change with it.
| Figure | Value | Where it comes from |
|---|---|---|
| Your recording's sample rate | 44,100 Hz | assumption: the reader's own recording; typical value from source [1] |
| Your recording's channel count | 2 | assumption: the reader's own recording; typical value from source [1] |
| Your recording's duration | 60 s | assumption: the reader's own recording; typical value from source [1] |
| Model's required sample rate (Whisper) | 16,000 Hz | source [1] |
| Model's required channel count (Whisper) | 1 | source [1] |
| Bit depth (PCM encoding) | 16 bits | assumption: typical uncompressed PCM from source [1] |
| Samples per channel in your recording | 2,646,000 | computed: Your recording's sample rate × Your recording's duration |
| Total samples in your recording | 5,292,000 | computed: Samples per channel in your recording × Your recording's channel count |
| Bytes per sample (PCM) | 2 bytes | computed: Bit depth (PCM encoding) ÷ 8 |
| Uncompressed file size of your recording | 10,584,000 bytes | computed: Total samples in your recording × Bytes per sample (PCM) |
| Highest frequency kept in your recording (Nyquist limit) | 22,050 Hz | computed: Your recording's sample rate ÷ 2 |
| Samples per channel after resampling to model rate | 960,000 | computed: Model's required sample rate (Whisper) × Your recording's duration |
| Total samples after resampling and downmixing | 960,000 | computed: Samples per channel after resampling to model rate × Model's required channel count (Whisper) |
| Highest frequency kept after resampling to model rate | 8,000 Hz | computed: Model's required sample rate (Whisper) ÷ 2 |
- Your recording's sample rate: 44100 Hz
- Your recording's channel count: 2
- Your recording's duration: 60 s
- Model's required sample rate (Whisper): 16000 Hz
- Model's required channel count (Whisper): 1
- Bit depth (PCM encoding): 16 bits
Works out: Samples per channel in your recording; Total samples in your recording; Bytes per sample (PCM); Uncompressed file size of your recording; Highest frequency kept in your recording (Nyquist limit); Samples per channel after resampling to model rate; Total samples after resampling and downmixing; Highest frequency kept after resampling to model rate
Sources
- Our own data, 15 Nov 2026: Why an audio model fails on your recording: a sourced checklist
- raw.githubusercontent.com
- (2212.04356) Robust Speech Recognition via Large-Scale Weak Supervision
- (2212.04356) Robust Speech Recognition via Large-Scale Weak Supervision
- GitHub - openai/whisper: Robust Speech Recognition via Large-Scale Weak Supervision · GitHub