Why an audio model fails on your recording: a sourced checklist
What we found
The Whisper source code sets the sample rate it works at: its audio module defines SAMPLE_RATE = 16000 and CHUNK_LENGTH = 30 (https://raw.githubusercontent.com/openai/whisper/main/whisper/audio.py). The same file's load_audio function opens with "Open an audio file and read as mono waveform, resampling as necessary" and calls ffmpeg with the arguments "-ac", "1" and "-ar", str(sr) (https://raw.githubusercontent.com/openai/whisper/main/whisper/audio.py).
The Whisper paper, "Robust Speech Recognition via Large-Scale Weak Supervision", states "All audio is re-sampled to 16,000 Hz" and "We break audio files into 30-second segments paired with the subset of the transcript" (https://ar5iv.labs.arxiv.org/html/2212.04356). Its abstract says the models were trained on "680,000 hours of multilingual and multitask supervision" (https://arxiv.org/abs/2212.04356).
The Whisper README says "Internally, the transcribe() method reads the entire file and processes the audio with a sliding 30-second window" and that the package "also requires the command-line tool ffmpeg to be installed on your system" (https://github.com/openai/whisper).
The Demucs README says "Audio is resampled on the fly if necessary" and that the output is "four stereo wav files sampled at 44.1 kHz" (https://github.com/facebookresearch/demucs). It also says "All audio formats supported by torchaudio can be processed", that the output is a wav file encoded as int16, and that on Windows torchaudio support is limited so it relies on ffmpeg (https://github.com/facebookresearch/demucs).
On clipping, the Demucs README says that by default it rescales each output stem to avoid clipping (which can change the relative volume between stems), that --clip-mode clamp gives hard clipping instead, and that reducing the input mixture's volume is an alternative (https://github.com/facebookresearch/demucs).
The MUSDB18 dataset page says "All signals are stereophonic and encoded at 44.1kHz" and describes "150 full lengths music tracks (~10h duration)" (https://sigsep.github.io/datasets/musdb.html). The Demucs README says its Hybrid Transformer Demucs model "achieves a SDR of 9.00 dB on the MUSDB HQ test set" (https://github.com/facebookresearch/demucs).
The SiSEC 2018 paper says it prepared "a new music separation database: MUSDB18, featuring close to 10h of audio", names the "Source to Distortion, to Artefact, to Interference ratios (SDR, SAR, SIR)", and says "For each track, the median value of the score was taken and used for the boxplots" (https://ar5iv.labs.arxiv.org/html/1804.06267).
In a worked example with an assumed 60-second recording stored as uncompressed 16-bit PCM (header ignored), the table below is calculated by our arithmetic. Nyquist: a sampled signal can represent frequencies only up to half the sampling rate, so 44,100 / 2 = 22,050 Hz and 16,000 / 2 = 8,000 Hz are the highest frequencies that can be kept at each rate (this is the standard sampling theorem, not a figure from the pages above).
| Quantity (calculated by our arithmetic) | Formula | 44.1 kHz stereo 16‑bit | 16 kHz mono 16‑bit |
|---|---|---|---|
| Samples per channel in 60 s | rate x 60 | 44,100 x 60 = 2,646,000 | 16,000 x 60 = 960,000 |
| Channels | given | 2 | 1 |
| Total samples | samples per channel x channels | 2,646,000 x 2 = 5,292,000 | 960,000 x 1 = 960,000 |
| Bytes per sample | 16 bits / 8 | 2 | 2 |
| File size in bytes | total samples x 2 | 5,292,000 x 2 = 10,584,000 | 960,000 x 2 = 1,920,000 |
| Highest frequency kept (Nyquist) | rate / 2 | 22,050 Hz | 8,000 Hz |
| Data rate in bytes per second | file size / 60 | 176,400 | 32,000 |
| Size ratio | 10,584,000 / 1,920,000 | 5.5125 times the 16 kHz mono size | 1 |
By our arithmetic, the conversion keeps 960,000 of 5,292,000 samples (about 18.1 percent) and discards everything between 8,000 Hz and 22,050 Hz.
Checklist: cause, check, fix
- Sample rate or channel mismatch. Cause: a model works at a fixed rate; Whisper uses 16,000 Hz mono (audio.py and paper above). Check: read your file's rate and channels (for example with ffprobe) and compare with the model's documented rate. Fix: let the tool resample (Whisper's load_audio and Demucs both resample on the fly per the sources), or convert yourself with ffmpeg; if you resample by hand, do it once, with a proper resampler.
- Missing or broken decoder. Cause: Whisper needs ffmpeg installed; Demucs on Windows relies on ffmpeg (READMEs above). Check: run ffmpeg -version, and play the file in a normal player. Fix: install ffmpeg, or convert the file to wav first.
- Clipping and loudness. Cause: a hot or distorted mixture can clip after separation; the Demucs README describes rescaling or hard clipping of stems. Check: look at the waveform for flat tops at full scale, and listen for crackle. Fix: lower the input volume before separation (the README's suggestion), and choose between the default rescale and
--clip-mode clampknowing the trade-off the README gives. - Bandwidth lost by downsampling. Cause: at 16 kHz the highest kept frequency is 8,000 Hz (Nyquist, worked above), so content above it cannot be used by a 16 kHz model. Check: compare the spectrum of the original and converted file. Fix: for tasks where high frequencies matter, use a model that works at a higher rate (Demucs outputs 44.1 kHz stems per its README).
- Domain mismatch. Cause: a model is trained on particular data; Whisper on 680,000 hours of multilingual audio (abstract), Demucs is benchmarked on MUSDB18, 150 stereo music tracks of about 10 hours (dataset page). Check: ask whether your audio resembles that data (language, noise, instruments, recording chain). Fix: try a model closer to your case, clean or trim the audio, or fine-tune if the project supports it.
- Long audio and 30-second windows. Cause: Whisper processes audio in 30-second windows (README, audio.py, paper), so context is cut at window edges. Check: see whether errors cluster at window boundaries. Fix: trim silence, or split at pauses before transcribing.
- Reading benchmark scores correctly. Cause: a quoted SDR of 9.00 dB is a score on MUSDB HQ test tracks (Demucs README), and SiSEC 2018 aggregates per track by the median. Check: see which dataset, which metric and which aggregation a number comes from. Fix: treat a benchmark figure as evidence for music like the test set, not a promise for your file.
Where this fails
Models differ: this checklist is built from the documentation of Whisper and Demucs and does not cover other models, whose input rules may differ. The order of the checklist is not a measured ranking of how often each cause occurs; no tests on recordings were run. The worked example assumes uncompressed 16-bit PCM, so compressed files (mp3, flac) have different sizes. The Nyquist limit is a theoretical ceiling; real resamplers roll off before it.
Sources
- https://raw.githubusercontent.com/openai/whisper/main/whisper/audio.py : Whisper audio loading code (SAMPLE_RATE, CHUNK_LENGTH, ffmpeg arguments), read 2026-11-15.
- https://github.com/openai/whisper : Whisper README (30-second window, ffmpeg), read 2026-11-15. The README does not mention 16 kHz; that comes from the code and paper.
- https://arxiv.org/abs/2212.04356 : Whisper paper abstract (680,000 hours), read 2026-11-15.
- https://ar5iv.labs.arxiv.org/html/2212.04356 : Whisper paper full text (16,000 Hz, 30-second segments), read 2026-11-15.
- https://github.com/facebookresearch/demucs : Demucs README (resampling, 44.1 kHz stereo output, clipping, SDR 9.00 dB), read 2026-11-15.
- https://sigsep.github.io/datasets/musdb.html : MUSDB18 dataset page (150 tracks, 44.1 kHz stereo), read 2026-11-15.
- https://ar5iv.labs.arxiv.org/html/1804.06267 : The 2018 Signal Separation Evaluation Campaign (SDR/SAR/SIR, median per track), read 2026-11-15.