Automatic music transcription detects notes from onsets
Jonas Petrak, Research writer (AI-assisted) · · updated
Automatic music transcription turns recordings—anything from a piano solo to a band rehearsal—into written notes. This process starts with detecting exactly when each note begins, then works out its pitch, using a mix of signal analysis and algorithms that try to mimic how humans hear music. Each step has its own challenges, especially when dealing with real-world audio rather than isolated tones.
Onset detection marks where notes begin
The first problem automatic transcription faces is knowing when each note starts. In audio, a note's start is called its onset. This is not always obvious: percussion, vocals, and overlapping instruments can hide or blur onsets. Computers do not hear like humans, so they must extract clues from the sound itself.
Onset detection algorithms look for sudden changes in the audio, such as:
- Increases in overall energy (the signal gets louder)
- Shifts in the energy distribution across frequencies (spectral flux)
- Changes in the phase of the signal
- Jumps in detected pitch, if pitch can be reliably tracked
Simple methods might flag every bump in volume, but this brings many false positives: a drum hit or noise could be confused for a melodic note. More advanced systems analyze the spectral content—how much energy is in each frequency band—and use machine learning or psychoacoustic rules to better match human perception [2]. For example, onset detection in a solo flute piece can be quite reliable, but in a dense rock mix, even state-of-the-art systems may miss or double-trigger some notes.
Pitch detection estimates the note for each onset
Once the system knows where a note begins, it needs to determine which note was played. This is the job of pitch detection algorithms. These work by analyzing the segment of audio following each detected onset.
There are two main families of pitch detection:
- Time-domain methods (like autocorrelation or the YIN algorithm) look for repeating patterns in the waveform. These work well for clean, monophonic (single-note) signals but struggle with chords or overlapping notes.
- Frequency-domain methods (like the periodogram, harmonic product spectrum, or cepstral analysis) convert the signal into its frequency components using the Fast Fourier Transform (FFT). These can better handle polyphonic (multi-note) passages, but are more computationally demanding and still face limits as notes overlap closely in pitch [3].
Both approaches face errors. For example, simple zero-crossing methods can mistake overtones or noise for the fundamental pitch, producing octave errors or spurious results [3]. More sophisticated algorithms combine time and frequency analysis, or use dynamic programming to track pitch changes smoothly across time.
A worked example: transcribing a piano melody
Suppose you record a simple piano melody at home and want to generate sheet music from it. Here’s how a typical transcription pipeline would approach this, step by step:
- The system first scans your recording for onsets—moments where a new note likely begins—using spectral flux. This method picks up sudden jumps in energy across frequency bands, such as when a piano hammer strikes a string.
- For each onset, the system grabs a short audio slice (for example, 50 ms) and runs an autocorrelation pitch detector. Piano note attacks are almost perfectly periodic in this window, so this technique reliably pulls out the fundamental frequency for single, clear notes.
- The detected frequency is mapped to the nearest piano key. The system logs this as a discrete note and writes it into a notation file, building up a digital score, one event at a time.
- The same process repeats for each onset, reconstructing the melody as a series of timestamped notes.
When the melody features overlapping notes—chords or pedal sustain—the autocorrelation method breaks down. The system must switch to a frequency-domain detector, looking for peaks in the spectrum that align with the combined harmonics of multiple piano notes. This step is more error-prone, especially when the notes are tightly packed in pitch, as the harmonics blend and the peaks become less distinct.
This example shows the method in principle; for benchmarks and accuracy figures on real piano recordings, see Hawthorne et al. (2018) and the MAPS dataset results in the Method and Sources.
The tools that do this, and how well
Current automatic music transcription tools combine onset detection with pitch detection algorithms, but their effectiveness depends sharply on the instrument and arrangement. For monophonic lines—think a solo saxophone or a cappella singer—most pitch detectors (like zero-crossing or autocorrelation-based methods) can track notes with few errors, since the waveform contains only one pitch at a time [3]. As soon as you add polyphony, such as a dense piano chord, even frequency-domain approaches built on FFTs or periodograms start to misfire: overlapping harmonics from multiple notes blur together, making reliable separation much harder [3].
Notation software that imports MIDI or attempts direct audio-to-score conversion is only as good as the clarity of the input. Clean, single-instrument MIDI tracks often transcribe with high accuracy. But as soon as you throw in the kind of noisy, layered material found in live jazz improvisation or birdsong (famously transcribed by Messiaen directly by ear), these tools struggle—usually outputting partial or error-prone results [1]. The bottleneck isn't only in onset or pitch detection in isolation, but in how well the tool can untangle overlapping frequencies and ambiguous note attacks.
How this changes what you can do
Knowing how automatic transcription breaks down—onsets first, then pitch—gives you leverage in real workflows. For solo flute or unaccompanied voice, most open-source systems will get you clean, nearly error-free lines straight out of the box; you’ll spend more time fixing expressive markings than actual notes. Move to a guitar solo, and the picture changes: bends and slides tend to confuse pitch estimators, so expect to nudge a handful of notes per phrase, especially in rapid runs. For a dense jazz combo, even the best current models (see the experiments linked above) will output a draft that’s more like a highlighter pass: you’ll recognize the structure, but you’ll still be listening and cross-checking chords and layered lines by hand.
This division isn’t just theoretical. It shows up when you’re deciding whether to trust a one-click plugin or to budget extra hours for manual cleanup—and how much you’ll need to explain to a collaborator who’s expecting “sheet music” from a mix. If you need full-score accuracy for publishing or arranging, every run ends with at least one pass by a skilled musician; no software, open or commercial, matches human ears on ensemble music yet. That line’s not moving fast, either, judging by the benchmarks cited in the model section.
Questions readers ask
How does onset detection differ between instruments?
Onset detection is often more reliable for percussive instruments (like piano or drums) because their note starts are sharp and distinct. For sustained or legato instruments (like strings or voice), onsets can be harder to detect, requiring more advanced signal analysis.
Can automatic transcription handle polyphony (chords)?
Frequency-domain pitch detection can estimate multiple pitches at once, but accuracy drops as the number of overlapping notes increases or if the notes are close in pitch. Chords on piano or guitar are much harder to transcribe automatically than solo melodies.
Does the system work for music with vocals and accompaniment?
Mixes with both vocals and instruments are challenging for automatic transcription. Onset and pitch detectors may confuse overlapping harmonics, leading to missed notes or incorrect assignments. Manual correction is usually needed for complex mixes.