AI Music Models Won't Listen? A Practical Guide to Browser-Side Audio Analysis

Turning a hummed melody into a full song sounds like an elegant AI pipeline: transcribe pitch, rhythm, and chords in the browser, hand them to a text-to-music model to generate the backing track, then layer the melody on top. That's exactly how SongUp AI's Hum to Song feature was designed—using Spotify's open-source Basic Pitch for transcription and Google's Lyria to generate the accompaniment.

But the team found that the second step was broken. Lyria doesn't accept audio input and can only be driven by prompts, which are unreliable. On a test track ("Twinkle" in A major at 96 BPM), the model followed the requested key only about a third of the time, the chord loop often started from its own bar, and one backing track even opened the verse on a IV chord. Tempo was mostly accurate, but there were deviations to 95 and 100 BPM.

The melody came in as planned at bar 5, in A, at 96 BPM—and the result was the melody and the band each doing their own thing. So the team changed strategy: stop trusting the prompt and instead measure the audio that comes back. All analysis happens in the browser's OfflineAudioContext, with no sample downloads and reproducible results.

Tempo Detection: A Comb Filter Locked Around the Target BPM

The first step in rhythm analysis is computing an onset envelope (with roughly 10 ms frame hop), then scoring each candidate tempo at each phase by accumulating envelope values. The search range is constrained to within ±8% of the requested BPM—because a free-tempo search on pop music easily locks onto half or double time.

The key fix was normalizing the score per beat. Without normalization, double tempo almost always wins. The code also handles half and double time, but only adopts them when their score is clearly stronger (more than 1.35×), since the melody should follow the tempo it was written at.

function findTempo(f: Features, targetBpm: number) {
  const evalBpm = (bpm: number) => {
    const period = 60 / bpm / f.frameSec;
    let bestPhase = 0, best = -1;
    for (let ph = 0; ph < period; ph += 0.5) {
      let s = 0;
      for (let p = ph; p < f.onset.length; p += period) s += sampleAt(f.onset, p);
      if (s > best) { best = s; bestPhase = ph; }
    }
    // Normalize per beat so faster tempos don't win just by having more beats
    return { bpm, phaseFrames: bestPhase, strength: best / (f.onset.length / period) };
  };
  // ...search within ±8% of target, then check half and double time
}

The onset envelope is the foundational signal for rhythm analysis. It computes the rate of change of audio energy frame by frame, highlighting the transients at note starts, typically as the positive half-wave rectification of $E[n] - E[n-1]$. The article uses a frame hop of about 10 ms, meaning roughly 100 frames per second, which ensures beat localization resolution on the order of 10 ms—enough to distinguish subtle tempo differences in pop music. The onset envelope is sensitive to transients like drum hits and plucked strings, but responds weakly to sustained tones (such as long string notes). This is why the subsequent scoring can make the true beat positions stand out instead of being muddled by the sustained energy of chords.

Where Does the Verse Start? Judge by Low-Frequency Energy, Not Loudness

Finding the downbeat (the beat every 4 that carries the kick drum and chord changes) is relatively straightforward: each candidate offset is scored by low-frequency energy and chroma change. The hard part is determining when the intro ends.

The most intuitive rule—"find the biggest jump in loudness"—failed on real backing tracks: the drum fill in the last intro bar was the biggest jump, one bar earlier than the correct position. Drum fills are loud but have no low end. So the team specified that a bar only counts as the start of the verse when both loudness and kick/bass energy reach the level present for most of the song.

Reference values are taken from the median of mid-song bars, so a loud final chorus doesn't skew the baseline.

Chroma change is a feature commonly used in key and section analysis. The chroma vector folds spectral energy onto 12 pitch classes, ignoring octave information. The cosine or Euclidean distance between chroma vectors of adjacent frames or bars measures how much the harmonic content changes. When identifying downbeats, chord changes are often accompanied by significant chroma shifts, so including chroma change as one of the scoring terms effectively distinguishes true section boundaries from mere drum fills or transitions—the latter have a sudden volume spike but no change in harmonic progression. This dual perspective of loudness and harmony avoids over-reliance on amplitude.

Key: Correlate Across 12 Rotations

Weight the melody's pitch classes by note duration, rotate by -6 to +5 semitones, and correlate each rotation with the average chroma of the backing track during the verse. The requested key gets a tiny 0.02 bonus as a tiebreaker. The best rotation gives the transposition amount.

This also explains why the final song might be a few semitones away from the pitch you hummed—the team would rather shift your melody than have it clash with the band.

Chords: One Goertzel Filter per Semitone

The model often plays its own chords rather than the requested ones. To keep the melody out of conflict, you need to know what the band is playing every half bar. FFT analysis falls short here: at an 11 kHz sample rate with 1024-sample frames, one frequency bin is about 11 Hz wide—wider than the spacing between adjacent notes below A3.

So the harmonic map instead uses one Goertzel filter per semitone from E2 to C6. Goertzel is a single-bin DFT that is cheap to compute for about 45 specific frequencies and can be tuned exactly to each note.

function goertzel(x: Float32Array, from: number, to: number, freq: number, rate: number) {
  const w = (2 * Math.PI * freq) / rate;
  const coeff = 2 * Math.cos(w);
  let s1 = 0, s2 = 0;
  for (let i = from; i < to; i++) {
    const s0 = x[i] + coeff * s1 - s2;
    s2 = s1; s1 = s0;
  }
  return s1 * s1 + s2 * s2 - coeff * s1 * s2; // power at that frequency
}

Fold the 45 powers into 12 pitch classes, normalize to a maximum of 1, and you get the chroma vector for each half bar.

The Goertzel algorithm is an efficient single-frequency implementation of the DFT that doesn't compute the whole spectrum—it only focuses on one target frequency. Its recurrence is $s[n] = x[n] + 2\cos(\omega)s[n-1] - s[n-2]$, and the power at that frequency is obtained as $s[N-1]^2 + s[N-2]^2 - 2\cos(\omega)s[N-1]s[N-2]$. When you only need to detect a few dozen specific pitches, Goertzel's computational cost is far lower than FFT, and its frequency resolution isn't limited by frame length—it can be aimed precisely at the center frequency of each semitone. For about 45 semitones from E2 to C6, running 45 Goertzel passes per half bar is fully acceptable in real time within the browser's OfflineAudioContext.

Phrase Shifting and Conflict Repair

The final two steps align the melody with the band:

• Phrase shifting: Try starting the melody at the bar where the band enters, or up to 3 bars later. Score by the proportion of melody duration that falls on pitch classes the band is playing. Only shift if it improves the score by at least 5%; otherwise keep the natural entry.
• Conflict repair: Only move a note when its pitch class is nearly absent in the band (<0.3) and a note one semitone away is strongly present (≥0.55). Move by one semitone; whole-tone neighbors stay put, so passing tones are preserved. Only minor-second friction gets repaired.

On a real backing track, these steps raised the consonance between melody and band from 0.50 to 0.63. The team also wrote a unit test suite with 60 checks to prevent future tweaks from silently breaking the math.

The consonance score here is defined as the proportion of melody duration where the melody note falls on a pitch class the backing track is playing. This is a simplified measure of harmonic consonance—it doesn't distinguish between perfect fifths, major thirds, or other intervals by degree of consonance, only whether it's a "hit" or not. Going from 0.50 to 0.63 means about 13% of the melody duration shifted from clashing with the backing to landing on chord tones. Though crude, this measure is directionally clear, easy to compute, and highly correlated with the human perception of "not clashing." The team locked down the math with 60 unit tests to ensure future tweaks wouldn't silently lower that score.

Three Tips for Generative Audio Developers

1. Treat model output as data to be measured, not a contract. A prompt is a request; the audio is the fact.
2. Naive prompts beat clever prompts. A simple looped chord prompt ("A | D | D | E, I–IV–IV–V in A major") was followed better than a detailed bar-by-bar timeline.
3. Let users hear the cheap step first. Before incurring any cost, Hum to Song plays the transcribed notes alone: "This is the melody we heard." If the transcription is wrong, no one pays to find out.

The general value of this approach: when a generative model can't be precisely controlled, post-hoc measurement is the source of reliability. For any team building generative audio products, treating model output as data to analyze rather than a promise to believe is the more pragmatic engineering path.

One-line Summary

Text-to-music models are unreliable; browser-side audio analysis can measure and fix melody-to-backing alignment.

Paragraph Summary

SongUp AI's Hum to Song feature uses Basic Pitch to transcribe hummed melodies and Lyria to generate backing tracks, but found Lyria's adherence to key, chords, and section structure highly unreliable—following the requested key only about a third of the time in tests. The team shifted to measuring the returned audio: implementing per-beat-normalized comb-filter tempo detection, intro-boundary detection based on both loudness and low-frequency energy, key detection via chroma rotation correlation, and per-semitone chord analysis with Goertzel filters—all in the browser's OfflineAudioContext. Through phrase shifting and conservative semitone conflict repair, they raised the consonance between melody and backing from 0.50 to 0.63. The core takeaway: treat generative model output as data to be measured, not a contract—and let users hear the free transcribed melody before paying for generation.