Turning Sound Into Light

I've been building a little personal project on the side: a visualizer that listens to whatever's playing on my machine and reacts to it. The graphics are the easy part. The bit I find interesting is the question underneath it. You start with nothing but a stream of raw audio samples, so how do you actually recover the bass, the kick drum, and the notes being played, fast enough to draw 90 frames a second?

Here's the gist.

Fourier: the exact core

All you start with is $x[n]$ , air pressure sampled 48,000 times a second. That tells you when the sound is loud, but nothing about which frequencies are in it. To get at those you change basis with the Discrete Fourier Transform:

X[k] \;=\; \sum_{n=0}^{N-1} x[n]\, e^{-j 2\pi k n / N}

Each $|X[k]|$ is exactly how much energy sits at frequency $f_k = k\,f_s/N$ . This is the one piece that's genuinely exact rather than a heuristic. Done the obvious way it's $O(N^2)$ and far too slow, but the FFT factors it down to $O(N\log N)$ , which is the only reason any of this runs in real time.

Catching the drums

Notes sustain, but drum hits appear out of nowhere. So I look at how much each frequency jumped since the last frame, and only count the rises:

\text{flux}_t \;=\; \sum_{k} \max\!\big(0,\; |X_t[k]| - |X_{t-1}[k]|\big)

That $\max(0,\cdot)$ is the whole trick. A note fading out shouldn't register as a hit, only an attack should, so anything going down gets thrown away. What's left spikes right when something gets struck.

Seeing the notes

The FFT is actually the wrong tool for pitch. Its bins are evenly spaced, but music is logarithmic, since every octave doubles in frequency. So instead I space the bins geometrically, one set per pitch:

f_k \;=\; f_{\min} \cdot 2^{\,k/b}

Fold those down onto the twelve note names, give each one a color, and the visuals light up in the hue of whatever's playing. Play a scale and the colors walk up in order. That's the part I was really after.

Underneath it's all the same move: get the audio into a form where the thing you want falls out on its own. The graphics are the last 5%. The math is the part that's actually listening.