Audio Feature Engineering

A detection or separation model only sees what the features give it, so audio work starts by turning waves into numbers. The page moves from the Fourier transform to a first read on speech feature extraction, then a Learning from Audio series, then spectrograms, and ends on Mel-frequency cepstral coefficients (MFCC).

The same notes are in Basics, Digital Signal Processing (DSP), Fourier Transform, and SIGNAL PROCESSING NN (FFT, WAVELETS, SHAPELETS).

Every audio feature below starts from the Fourier transform. DataMListic's Fourier Transform Formula Explained on YouTube is the short video on the formula itself.

With the transform in place, Read this first, by Jonathan Hui — the speech recognition feature extraction pipeline end to end: A/D, boost, Mel-frequency cepstral coefficients (MFCC), Perceptual Linear Prediction (PLP), Hanning vs Hamming, window, discrete Fourier transform (DFT), IDFT.

The Learning from Audio series by mlearnere then builds the same features one step at a time. It starts with Learning from Audio: Wave Forms, then stays in the time domain with Learning from Audio: Time Domain Features, which argues that while deep learning often works in the frequency domain, features extracted from the amplitude of each sample still describe the wave form's properties. It crosses into frequency with Learning from Audio: Fourier Transformation, reaches Learning from Audio: The Mel Scale, Mel Spectrograms, and Mel Frequency Cepstral Coefficients, and ends with Learning from Audio: Pitch and Chromagrams, which goes beyond frequency over time once the spectrogram and MFCCs are understood.

The spectrogram is the feature most audio models actually consume, so it gets its own reading. Understanding the Mel Spectrogram is the first explanation. 2 is Getting to Know the Mel Spectrogram: read this short post if you want to be like Neo and know all about the Mel Spectrogram, or maybe not all, but at least a little. 3 is Thomas Christiansen's beginner's guide to visualizing audio as a spectrogram in Python, treating audio as something to see rather than only hear. 4 is Christopher Lewis on how to create and understand Mel-spectrograms, where the horizontal axis is time, the vertical axis is frequency, and color intensity is the amplitude of a frequency at a point in time. Frames of a spectrogram also change over time, and DataMListic's Spectral Features - Deltas and Delta-Deltas Explained, delta deltas, covers those first and second differences.

The last step compresses the Mel spectrogram into Mel-frequency cepstral coefficients (MFCC). DataMListic's Mel Frequency Cepstral Coefficients (MFCC) Explained on YouTube is the video for it.