Deep Neural Audio

Sound has to become something a network can read, usually a spectrogram, before a model can recognize speech or classify a sound. The page follows that path: how speech recognition and audio features work, a series that builds audio deep learning step by step, then pretrained speech models like Wav2Vec and Whisper, and finally audio generation with Neural Amp Modeler and the Echo State Network idea.

The same notes are in Other Tools and Speech.

The first question is what a speech system does. Audio Deep Learning Made Simple: Automatic Speech Recognition (ASR), How it Works is Ketan Doshi on the class of applications, like voice assistants, that start with a clip of spoken audio and extract the words as text. Beyond frequency over time, Learning from Audio: Pitch and Chromagrams goes deeper into structures past the spectrogram and the Mel scale.

The fuller walk is Audio deep learning made simple by Ketan Doshi, a series meant to be read in order:

1. State-of-the-Art Techniques (What is sound and how it is digitized. What problems is audio deep learning solving in our daily lives. What are Spectrograms and why they are all-important.)

2. Why Mel Spectrograms perform better (Processing audio data in Python. What are Mel Spectrograms and how to generate them)

3. Data Preparation and Augmentation (Enhance Spectrograms features for optimal performance by hyper-parameter tuning and data augmentation)

4. Sound Classification (End-to-end example and architecture to classify ordinary sounds. Foundational application for a range of scenarios.)

5. Beam Search (Algorithm commonly used by Speech-to-Text and NLP applications to enhance predictions)

After building a pipeline by hand, pretrained speech models skip most of it. For Wav2Vec — paper, the YouTube video explains the Wav2Vec paper, the framework for self-supervised learning of speech representations. Whisper is openai/whisper, robust speech recognition via large-scale weak supervision, and on top of it ZAC (Zero-shot Audio Classification using Whisper) allows you to assign audio files to ANY class you want without training.

Models can also produce sound, not only label it. Neural Amp Modeler — an open-source project that uses deep learning to create models of guitar amplifiers and pedals with state-of-the-art accuracy. Its Training notebook is where you train a model, and NAM uses WaveNet, the paper "WaveNet: A Generative Model for Raw Audio".

The author's last note is the Echo State Network. The YouTube talk is Farshad Noravesh's part 1 on echo state networks. Gentle introduction to Echo State Networks is Madalina Ciortan on what they are, why and when to use one, and a simple implementation in Python, and the GitHub code is the notebook for it. Back on the amp side, Neural Amp Modeler users is the page for getting into NAM, from making your own models to connecting with other musicians.