> For the complete documentation index, see [llms.txt](https://www.mlcompendium.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.mlcompendium.com/deep-learning/deep-neural-audio.md).

# Deep Neural Audio

Sound has to become something a network can read, usually a spectrogram, before a model can recognize speech or classify a sound. The page follows that path: how speech recognition and audio features work, a series that builds audio deep learning step by step, then pretrained speech models like Wav2Vec and Whisper, and finally audio generation with Neural Amp Modeler and the Echo State Network idea.

The same notes are in [Other Tools](/predictive-ml/audio-algorithms.md#other-tools) and [Speech](/generative-ai/speech.md).

The first question is what a speech system does. [Audio Deep Learning Made Simple: Automatic Speech Recognition (ASR), How it Works](https://towardsdatascience.medium.com/audio-deep-learning-made-simple-automatic-speech-recognition-asr-how-it-works-716cfce4c706) is Ketan Doshi on the class of applications, like voice assistants, that start with a clip of spoken audio and extract the words as text. Beyond frequency over time, [Learning from Audio: Pitch and Chromagrams](https://towardsdatascience.medium.com/learning-from-audio-pitch-and-chromagrams-5158028a505) goes deeper into structures past the spectrogram and the Mel scale.

The fuller walk is Audio deep learning made simple by Ketan Doshi, a series meant to be read in order:

1. [State-of-the-Art Techniques](https://towardsdatascience.medium.com/audio-deep-learning-made-simple-part-1-state-of-the-art-techniques-da1d3dff2504) *(What is sound and how it is digitized. What problems is audio deep learning solving in our daily lives. What are Spectrograms and why they are all-important.)*
2. [Why Mel Spectrograms perform better](https://towardsdatascience.medium.com/audio-deep-learning-made-simple-part-2-why-mel-spectrograms-perform-better-aad889a93505) *(Processing audio data in Python. What are Mel Spectrograms and how to generate them)*
3. [Data Preparation and Augmentation](https://towardsdatascience.medium.com/audio-deep-learning-made-simple-part-3-data-preparation-and-augmentation-24c6e1f6b52) *(Enhance Spectrograms features for optimal performance by hyper-parameter tuning and data augmentation)*
4. [Sound Classification](https://towardsdatascience.medium.com/audio-deep-learning-made-simple-sound-classification-step-by-step-cebc936bbe5) *(End-to-end example and architecture to classify ordinary sounds. Foundational application for a range of scenarios.)*
5. [Beam Search](https://towardsdatascience.medium.com/foundations-of-nlp-explained-visually-beam-search-how-it-works-1586b9849a24) *(Algorithm commonly used by Speech-to-Text and NLP applications to enhance predictions)*

After building a pipeline by hand, pretrained speech models skip most of it. For *Wav2Vec — paper*, the [*YouTube*](https://www.youtube.com/watch?v=fMqYul2TvBE) video explains the Wav2Vec paper, the framework for self-supervised learning of speech representations. [Whisper](https://github.com/openai/whisper) is openai/whisper, robust speech recognition via large-scale weak supervision, and on top of it [ZAC](https://github.com/jumon/zac) (Zero-shot Audio Classification using Whisper) allows you to assign audio files to ANY class you want without training.

Models can also produce sound, not only label it. [Neural Amp Modeler](https://www.neuralampmodeler.com/) — an open-source project that uses deep learning to create models of guitar amplifiers and pedals with state-of-the-art accuracy. Its [Training notebook](https://colab.research.google.com/github/sdatkinson/neural-amp-modeler/blob/main/bin/train/easy_colab.ipynb#scrollTo=zrXbQY7vjZjk) is where you train a model, and [NAM uses WaveNet](https://arxiv.org/abs/1609.03499), the paper "WaveNet: A Generative Model for Raw Audio".

The author's last note is the Echo State Network. The [YouTube](https://www.youtube.com/watch?v=uF4i9_7IQlI) talk is Farshad Noravesh's part 1 on echo state networks. [Gentle introduction to Echo State Networks](https://towardsdatascience.medium.com/gentle-introduction-to-echo-state-networks-af99e5373c68) is Madalina Ciortan on what they are, why and when to use one, and a simple implementation in Python, and the [GitHub code](https://github.com/ciortanmadalina/EchoStateNetwork/blob/master/EchoStateNetwork.ipynb) is the notebook for it. Back on the amp side, [Neural Amp Modeler users](https://www.neuralampmodeler.com/users) is the page for getting into NAM, from making your own models to connecting with other musicians.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://www.mlcompendium.com/deep-learning/deep-neural-audio.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.