Autoencoders

An autoencoder learns to squeeze data into a small code and rebuild it, which makes it a nonlinear answer to the dimensionality reduction that PCA and ICA already do. This page starts with the basic autoencoder, its comparison to PCA, and its reconstruction uses such as denoising and anomaly detection, then moves to variational autoencoders.

The reconstruction error is also what makes autoencoders useful for spotting unusual rows. The same notes are in Anomaly Detection and Fraud Detection.

We often use ICA or PCA to extract features from the high-dimensional data, and the autoencoder is another way to achieve the same purpose in deep learning. How to use AE for dimensionality reduction + code shows it using keras’ functional API. The broad tour is the Keras.io blog post about AE’s, Building Autoencoders in Keras - regular, deep, sparse, regularized, cnn, variational. A keras.io replicate post but explains AE quite nicely, and Examples of vanilla, multi layer, CNN and sparse AE’s used to follow it; that address is kept at the end of the page.

Convolutional autoencoders get their own walkthrough. Another example of CNN-AE is Autoencoders, Deep Learning bits #1, featuring data compression, image reconstruction and segmentation (with examples!). Another AE tutorial sat beside it.

The comparison with PCA is where the idea becomes clear. Hinton’s coursera course on PCA vs AE, basically some info about what PCA does - maximizing variance and projecting and then what AE does and can do to achieve similar but non-linear dense representations; the lecture address is kept at the end. To see the difference, A great tutorial on how does the clusters look like after applying PCA/ICA/AE is a Kaggle notebook doing 2D visualization with PCA and ICA against autoencoders on the Mercedes-Benz Greener Manufacturing data. Another great presentation on PCA vs AE, summarized in the KPCA section of this notebook, is kept at the end too. +another one is the Urtasun and Zemel CSC 411 lecture on PCA and autoencoders, and +StackExchange asks what the difference between factor analysis and autoencoders is, and when to use one and not the other.

From comparison back to code: autoencoders have become an intriguing tool for data compression, and implementing them in Keras is surprisingly straightforward, which is Trujillo Herman's point in the Autoencoder tutorial with python code and how to encode after. The same encoder-decoder shape carries over to sequences, and Jason Brownlee's mastery post shows how to develop an encoder-decoder model with attention in Keras, where attention addresses the limits of the architecture on long sequences. When the code has to be very small, Git code for low dimensional auto encoder is a Keras model training method that allows to get good results for heavily dimensionality reduction.

Denoising is the same idea at language scale. Bart denoising AE, sequence to sequence pre training for NL generation translation and comprehension. An attention based seq to seq auto encoder has its code in the git repo that trains a machine to write Katakana using the sequence-to-sequence technique; its write-up is kept at the end.

The reconstruction error then becomes a detector: AE for anomaly detection, fraud detection is Venelin Valkov's credit card fraud detection using autoencoders in Keras, part VII of TensorFlow for Hackers.

Variational AE

A plain autoencoder compresses, but it cannot be asked to generate something specific, and the variational autoencoder builds on the material above to fix that.

The Simple explanation is still marked Unread; it starts from GANs that generate random faces and asks how to generate a specific one. Pixel art VAE is the VAE category of Machine Learning Insights. Unread - another VAE sat next to it. Pixel GAN VAE is Synced on the PixelGAN Autoencoder, whose generative path is a convolutional autoregressive network on pixels conditioned on a latent code, with a GAN imposing a prior on that code. Disentangled VAE - improves VAE, and the video is Arxiv Insights on variational autoencoders.

At language scale, Optimus - pretrained VAE is the first large-scale pre-trained VAE language model. Its paper is Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space, and the Microsoft blog places Optimus next to FQ-GAN and Prevalent as advances in large-scale deep generative models.

See also Cleaning Data With AI Denoisers (October 2024), on reducing noise in images, audio, and video beyond traditional filters and statistical methods.

For a longer look inside the model, the Towards Data Science post deep-inside-autoencoders-7e41f319999f is at https://towardsdatascience.com/deep-inside-autoencoders-7e41f319999f.

Deprecated links

These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.

- Towards Data Science: how-to-reduce-image-noises-by-autoencoder-65d5e6de543. This address no longer opens: https://towardsdatascience.com/how-to-reduce-image-noises-by-autoencoder-65d5e6de543 - Towards Data Science: teaching-a-variational-autoencoder-vae-to-draw-mnist-characters-978675c95776. This address no longer opens: https://towardsdatascience.com/teaching-a-variational-autoencoder-vae-to-draw-mnist-characters-978675c95776 - Examples of vanilla, multi layer, CNN and sparse AE’s. This address no longer opens: https://wiseodd.github.io/techblog/2016/12/03/autoencoders/ - Hinton’s coursera course. This address no longer opens: https://www.coursera.org/learn/neural-networks/lecture/JiT1i/from-pca-to-autoencoders-5-mins - Another great presentation on PCA vs AE,. This address no longer opens: https://web.cs.hacettepe.edu.tr/~aykut/classes/fall2016/bbm406/slides/l25-kernel_pca.pdf - Attention based seq to seq auto encoder. This address no longer opens: https://wanasit.github.io/attention-based-sequence-to-sequence-in-keras.html