Feature Engineering

Raw columns are rarely what a model needs, so this page is the work of building features, filling gaps, and measuring how alike two rows are. It moves from feature engineering and imputing, through representation learning, into similarity, distance, and the distance tools.

FEATURE ENGINEERING

Building a feature starts with looking at the data, and then with one guide per data type: numbers, categories, and text. The same notes are in Auto Feature Engineering, Box Cox, and Feature engineering.

Sunil Ray's comprehensive guide to data exploration is the starting point, a tutorial that comprises missing value imputation, outliers, feature engineering, and variable creation: Vidhya on FE, anomalies, engineering, imputing.

From there, Dipanjan's Understanding Feature Engineering series takes one data type per part. Part 1, Continuous Numeric Data, starts from the idea that data makes the world go round and is Many types of FE, including log and box cox transform - a very useful explanation. Part 2 moves from continuous numbers to categorical data, the discrete kind of structured data: Categorical Data, where the hashing is really cool, and the same article is the Dummy variables and feature hashing reference. Part 3, Traditional Methods for Text Data, is Text data - unigrams, bag of words, N-grams (2,3,..), tfidf matrix, cosine_similarity(tfidf) ontop of a tfidf matrix, unsupervised hierarchical clustering with similarity measures on top of (cosine_similarity), LDA for topic modelling in sklearn - pretty awesome, Kmeans(lda),. That is Dipanjan feature engineering across cont numeric, categorical, and traditional methods.

Part 4 is the hands-on, intuitive approach to deep learning methods for text data (Word2Vec, GloVe and FastText), for turning noisy, unstructured text into structured, vectorized formats any machine learning algorithm can use. It is both Deep learning data for FE and Word embedding using keras, continuous BOW - CBOW, SKIPGRAM, word2vec - really good. Christine Doig's PyGotham 2015 Introduction to Topic Modeling in Python is Topic Modelling - a fantastic slide show about topic modelling using LDA etc.

The same notes are in LDA (Latent Dirichlet Allocation).

For quick reference, Dipanjan on feature engineering is parts 1 - cont numeric, 2 - categorical, and 3 for text. Categories also need encoders beyond dummies, and the two Python packages for that are Target encoding git and Category encoding git.

FEATURE IMPUTING

A built feature is only usable if its gaps are filled. The same data exploration guide by Sunil Ray covers missing value imputation alongside outliers and variable creation: Vidhya on FE, anomalies, engineering, imputing. The library for doing it is the Python package Fancy impute.

REPRESENTATION LEARNING

Hand-built and imputed features have an alternative: learning the representation itself. Representation Learning with Contrastive Predictive Coding is the paper that proposes a universal unsupervised approach to extract useful representations from high-dimensional data, since unsupervised learning has not seen the adoption supervised learning has.

SIMILARITY

Once rows are vectors, whether hand-built or learned, the next question is how alike two of them are. The same notes are in Recommender Systems and String Matching.

Cosine similarity is the default measure. Christian S. Perone's Cosine Similarity for Vector Space Models (Part III) is the continuation of his TF-IDF tutorial (Part I and Part II), and it is the Cosine similarity tutorial. The Stack Exchange question Cosine vs dot product notes that cosine similarity is just the dot product scaled by the product of the magnitudes, and asks when each makes the better distance metric; the same Perone tutorial on its https address is Cosine vs dot product 2. When speed matters, Fast cosine similarity implementation is the Stack Overflow thread on speeding up scipy's cosine distance with numpy and cython.

Words and strings need their own measures. Edit distance similarity is one. Diff lib similarity and soundex is the question about a Python library to identify the similarity between two words or sentences, for comparing audio-to-text output against known words such as a person or company name. Selva Prabhakaran's Gensim tutorial, for the package billed as 'Topic Modeling for Humans' that also works with word vector models such as Word2Vec and FastText, covers Soft cosine and cosine. Correlation is a similarity measure too: MachineLearningMastery's How to Calculate Correlation Between Variables in Python is the Pearson also used to detect similar vectors link.

Distance

Similarity has a mirror image in distance, and some models need the distance form. The same notes are in Dynamic Time Warping (DTW).

MachineLearningMastery's 4 Distance Measures for Machine Learning is the Mastery on distance formulas link. It covers the role of distance measures, then Hamming Distance, Euclidean Distance, Manhattan Distance (Taxicab or City Block), and Minkowski Distance. The bridge back to the section above is that cosine distance = 1 - cosine similarity. Geo data needs one more measure: working with Geo data is fun once the data is clean and loaded to a dataframe or an array, and the real work starts when you have to find distances between two coordinates or cities and generate a distance matrix, which is where Haversine distance with Scipy spatial and pandas comes in.

Distance Tools

For geographic distance at scale, the tool is GeoPandas, whose GeoPandas 1.1.4 documentation is the reference.

TF-IDF notes from this page are on the TF-IDF page.