# The Machine & Deep Learning Compendium
Covering approximately **500 topics**, the ML & DL Compendium includes summaries, links, and articles across a wide array of subjects, including LLMs. These range from modern machine learning algorithms and deep learning techniques to specialized areas like NLP, audio processing, computer vision (classic and deep), time-series analysis, anomaly detection, and graphs. It also deep dives into strategic themes like data science management, team building, and practical essentials like product management, design, and technology stacks from a data science perspective.
The ML & DL Compendium is completely open and now lives on [GitHub](https://github.com/orico/www.mlcompendium.com/) (please star it!). Driven by my belief in knowledge-sharing and education, this project will always remain not-for-profit and free.
{% embed url="" %}
The ML & DL Compendium Official GitHub repo
{% endembed %}
The Machine & Deep Learning Compendium began as a personal project—a curated list of resources I maintained in a private Google document for my own learning. That document has now evolved into this new interface, and I’m excited to share it as an educational tool to help others learn and connect with the brilliant authors I’ve summarized, quoted, and referenced.
I envision it as a go-to resource for learners of all levels—whether you're an industry data scientist, an academic, or just starting out. It’s designed to save you countless hours of searching and filtering through content, providing a streamlined path to invaluable authors and resources you can further support.
Let’s work together to support the community, amplify the voices of authors, and democratize education! If you spot something that could be improved, feel free to contribute via [GitHub](https://github.com/orico/www.mlcompendium.com/tree/master) or [reach out to](https://www.linkedin.com/in/cohenori/) me directly.
{% embed url="" %}
The ML Compendium Article
{% endembed %}
Many Thanks, \
Dr. Ori Cohen
[My Website](https://www.oricohen.com/) |[ Medium](https://medium.com/@cohenori) |[ LinkedIn](https://www.linkedin.com/in/cohenori/) | [ML Compendium](http://www.mlcompendium.com/) | [Ops Compendium](https://www.opscompendium.com/) | [State of GenAI](https://stateofgenai.com/) | [State Of MLOps](https://stateofmlops.com/) |
# Thanks Page
I would like to thank the following contributors: [Samuel Jefroykin](https://www.linkedin.com/in/samueljefroykin/), [Sefi Keller](https://www.linkedin.com/in/sefikeller/?originalSubdomain=il)
# Overview
# Model Families
1. [A wonderful introduction into machine learning, and how to choose the right algorithm or family of algorithms for the task at hand.](https://blogs.sas.com/content/subconsciousmusings/2017/04/12/machine-learning-algorithm-use/?utm_source=facebook\&utm_medium=cpc\&utm_campaign=analytics-global\&utm_content=US_interests-conversions)
2. [Stanford cs221](https://stanford.edu/~shervine/teaching/cs-221/) - reflex, variable, state, logic
# Weakly Supervised
[Text classification with extremely small datasets](https://towardsdatascience.com/text-classification-with-extremely-small-datasets-333d322caee2), relies heavily on feature engineering methods such as number of hashtags, number of punctuations and other insights that are really good for this type of text.
1. A great [review paper](https://pdfs.semanticscholar.org/3adc/fd254b271bcc2fb7e2a62d750db17e6c2c08.pdf) for weakly supervision, discusses:
1. Incomplete supervision
2. Inaccurate
3. Inexact
4. Active learning
2. [Stanford on](https://dawn.cs.stanford.edu/2017/07/16/weak-supervision/) weakly
3. [Stanford ai on snorkel](http://ai.stanford.edu/blog/weak-supervision/)
4. [Intro to Snorkel](https://medium.com/@towardsai/data-centric-ai-with-snorkel-ai-the-enterprise-ai-platform-a8ed0803c24c)
5. [Hazy research on weak and snorkel](https://hazyresearch.github.io/snorkel/blog/ws_blog_post.html)
6. [Out of distribution generalization using test-time training](https://arxiv.org/abs/1909.13231) - "Test-time training turns a single unlabeled test instance into a self-supervised learning problem, on which we update the model parameters before making a prediction on this instance. "
7. [Learning Deep Networks from Noisy Labels with Dropout Regularization](https://arxiv.org/pdf/1705.03419.pdf) - "Large datasets often have unreliable labels—such as those obtained from Amazon’s Mechanical Turk or social media platforms—and classifiers trained on mislabeled datasets often exhibit poor performance. We present a simple, effective technique for accounting for label noise when training deep neural networks. We augment a standard deep network with a softmax layer that models the label noise statistics. Then, we train the deep network and noise model jointly via end-to-end stochastic gradient descent on the (perhaps mislabeled) dataset. The augmented model is overdetermined, so in order to encourage the learning of a non-trivial noise model, we apply dropout regularization to the weights of the noise model during training. Numerical experiments on noisy versions of the CIFAR-10 and MNIST datasets show that the proposed dropout technique outperforms state-of-the-art methods."
8. [Distill to label weakly supervised instance labeling using knowledge distillation](https://arxiv.org/pdf/1907.12926.pdf) - “Weakly supervised instance labeling using only image-level labels, in lieu of expensive fine-grained pixel annotations, is crucial in several applications including medical image analysis. In contrast to conventional instance segmentation scenarios in computer vision, the problems that we consider are characterized by a small number of training images and non-local patterns that lead to the diagnosis. In this paper, we explore the use of multiple instance learning (MIL) to design an instance label generator under this weakly supervised setting. Motivated by the observation that an MIL model can handle bags of varying sizes, we propose to repurpose an MIL model originally trained for bag-level classification to produce reliable predictions for single instances, i.e., bags of size 1. To this end, we introduce a novel regularization strategy based on virtual adversarial training for improving MIL training, and subsequently develop a knowledge distillation technique for repurposing the trained MIL model. Using empirical studies on colon cancer and breast cancer detection from histopathological images, we show that the proposed approach produces high-quality instance-level prediction and significantly outperforms state-of-the MIL methods.”
9. [Yet another article summarising FAIR](https://neurohive.io/en/state-of-the-art/semi-weakly-supervised-learning-increasing-classification-accuracy-with-billion-scale-unlabeled-images/)
# Semi Supervised
1. [Paper review](https://pdfs.semanticscholar.org/3adc/fd254b271bcc2fb7e2a62d750db17e6c2c08.pdf)
2. [Ruder an overview of proxy labeled for semi supervised (AMAZING)](https://ruder.io/semi-supervised/)
3. Self training
1. [Self training and tri training](https://github.com/zidik/Self-labeled-techniques-for-semi-supervised-learning)
2. [Confidence regularized self training](https://github.com/yzou2/CRST)
3. [Domain adaptation for semantic segmentation using class balanced self-training](https://github.com/yzou2/CBST)
4. [Self labeled techniques for semi supervised learning](https://github.com/zidik/Self-labeled-techniques-for-semi-supervised-learning)
4. Tri training
1. [Trinet for semi supervised Deep learning](https://www.ijcai.org/Proceedings/2018/0278.pdf)
2. [Tri training exploiting unlabeled data using 3 classes](https://www.researchgate.net/publication/3297469_Tri-training_Exploiting_unlabeled_data_using_three_classifiers), [paper](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.487.2431\&rep=rep1\&type=pdf)
3. [Improving tri training with unlabeled data](https://link.springer.com/chapter/10.1007/978-3-642-25349-2_19)
4. [Tri training using NN ensemble](https://link.springer.com/chapter/10.1007/978-3-642-31919-8_6)
5. [Asymmetric try training for unsupervised domain adaptation](https://github.com/corenel/pytorch-atda), [another implementation](https://github.com/vtddggg/ATDA), [another](https://github.com/ksaito-ut/atda), [paper](https://arxiv.org/abs/1702.08400)
6. [Tri training git](https://github.com/LiangjunFeng/Tri-training)
5. [Fast ai forums](https://forums.fast.ai/t/semi-supervised-learning-ssl-uda-mixmatch-s4l/56826)
6. [UDA GIT](https://github.com/google-research/uda), [paper](https://arxiv.org/abs/1904.12848), [medium\*](https://medium.com/syncedreview/google-brain-cmu-advance-unsupervised-data-augmentation-for-ssl-c0a6157505ce), medium 2 ([has data augmentation articles)](https://medium.com/towards-artificial-intelligence/unsupervised-data-augmentation-6760456db143)
7. [s4l](https://arxiv.org/abs/1905.03670)
8. [Google’s UDM and MixMatch dissected](https://mlexplained.com/2019/06/02/papers-dissected-mixmatch-a-holistic-approach-to-semi-supervised-learning-and-unsupervised-data-augmentation-explained/)- For text classification, the authors used a combination of back translation and a new method called TF-IDF based word replacing.
Back translation consists of translating a sentence into some other intermediate language (e.g. French) and then translating it back to the original language (English in this case). The authors trained an English-to-French and French-to-English system on the WMT 14 corpus.
TF-IDF word replacement replaces words in a sentence at random based on the TF-IDF scores of each word (words with a lower TF-IDF have a higher probability of being replaced).
1. [MixMatch](https://arxiv.org/abs/1905.02249), [medium](https://towardsdatascience.com/a-fastai-pytorch-implementation-of-mixmatch-314bb30d0f99), [2](https://medium.com/@sanjeev.vadiraj/eureka-mixmatch-a-holistic-approach-to-semi-supervised-learning-125b14e82d2f), [3](https://medium.com/@sshleifer/mixmatch-paper-summary-1995f3d11cf), [4](https://medium.com/@literallywords/tl-dr-papers-mixmatch-9dc4cd217121), that works by guessing low-entropy labels for data-augmented unlabeled examples and mixing labeled and unlabeled data using MixUp. We show that MixMatch obtains state-of-the-art results by a large margin across many datasets and labeled data amounts
2. ReMixMatch - [paper](https://arxiv.org/pdf/1911.09785.pdf) is really good. “We improve the recently-proposed “MixMatch” semi-supervised learning algorithm by introducing two new techniques: distribution alignment and augmentation anchoring”
3. [FixMatch](https://amitness.com/2020/03/fixmatch-semi-supervised/) - FixMatch is a recent semi-supervised approach by Sohn et al. from Google Brain that improved the state of the art in semi-supervised learning(SSL). It is a simpler combination of previous methods such as UDA and ReMixMatch.\
*Image via* [Amit Chaudhary](https://amitness.com/) *wrong credit?* [*let me know*](mailto:ori@oricohen.com)
4. [Curriculum Labeling: Self-paced Pseudo-Labeling for Semi-Supervised Learning](https://arxiv.org/pdf/2001.06001.pdf)
5. [FAIR](https://ai.facebook.com/blog/billion-scale-semi-supervised-learning/) [2](https://ai.facebook.com/blog/mapping-the-world-to-help-aid-workers-with-weakly-semi-supervised-learning/) original, [Summarization of FAIR’s student teacher weak/ semi supervision](https://analyticsindiamag.com/how-to-do-machine-learning-when-data-is-unlabelled/)
6. [Leveraging Just a Few Keywords for Fine-Grained Aspect Detection Through Weakly Supervised Co-Training](https://www.aclweb.org/anthology/D19-1468.pdf)
7. [Fidelity-Weighted](https://openreview.net/forum?id=B1X0mzZCW) Learning - “fidelity-weighted learning” (FWL), a semi-supervised student- teacher approach for training deep neural networks using weakly-labeled data. FWL modulates the parameter updates to a student network (trained on the task we care about) on a per-sample basis according to the posterior confidence of its label-quality estimated by a teacher (who has access to the high-quality labels). Both student and teacher are learned from the data."
8. [Unproven student teacher git](https://github.com/EricHe98/Teacher-Student-Training)
9. [A really nice student teacher git with examples](https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation).
10\. [Teacher student for tri training for unlabeled data exploitation](https://arxiv.org/abs/1909.11233)
# Active Learning
1. If you need to start somewhere start [here](https://www.datacamp.com/community/tutorials/active-learning) - types of AL, the methodology, examples, sample selection functions.
2. A thorough [review paper](http://burrsettles.com/pub/settles.activelearning.pdf) about AL
3. [The book on AL](http://burrsettles.com/pub/settles.activelearning.pdf)
4. [Choose your model first, then do AL, from lighttag](https://www.lighttag.io/blog/active-learning-optimization-is-not-imporvement/)
1. The alternative is Query by committee - Importantly, the active learning method we presented above is the most naive form of what is called "uncertainty sampling" where we chose to sample based on how uncertain our model was. An alternative approach, called Query by Committee, maintains a collection of models (the committee) and selecting the most "controversial" data point to label next, that is one where the models disagreed on. Using such a committee may allow us to overcome the restricted hypothesis a single model can express, though at the onset of a task we still have no way of knowing what hypothesis we should be using.
2. [Paper](https://arxiv.org/pdf/1807.04801.pdf): warning against transferring actively sampled datasets to other models
5. [How to increase accuracy with AL ](http://www.ijcte.org/papers/910-AC0013.pdf)
6. [AL with model selection](http://www.alnurali.com/papers/paper_aaai_2014.pdf) - paper
7. Using weak and strong oracle in AL, [paper](http://publications.lib.chalmers.se/records/fulltext/248447/248447.pdf).
8. [The pitfalls of AL](http://www.kdd.org/exploration_files/v12-02-9-UR-Attenberg.pdf) - how to choose (cost-effectively) the active learning technique when one starts without the labeled data needed for methods like cross-validation; 2. how to choose (cost-effectively) the base learning technique when one starts without the labeled data needed for methods like cross-validation, given that we know that learning curves cross, and given possible interactions between active learning technique and base learner; 3. how to deal with highly skewed class distributions, where active learning strategies find few (or no) instances of rare classes; 4. how to deal with concepts including very small subconcepts (“disjuncts”)—which are hard enough to find with random sampling (because of their rarity), but active learning strategies can actually avoid finding them if they are misclassified strongly to begin with; 5. how best to address the cold-start problem, and especially 6. whether and what alternatives exist for using human resources to improve learning, that may be more cost efficient than using humans simply for labeling selected cases, such as guided learning \[3], active dual supervision \[2], guided feature labeling \[1], etc.
9. [Confidence based stopping criteria paper](http://www.cs.cmu.edu/~./hovy/papers/10ACMjournal-activelearning-stopping.pdf)
10. A great [tutorial ](http://hunch.net/~active_learning/active_learning_icml09.pdf)
11. [AWS Sagemaker Active Learning](https://youtu.be/8J7y513oSsE?t=435), using annotation consolidation that finds outliers and weights accordingly, then takes that data, trains a model with the annotation + training data, if labeled with high probability, will use those labels, otherwise will re-annotate.
12. [An ok video](https://www.youtube.com/watch?v=Et7h1A1j4ns\&feature=youtu.be)
13. [Active learning framework in python](https://github.com/bwallace/curious_snake)
14. [Active Learning Using Pre-clustering](https://www.researchgate.net/profile/Arnold_Smeulders/publication/221345455_Active_learning_using_pre-clustering/links/54c3cc440cf2911c7a4cc74a/Active-learning-using-pre-clustering.pdf)
15. [A literature survey of active machine learning in the context of natural language processing](http://eprints.sics.se/3600/)
16. [Mnist competition (unpublished) using AL](http://dag.cvc.uab.es/mnist/statistics/)
17. [Practical Online Active Learning for Classification](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.87.5536\&rep=rep1\&type=pdf)
18. [Video 2](https://www.youtube.com/watch?v=8Jwp4_WbRio\&index=7\&list=PLegWUnz91Wfsn6skGOofRoeFoOyfdqSyN)
19. [Active learning in R - code](https://github.com/gsimchoni/ActiveLearningExercise)
20. [Deep bayesian active learning with image data](https://arxiv.org/pdf/1703.02910.pdf)
21. [Medium on AL](https://news.voyage.auto/active-learning-and-why-not-all-data-is-created-equal-8a43a758c6f9)\*\*\*
22. [Integrating Human-in-the-Loop (HITL) in machine learning is a necessity, not a choice. Here’s why?](https://medium.com/@supriya2211/integrating-human-in-the-loop-hitl-in-machine-learning-application-is-a-necessity-not-a-choice-f25e131ca84e) By Supriya Ghosh
#### Human In The loop ML book by [Robert munro](https://www.manning.com/books/human-in-the-loop-machine-learning#ref)
1. [GIT](https://github.com/rmunro/pytorch_active_learning)
2. [Active transfer learning](https://medium.com/pytorch/active-transfer-learning-with-pytorch-71ed889f08c1)
3. [Uncertainty sampling](https://towardsdatascience.com/uncertainty-sampling-cheatsheet-ec57bc067c0b)
1. Least Confidence: difference between the most confident prediction and 100% confidence
2. Margin of Confidence: difference between the top two most confident predictions
3. Ratio of Confidence: ratio between the top two most confident predictions
4. Entropy: difference between all predictions, as defined by information theory
[Diversity sampling](https://towardsdatascience.com/https-towardsdatascience-com-diversity-sampling-cheatsheet-32619693c304) - you want to make sure that it covers as diverse a set of data and real-world demographics as possible.
1. Model-based Outliers: sampling for low activation in your logits and hidden layers to find items that are confusing to your model because of lack of information
2. Cluster-based Sampling: using Unsupervised Machine Learning to sample data from all the meaningful trends in your data’s feature-space
3. Representative Sampling: sampling items that are the most representative of the target domain for your model, relative to your current training data
4. Real-world diversity: using sampling strategies that increase fairness when trying to support real-world diversity
[Combine uncertainty sampling and diversity sampling](https://towardsdatascience.com/advanced-active-learning-cheatsheet-d6710cba7667)
1. Least Confidence Sampling with Clustering-based Sampling: sample items that are confusing to your model and then cluster those items to ensure a diverse sample (see diagram below).
2. Uncertainty Sampling with Model-based Outliers: sample items that are confusing to your model and within those find items with low activation in the model.
3. Uncertainty Sampling with Model-based Outliers and Clustering: combine methods 1 and 2.
4. Representative Cluster-based Sampling: cluster your data to capture multinodal distributions and sample items that are most like your target domain (see diagram below).
5. Sampling from the Highest Entropy Cluster: cluster your unlabeled data and find the cluster with the highest average confusion for your model.
6. Uncertainty Sampling and Representative Sampling: sample items that are both confusing to your current model and the most like your target domain.
7. Model-based Outliers and Representative Sampling: sample items that have low activation in your model but are relatively common in your target domain.
8. Clustering with itself for hierarchical clusters: recursively cluster to maximize the diversity.
9. Sampling from the Highest Entropy Cluster with Margin of Confidence Sampling: find the cluster with the most confusion and then sample for the maximum pairwise label confusion within that cluster.
10. Combining Ensemble Methods and Dropouts with individual strategies: aggregate results that come from multiple models or multiple predictions from one model via Monte-Carlo Dropouts aka Bayesian Deep Learning.
Active transfer learning.
Machine in the loop
1. [Similar to AL, just a machine / model / algo adds suggestions. This is obviously a tradeoff of bias and clean dataset](https://www.lighttag.io/blog/when-to-use-machine-in-the-loop/)
###
# Online Learning
### Online (Classical) Learning
1. If you want to start with OL - [start here](https://dziganto.github.io/data%20science/online%20learning/python/scikit-learn/An-Introduction-To-Online-Machine-Learning/) & [here](https://www.analyticsvidhya.com/blog/2015/01/introduction-online-machine-learning-simplified-2/)
2. Shay Shalev - [A thesis about online learning](http://ttic.uchicago.edu/~shai/papers/ShalevThesis07.pdf)
3. [Some answers about what is OL,](https://www.quora.com/What-is-the-best-way-to-learn-online-machine-learning) the first one actually talks about S.Shalev’s [other paper.](http://www.cs.huji.ac.il/~shais/papers/OLsurvey.pdf)
4. Online learning - Andrew Ng - [coursera](https://www.coursera.org/learn/machine-learning/lecture/ABO2q/online-learning)
5. [Chip Huyen on online prediction & learning](https://huyenchip.com/2020/12/27/real-time-machine-learning.html)
6. [River](https://github.com/online-ml/river/) is a Python library for [online machine learning](https://www.wikiwand.com/en/Online_machine_learning)\
7.
### Online Deep Learning (ODL)
1. [Hedge back propagation (HDP), Autonomous DL, Qactor](https://towardsdatascience.com/online-deep-learning-odl-and-hedge-back-propagation-277f338a14b2) - online AL for noisy labeled stream data.
# N-Shot Learning
### N-SHOT LEARNING
1. [Zero shot, one shot, few shot](https://blog.floydhub.com/n-shot-learning/) (siamese is one shot)
### ZERO SHOT LEARNING
1. [Instead of using class labels](https://www.youtube.com/watch?v=jBnCcr-3bXc), we use some kind of vector representation for the classes, taken from a co-occurrence-after-svd or word2vec. - quite clever. This enables us to figure out if a new unseen class is near one of the known supervised classes. KNN can be used or some other distance-based classifier. Can we use word2vec for similarity measurements of new classes?\
\
Image by [Dr. Timothy Hospedales, Yandex](https://www.youtube.com/watch?v=jBnCcr-3bXc)
\
for classification, we can use nearest neighbour or manifold-based labeling propagation.\
\
Image by [Dr. Timothy Hospedales, Yandex](https://www.youtube.com/watch?v=jBnCcr-3bXc)\
Multiple category vectors? Multilabel zero-shot also in the video
2. [with siamese networks](https://towardsdatascience.com/zero-shot-intent-classification-with-siamese-networks-35900471c7fd)
#### GPT3 is ZERO, ONE, FEW
1. [Prompt Engineering Tips & Tricks](https://blog.andrewcantino.com/blog/2021/04/21/prompt-engineering-tips-and-tricks/)
2. [Open GPT3 prompt engineering](https://medium.com/swlh/openai-gpt-3-and-prompt-engineering-dcdc2c5fcd29)
# Unlearning
Machine unlearning (MU) refers to the challenge of erasing a data point's influence on the input-output mapping of an ML model.
## Papers
1. (really good) A [survey](https://arxiv.org/abs/2209.02299) of [MU](https://arxiv.org/pdf/2209.02299.pdf).
2. [Existing literature on MU](https://github.com/jjbrophy47/machine_unlearning)
3. [Machine Unlearning: The Right to be Forgotten](https://www.kaggle.com/code/tamlhp/machine-unlearning-the-right-to-be-forgotten#sec:algorithms)
4. (really good) [Awesome MU on Github](https://github.com/tamlhp/awesome-machine-unlearning?tab=readme-ov-file#type-image) ([website](https://awesome-machine-unlearning.github.io/))- a collection of academic articles, published methodology, and datasets on the subject of machine unlearning. model agnostic, intrinsic, and data-driven approaches, evaluation metrics, and datasets.
5. Who's Harry Potter? Approximate Unlearning in LLMs. [Arxiv](https://arxiv.org/abs/2310.02238), [paper](https://browse.arxiv.org/pdf/2310.02238), [Microsoft](https://www.microsoft.com/en-us/research/project/physics-of-agi/articles/whos-harry-potter-making-llms-forget-2/), [medium](https://pub.towardsai.net/who-is-harry-potter-inside-microsoft-researchs-fine-tuning-method-for-unlearning-concepts-in-llms-33dfe8e742a9)
6. [fast yet effective MU](https://arxiv.org/pdf/2111.08947.pdf)
7. [one shot MU](https://arxiv.org/pdf/2201.05629.pdf)
8. [A review on MU](https://link.springer.com/article/10.1007/s42979-023-01767-4), Zhang et al.
9. [Machine Un-learning: An Overview of Techniques, Applications, and Future Directions](https://link.springer.com/article/10.1007/s12559-023-10219-3), Siva et al.
10. [Machine Unlearning](https://arxiv.org/abs/1912.03817)
## Medium
1. [A fresh perspective on machine unlearning, with a real-world solution!](https://medium.com/@aliborji/a-fresh-perspective-on-machine-unlearning-with-a-real-world-solution-203821dd01c0) a solution that uses three of the following approaches. Data Augmentation, Weight Decay, Fine-Tuning, Selective Retraining, and Neural Architecture Modifications.
2. What is MU? Part [1](https://medium.com/@choquette.christopher/what-is-machine-unlearning-pt-1-933ff53dc9a6), [2](https://medium.com/@choquette.christopher/how-to-do-machine-unlearning-pt-2-ae32cb6ca2f1)
## GitHub
1. [search results](https://github.com/search?q=unlearning\&type=repositories)
2. This [repository](https://github.com/cleverhans-lab/machine-unlearning) contains the core code used in the SISA experiments of our [Machine Unlearning](https://arxiv.org/abs/1912.03817) paper along with some example scripts.
3. [Implementations of various data deletion methods.](https://github.com/ChrisWaites/data-deletion?tab=readme-ov-file) \
[Evaluation Doc.](https://docs.google.com/document/d/14B_aLihLTNE7a2yRQHNRRVwvSOkttakVYFhlayZBNkE/edit)
4. This [repository](https://github.com/shash42/Evaluating-Inexact-Unlearning/tree/master) contains the code used in our experiments of our paper on [Evaluating Machine Unlearning](https://arxiv.org/abs/2201.06640) in the src/ folder along with some sample scripts in the scripts/ folder.
5. #### [This](https://github.com/meghdadk/SCRUB) is a Python implementation of "Towards Unbounded Machine Unlearning"
6. #### [data deletion](https://github.com/ChrisWaites/data-deletion?tab=readme-ov-file)
## Community
1. NeurIPS 2023 [Kaggle](https://www.kaggle.com/competitions/neurips-2023-machine-unlearning/leaderboard) - Machine Unlearning Erase the influence of requested samples without hurting accuracy
# Data Science
### **LIFE CYCLE**
[**Microsoft on Team DS Lifecycle**](https://docs.microsoft.com/en-us/azure/architecture/data-science-process/overview) **- "**The Team Data Science Process (TDSP) is an agile, iterative data science methodology to deliver predictive analytics solutions and intelligent applications efficiently. TDSP helps improve team collaboration and learning by suggesting how team roles work best together. TDSP includes best practices and structures from Microsoft and other industry leaders to help toward successful implementation of data science initiatives. The goal is to help companies fully realize the benefits of their analytics program.
This article provides an overview of TDSP and its main components. We provide a generic description of the process here that can be implemented with different kinds of tools. A more detailed description of the project tasks and roles involved in the lifecycle of the process is provided in additional linked topics. Guidance on how to implement the TDSP using a specific set of Microsoft tools and infrastructure that we use to implement the TDSP in our teams is also provided."
[**Google’s famous MLops**](https://cloud.google.com/solutions/machine-learning/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning#mlops_level_0_manual_process)
[**Fast ai project checklist**](https://www.fast.ai/2020/01/07/data-questionnaire/?fbclid=IwAR2M_kdqKGSQ9uOFfdTncA6415K31V_flN203T1vHwNJOYg83XY2a9c-Jgg)
**"When I used to do consulting, I’d always seek to understand an organization’s context for developing data projects, based on these considerations:**
* **Strategy: What is the organization trying to do (objective) and what can it change to do it better (levers)?**
* **Data: Is the organization capturing necessary data and making it available?**
* **Analytics: What kinds of insights would be useful to the organization?**
* **Implementation: What organizational capabilities does it have?**
* **Maintenance: What systems are in place to track changes in the operational environment?**
* **Constraints: What constraints need to be considered in each of the above areas?"**
### **WORKFLOWS**
1. [**kaggle**](https://towardsdatascience.com/my-secret-sauce-to-be-in-top-2-of-a-kaggle-competition-57cff0677d3c?fbclid=IwAR3Iei5OmwswIMbbqcz2dNr5rLsWS-iuuaAuOjmhCELTTEBTPmSM85mTw7U)
### **PLATFORMS**
1. [**Uber, google, netflix, airbnb, etc**](https://databaseline.tech/a-tour-of-end-to-end-ml-platforms/)
### **STACK**
1. [**Medium on canonical stack**](https://towardsdatascience.com/rise-of-the-canonical-stack-in-machine-learning-724e7d2faa75)
### **Being a DS / Researcher**
1. [**A day in a life**](https://towardsdatascience.com/12-things-i-learned-during-my-first-year-as-a-machine-learning-engineer-2991573a9195)
2. [**Advice for a ds**](https://medium.com/the-data-experience/building-a-data-pipeline-from-scratch-32b712cfb1db)**, business kpi are not research kpi, etc**
3. [**Review of deep learning papers and co authorship**](https://neurovenge.antonomase.fr/)
4. **Full stack DS** [**Uri Weiss**](https://linkedin.com/in/uriweiss)
**by** [**Uri Weiss**](https://linkedin.com/in/uriweiss)**. wrong credits?** [**please contact me**](mailto:ori@oricohen.com)**.**
5. [**ML practices for a DS**](https://se-ml.github.io/)
### **Team Building / Group Cohesion**
1. [**DS vs DA vs MLE**](https://medium.com/@meightpc_14421/data-scientist-vs-data-analysis-vs-ml-engineer-which-job-is-most-suited-for-you-def7b12b3256) **- the most intensive diagram post ever. This is the motherload of figure references.**
**References:**
[**1**](https://medium.com/@rdavila01/a-team-development-roadmap-ce5247127037)**,** [**2**](https://medium.com/swlh/team-development-stages-51df5606c0a2)**,** [**3**](https://medium.com/unexpected-leadership/forming-storming-norming-and-performing-5d06d021a969)**,** [**4**](https://medium.com/@RiterApp/8-models-of-team-effectiveness-3a3b84efb3ae)**,** [**5**](https://medium.com/@warren2lynch/traditional-to-scrum-team-forming-storming-norming-and-performing-3fd5fd1f5ea9)**,** [**6**](https://medium.com/@pallawi.ds/new-employee-best-practices-to-perform-with-the-team-tuckmans-stages-of-group-development-c656ca295bee)**,** [**7**](https://medium.com/agilegreat/tuckman-model-for-building-great-teams-7b3203d7a9e3)**,** [**8**](https://medium.com/simply-agile/agile-leader-pattern-2-for-building-awesome-teams-stabilize-teams-32785b70868c)**,** [**9**](https://medium.com/hackernoon/team-building-mental-models-1f431ae29361)**, 10**
[**Why data science needs generalists not specialists** ](https://hbr.org/2019/03/why-data-science-teams-need-generalists-not-specialists)
1. **(good advice)** [**Building a DS function (team)**](https://medium.com/ww-tech-blog/from-0-to-60-models-in-two-years-building-out-an-impactful-data-science-function-9ef86abb9605)
### Culture
1. [Netflix](https://jobs.netflix.com/culture) culture
2. [Reed hastings on netflix' keeper test](https://hrtechx.com/2020/11/20/netflixs-keeper-test-is-the-secret-to-a-successful-workforce/) - "netflixs-keeper-test-is-the-secret-to-a-successful-workforce"
1. [response 1](https://www.highlights.lornerubis.com/page/83/)
3.
### **Agile for data-science-research**
1. [**How to manage a data science research team using agile methodology, not scrum and not kanban**](https://towardsdatascience.com/data-science-agile-cycles-my-method-for-managing-data-science-projects-in-the-hi-tech-industry-b289e8a72818)
2. [**Workflow for data science research projects**](https://towardsdatascience.com/data-science-project-flow-for-startups-282a93d4508d)
3. [**Tips for data science research management**](https://towardsdatascience.com/my-best-tips-for-agile-data-science-research-b40365cc979d)
4. [**IMO a really bad implementation of agile for data-science-projects**](https://www.locallyoptimistic.com/post/agile-analytics-p1/)
### **SOTA AND CURRENT TRENDS SUMMARIES**
1. [**ICLR 2019**](https://huyenchip.com/2019/05/12/top-8-trends-from-iclr-2019.html?fbclid=IwAR28Ez8Hs-XMSxcQb2NHfLQvZ5m4C8b4NIZPue00u6MZzrlI90Oqx8TExuU)
2. [**Medium**](https://medium.com/huggingface/the-best-and-most-current-of-modern-natural-language-processing-5055f409a1d1?fbclid=IwAR22vuGFXHil1Nz4vJr4uhueiKPRMz2T-BSwPXl8kg5iQZ54ppHe5ffecqI)
3. [**State of ai, a yearly report**](https://www.stateof.ai/)
### **Building Data/DS teams**
1. [**(great) the data team a short story by erik bern**](https://erikbern.com/2021/07/07/the-data-team-a-short-story.html)
2. [Guilds / Gangs / Squads](https://aviranm.medium.com/the-evolution-of-a-guild-a6c7d1927610) by Aviran Mordo
[ Squads, Tribes, Guilds, dont be like Spotify](https://uxdesign.cc/squads-tribes-guild-to-be-like-spotify-or-not-13ecf690fd36)
1. [Discover the Spotify Model](https://www.atlassian.com/agile/agile-at-scale/spotify)
### **YOUTUBE COURSES**
* [**DEEPNET.TV YOUTUBE (excellent)**](https://www.youtube.com/channel/UC9OeZkIwhzfv-_Cb7fCikLQ)
* [**Mitchel ML Lectures (too long)**](http://www.cs.cmu.edu/~ninamf/courses/601sp15/lectures.shtml)
* [**Quoc Les (google) wrote DNN tutorials and 3H video (not intuitive)**](http://cs.stanford.edu/~quocle/)
* [**KDnuggets: numpy, panda, scikit, tutorials.**](http://www.kdnuggets.com/2015/11/seven-steps-machine-learning-python.html)
* [**Deep learning online book (too wordy)**](http://neuralnetworksanddeeplearning.com/)
* [**Genetic Algorithms - grid search hyper params better than brute force.. obviously**](https://medium.com/@harvitronix/lets-evolve-a-neural-network-with-a-genetic-algorithm-code-included-8809bece164)
*
* [**CNN tutorial**](http://mccormickml.com/2015/01/10/understanding-the-deeplearntoolbox-cnn-example/)
* [**Introduction to programming in scikit**](http://nbviewer.jupyter.org/github/donnemartin/data-science-ipython-notebooks/blob/master/scikit-learn/scikit-learn-intro.ipynb)
* [**SVM in scikit python**](https://github.com/jakevdp/sklearn_pycon2015/blob/master/notebooks/03.1-Classification-SVMs.ipynb)
* [**Sklearn scipy PCA tutorial**](https://github.com/jakevdp/sklearn_pycon2015/blob/master/notebooks/04.1-Dimensionality-PCA.ipynb)
* [**RNN** ](http://colah.github.io/posts/2015-08-Understanding-LSTMs/)
* [**Matrix Multiplication**](http://www.mathwarehouse.com/algebra/matrix/multiply-matrix.php) **- linear algebra**
### **Deep learning Course**
1. [**Kadenze - deep learning tensor flow**](https://www.kadenze.com/courses/creative-applications-of-deep-learning-with-tensorflow-iv/sessions/introduction-to-tensorflow) **- Histograms for (Image distribution - mean distribution) / std dev, are looking quite good.**
2. [**deep learning with keras**](https://github.com/fchollet/deep-learning-with-python-notebooks)
### **Machine Learning Courses**
1. [**Recommended: Udacity includes ML and DL** ](https://classroom.udacity.com/courses/ud188/lessons/b4ca7aaa-b346-43b1-ae7d-20d27b2eab65/concepts/4b7026be-06e3-49de-a362-ce109172659e)
2. [**Week1: Introduction Lesson 4: Supervised, unsupervised.**](https://www.coursera.org/learn/machine-learning/lecture/1VkCb/supervised-learning)
3. [**Lesson 6: model regression, cost function**](https://www.coursera.org/learn/machine-learning/lecture/db3jS/model-representation)
4. [**Lesson 71: optimization objective, large margin classification**](https://www.coursera.org/learn/machine-learning/lecture/sHfVT/optimization-objective)
5. [**PCA at coursera #1**](https://www.coursera.org/learn/machine-learning/lecture/GBFTt/principal-component-analysis-problem-formulation)
6. [**PCA at coursera**](https://www.coursera.org/learn/machine-learning/lecture/ZYIPa/principal-component-analysis-algorithm) **#2**
7. [**PCA #3**](https://www.coursera.org/learn/machine-learning/lecture/S1bq1/choosing-the-number-of-principal-components)
8. [**SVM at coursera #1 - simplified**](https://www.coursera.org/learn/predictive-analytics/lecture/2Qh1o/support-vector-machine-example)
### NLP Courses
1. [spacy](https://spacy.io/usage/spacy-1) 101
2. [gensim](https://www.machinelearningplus.com/nlp/gensim-tutorial/), [2](https://radimrehurek.com/gensim/auto_examples/), gensim notebooks
3. [nltk](https://realpython.com/nltk-nlp-python/), [2](https://www.tutorialspoint.com/natural_language_toolkit/index.htm)
4. [yandex](#life-cycle)
5. lena [voita](https://lena-voita.github.io/nlp_course.html)
[ ](<
https://spacy.io/usage/spacy-101
https://www.machinelearningplus.com/nlp/gensim-tutorial/
https://radimrehurek.com/gensim/auto_examples/
https://realpython.com/nltk-nlp-python/
https://www.tutorialspoint.com/natural_language_toolkit/index.htm
https://github.com/RaRe-Technologies/gensim/tree/develop/docs/notebooks
https://github.com/yandexdataschool/nlp_course
https://lena-voita.github.io/nlp_course.html
https://github.com/fchollet/deep-learning-with-python-notebooks>)
### **Predictive Analytics Course**
[**Syllabus**](https://www.coursera.org/learn/predictive-analytics)
[**Week 2: Lesson 29: supervised learning** ](https://www.coursera.org/learn/predictive-analytics/lecture/qzrx8/statistics-vs-machine-learning)
[**Lesson 36: From rules to trees**](https://www.coursera.org/learn/predictive-analytics/lecture/qTN05/from-rules-to-trees)
[**Lesson 43: overfitting, then validation, then accuracy**](https://www.coursera.org/learn/predictive-analytics/lecture/cnLwv/overfitting)
[**Lesson 46: bootstrap, bagging, boosting, random forests.**](https://www.coursera.org/learn/predictive-analytics/lecture/ZUJqG/bootstrap-revisited)
[**Lesson 52: NN**](https://www.coursera.org/learn/predictive-analytics/lecture/6uyga/nearest-neighbor)
[**Lesson 55: Gradient Descent**](https://www.coursera.org/learn/predictive-analytics/lecture/68oAE/optimization-by-gradient-descent)
[**Lesson 59: Logistic regression, SVM, Regularization, Lasso, Ridge regression**](https://www.coursera.org/learn/predictive-analytics/lecture/FecmG/intuition-for-logistic-regression)
[**Lesson 64: gradient descent, stochastic, parallel, batch.**](https://www.coursera.org/learn/predictive-analytics/lecture/eCynR/stochastic-and-batched-gradient-descent)
[**Unsupervised: Lesson X K-means, DBscan**](https://www.coursera.org/learn/predictive-analytics/lecture/WWiiy/introduction-to-unsupervised-learning)
### **BOOKS & NOTEBOOKS**
1. [**Machine learning design patterns**](https://www.oreilly.com/library/view/machine-learning-design/9781098115777/)**,** [**git**](https://github.com/GoogleCloudPlatform/ml-design-patterns) **notebooks!,** [**medium**](https://lakshmanok.medium.com/machine-learning-design-patterns-58e6ecb013d7)
1. [**DP1 - transform**](https://medium.com/swlh/ml-design-pattern-1-transform-9e82ccbc3209) **Moving an ML model to production is much easier if you keep inputs, features, and transforms separate**
2. [**DP2 - checkpoints**](https://towardsdatascience.com/ml-design-pattern-2-checkpoints-e6ca25a4c5fe) **Saving the intermediate weights of your model during training provides resilience, generalization, and tunability**
3. [**DP3 - virtual epochs**](https://medium.com/google-cloud/ml-design-pattern-3-virtual-epochs-f842296de730) **Base machine learning model training and evaluation on total number of examples, not on epochs or steps**
4. [**DP4 - keyed predictions**](https://towardsdatascience.com/ml-design-pattern-4-keyed-predictions-a8de67d9c0f4) **Export your model so that it passes through client keys**
5. [**DP5 - repeatable sampling**](https://towardsdatascience.com/ml-design-pattern-5-repeatable-sampling-c0ccb2889f39) **use the hash of a well distributed column to split your data into training, validation, and testing**
2. [**Gensim notebooks**](https://github.com/RaRe-Technologies/gensim/tree/develop/docs/notebooks) **- from w2v, doc2vec to nmf, lda, pca, sklearn api, cosine, topic modeling, tsne, etc.**
3. [**Deep learning with python**](https://www.manning.com/books/deep-learning-with-python) **- francois chollet, deep learning & vision** [**git notebooks!**](https://github.com/fchollet/deep-learning-with-python-notebooks)**,** [**official notebooks**](https://github.com/PacktPublishing/Deep-Learning-with-Keras)**.**
4. **Yandex school,** [**nlp notebooks**](https://github.com/yandexdataschool/nlp_course)
5. [**Machine learning engineering book**](http://www.mlebook.com/wiki/doku.php) **(i.e., data science)**
6. [**Interpretable Machine Learning book**](https://christophm.github.io/interpretable-ml-book/)
7.
### **COST**
1. [**GPT2/3**](https://medium.com/modern-nlp/estimating-gpt3-api-cost-50282f869ab8)
### **Patents**
1. [**Method Patent Exceptionalism**](https://ilr.law.uiowa.edu/print/volume-102-issue-3/method-patent-exceptionalism)
### **General Advice**
**(really good)** [**Practical advice for analysis of large, complex data sets**](https://www.unofficialgoogledatascience.com/2016/10/practical-advice-for-analysis-of-large.html) **- distributions, outliers, examples, slices, metric significance, consistency over time, validation, description, evaluation, robustness in measurement, reproducibility, etc.**
# Data Science Tools
## Python
1. [How to use better OOP in python.](https://hackernoon.com/improve-your-python-python-classes-and-object-oriented-programming-d09ff461168d)
2. [Best practices programming python classes - a great lecture.](https://www.youtube.com/watch?v=HTLu2DFOdTg)
3. [How to know pip packages size’](https://stackoverflow.com/questions/34266159/how-to-see-pip-package-sizes-installed) good for removal
4. [Python type checking tutorial](https://medium.com/@ageitgey/learn-how-to-use-static-type-checking-in-python-3-6-in-10-minutes-12c86d72677b)
5. [Import click - command line interface](https://zetcode.com/python/click/)
6. [Concurrency vs Parallelism (great)](https://stackoverflow.com/questions/1050222/what-is-the-difference-between-concurrency-and-parallelism#:~:text=Concurrency%20is%20when%20two%20or,e.g.%2C%20on%20a%20multicore%20processor.)
7. [Async in python](https://medium.com/velotio-perspectives/an-introduction-to-asynchronous-programming-in-python-af0189a88bbb)
8. [Coroutines vs futures](https://stackoverflow.com/questions/34753401/difference-between-coroutine-and-future-task-in-python-3-5)
9. Coroutines [generators async wait](https://masnun.com/2015/11/13/python-generators-coroutines-native-coroutines-and-async-await.html)
10. [Intro to concurrent,futures](http://masnun.com/2016/03/29/python-a-quick-introduction-to-the-concurrent-futures-module.html)
11. [Future task event loop](https://masnun.com/2015/11/20/python-asyncio-future-task-and-the-event-loop.html)
Async io
1. [Intro](https://realpython.com/lessons/what-asyncio/)
2. [complete](https://realpython.com/async-io-python/)
Clean code:
* [Clean code in python git](https://github.com/zedr/clean-code-python)
* [About the book](https://medium.com/@m_mcclarty/tech-book-talk-clean-code-in-python-aa2c92c6564f)
### Virtual Environments
* [stack overflow on pyenv / venv / etc](https://stackoverflow.com/questions/41573587/what-is-the-difference-between-venv-pyvenv-pyenv-virtualenv-virtualenvwrappe)
* [Guide to pyenv & pyenv virtualenv](https://medium.com/swlh/a-guide-to-python-virtual-environments-8af34aa106ac)
* [Managing virtual env with pyenv](https://towardsdatascience.com/managing-virtual-environment-with-pyenv-ae6f3fb835f8)
* [Just use venv](https://towardsdatascience.com/all-you-need-to-know-about-python-virtual-environments-9b4aae690f97)
* [Summary on all the \*envs](https://stackoverflow.com/questions/41573587/what-is-the-difference-between-venv-pyvenv-pyenv-virtualenv-virtualenvwrappe)
* [A really good primer on virtual environments](https://realpython.com/python-virtual-environments-a-primer/)
* [Introduction to venv](http://cewing.github.io/training.python_web/html/presentations/venv_intro.html) complementary to the above
* [Pipenv ](https://pipenv.readthedocs.io/en/latest/)
* [A great intro to pipenv](https://realpython.com/pipenv-guide/)
* [A complementary to pipenv above](https://robots.thoughtbot.com/how-to-manage-your-python-projects-with-pipenv)
* [Comparison between all \*env](https://stackoverflow.com/questions/41573587/what-is-the-difference-between-venv-pyvenv-pyenv-virtualenv-virtualenvwrappe)
* [pyenv, virtualenv and using them with Jupyter](https://albertauyeung.github.io/2020/08/17/pyenv-jupyter.html/) - a make sense tutorial and instructions on how to use all.
* [Create isolated Jupyter ipython kernels with pyenv and virtualenv](https://www.alfredo.motta.name/create-isolated-jupyter-ipython-kernels-with-pyenv-and-virtualenv/) by alfredo motta
* [Jupyter Notebook in a virtual env](https://towardsdatascience.com/jupyter-notebooks-i-getting-started-with-jupyter-notebooks-f529449797d2) by Christine Egan
#### PYENV
1. [Installing pyenv](https://bgasparotto.com/install-pyenv-ubuntu-debian)
2. [Intro to pyenv](https://realpython.com/intro-to-pyenv/)
3. [Pyenv tutorial and finding where it is](https://anil.io/blog/python/pyenv/using-pyenv-to-install-multiple-python-versions-tox/) *\*\**
4. [Pyenv override system python on mac](https://github.com/pyenv/pyenv/issues/660)
5. pyenv virtualenv
## JUPYTER
* [Cloud GPUS cheap](https://www.paperspace.com/gradient)
* [Importing a notebook as a module](http://jupyter-notebook.readthedocs.io/en/latest/examples/Notebook/Importing%20Notebooks.html)
* Important [colaboratory commands for jupytr ](https://medium.com/deep-learning-turkey/google-colab-free-gpu-tutorial-e113627b9f5d)
* [Timing and profiling in Jupyter](http://pynash.org/2013/03/06/timing-and-profiling/)
* ([Debugging in Jupyter, how?)](https://kawahara.ca/how-to-debug-a-jupyter-ipython-notebook/) - put a one liner before the code and query the variables inside a function.
* [28 tips n tricks for jupyter ](https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/)
* Jupyter notebooks as a module
1. [Nbdev](https://github.com/fastai/nbdev), on [fast.ai](https://www.fast.ai/2019/12/02/nbdev/)
2. [jupytext](https://github.com/mwouts/jupytext)
* [Virtual environments in jupyter](https://anbasile.github.io/programming/2017/06/25/jupyter-venv/)
1. Enter your project directory
2. $ python -m venv projectname
3. $ source projectname/bin/activate
4. (venv) $ pip install ipykernel
5. (venv) $ ipython kernel install --user --name=projectname
6. Run jupyter notebook \* (not entirely sure how this works out when you have multiple notebook processes, can we just reuse the same server?)
7. Connect to the new server at port 8889
8.
* [Virtual env with jupyter ](https://janakiev.com/til/jupyter-virtual-envs/)
([how does reshape work?)](http://anie.me/numpy-reshape-transpose-theano-dimshuffle/) - a shape of (2,4,6) is like a tree of 2->4 and each one has more leaves 4->6.
As far as i can tell, reshape effectively flattens the tree and divide it again to a new tree, but the total amount of inputs needs to stay the same. 2\*4\*6 = 4\*2\*3\*2 for example
code:\
`import numpy` \
`rng = numpy.random.RandomState(234)` \
`a = rng.randn(2,3,10)` \
`print(a.shape)` \
`print(a)` \
`b = numpy.reshape(a, (3,5,-1))` \
`print(b.shape)` \
`print (b)`
\*\*\* A tutorial for [Google Colaboratory - free Tesla K80 with Jup-notebook](https://www.kdnuggets.com/2018/02/google-colab-free-gpu-tutorial-tensorflow-keras-pytorch.html/2)
[Jupyter on Amazon AWS](https://blog.keras.io/running-jupyter-notebooks-on-gpu-on-aws-a-starter-guide.html)
How to add extensions to jupyter: [extensions](https://codeburst.io/jupyter-notebook-tricks-for-data-science-that-enhance-your-efficiency-95f98d3adee4)
[Connecting from COLAB to MS AZURE](https://medium.com/@d.sakryukin/simple-cryptocurrency-trading-data-preparation-in-15-minutes-using-ms-azure-and-google-colab-44872b023d11)
[Streamlit vs. Dash vs. Shiny vs. Voila vs. Flask vs. Jupyter](https://towardsdatascience.com/streamlit-vs-dash-vs-shiny-vs-voila-vs-flask-vs-jupyter-24739ab5d569)
## SCIPY
1. [Optimization problems, a nice tutorial](http://scipy-lectures.org/advanced/mathematical_optimization/) to finding the minima
2. [Minima / maxima](https://stackoverflow.com/questions/4624970/finding-local-maxima-minima-with-numpy-in-a-1d-numpy-array) finding it in a 1d numpy array
## NUMPY
[Using numpy efficiently](https://speakerdeck.com/cournape/using-numpy-efficiently) - explaining why vectors work faster.\
[Fast vector calculation, a benchmark](https://towardsdatascience.com/data-science-with-python-turn-your-conditional-loops-to-numpy-vectors-9484ff9c622e) between list, map, vectorize. Vectorize wins. The idea is to use vectorize and a function that does something that may involve if conditions on a vector, and do it as fast as possible.
## PANDAS
1. [Great introductory tutorial](http://nikgrozev.com/2015/12/27/pandas-in-jupyter-quickstart-and-useful-snippets/#loading_csv_files) about using pandas, loading, loading from zip, seeing the table’s features, accessing rows & columns, boolean operations, calculating on a whole row\column with a simple function and on two columns even, dealing with time\date parsing.
2. [Visualizing pandas pivoting and reshaping functions by Jay Alammar](http://jalammar.github.io/visualizing-pandas-pivoting-and-reshaping/) - pivot melt stack unstack
3. [How to beautify pandas dataframe using html display](https://stackoverflow.com/questions/26873127/show-dataframe-as-table-in-ipython-notebook)
4. [Speeding up pandas ](https://realpython.com/fast-flexible-pandas/)
5. [The fastest way to select rows by columns, by using masked values](https://stackoverflow.com/questions/17071871/select-rows-from-a-dataframe-based-on-values-in-a-column-in-pandas) (benchmarked):
6. def mask\_with\_values(df): mask = df\['A'].values == 'foo' return df\[mask]
7. [Parallelism, pools, threads, dask](https://towardsdatascience.com/speed-up-your-algorithms-part-3-parallelization-4d95c0888748#7e6e)
8. [Accessing dataframe rows, columns and cells](http://pythonhow.com/accessing-dataframe-columns-rows-and-cells/)- by name, by index, by python methods.
9. [Looping through pandas](https://medium.com/swlh/how-to-efficiently-loop-through-pandas-dataframe-660e4660125d)
10. [How to inject headers into a headless CSV file](http://pythonforengineers.com/introduction-to-pandas/) -
11. [Dealing with time series](http://pandas.pydata.org/pandas-docs/stable/timeseries.html) in pandas,
1. [Create a new column](https://stackoverflow.com/questions/25570147/add-new-column-based-on-boolean-values-in-a-different-column) based on a (boolean or not) column and calculation:
2. Using python (map)
3. Using numpy
4. using a function (not as pretty)
12. Given a DataFrame, the [shift](http://machinelearningmastery.com/convert-time-series-supervised-learning-problem-python/)() function can be used to create copies of columns that are pushed forward (rows of NaN values added to the front) or pulled back (rows of NaN values added to the end).
1. df\['t'] = \[x for x in range(10)]
2. df\['t-1'] = df\['t'].shift(1)
3. df\['t-1'] = df\['t'].shift(-1)
13. [Row and column sum in pandas and numpy](http://blog.mathandpencil.com/column-and-row-sums)
14. [Dataframe Validation In Python](https://www.youtube.com/watch?time_continue=905\&v=1fHGXOfiDO0) - A Practical Introduction - Yotam Perkal - PyCon Israel 2018
15. In this talk, I will present the problem and give a practical overview (accompanied by Jupyter Notebook code examples) of three libraries that aim to address it: Voluptuous - Which uses Schema definitions in order to validate data \[[https://github.com/alecthomas/voluptuous](https://www.youtube.com/redirect?v=1fHGXOfiDO0\&event=video_description\&redir_token=jIIzdRAEjZBpzVhRYfFzTcx52358MTU0NzQ0ODY3N0AxNTQ3MzYyMjc3\&q=https%3A%2F%2Fgithub.com%2Falecthomas%2Fvoluptuous)] Engarde - A lightweight way to explicitly state your assumptions about the data and check that they're actually true \[[https://github.com/TomAugspurger/engarde](https://www.youtube.com/redirect?v=1fHGXOfiDO0\&event=video_description\&redir_token=jIIzdRAEjZBpzVhRYfFzTcx52358MTU0NzQ0ODY3N0AxNTQ3MzYyMjc3\&q=https%3A%2F%2Fgithub.com%2FTomAugspurger%2Fengarde)] \* TDDA - Test Driven Data Analysis \[ ]. By the end of this talk, you will understand the Importance of data validation and get a sense of how to integrate data validation principles as part of the ML pipeline.
16. [Stop using itterows](https://medium.com/@rtjeannier/pandas-101-cont-9d061cb73bfc), use apply.
17. [(great) Group and Aggregate by One or More Columns in Pandas](https://jamesrledoux.com/code/group-by-aggregate-pandas)
18. [Pandas Groupby: Summarising, Aggregating, and Grouping data in Python](https://www.shanelynn.ie/summarising-aggregation-and-grouping-data-in-python-pandas/#applying-multiple-functions-to-columns-in-groups)
19. [pandas function you didnt know about](https://towardsdatascience.com/25-pandas-functions-you-didnt-know-existed-p-guarantee-0-8-1a05dcaad5d0)
20. [json\_normalize()](https://towardsdatascience.com/all-pandas-json-normalize-you-should-know-for-flattening-json-13eae1dfb7dd)
### Exploratory Data Analysis (EDA)
1. [Pandas summary](https://github.com/mouradmourafiq/pandas-summary)
2. [Pandas html profiling](https://github.com/pandas-profiling/pandas-profiling)
3. [Sweetviz](https://github.com/fbdesignpro/sweetviz) - "Sweetviz is an open-source Python library that generates beautiful, high-density visualizations to kickstart EDA (Exploratory Data Analysis) with just two lines of code. Output is a fully self-contained HTML application.
The system is built around quickly visualizing target values and comparing datasets. Its goal is to help quick analysis of target characteristics, training vs testing data, and other such data characterization tasks."
### TIMESERIES
1. (good) [Pandas time series manipulation](https://towardsdatascience.com/practical-guide-for-time-series-analysis-with-pandas-196b8b46858f)
2. [Using resample](https://towardsdatascience.com/using-the-pandas-resample-function-a231144194c4)
1. [Basic TS manipulation](https://towardsdatascience.com/basic-time-series-manipulation-with-pandas-4432afee64ea)
2. [Fill missing ts gaps, or how to resample](https://stackoverflow.com/questions/32241692/fill-missing-timeseries-data-using-pandas-or-numpy)
3. SCI-KIT LEARN
4. Pipeline t[o json 1](https://cmry.github.io/notes/serialize), [2](https://cmry.github.io/notes/serialize-sk)
5. [cuML](https://github.com/rapidsai/cuml) - Multi gpu, multi node-gpu alternative for SKLEARN algorithms
6. [Gpu TSNE ^](https://www.reddit.com/r/MachineLearning/comments/e0j9cb/p_2000x_faster_rapids_tsne_3_hours_down_to_5/?utm_source=share\&utm_medium=ios_app\&utm_name=iossmf)
7. [Awesome code examples](http://machinelearningmastery.com/get-your-hands-dirty-with-scikit-learn-now/) about using svm\knn\naive\log regression in sklearn in python, i.e., “fitting a model onto the data”
8. [Parallelism of numpy, pandas and sklearn using dask and clusters](https://github.com/dask/dask). [Webpage](https://dask.pydata.org/en/latest/), [docs](http://dask-ml.readthedocs.io/en/latest/index.html), [example in jupyter](https://hub.mybinder.org/user/dask-dask-examples-6bi4j3qj/notebooks/machine-learning.ipynb).
Also Insanely fast, [see here](https://www.youtube.com/watch?v=5Zf6DQaf7jk).
1. [Functional api for sk learn](https://scikit-lego.readthedocs.io/en/latest/preprocessing.html), using pipelines. thank you sk-lego.
2. 
Images by [SK-Lego](https://medium.com/@jeremyrchow)
## FAST.AI
1. [Medium](https://medium.com/@hiromi_suenaga/deep-learning-2-part-1-lesson-1-602f73869197) on all fast.ai courses, 14 posts
## PYCARET
[1. What is? by vidhaya](https://www.analyticsvidhya.com/blog/2020/05/pycaret-machine-learning-model-seconds/?utm_source=AVFacebook\&utm_medium=post\&utm_campaign=19_june_intermediate_article\&fbclid=IwAR0NZV5fJgXtpoCBfmauCiGQC_QOK0cbQrpuhhpDBAtEngGG_NBsRlcVRas) - [PyCaret](https://pycaret.org/) is an open-source, machine learning library in Python that helps you from data preparation to model deployment. It is easy to use and you can do almost every data science project task with just one line of code.
## NVIDIA TF CUDA CUDNN
* [Install TF](https://www.tensorflow.org/install/install_linux#NVIDIARequirements)
* [Install cuda on ubuntu](https://devtalk.nvidia.com/default/topic/1030495/cuda-setup-and-installation/install-a-specific-cuda-version-for-ubuntu-16-04/), [official linux](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/)
* [Replace cuda version](https://askubuntu.com/questions/959835/how-to-remove-cuda-9-0-and-install-cuda-8-0-instead) *\*\**
* [Cuda 9 download](https://developer.nvidia.com/cuda-90-download-archive?target_os=Linux\&target_arch=x86_64\&target_distro=Ubuntu\&target_version=1704\&target_type=runfilelocal)
* [Install cudnn](https://askubuntu.com/questions/1033489/the-easy-way-install-nvidia-drivers-cuda-cudnn-and-tensorflow-gpu-on-ubuntu-1)
* [Installing everything easily](https://askubuntu.com/questions/1033489/the-easy-way-install-nvidia-drivers-cuda-cudnn-and-tensorflow-gpu-on-ubuntu-1)
* [Failed](https://stackoverflow.com/questions/43022843/nvidia-nvml-driver-library-version-mismatch) to initialize NVML: Driver/library version mismatch
## GCP
[Resize google disk size](https://medium.com/google-cloud/resize-your-persist-disk-on-google-cloud-on-the-fly-b3491277b718), [1,](https://cloud.google.com/compute/docs/disks/add-persistent-disk) \*\*\[2]\(),\*\*
## GIT / Bitbucket
1. [understanding git](https://learngitbranching.js.org/)
2. [pre-commit](https://pre-commit.com/)
3. [Rewrite git history, all the commands](https://www.youtube.com/watch?v=ElRzTuYln0M)
4. [Installing git LFS](https://askubuntu.com/questions/799341/how-to-install-git-lfs-on-ubuntu-16-04)
5. [Use git lfs](https://confluence.atlassian.com/bitbucket/use-git-lfs-with-bitbucket-828781636.html)
6. [Download git-lfs](https://git-lfs.github.com/)
7. [Git wip](https://carolynvanslyck.com/blog/2020/12/git-wip/) (great)
by [Carolyn Van Slyck](https://carolynvanslyck.com/)
# Management
## OKRs & KPIs
1. [Metrics vs KRs](https://www.perdoo.com/resources/the-difference-between-metrics-kpis-key-results/#:~:text=A%20KPI%20is%20a%20tool,makes%20use%20of%20a%20metric.) - boils down to a starting value.
2. [OKRs vs KPIs](https://medium.com/@meetfelipe/okr-vs-kpis-what-is-the-difference-ffa54673fcf1) by filipe castro, [1](https://weekdone.com/okr-comparison/okr-vs-kpi), [2](https://www.wrike.com/blog/kpis-vs-okrs-compare-need-successful/), [3](https://www.whatmatters.com/resources/difference-between-okr-kpi)
### Data Science OKR KPI
1. [OKR vs KPI](https://www.clearpointstrategy.com/okrs-vs-kpis/), strategic vs tactical
2. [Difference between KPI targets and goals](https://bernardmarr.com/default.asp?contentID=1346#:~:text=The%20terms%20key%20performance%20indicator,NOT%20the%20same%20as%20goals.)
3. [Comet ml on medium](https://medium.com/comet-ml/a-data-scientists-guide-to-communicating-results-c79a5ef3e9f1)
1. [For the Data Driven manager (not ds)](https://www.klipfolio.com/blog/17-kpi-management-data-driven-manager)
2. [Measuring DS business value](https://blog.dominodatalab.com/measuring-data-science-business-value/)
3. [Best KPIS for DS - the best is what not to do](https://www.quora.com/What-are-the-best-KPIs-for-Data-Science-team)
## Management
1. [Important Traits To Help You Become A Better Data-Science Manager](https://towardsdatascience.com/important-traits-to-help-you-become-a-better-data-science-manager-dc0de3a37961), by Dr. Ori Cohen
2. [7 management styles and how to use them](https://www.breathehr.com/en-gb/blog/topic/business-leadership/best-management-styles-and-how-to-use-them)
3. 7 leadership styles (similar to the above)
4. [The secret sauce of DS management](https://www.youtube.com/watch?v=qO7sl8_YtJM) by Shir Meir Lador
5. rework by Google - [what makes a great manager](https://rework.withgoogle.com/guides/managers-identify-what-makes-a-great-manager/steps/learn-about-googles-manager-research/)
## Project Management
1. [Data-science? Agile? Cycles? My method for managing data-science projects in the Hi-tech industry](https://towardsdatascience.com/data-science-agile-cycles-my-method-for-managing-data-science-projects-in-the-hi-tech-industry-b289e8a72818), by Dr. Ori Cohen
2. [Lessons learned leading AI teams](https://blogs.intuit.com/blog/2020/06/23/lessons-learned-leading-ai-teams/), by Shir Meir Lador
3. How to avoid conflicts and delays in the AI development [Part 1](https://blogs.intuit.com/blog/2020/12/08/how-to-avoid-conflicts-and-delays-in-the-ai-development-process-part-i/), [Part 2](https://blogs.intuit.com/blog/2021/01/06/how-to-avoid-conflicts-and-delays-in-the-ai-development-process-part-ii/), by Shir Meir Lador
## Building Teams
1. rework by google - [understanding team effectiveness](https://rework.withgoogle.com/guides/understanding-team-effectiveness/steps/introduction/)
2. Conway's law "Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations."
3. [team topologies](https://teamtopologies.com/), [youtube](https://www.youtube.com/c/TeamTopologies/videos)
1. [key concepts](https://teamtopologies.com/key-concepts)
2. DS are "Complicated Subsystem team: Phd Level, great expertise, in depth knowledge.
3. feature teams are "Stream-aligned team"
4. enabling teams help bridge the gap in knowledge for feature teams, such as architecture
5. platform team - providing a platform to speed up feature teams.
6. [team topologies article](https://www.scaledagileframework.com/organizing-agile-teams-and-arts-team-topologies-at-scale/) - A complicated-subsystem team is responsible for building and maintaining a part of the system that depends heavily on specialist knowledge, to the extent that most team members must be specialists in that area of knowledge in order to understand and make changes to the subsystem. \[1]
7. [team topology for ML](https://towardsdatascience.com/team-topology-for-machine-learning-45bddba626e3)
8. [team topologies for data engineering](https://medium.com/data-arena/team-topologies-for-data-engineering-teams-a15c5eb3849c)
9. [towards data mesh: data domains and team topologies](https://francois-nguyen.blog/2021/03/07/towards-a-data-mesh-part-1-data-domains-and-teams-topologies/)
4. [atlassian ](https://www.atlassian.com/devops/frameworks/team-structure)- "it's important to understand that not every team shares the same goals, or will use the same practices and tools. Even the way a team is composed shouldn’t be standardized. Different teams require different structures, depending on the greater context of the company and its appetite for change. "
5. [good article](https://betterprogramming.pub/team-topologies-a-new-way-of-thinking-about-teams-8f4853038509) that talks about conway's law and team topologies by mark mishaev
1. Quote "The goal of this team is to reduce the cognitive load of stream-aligned teams working on systems that include or use the complicated subsystem. The team handles the subsystem complexity via specific capabilities and expertise that are typically hard to find or grow.
Examples of complicated subsystems might include face-recognition algorithms, machine learning approaches, real-time devices drivers, digital signal processing, or any other expertise-based capability that would be hard to embed directly within the stream-aligned team"
6. [team patterns building an eng team](https://www.kennethlange.com/team-patterns-how-to-structure-an-engineering-team/) by Kenneth Lange - an alternative to team topologies?
"In my experience there are four general team patterns that most companies follow. Yes, they have tweaked them to fit their circumstances, but the overall idea behind the pattern remains the same:
1. **Technology Team:** The team is formed around a technology, such as Android. For example, a team of mobile developers who build and maintain a mobile app.
2. **Matrix Team:** The developers report to a Development Manager, but they are “lend out” to cross-functional product or project teams where they do their daily work.
3. **Product Team:** The team is oriented around a product area, such as billing. It’s cross-functional, but all people on the team, regardless of their specialization, report to the same line manager.
4. **Self-Managed Product Team:** The team is oriented around a product area. But the management of the team is divided into technical leadership, typically handled by an Engineering Lead on the team, and people management, typically handled by an Engineering Manager outside the team."
7. [another good article](https://betterprogramming.pub/your-team-structures-aint-working-let-s-apply-team-topologies-470e8d4f7fe5) by Ryan Dawson
> “Organizations not only need to strive for autonomous teams, they also need to continuously think about and evolve themselves in order to deliver value quickly to customers” — *Team Topologies*
\[1] Book: Skelton, Matthew, and Manuel Pais. Team Topologies: Organizing Business and Technology Teams for Fast Flow. IT Revolution Press, 2019.
8. [Full cycle DS](https://towardsdatascience.com/fcds-b2d2e6b08d34)
## Scaling Agile - Agile Approaches
1. The spotify "model" - squads tribes chapters guilds
1. [Scaling agile snapshot 2012](https://blog.crisp.se/wp-content/uploads/2012/11/SpotifyScaling.pdf)
2. [Scaling Agile at Spotify](https://www.youtube.com/watch?v=SUR9q_Qcrk4) 2014 - Joakim Sunden and Anders Ivarsson
3. [inside Spotify by Andres Ivarsson](https://theagilerevolution.com/2016/07/06/episode-112-inside-spotify-with-anders-ivarsson/) (spotify) 2016
4. Spotify eng culture [p1](https://engineering.atspotify.com/2014/03/spotify-engineering-culture-part-1/) [p2](https://engineering.atspotify.com/2014/09/spotify-engineering-culture-part-2/) 2014 [youtube](https://www.youtube.com/watch?v=4GK1NDTWbkY) 2017 by Henrik Kniberg
5. [Spotify engineering colture](https://www.youtube.com/watch?v=4GK1NDTWbkY) 2017 youtube
6. [how things dont work in spotify and we are trying to solve them](https://www.slideshare.net/jchyip/how-things-still-dont-quite-work-at-spotify-and-how-were-trying-to-solve-it) 2017 and [youtube](https://www.youtube.com/watch?v=VZMf8QJmB98)
7. [you can do better than the spotify model](https://agile2017.sched.com/event/ATal/you-can-do-better-than-the-spotify-model-joakim-sunden-catherine-peck-phillips?ref=JeremiahLee), [video](https://vimeo.com/240125835) - agile 2017 - Joakim Sundén
"Spotify is used as a framework/model copied by others, but Spotify's model isn't without challenges even for Spotify
Encouragement that it's always hard AND it's always possible to improve
It's great to be inspired by others but at the end of the day you need to face your difficulties and solve your problems yourself
You can succeed with autonomy by never giving up; it comes with challenges and benefits"
8. [failed squad goals](https://www.jeremiahlee.com/posts/failed-squad-goals/) 2020, listen on [spotify](https://anchor.fm/jeremiah-oral-lee/episodes/Spotifys-Failed-SquadGoals-edia0p), [blowback response](https://www.jeremiahlee.com/posts/failed-squad-goals/comments/)
9. [there is no spotify model for scaling agile](https://vitalitychicago.com/blog/there-is-no-spotify-model-for-scaling-agile/)
10. [spotify model sucks](https://www.linkedin.com/pulse/spotify-sucks-erwin-verweij/) by erwin verweij
11. [how to structure eng team](https://www.linkedin.com/pulse/how-structure-engineering-team-scale-yotam-hadass) by yotam hadas
12. [spotify model - I dont think it means what you think it means](https://medium.com/serious-scrum/you-want-to-adopt-the-spotify-model-i-dont-think-it-means-what-you-think-it-means-7df4316081f) - "Don’t fool yourself and others. The Spotify engineering culture is NOT about their organisational structure. It is how people are allowed to determine what to do. It’s about autonomy. It’s about having a culture of safety. Among others. I advise you to revisit the videos so that you can experience it yourself." - Willem Jan Ageling
13. [balancing autonomy with accountability](https://www.scrum.org/resources/blog/balancing-autonomy-accountability) - edwin dando
2. ["shape up"](https://basecamp.com/shapeup?ref=JeremiahLee) book (under 200)
3. [SAFe](https://www.scaledagileframework.com/?ref=JeremiahLee) 5 - scaled agile framework
4. [Safe agile principles](https://scaledagileframework.com/safe-lean-agile-principles/)
## Working with partners
## Culture building
## Psychological Safety
1. (great) has a lot of tips on how to measure - [high performing teams need PS](https://www.fearlessculture.design/blog-posts/high-performing-teams-need-psychological-safety)
2. [high performing teams need psychological safety](https://workplaceinsights.microsoft.com/productivity/high-performing-teams-need-psychological-safety-heres-how-to-create-it/) a summary by microsoft
3. [five keys to successful google team](https://rework.withgoogle.com/blog/five-keys-to-a-successful-google-team/), 1st one is PS
## Settings standarts
## Career development
1. [development plan for managers](https://www.indeed.com/career-advice/career-development/development-plan-for-managers#:~:text=What%20is%20a%20development%20plan,specific%20function%20within%20an%20organization.)
2. [for junior DS](https://medium.com/@mbsahar4/my-development-plan-as-a-junior-data-scientist-ec3c68a2b641)
## **Books**
1. **People management**
1. **(good) the effective manager**
2. **radical candor**
3. **managing humans**
2. **Company Management**
1. **The CEO within**
2. **business without the bullshit**
3. **Collaborations and influence**
1. **(good) crucial conversations,** [**1**](https://wikisummaries.org/crucial-conversations-tools-for-talking-when-stakes-are-high/#:~:text=A%20crucial%20conversation%20is%20a,them%20and%20handle%20them%20well.)**,** [**2**](https://slooowdown.wordpress.com/2013/06/09/summary-of-crucial-conversations-tools-for-talking-when-the-stakes-are-high-by-kerry-patterson-joseph-grenny-ron-mcmillan-and-al-swizler/)**,** [**3**](https://fourminutebooks.com/crucial-conversations-summary/)
4. **Negotiations**
1. **never split the difference.** [**TLDR**](https://www.linkedin.com/pulse/never-split-difference-tldr-john-dziedzic/)**,** [**summary**](https://www.samuelthomasdavies.com/book-summaries/business/never-split-the-difference/)**,** [**summary & commentary**](https://growth.me/books/never-split-the-difference/) **,** [**summary**](https://www.oberlo.com/blog/never-split-the-difference-by-chris-voss-summary#:~:text=At%20its%20core%2C%20negotiation%20is,influence%20your%20life%20and%20career.)**,** [**2**](https://www.freshworks.com/crm/sales/sdr-sales-development-reps/summary-of-never-split-the-difference-blog/)**,** [**3**](https://www.samuelthomasdavies.com/book-summaries/business/never-split-the-difference/)**,** [**4**](https://medium.com/@highperformancelifestyle/never-split-the-difference-summary-review-animated-c32f72a36608)**,** [**youtube**](https://www.youtube.com/watch?v=OaEw7ZFs5sU)**,** [**chris voss**](https://www.youtube.com/watch?v=yPsvgmZlVuQ)**,** [**2**](https://www.youtube.com/watch?v=guZa7mQV1l0)**,** [**3**](https://www.youtube.com/watch?v=YNqpQ3zi8iQ)
5. **Manipulations**
1. **The prince,** [**1**](https://www.sparknotes.com/philosophy/prince/section3/)**,** [**2**](https://www.cliffsnotes.com/literature/p/the-prince/book-summary#:~:text=The%20Prince%20is%20an%20extended,the%20principles%20of%20princely%20government.)**, 3**
2. [**The 48 Laws of Power**](https://www.amazon.com/48-Laws-Power-Robert-Greene/dp/0140280197) **- “Amoral, cunning, ruthless, and instructive, this multi-million-copy New York Times bestseller is the definitive manual for anyone interested in gaining, observing, or defending against ultimate control – from the author of The Laws of Human Nature.**
6. **Others**
1. **(good) High output management**
2. **multipliers,**
3. **radical candor,**
4. **Trillion dollar coach,**
5. **The HP way,**
6. **How to measure anything,**
7. **Mindset,**
8. **(good) The hard thing about hard things**
9. [**principles life & work**](https://www.amazon.com/Principles-Life-Work-Ray-Dalio/dp/1501124021)**,** [**summary**](https://readingraphics.com/book-summary-principles-ray-dalio/)
# Project & Program Management
1. [How Big Tech Runs Tech Projects and the Curious Absence of Scrum](https://newsletter.pragmaticengineer.com/p/project-management-in-tech) - by Gergely Orosz
2. [Product manager vs project manager vs technical program manager](https://medium.com/swlh/product-manager-vs-project-manager-vs-technical-program-manager-e45d0cf32e01) - by Julius UY
3. [Tech Lead vs Team Lead](https://www.rubick.com/engineering-manager-vs-tech-lead/) - by jade Rubick
4. [Single Threaded Owner](https://www.rubick.com/implementing-amazons-single-threaded-owner-model/), i.e., "a team focusing on a single thing at a time"- by jade Rubick
# Data Science Management
## **INTERVIEW Qs**
1. [**40 questions on ensembles**](https://www.analyticsvidhya.com/blog/2017/02/40-questions-to-ask-a-data-scientist-on-ensemble-modeling-techniques-skilltest-solution/?utm_source=facebook.com\&utm_medium=social)
2. [**30 on trees**](https://www.analyticsvidhya.com/blog/2017/09/30-questions-test-tree-based-models/?utm_source=facebook.com\&utm_medium=social)
3. [**30 on knns**](https://www.analyticsvidhya.com/blog/2017/09/30-questions-test-k-nearest-neighbors-algorithm/?utm_source=facebook.com\&utm_medium=social\&fbclid=IwAR0JgeXKfyLGndL2_eMX7R6HLVY9la97V6QMIYb_4LnG56N-x1Oe5DsdhqE)
## **Politics**
1. [**The most difficult thing in ds, politics**](https://towardsdatascience.com/the-most-difficult-thing-in-data-science-politics-ca6dd781da56)
## **HIRING / RECRUITING**
1. [**Data engineer skills**](https://medium.com/@m_mcclarty/data-engineering-interview-guide-7a14d10887dd) **on medium**
1. **Coding (Typically Python)**
2. **SQL**
3. **Database design**
4. **Data architecture/big data technologies**
5. **Soft skills**
## **WRITING DOCS**
1. [**Design docs at google**](https://www.industrialempathy.com/posts/design-docs-at-google/)
## **LEGAL & CONTRACTS**
1. [**(FAST) Advisory board saas agreement**](https://fi.co/fast)
## General
1. [**The Care and Feeding of Data Scientists - O'reilly**](https://oreilly-ds-report.s3.amazonaws.com/Care_and_Feeding_of_Data_Scientists.pdf)
## Due Diligence
1. [by Inbal Budowski Tal ](https://inbal-budo.medium.com/how-to-run-a-machine-learning-technical-due-diligence-415970f77d5f#)
# Calculus
[**Start here!**](http://parrt.cs.usfca.edu/doc/matrix-calculus/)
**Help needed to make this page great,** [**please contact me**](mailto:ori@oric)**.**
# Probability & Statistics
[**Coursera course**](https://www.youtube.com/watch?v=WkOinijQmPU\&list=PLpl-gQkQivXiBmGyzLrUjzsblmQsLtkzJ\&index=1) **on probabilities - for data science, actually quite good in explaining a lot of the basic tools,prob, conditional, distributions, sampling, CI, hypothesis, etc.**
* [**A great resource for proba/bayes/b-networks/etc**](https://metacademy.org/graphs/concepts/bayesian_networks#focus=i9mo2e09\&mode=learn) **(adam bali)**
### [**Difference between**](https://stats.stackexchange.com/questions/665/whats-the-difference-between-probability-and-statistics)
* **I.e, Probability deals with predicting the likelihood of future events, while statistics involves the analysis of the frequency of past events.**
* **The problems considered by probability and statistics are inverse to each other.**
* **In probability theory we consider some underlying process which has some randomness or uncertainty modeled by random variables, and we figure out what happens.**
**=> Underlying process + randomness and random variables -> what happens next?**
* **In statistics we observe something that has happened, and try to figure out what underlying process would explain those observations.**
**=> observe what happened -> what is the underlying process?**
* **Finally, probability theory is mainly concerned with the deductive part, statistics with the inductive part of modeling processes with uncertainty**
### **Introduction to statistics**
1. [**Table of**](https://www.mathsisfun.com/data/index.html#stats) **content**
2. [**Median**](https://www.mathsisfun.com/median.html)
3. [**Mode**](https://www.mathsisfun.com/mode.html) **- most freq**
4. [**Weighted mean**](https://www.mathsisfun.com/data/weighted-mean.html)
5. [**Geometric mean**](https://www.mathsisfun.com/numbers/geometric-mean.html)
6. [**Harmonic mean**](https://www.mathsisfun.com/numbers/harmonic-mean.html)
7. [**Percentiles**](https://www.mathsisfun.com/data/percentiles.html)
8. [**Mean deviation**](https://www.mathsisfun.com/data/mean-deviation.html)
9. [**Correlation**](https://www.mathsisfun.com/data/correlation.html)
10. [**Standard deviation**](https://www.mathsisfun.com/data/standard-deviation.html)**,** [**formula**](https://www.mathsisfun.com/data/standard-deviation-formulas.html)
11. [**Standard normal distribution**](https://www.mathsisfun.com/data/standard-normal-distribution.html)
12. [**Skewness of distribution**](https://www.mathsisfun.com/data/skewness.html)
13. [**Confidence intervals (using std)**](https://www.mathsisfun.com/data/confidence-interval.html)
14. [**Accuracy vs precision (accurate vs hitting closely or density)**](https://www.mathsisfun.com/accuracy-precision.html)
15. [**Probability**](https://www.mathsisfun.com/data/probability.html)
16. [**Probability complement**](https://www.mathsisfun.com/data/probability-complement.html)
17. [**Chi-square test, p\_value, independent, dependent, significance**](https://www.mathsisfun.com/data/chi-square-test.html)
18. [**Variation vs variance**](https://stats.stackexchange.com/questions/88348/is-variation-the-same-as-variance) **- a private case**
19. [**Std vs variance**](https://www.investopedia.com/ask/answers/021215/what-difference-between-standard-deviation-and-variance.asp) **- std is in the same metric as the mean, is the root of variance., allows outliers to influence, will not result in samples cancelling each other without the square root in the formula.**
### **Introduction to Probability**
1. [**Types of events**](https://www.mathsisfun.com/data/probability-events-types.html)
2. [**Independent events**](https://www.mathsisfun.com/data/probability-events-independent.html)
3. [**Conditional proba**](https://www.mathsisfun.com/data/probability-events-conditional.html)
4. [**Proba tree diagrams**](https://www.mathsisfun.com/data/probability-tree-diagrams.html)
5. [**Mutually exclusive events**](https://www.mathsisfun.com/data/probability-events-mutually-exclusive.html)
6. [**Combination and permutations**](https://www.mathsisfun.com/combinatorics/combinations-permutations.html)
7. [**Bayes**](https://www.mathsisfun.com/data/bayes-theorem.html)
8. [**Least squares regresssion**](https://www.mathsisfun.com/data/least-squares-regression.html) **It works by making the total of the square of the errors as small as possible (that is why it is called "least squares"**
9. [**Random variables**](https://www.mathsisfun.com/data/random-variables.html)
10. [**Continuous random variables**](https://www.mathsisfun.com/data/random-variables-continuous.html)
11. [**Random vars mean, std, variance**](https://www.mathsisfun.com/data/random-variables-mean-variance.html)
### **More on Statistics**
1. [**25 concepts**](https://www.datasciencecentral.com/profiles/blogs/25-statistical-concepts-explained-in-simple-english-part-2) **(part 2),** [**29 more concepts**](https://www.datasciencecentral.com/profiles/blogs/29-statistical-concepts-explained-in-simple-english-part-1) **(part1) &** [**part 3**](https://www.datasciencecentral.com/profiles/blogs/29-statistical-concepts-explained-in-simple-english-part-2?fbclid=IwAR0VQFeBaJsm3ouEf7sV5WAupE1cI3PXhzWe9-lUYkZ_XCCF72_3r8w5hrI) **in statistics.**
### **Wiki**
1. [**Marginal probability**](https://en.wikipedia.org/wiki/Marginal_distribution)
2. [**Joint probability**](https://en.wikipedia.org/wiki/Joint_probability_distribution)
3. [**Conditional probability**](https://en.wikipedia.org/wiki/Probability)
4. [**Chain rule**](https://en.wikipedia.org/wiki/Chain_rule_\(probability\)) **- derivatives using the chain rule, on** [**khan**](https://www.khanacademy.org/math/ap-calculus-ab/ab-differentiation-2-new/ab-3-1a/v/chain-rule-introduction)
### **Recommended Courses**
1. [**Another great course on probability**](http://legacydirs.umiacs.umd.edu/~jbg/teaching/INST_414/)**, distribution types, conditional, joint, chain, etc.**
2. [**Kahn**](https://www.khanacademy.org/math/precalculus/prob-comb) **academy**
3. [**A really good intro**](https://www.youtube.com/watch?v=5NMxiOGL39M) **to probability, conditional, joint, etc.**
* [**What are confidence intervals?**](https://towardsdatascience.com/a-very-friendly-introduction-to-confidence-intervals-9add126e714)
**(another angle)** [**The main difference between probability and statistics has to do with knowledge**](https://www.thoughtco.com/probability-vs-statistics-3126368)
* **what are the known facts? Inherent in both probability and statistics is a** [**population**](https://www.thoughtco.com/what-is-a-population-in-statistics-3126308)**,**
* **every individual we are interested in studying, and a sample, consisting of the individuals that are selected from the population.**
* **in probability: would start with us knowing everything about the composition of a population, and then would ask, “What is the likelihood that a selection, or sample, from the population, has certain characteristics?”**
* **In statistics: we have no knowledge about the types of socks in the drawer. we infer properties about the population on the basis of a random sample.**
**Some** [**calculations**](https://www.mathsisfun.com/data/probability.html) **to get you into probability:**
* **Finding out the probability of an event**
* **Of two consecutive events (multiplication)**
* **Of several events (sum)**
* **Etc..**
### **STATISTICAL SAMPLING AND RESAMPLING**
1. [**What is? Method for sampling/resampling, and sampling errors explained.**](https://machinelearningmastery.com/statistical-sampling-and-resampling/) **(cross validation etc)**
# Probability
### **PDF (PROBABILITY DENSITY FUNCTION)**
1. [**Tutorial in scipy**](https://oneau.wordpress.com/2011/02/28/simple-statistics-with-scipy/)
2. [**Array-based tutorial in python with PDF and KDE**](http://firsttimeprogrammer.blogspot.co.il/2015/01/how-to-estimate-probability-density.html)
3. [**Summary of univariate distribution including pdf methods**](https://www.johndcook.com/blog/distributions_scipy/)
### **Kernel Density Estimation**
**This** [**tutorial**](https://mglerner.github.io/posts/histograms-and-kernel-density-estimation-kde-2.html?p=28) **actually explains why we should use KDE over a Histogram, it explains the cons of histograms and how KDE helps solve some issue that we usually encounter in ‘Sparse’ histograms where the distribution is hard to figure out.**
* **Supposedly a better** [**implementation**](https://github.com/Daniel-B-Smith/KDE-for-SciPy) **of KDE than SCIPY**
**How to use KDE? A** [**tutorial**](http://pythonhosted.org/PyQt-Fit/KDE_tut.html) **about kernel density and how to use it in python. Has several good graphs and shows use cases.**
**Video tutorials about Kernel Density:**
1. [**KDE** ](https://www.youtube.com/watch?v=gPWsDh59zdo)
2. **Non parametric** [**Kernel Regression Estimation**](https://www.youtube.com/watch?v=ncF7ArjJFqM)
3. **Non parametric** [**Sieve Estimation**](https://www.youtube.com/watch?v=cqecz-DL-jI)
4. [**Semi- nonparametric estimation**](https://www.youtube.com/watch?v=G1N53K530To)
[**Udacity Video Tutorial**](https://www.youtube.com/watch?v=MEP35FcrQGs\&list=PLAwxTw4SYaPn-ttWkPiUL7NP3lLRdUniJ\&index=80) **- pretty good**
1. **IMPORTANT:** [**Comparison and benchmarks of various KDE algo’s**](https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/)
2. [**Histograms and density plots**](https://towardsdatascience.com/histograms-and-density-plots-in-python-f6bda88f5ac0)
3. [**SK LEARN**](http://scikit-learn.org/stable/modules/density.html#kernel-density-estimation)
4. [**Gaussian KDE in scipy, version 2**](https://www.youtube.com/watch?v=MEP35FcrQGs\&list=PLAwxTw4SYaPn-ttWkPiUL7NP3lLRdUniJ\&index=80)
# Hypothesis Testing
1. (good) [HT explained](https://towardsdatascience.com/hypothesis-testing-explained-how-i-wish-it-was-explained-to-me-14379f7a41c3)
2. [two sample HT with python](https://levelup.gitconnected.com/two-sample-hypothesis-tests-with-python-43e1b8c52306)
# Feature Types
[**Feature Types**](http://www.biostat.umn.edu/~will/6470stuff/Class09-12/Handout09.pdf) **- no permission doc**
**Discrete**
* **Numbers**
* **Categorical**
* **Categorical data are variables that contain label values rather than numeric values.**
**The number of possible values is often limited to a fixed set.**
* **Categorical variables are often called** [**nominal**](https://en.wikipedia.org/wiki/Nominal_category)**.**
* **labels, usually discrete values such as gender, country of origin, marital status, high-school graduate**
**Continuous (the opposite of discrete): real-number values, measured on a continuous scale: height, weight.**
**In order to compute a regression, categorical predictors must be re-expressed as numeric: some form of indicator variables (0/1) with a separate indicator for each level of the factor.**
**Discrete with many values are often treated as continuous, i.e. zone numbers - > binary**
[**Variable types:**](http://www.socialresearchmethods.net/kb/measlevl.php) **Nominal(weather), ordinal(order var 1,2,3), interval(range),**
# Multi Label Classification
**(what is?)** [**Multilabel classification**](https://mlr-org.github.io/mlr-tutorial/devel/html/multilabel/index.html) **is a classification problem where multiple target labels can be assigned to each observation instead of only one like in multiclass classification.**
**Two different approaches exist for multilabel classification:**
* **Problem transformation methods try to transform the multilabel classification into binary or multiclass classification problems.**
* **Algorithm adaptation methods adapt multiclass algorithms so they can be applied directly to the problem.**
**I.e., the** [**Two approaches**](https://mlr-org.github.io/mlr-tutorial/devel/html/multilabel/index.html) **are:**
* **Use a classifier that does multi label**
* **Use any classifier with a wrapper that compares each two labels**
**great** [**PDF**](https://users.ics.aalto.fi/jesse/talks/Multilabel-Part01.pdf) **that explains about multi label classification and especially metrics,** [**part 2 here**](https://users.ics.aalto.fi/jesse/talks/Multilabel-Part02.pdf)
[**An awesome Paper**](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.104.9401\&rep=rep1\&type=pdf) **that explains all of these methods in detail, also available** [**here**](https://www.researchgate.net/publication/273859036_Multi-Label_Classification_An_Overview)**!**
**PT1: for each sample select one label, remove all others.**
**PT2: remove every sample which has multi labels.**
**PT3: for every combo of labels create a single-label, i.e. A\&B, A\&C etc..**
**PT4: (most common) create L datasets, for each label learn a binary representation, i.e., is it there or not.**
**PT5: duplicate each sample with only one of its labels**
**PT6: read the paper**
**There are other approaches for doing it within algorithms, they rely on the ideas PT3\4\5\6 implemented in the algorithms, or other tricks.**
**They also introduce Label cardinality and label density.**
[**Efficient net**](https://medium.com/gumgum-tech/multi-label-classification-for-threat-detection-part-1-60318b90ce11)**,** [**part 2**](https://medium.com/gumgum-tech/multi-label-image-classifier-for-threat-detection-with-fp16-inference-part-2-40fe0f9a93b3) **- EfficientNet is based on a network derived from a neural architecture search and novel compound scaling method is applied to iteratively build more complex network which achieves state of the art accuracy on multiclass classification tasks. Compound scaling refers to increasing the network dimensions in all three scaling formats using a novel strategy.**\
\
**Multi label confusion matrices with sklearn**
[ **Scikit multilearn package**](http://scikit.ml/index.html)
# Distribution
### **TYPES**
**(What are?) probabilities in a distribution always add up to 1.**
* [**More distribution explanations**](https://machinelearningmastery.com/statistical-data-distributions/)
* [**A very good explanation**](https://blog.cloudera.com/blog/2015/12/common-probability-distributions-the-data-scientists-crib-sheet/)
* [**A very wordy explanation**](http://people.stern.nyu.edu/adamodar/New_Home_Page/StatFile/statdistns.htm) **(figure2)**
1. [**Poison and poison process**](https://towardsdatascience.com/the-poisson-distribution-and-poisson-process-explained-4e2cb17d459)
**Comparing distributions:**
1. [**Kolmogorov smirnov not good for categoricals.**](https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test)
2. [**Comparing two**](https://math.stackexchange.com/questions/159940/comparing-distribution-of-two-data-sets)
3. [**Khan academy**](https://www.khanacademy.org/math/ap-statistics/quantitative-data-ap/describing-comparing-distributions/v/comparing-distributions)
4. [**Visually**](https://www.stat.auckland.ac.nz/~ihaka/787/lectures-distrib.pdf)
5. [**When they are not normal**](https://www.quora.com/Which-statistical-test-to-use-to-quantify-the-similarity-between-two-distributions-when-they-are-not-normal)
6. [**Using train / test trick**](https://towardsdatascience.com/how-dis-similar-are-my-train-and-test-data-56af3923de9b)
7. [**Code for Identifying distribution type and params, based on best fit.**](https://stackoverflow.com/questions/37487830/how-to-find-probability-distribution-and-parameters-for-real-data-python-3)
8.
### **Gaussian \ Normal Distribution**
[**“ if you collect data and it is not normal, “you need to collect more data”**](https://www.isixsigma.com/topic/normal-distributions-why-does-it-matter/)
[**Beautiful graphs**](https://stats.stackexchange.com/questions/116550/why-do-we-have-to-assume-normality-for-a-one-sample-t-test)
[**The normal distribution is popular for two reasons:**](https://www.quora.com/Why-do-we-use-the-normal-distribution-The-normal-is-an-approximation-Why-dont-we-use-a-simpler-distribution-with-simpler-numbers-to-memorize-If-it-is-an-approximation-does-it-have-to-be-so-specific)
1. **It is the most common distribution in nature (as distributions go)**
2. **An enormous number of statistical relationships become clear and tractable if one assumes the normal.**
**Sure, nothing in real life exactly matches the Normal. But it is uncanny how many things come close.**
**this is partly due to the Central Limit Theorem, which says that if you average enough unrelated things, you eventually get the Normal.**
* **the Normal distribution in statistics is a special world in which the math is straightforward and all the parts fit together in a way that is easy to understand and interpret.**
* **It may not exactly match the real world, but it is close enough that this one simplifying assumption allows you to predict lots of things, and the predictions are often pretty reasonable.**
* **statistically convenient.**
* **represented by basic statistics**
* **average**
* **variance (or standard deviation) - the average of what's left when you take away the average, but to the power of 2.**
**In a statistical test, you need the data to be normal to guarantee that your p-values are accurate with your given sample size.**
**If the data are not normal, your sample size may or may not be adequate, and it may be difficult for you to know which is true.**
### **COMPARING DISTRIBUTIONS**
1. **Categorical data can be transformed to a histogram i.e., #class / total and then measured for distance between two histograms’, e.g., train and production. Using earth mover distance** [**python**](https://jeremykun.com/2018/03/05/earthmover-distance/) [**git wrapper to c**](https://github.com/pdinges/python-emd)**, linear programming, so its slow.**
2. [**Earth movers**](https://towardsdatascience.com/earth-movers-distance-68fff0363ef2)**.**
3. [**EMD paper**](http://infolab.stanford.edu/pub/cstr/reports/cs/tr/99/1620/CS-TR-99-1620.ch4.pdf)
4. **Also check KL DIVERGENCE in the information theory section.**
5. [**Bengio**](https://arxiv.org/abs/1901.10912) **et al, transfer objective for learning to disentangle casual mechanisms - We propose to meta-learn causal structures based on how fast a learner adapts to new distributions arising from sparse distributional changes**
# Distribution Transformation
[**Top 3 methods for handling skewed data**](https://towardsdatascience.com/top-3-methods-for-handling-skewed-data-1334e0debf45)**. Log, square root, box cox transformations**
### **BOX COX**
[**Power transformations**](https://machinelearningmastery.com/power-transforms-with-scikit-learn/?fbclid=IwAR37SGKEXWQ_39qZLKAQ5WunSECo0JXsd3qgz3dPGITTGcVwHJla-_7GLKg)
**(What is the Box-Cox Power Transformation?)**
* **a procedure to identify an appropriate exponent (Lambda = l) to use to transform data into a “normal shape.”**
* **The Lambda value indicates the power to which all data should be raised.**
[**The Box-Cox transformation is a useful family of transformations.** ](http://www.itl.nist.gov/div898/handbook/eda/section3/eda336.htm)
* **Many statistical tests and intervals are based on the assumption of normality.**
* **The assumption of normality often leads to tests that are simple, mathematically tractable, and powerful compared to tests that do not make the normality assumption.**
* **Unfortunately, many real data sets are in fact not approximately normal.**
* **However, an appropriate transformation of a data set can often yield a data set that does follow approximately a normal distribution.**
* **This increases the applicability and usefulness of statistical techniques based on the normality assumption.**
**IMPORTANT:!! After a transformation (c), we need to measure the normality of the resulting transformation (d) .**
* **One measure is to compute the correlation coefficient of a** [**normal probability plot**](http://www.itl.nist.gov/div898/handbook/eda/section3/normprpl.htm) **=> (d).**
* **The correlation is computed between the vertical and horizontal axis variables of the probability plot and is a convenient measure of the linearity of the probability plot**
* **In other words: the more linear the probability plot, the better a normal distribution fits the data!**
[**\*NOTE: another useful link that explains it with figures, but i did not read it.**](http://blog.minitab.com/blog/applying-statistics-in-quality-projects/how-could-you-benefit-from-a-box-cox-transformation)
**GUARANTEED NORMALITY?**
* **NO!**
* **This is because it actually does not really check for normality;**
* **the method checks for the smallest standard deviation.**
* **The assumption is that among all transformations with Lambda values between -5 and +5, transformed data has the highest likelihood – but not a guarantee – to be normally distributed when standard deviation is the smallest.**
* **it is absolutely necessary to always check the transformed data for normality using a probability plot. (d)**
**+ Additionally, the Box-Cox Power transformation only works if all the data is positive and greater than 0.**
**+ achieved easily by adding a constant ‘c’ to all data such that it all becomes positive before it is transformed. The transformation equation is then:**
[**COMMON TRANSFORMATION FORMULAS (based on the actual formula)**](http://www.statisticshowto.com/box-cox-transformation/)
**Finally: An awesome** [**tutorial (dead),**](http://www.kmdatascience.com/2017/07/box-cox-transformations-in-python.html) [**here is a new one**](https://towardsdatascience.com/box-cox-transformation-explained-51d745e34203#:~:text=scipy.stats.boxcox\(\),the%2095%25%20confidence%20interval\).) **in python with** [**code examples**](https://github.com/kentmacdonald2/Box-Cox-Transformation-Python-Example)**, there is also another code example** [**here** ](https://stackoverflow.com/questions/33944129/python-library-for-data-scaling-centering-and-box-cox-transformation)**“Simply pass a 1-D array into the function and it will return the Box-Cox transformed array and the optimal value for lambda. You can also specify a number, alpha, which calculates the confidence interval for that value. (For example, alpha = 0.05 gives the 95% confidence interval).”**
**\* Maybe there is a slight problem in the python vs R code,** [**details here**](http://shahramabyari.com/2015/12/21/data-preparation-for-predictive-modeling-resolving-skewness/)**, but needs investigating.**
### **MANN-WHITNEY U TEST**
**(**[**what is?**](https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test)**) - the Mann–Whitney U test is a** [**nonparametric**](https://en.wikipedia.org/wiki/Nonparametric_statistics) [**test**](https://en.wikipedia.org/wiki/Statistical_hypothesis_test) **of the** [**null hypothesis**](https://en.wikipedia.org/wiki/Null_hypothesis) **that it is equally likely that a randomly selected value from one sample will be less than or greater than a randomly selected value from a second sample.**
**In other words: This test can be used to determine whether two independent samples were selected from populations having the same distribution.**
**Unlike the** [**t-test**](https://en.wikipedia.org/wiki/T-test) **it does not require the assumption of** [**normal distributions**](https://en.wikipedia.org/wiki/Normal_distribution)**. It is nearly as efficient as the t-test on normal distributions.**
### **NULL HYPOTHESIS**
1. [**What is chi-square and what is a null hypothesis, and how do we calculate observed vs expected and check if we can reject the null and get significant difference.**](https://medium.com/greyatom/goodness-of-fit-using-chi-square-be5bba375caf)
2. **Analytics vidhya**
1. [**What is hypothesis testing** ](https://www.analyticsvidhya.com/blog/2015/09/hypothesis-testing-explained/)
2. [**Intro to t-tests analytics vidhya**](https://www.analyticsvidhya.com/blog/2019/05/statistics-t-test-introduction-r-implementation/?utm_source=facebook.com\&utm_medium=social) **- always good**
3. [**Anova analysis of variance**](https://www.analyticsvidhya.com/blog/2018/01/anova-analysis-of-variance/?utm_source=facebook.com\&utm_medium=social\&fbclid=IwAR1lMhaoKevShaIDpNoRNPL-V7y_LMscZSPG_0Dp1qvCkhDoJgzyt4fMDKM)**, one way, two way, manova**
1. **if the means of two or more groups are significantly different from each other. ANOVA checks the impact of one or more factors by comparing the means of different samples.**
2. **A one-way ANOVA tells us that at least two groups are different from each other. But it won’t tell us which groups are different.**
3. **For such cases, when the outcome or dependent variable (in our case the test scores) is affected by two independent variables/factors we use a slightly modified technique called two-way ANOVA.**
3. **multivariate case and the technique we will use to solve it is known as MANOVA.**
# Normalization & Scaling
1. [**A comparison of normalization / scaling techniques in sklearn**](http://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html#sphx-glr-auto-examples-preprocessing-plot-all-scaling-py)
2. [**Another great explanation on sklearn and (general) scaling**](http://benalexkeen.com/feature-scaling-with-scikit-learn/) **- normal, min max, etc..**
3. [**Normalization\standardize features** ](http://machinelearningmastery.com/normalize-standardize-machine-learning-data-weka/)
* **data has varying scales**
* **Normalize between range 0 to 1.**
* **When the algorithm you are using does not make assumptions about the distribution of your data, such as k-nearest neighbors and artificial neural networks.**
* **Standardize, mean of 0 and a std of 1:**
* **When the algorithm assumes a gaussian dist, such as linear regression, logistic regression and linear discriminant analysis. LR, LogR, LDA**
**\*\*Generally, it is a good idea to standardize data that has a Gaussian (bell curve) distribution and normalize otherwise.4. In general terms, we should test 0,1 or -1,1 empirically and possibly match the range to the NN gates/activation function etc.**
# Regularization
[**Watch this.**](https://www.youtube.com/watch?v=sO4ZirJh9ds) **Also explains about ISO surfaces, lp norm, sparseness.**
**(what is?)** [**Regularization (in linear regression**](https://datanice.github.io/machine-learning-101-what-is-regularization-interactive.html)**) - to find the best model we define a loss or cost function that describes how well the model fits the data, and try minimize it. For a complex model that fits even the noise, i.e., over fitted, we penalize it by adding a complexity term that would add BIGGER LOSS for more complex models.**
* **Bigger lambda -> high complexity models (deg 3) are ruled out, more punishment.**
* **Smaller lambda -> models with high training error are rules out. I.e., linear model on non linear data?, i.e., deg 1.**
* **Optimal is in between (deg 2)**
[**L1 - for sparse models,** ](https://stats.stackexchange.com/questions/45643/why-l1-norm-for-sparse-models)
[**L1 vs L2, some formula**](https://towardsdatascience.com/l1-and-l2-regularization-methods-ce25e7fc831c)
[**Rehearsal on vector normalization**](http://mathworld.wolfram.com/VectorNorm.html) **- for l1,l2,l3,l4 etc, what is the norm? (absolute value in certain cases)**
**(Difference between? And features of)** [**L1 vs L2**](http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/) **as loss function and regularization.**
* **L1 - moves the regressor faster, feature selection by sparsing coefficients (zeroing them), with sparse algorithms it is computationally efficient, with others no, so use L2.**
* **L2 - moves slower, doesn't sparse, computationally efficient.**
**Why does L1 lead to sparity?**
* [**Intuition**](https://www.quora.com/Why-is-L1-regularization-supposed-to-lead-to-sparsity-than-L2) **+** [**some mathematical info**](https://www.quora.com/What-is-the-difference-between-L1-and-L2-regularization)
* **L1 & L2 regularization add constraints to the optimization problem. The curve H0 is the hypothesis. The solution is a set of points where the H0 meets the constraints.**
* **In L2 the the hypothesis is tangential to the ||w||\_2. The point of intersection has both x1 and x2 components. On the other hand, in L1, due to the nature of ||w||\_1, the viable solutions are limited to the corners of the axis, i.e., x1. So that the value of x2 = 0. This means that the solution has eliminated the role of x2 leading to sparsity.**
* **This can be extended to a higher dimensions and you can see why L1 regularization leads to solutions to the optimization problem where many of the variables have value 0.**
* **In other words, L1 regularization leads to sparsity.**
* **Also considered feature selection - although with LibSVM the recommendation is to feature select prior to using the SVM and use L2 instead.**
[**L1 sparsity - intuition #2**](https://www.quora.com/What-is-the-difference-between-L1-and-L2-regularization)
* **For simplicity, let's just consider the 1-dimensional case.**
* **L2:**
* **L2-regularized loss function F(x)=f(x)+λ∥x∥^2 is smooth.**
* **This means that the optimum is the stationary point (0-derivative point).**
* **The stationary point of F can get very small when you increase λ, but it will still won't be 0 unless f′(0)=0.**
* **L1:**
* **regularized loss function F(x)=f(x)+λ∥x∥ is non-smooth, i.e., a min knee of 0.**
* **It's not differentiable at 0.**
* **Optimization theory says that the optimum of a function is either the point with 0-derivative or one of the irregularities (corners, kinks, etc.). So, it's possible that the optimal point of F is 0 even if 0 isn't the stationary point of f.**
* **In fact, it would be 0 if λ is large enough (stronger regularization effect). Below is a graphical illustration.**
**In multi-dimensional settings: if a feature is not important, the loss contributed by it is small and hence the (non-differentiable) regularization effect would turn it off.**
[**Intuition + formulation, which is pretty good:**](https://stats.stackexchange.com/questions/45643/why-l1-norm-for-sparse-models)
**(**[**did not watch**](https://www.coursera.org/learn/machine-learning/lecture/db3jS/model-representation)**) but here is andrew ng talks about cost functions.**
**L2 regularization** [**equivalent to Gaussian prior**](https://stats.stackexchange.com/questions/163388/l2-regularization-is-equivalent-to-gaussian-prior)
**L1 regularization** [**equivalent to a Laplacean Prior**](https://stats.stackexchange.com/questions/163388/l2-regularization-is-equivalent-to-gaussian-prior)**(same link as above) - “Similarly the relationship between L1 norm and the Laplace prior can be undestood in the same fashion. Take instead of a Gaussian prior, a Laplace prior combine it with your likelihood and take the logarithm.“**\
[**How does regularization look like in SVM**](https://datascience.stackexchange.com/questions/4943/intuition-for-the-regularization-parameter-in-svm) **- controlling ‘C’**
# Information Theory
### **ENTROPY / INFORMATION GAIN**
1. [**Shannon entropy in python, basically entropy(value counts)**](https://www.kite.com/python/answers/how-to-calculate-shannon-entropy-in-python)
2. [**Mastery on plogp entropy function**](https://machinelearningmastery.com/what-is-information-entropy/)
3. [**Entropy functions**](https://gist.github.com/jaradc/eeddf20932c0347928d0da5a09298147)
### Tools
1. [EntroPy](https://raphaelvallat.com/entropy/build/html/index.html#) / [AntroPy](https://raphaelvallat.com/antropy/build/html/index.html) \[[Git](https://github.com/raphaelvallat/antropy)]
2. [PyInform](https://github.com/ELIFE-ASU/PyInform) \[[Docs](https://elife-asu.github.io/PyInform/index.html)]- PyInform is a python library of information-theoretic measures for time series data. PyInform is backed by the [Inform](https://github.com/elife-asu/inform) C library.
3. [PyEntropy](https://github.com/nikdon/pyEntropy)
### Tutorials
[**Great tutorial on all of these topics**](https://www.bogotobogo.com/python/scikit-learn/scikt_machine_learning_Decision_Tree_Learning_Informatioin_Gain_IG_Impurity_Entropy_Gini_Classification_Error.php)**\*\*\***
[**Entropy**](https://www.techleer.com/articles/496-a-short-introduction-to-entropy-cross-entropy-and-kl-divergence-aurelien-geron/) **- lack of order or lack of predictability (**[**excellent slide lecture by Aurelien Geron**](https://www.youtube.com/watch?time_continue=3\&v=ErfnhcEV1O8)**)**
**Cross entropy will be equal to entropy if the probability distributions of p (true) and q(predicted) are the same. However, if cross entropy is bigger (known as relative\_entropy or kullback leibler divergence)**
**In this example we want the cross entropy loss to be zero, i.e., when we have a one hot vector and a predicted vector which are identical, i.e., 100% in the same class for predicted and true, we get 0. In all other cases we get some number that gets larger if the predicted class probability is lower than zero as seen here:**
**Formula for 2 classes:**
**NOTE: Entropy can be generalized as a formula for N > 2 classes:**
**(We want to grow a simple tree)** [**awesome pdf tutorial**](http://www.ke.tu-darmstadt.de/lehre/archiv/ws0809/mldm/dt.pdf)**→ a good attribute prefers attributes that split the data so that each successor node is as pure as possible**
* **i.e., the distribution of examples in each node is so that it mostly contains examples of a single class**
* **In other words: We want a measure that prefers attributes that have a high degree of „order“:**
* **Maximum order: All examples are of the same class**
* **Minimum order: All classes are equally likely → Entropy is a measure for (un-)orderedness Another interpretation:**
* **Entropy is the amount of information that is contained**
* **all examples of the same class → no information**
**Entropy is the amount of unorderedness in the class distribution of S**
**IMAGE above:**
* **Maximal value when the equal class distribution**
* **Minimal value when only one class is in S**
**So basically if we have the outlook attribute and it has 3 categories, we calculate the entropy for E(feature=category) for all 3.**
**INFORMATION: The I(S,A) formula below.**
**What we actually want is the average entropy of the entire split, that corresponds to an entire attribute, i.e., OUTLOOK (sunny & overcast & rainy)**
**Information Gain: is actually what we gain by subtracting information from the entropy.**
**In other words we find the attributes that maximizes that difference, in other other words, the attribute that reduces the unorderness / lack of order / lack of predictability.**
**The BIGGER GAIN is selected.**
**There are some properties to Entropy that influence INFO GAIN (?):**
**There are some disadvantages with INFO GAIN, done use it when an attribute has many number values, such as “day” (date wise) 05/07, 06/07, 07/07..31/07 etc.**
**Information gain is biased towards choosing attributes with a large number of values and causes:**
* **Overfitting**
* **fragmentation**
**We measure Intrinsic information of an attribute, i.e., Attributes with higher intrinsic information are less useful.**
**We define Gain Ratio as info-gain with less bias toward multi value attributes, ie., “days”**
**NOTE: Day attribute would still win with the Gain Ratio, Nevertheless: Gain ratio is more reliable than Information Gain**
**Therefore, we define the alternative, which is the GINI INDEX. It measures impurity, we define the average Gini, and the Gini Gain.**
[**FINALLY, further reading about decision trees and examples of INFOGAIN and GINI here.**](http://www.ke.tu-darmstadt.de/lehre/archiv/ws0809/mldm/dt.pdf)
[**Variational bounds on mutual informati**](https://arxiv.org/abs/1905.06922v1)**on**
### **CROSS ENTROPY, RELATIVE ENT, KL-D, JS-D, SOFT MAX**
1. [A really good explanation on all of them](https://www.countbayesie.com/blog/2017/5/9/kullback-leibler-divergence-explained)
2. [Another good one on all of them](https://gombru.github.io/2018/05/23/cross_entropy_loss/)
3. [mastery on a gentle intro to CE](https://machinelearningmastery.com/cross-entropy-for-machine-learning/)
4. [Mastery on entropy](https://machinelearningmastery.com/divergence-between-probability-distributions/), kullback leibler divergence (asymmetry), jensen-shannon divergence (symmetry) (has code)
5. [Entropy, mutual information and KL Divergence by AurelienGeron](https://www.techleer.com/articles/496-a-short-introduction-to-entropy-cross-entropy-and-kl-divergence-aurelien-geron/)
6. [Gensim on divergence](https://radimrehurek.com/gensim/auto_examples/tutorials/run_distance_metrics.html#sphx-glr-auto-examples-tutorials-run-distance-metrics-py) metrics such as KL jaccard etc, pros and cons, lda is a mess on small data.
7. [Advise on KLD](https://datascience.stackexchange.com/questions/9262/calculating-kl-divergence-in-python)ivergence
8. [Neural machine translation using pytorch and CE](https://towardsdatascience.com/neural-machine-translation-15ecf6b0b)
### SOF**TMAX**
1. [**Understanding softmax**](https://medium.com/data-science-bootcamp/understand-the-softmax-function-in-minutes-f3a59641e86d)
2. [**Softmax and negative likelihood (NLL)**](https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/)
3. [**Softmax vs cross entropy**](https://www.quora.com/Is-the-softmax-loss-the-same-as-the-cross-entropy-loss#) **- Softmax loss and cross-entropy loss terms are used interchangeably in industry. Technically, there is no term as such Softmax loss. people use the term "softmax loss" when referring to "cross-entropy loss". The softmax classifier is a linear classifier that uses the cross-entropy loss function. In other words, the gradient of the above function tells a softmax classifier how exactly to update its weights using some optimization like** [**gradient descent**](https://en.wikipedia.org/wiki/Gradient_descent)**.**
**The softmax() part simply normalises your network predictions so that they can be interpreted as probabilities. Once your network is predicting a probability distribution over labels for each input, the log loss is equivalent to the cross entropy between the true label distribution and the network predictions. As the name suggests, softmax function is a “soft” version of max function. Instead of selecting one maximum value, it breaks the whole (1) with maximal element getting the largest portion of the distribution, but other smaller elements getting some of it as well.**
**This property of softmax function that it outputs a probability distribution makes it suitable for probabilistic interpretation in classification tasks.**
**Cross entropy indicates the distance between what the model believes the output distribution should be, and what the original distribution is. Cross entropy measure is a widely used alternative of squared error. It is used when node activations can be understood as representing the probability that each hypothesis might be true, i.e. when the output is a probability distribution. Thus it is used as a loss function in neural networks which have softmax activations in the output layer.**
### **TIME SERIES ENTROPY**
1. [**entroPY**](https://raphaelvallat.com/entropy/build/html/index.html) **- EntroPy is a Python 3 package providing several time-efficient algorithms for computing the complexity of one-dimensional time-series. It can be used for example to extract features from EEG signals.**
[**Approximate entropy paper**\ ](https://journals.physiology.org/doi/pdf/10.1152/ajpheart.2000.278.6.H2039)
**print(perm\_entropy(x, order=3, normalize=True)) # Permutation entropy**
**print(spectral\_entropy(x, 100, method='welch', normalize=True)) # Spectral entropy**
**print(svd\_entropy(x, order=3, delay=1, normalize=True)) # Singular value decomposition entropy**
**print(app\_entropy(x, order=2, metric='chebyshev')) # Approximate entropy**
**print(sample\_entropy(x, order=2, metric='chebyshev')) # Sample entropy**
**print(lziv\_complexity('01111000011001', normalize=True)) # Lempel-Ziv complexity**
1. [**PyInform**](https://elife-asu.github.io/PyInform/index.html)
### **Complement Objective Training**
1. **Article by** [**LightTag**](https://www.lighttag.io/blog/complement-objective-training-with-pytorch-lightning/)**,** [**paper**](https://arxiv.org/pdf/1903.01182.pdf) **-**
**COT is a technique to effectively provide explicit negative feedback to our model. The technique gives us non-zero gradients with respect to incorrect classes, which are used to update the model's parameters.**
**COT doesn't replace cross-entropy. It's used as a second training step as follows: We run cross-entropy, and then we do a COT step. We minimize the cross-entropy between our target distribution. That's equivalent to maximizing the likelihood of the correct class. During the COT step, we maximize the entropy of the complement distribution. We pretend that the correct class isn't an option and make the remaining classes equally likely.**
**But, since the true class is an option, and we're training for it explicitly, maximizing the true classes probability and pushing the remaining classes to be equally likely is actually pushing their probabilities to 0 explicitly, which provides explicit gradients to propagate through our model.**
# Game Theory
[**In computer science**](https://www.analyticsvidhya.com/blog/2019/11/game-theory-ai/)
# Multi CPU Processing
[**Numpy**](https://gitlab.com/tenzing/shared-array) **on multi process, and** [**how to use it.**](https://medium.com/analytics-vidhya/multiprocessing-for-data-scientists-in-python-427b2ff93af1)
1. [**Pandas on multi process**](https://github.com/nalepae/pandarallel)
2. **\*\*\***[**Dask**](https://docs.dask.org/en/latest/) **- youtube** [**intros**](https://www.youtube.com/channel/UCj9eavqmvwaCyKhIlu2GaoA)
1. **Diagnostic** [**dashboards**](https://www.youtube.com/watch?v=N_GqzcuGLCY)
2. [**Ditributed sklearn**](https://www.youtube.com/watch?v=5Zf6DQaf7jk) **(amazing)**
3. [**Dask vs swifter vs vectorize**](https://gdcoder.com/speed-up-pandas-apply-function-using-dask-or-swifter-tutorial/)
1. **Dask is dask**
2. **Swifter will attempt to understand if dask or pandas apply should be used, looks like its using multi cpu so it may not be just using dask on the backend?**
3. **Vectorize is just another option**
4. [**Multi process cpu example**](https://datascience.blog.wzb.eu/2017/06/19/speeding-up-nltk-with-parallel-processing/)
5. [**Medium on MP, using MP pool, Ray etc.**](https://medium.com/distributed-computing-with-ray/how-to-scale-python-multiprocessing-to-a-cluster-with-one-line-of-code-d19f242f60ff)
6. [**Async (multi process/thread/coroutines/asyncio)**](https://medium.com/velotio-perspectives/an-introduction-to-asynchronous-programming-in-python-af0189a88bbb)
# Benchmarking
## Algorithms
1. [scikit bench](https://github.com/IntelPython/scikit-learn_bench) - "scikit-learn\_bench benchmarks various implementations of machine learning algorithms across data analytics frameworks. It currently support the scikit-learn, DAAL4PY, cuML, and XGBoost frameworks for commonly used machine learning algorithms."
**Numpy Blas:**
1. [**How do i know which version of blas is installed**](https://stackoverflow.com/questions/37184618/find-out-if-which-blas-library-is-used-by-numpy)
2. [**Benchmark OpenBLAS, Intel MKL vs ATLAS**](https://github.com/tmolteno/necpp/issues/18)
1. [**Another comparison**](http://markus-beuckelmann.de/blog/boosting-numpy-blas.html)
2. 
**GLUE:**
1. [**Glue / super glue** ](https://gluebenchmark.com/leaderboard/?fbclid=IwAR17Xo2pgpDVE_ZuwITDSi07FLM6S2f1VTXiLywwr2NnUGqS8AdndZLQpXI)
**State of the art in AI:**
1. **In terms of** [**domain X datasets**](https://www.stateoftheart.ai/)
**Cloud providers:**
* [**Part 1**](https://rare-technologies.com/machine-learning-hardware-benchmarks/)**,** [**part 2 y gensim**](https://rare-technologies.com/machine-learning-benchmarks-hardware-providers-gpu-part-2/)
*
**Datasets:**
* [**EFF FF Benchmarks in AI**](https://www.eff.org/ai/metrics)
**Hardware:**
* [**Nvidia**](https://www.phoronix.com/scan.php?page=article\&item=nvidia-rtx2080ti-tensorflow\&num=1) **1070 vs 1080 vs 2080**
* [**Cpu vs GPU benchmarking for CNN\Test\LTSM\BDLTSM**](http://minimaxir.com/2017/07/cpu-or-gpu/) **- google and amazon vs gpu**
* [**Nvidia GPUs**](https://www.pugetsystems.com/labs/hpc/TitanXp-vs-GTX1080Ti-for-Machine-Learning-937/) **- titax Xp\1080TI\1070 on googlenet**
* **March\17 -** [**Nvidia GPUs for desktop**](https://medium.com/@timcamber/deep-learning-pc-build-5cffa71ad97)**, in terms of price and cuda units, the bottom line is 1060-1080.**
* [**Another bench up to 2013**](http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/) **- regarding many GPUS vs CPUs in terms of BW**
**Platforms**
* [**Cntk vs tensorflow**](http://minimaxir.com/2017/06/keras-cntk/)
* [**CNTK, TEnsor, torch, etc on cpu and gpu**](https://arxiv.org/pdf/1608.07249.pdf)
**Algorithms:**
* [**Comparing**](https://martin-thoma.com/comparing-classifiers/) **accuracy, speed, memory and 2D visualization of classifiers:**
[**SVM,**](http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) [**k-nearest neighbors,**](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html) [**Random Forest,**](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) [**AdaBoost Classifier,**](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html) [**Gradient Boosting,**](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html) [**Naive, Bayes,**](http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.html) [**LDA,**](http://scikit-learn.org/0.16/modules/generated/sklearn.lda.LDA.html) [**QDA,**](http://scikit-learn.org/0.16/modules/generated/sklearn.qda.QDA.html) [**RBMs,**](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.BernoulliRBM.html) [**Logistic Regression,**](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) [**RBM**](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.BernoulliRBM.html) **+ Logistic Regression Classifier**
* [**LSTM vs cuDNN LS1TM**](https://chainer.org/general/2017/03/15/Performance-of-LSTM-Using-CuDNN-v5.html) **- batch size of power 2 matters, the latter is faster.**
**Scaling networks and predicting performance of NN:**
* [**A great overview of NN type**](https://www.youtube.com/watch?v=lgK0BlXdOCw\&feature=youtu.be)**s, but the idea behind the video is to create a system that can predict train time and possibly accuracy when scaling networks using multiple GPUs, there is also a nice slide about general hardware recommendations.**
**NLP**
* [**XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization**](https://github.com/google-research/xtreme/blob/master/README.md)
#### Multi-Task Learning
1. [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://arxiv.org/abs/1705.07115) (Yarin Gal) [GitHub](https://github.com/ranandalon/mtl) - "In this paper we make the observation that the performance of such systems is strongly dependent on the relative weighting between each task’s loss. Tuning these weights by hand is a difficult and expensive process, making multi-task learning prohibitive in practice. We propose a principled approach to multi-task deep learning which weighs multiple loss functions by considering the homoscedastic uncertainty of each task. "
2. [Ruder on Multi Task Learning](https://ruder.io/multi-task/) - "By sharing representations between related tasks, we can enable our model to generalize better on our original task. This approach is called Multi-Task Learning (MTL) and will be the topic of this blog post."
# Features
### **CORRELATION**
1. [**Pearson**](https://machinelearningmastery.com/how-to-use-correlation-to-understand-the-relationship-between-variables/)
#### **CORRELATION VS COVARIANCE**
1. [**Correlation is between -1 to 1, covariance is -inf to inf, units in covariance affect the scale, so correlation is preferred, it is normalized.** ](https://towardsdatascience.com/correlation-coefficient-clearly-explained-f034d00b66ac)**Correlation is a measure of association. Correlation is used for bivariate analysis. It is a measure of how well the two variables are related.**\
**Covariance is also a measure of association. Covariance is a measure of the relationship between two random variables.**
2.
#### **CORRELATION BETWEEN FEATURE TYPES**
1. **Association vs correlation - correlation is a measure of association and a yes no question without assuming linearity**
2. [**A great article in medium**](https://medium.com/@outside2SDs/an-overview-of-correlation-measures-between-categorical-and-continuous-variables-4c7f85610365)**, covering just about everything with great detail and explaining all the methods plus references.**
3. **Heat maps for categorical vs target - groupby count per class, normalize by total count to see if you get more grouping in a certain combination of cat/target than others.**
4. [**Anova**](https://www.researchgate.net/post/Which_test_do_I_use_to_estimate_the_correlation_between_an_independent_categorical_variable_and_a_dependent_continuous_variable)**/**[**log regression**](https://www.statalist.org/forums/forum/general-stata-discussion/general/1470627-correlation-between-continous-and-categorical-variable) [**2\*,**](https://dzone.com/articles/correlation-between-categorical-and-continuous-var-1) [**git**](https://github.com/ShitalKat/Correlation/blob/master/Correlation%20between%20categorical%20and%20continuous%20variables.ipynb)**,** [**3**](https://www.edvancer.in/DESCRIPTIVE+STATISTICS+FOR+DATA+SCIENCE-2)**, for numeric/**[**cont vs categorical**](https://www.quora.com/How-can-I-measure-the-correlation-between-continuous-and-categorical-variables) **- high F score from anova hints about association between a feature and a target, i.e., the importance of the feature to separating the target.**
5. **Anova youtube** [**1**](https://www.youtube.com/watch?v=ITf4vHhyGpc)**,** [**2**](https://www.youtube.com/watch?v=-yQb_ZJnFXw)
**image by multiple possible sources,** [**rayhanul islam**](https://www.quora.com/How-can-I-measure-the-correlation-between-continuous-and-categorical-variables)**,** [**statistics for fun**](https://www.facebook.com/statneil/photos/a.787373884990868/839856346409288/?type=3)**.**
6. [**Cat vs cat**](https://towardsdatascience.com/the-search-for-categorical-correlation-a1cf7f1888c9)**, many metrics - on medium**
#### **CORRELATION VISUALIZATION**
[**Feature space**](https://towardsdatascience.com/escape-the-correlation-matrix-into-feature-space-4d71c51f25e5)
### **PREDICTIVE POWER SCORE (PPS)**
[**Is an asymmetric, data-type-agnostic score for predictive relationships between two columns that ranges from 0 to 1.**](https://towardsdatascience.com/rip-correlation-introducing-the-predictive-power-score-3d90808b9598) [**github**](https://github.com/8080labs/ppscore)
**Too many scenarios where the correlation is 0. This makes me wonder if I missed something… (Excerpt from the** [**image by Denis Boigelot**](https://en.wikipedia.org/wiki/Correlation_and_dependence)**)**
**Regression**
**In case of an regression, the ppscore uses the mean absolute error (MAE) as the underlying evaluation metric (MAE\_model). The best possible score of the MAE is 0 and higher is worse. As a baseline score, we calculate the MAE of a naive model (MAE\_naive) that always predicts the median of the target column. The PPS is the result of the following normalization (and never smaller than 0):**
**PPS = 1 - (MAE\_model / MAE\_naive)**
**Classification**
**If the task is a classification, we compute the weighted F1 score (wF1) as the underlying evaluation metric (F1\_model). The F1 score can be interpreted as a weighted average of the precision and recall, where an F1 score reaches its best value at 1 and worst score at 0. The relative contribution of precision and recall to the F1 score are equal. The weighted F1 takes into account the precision and recall of all classes weighted by their support as described** [**here**](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html)**. As a baseline score (F1\_naive), we calculate the weighted F1 score for a model that always predicts the most common class of the target column (F1\_most\_common) and a model that predicts random values (F1\_random). F1\_naive is set to the maximum of F1\_most\_common and F1\_random. The PPS is the result of the following normalization (and never smaller than 0):**
**PPS = (F1\_model - F1\_naive) / (1 - F1\_naive)**
### **MUTUAL INFORMATION COEFFICIENT**
[**Paper**](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3325791/) **- we present a measure of dependence for two-variable relationships: the maximal information coefficient (MIC). MIC captures a wide range of associations both functional and not, and for functional relationships provides a score that roughly equals the coefficient of determination (R2) of the data relative to the regression function.**
**Computing MIC**\
**(A) For each pair (x,y), the MIC algorithm finds the x-by-y grid with the highest induced mutual information. (B) The algorithm normalizes the mutual information scores and compiles a matrix that stores, for each resolution, the best grid at that resolution and its normalized score. (C) The normalized scores form the characteristic matrix, which can be visualized as a surface; MIC corresponds to the highest point on this surface.**
**In this example, there are many grids that achieve the highest score. The star in (B) marks a sample grid achieving this score, and the star in (C) marks that grid's corresponding location on the surface.**
[**Mutual information classifier**](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html) **- Estimate mutual information for a discrete target variable.**
**Mutual information (MI)** [**\[1\]**](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html#r50b872b699c4-1) **between two random variables is a non-negative value, which measures the dependency between the variables. It is equal to zero if and only if two random variables are independent, and higher values mean higher dependency.**
**The function relies on nonparametric methods based on entropy estimation from k-nearest neighbors distances as described in** [**\[2\]**](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html#r50b872b699c4-2) **and** [**\[3\]**](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html#r50b872b699c4-3)**. Both methods are based on the idea originally proposed in** [**\[4\]**](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html#r50b872b699c4-4)**.**
[**MI score**](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mutual_info_score.html) **- Mutual Information between two clusterings.**
**The Mutual Information is a measure of the similarity between two labels of the same data.**
[**Adjusted MI score**](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.adjusted_mutual_info_score.html#sklearn.metrics.adjusted_mutual_info_score) **- Adjusted Mutual Information between two clusterings.**
**Adjusted Mutual Information (AMI) is an adjustment of the Mutual Information (MI) score to account for chance. It accounts for the fact that the MI is generally higher for two clusterings with a larger number of clusters, regardless of whether there is actually more information shared.**
**This metric is furthermore symmetric: switching label\_true with label\_pred will return the same score value. This can be useful to measure the agreement of two independent label assignments strategies on the same dataset when the real ground truth is not known**
[**Normalized MI score**](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.normalized_mutual_info_score.html#sklearn.metrics.normalized_mutual_info_score) **- Normalized Mutual Information (NMI) is a normalization of the Mutual Information (MI) score to scale the results between 0 (no mutual information) and 1 (perfect correlation). In this function, mutual information is normalized by some generalized mean of H(labels\_true) and H(labels\_pred)), defined by the average\_method.**
### **CRAMER’S COEFFICIENT**
[**Calculating** ](https://stackoverflow.com/questions/20892799/using-pandas-calculate-cram%C3%A9rs-coefficient-matrix)
##
### **FEATURE SELECTION**
**A series of good articles that explain about several techniques for feature selection**
1. [**How to parallelize feature selection on several CPUs,**](https://stackoverflow.com/questions/37037450/multi-label-feature-selection-using-sklearn) **do it per label on each cpu and average the results.**
2. [**A great notebook about feature correlation and manytypes of visualization, what to drop what to keep, using many feature reduction and selection methods (quite a lot actually). Its a really good intro**](https://www.kaggle.com/kanncaa1/feature-selection-and-data-visualization)
3. [**Multi class classification, feature selection, model selection, co-feature analysis**](https://towardsdatascience.com/multi-class-text-classification-with-scikit-learn-12f1e60e0a9f)
4. [**Text analysis for sentiment, doing feature selection**](https://streamhacker.com/tag/chi-square/) **a tutorial with chi2(IG?),** [**part 2 with bi-gram collocation in ntlk**](https://streamhacker.com/2010/05/24/text-classification-sentiment-analysis-stopwords-collocations/)
5. **What is collocation? - “the habitual juxtaposition of a particular word with another word or words with a frequency greater than chance.”**
6. [**Sklearn feature selection methods (4) - youtube**](https://www.youtube.com/watch?v=wjKvyk8xStg)
7. [**Univariate**](http://blog.datadive.net/selecting-good-features-part-i-univariate-selection/) **and independent features**
8. [**Linear models and regularization,**](http://blog.datadive.net/selecting-good-features-part-ii-linear-models-and-regularization/) **doing feature ranking**
9. [**Random forests and feature ranking**](http://blog.datadive.net/selecting-good-features-part-iii-random-forests/)
10. [**Random Search for focus and only then grid search for Random Forest**](https://towardsdatascience.com/hyperparameter-tuning-the-random-forest-in-python-using-scikit-learn-28d2aa77dd74)**,** [**code**](https://github.com/WillKoehrsen/Machine-Learning-Projects/blob/master/random_forest_explained/Improving%20Random%20Forest%20Part%202.ipynb)
11. [**Stability selection and recursive feature elimination (RFE).**](http://blog.datadive.net/selecting-good-features-part-iv-stability-selection-rfe-and-everything-side-by-side/) **are wrapper methods in sklearn for the purpose of feature selection.** [**RFE in sklearn**](http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html)
12. [**Kernel feature selection via conditional covariance minimization**](http://bair.berkeley.edu/blog/2018/01/23/kernels/) **(netanel d.)**
13. [**Github class that does the following**](https://towardsdatascience.com/a-feature-selection-tool-for-machine-learning-in-python-b64dd23710f0)**:**
1. **Features with a high percentage of missing values**
2. **Collinear (highly correlated) features**
3. **Features with zero importance in a tree-based model**
4. **Features with low importance**
5. **Features with a single unique value**
14. [**Machinelearning mastery on FS**](https://machinelearningmastery.com/feature-selection-machine-learning-python/)**:**
1. **Univariate Selection.**
2. **Recursive Feature Elimination.**
3. **Principle Component Analysis.**
4. **Feature Importance.**
15. [**Sklearn tutorial on FS:**](http://scikit-learn.org/stable/modules/feature_selection.html)
1. **Low variance**
2. **Univariate kbest**
3. **RFE**
4. **selectFromModel using \_coef \_important\_features**
5. **Linear models with L1 (svm recommended L2)**
6. **Tree based importance**
16. [**A complete overview of many methods**](https://www.analyticsvidhya.com/blog/2016/12/introduction-to-feature-selection-methods-with-an-example-or-how-to-select-the-right-variables/)
1. **(reduction) LDA: Linear discriminant analysis is used to find a linear combination of features that characterizes or separates two or more classes (or levels) of a categorical variable.**
2. **(selection) ANOVA: ANOVA stands for Analysis of variance. It is similar to LDA except for the fact that it is operated using one or more categorical independent features and one continuous dependent feature. It provides a statistical test of whether the means of several groups are equal or not.**
3. **(Selection) Chi-Square: It is a is a statistical test applied to the groups of categorical features to evaluate the likelihood of correlation or association between them using their frequency distribution.**
4. **Wrapper methods:**
1. **Forward Selection: Forward selection is an iterative method in which we start with having no feature in the model. In each iteration, we keep adding the feature which best improves our model till an addition of a new variable does not improve the performance of the model.**
2. **Backward Elimination: In backward elimination, we start with all the features and removes the least significant feature at each iteration which improves the performance of the model. We repeat this until no improvement is observed on removal of features.**
3. **Recursive Feature elimination: It is a greedy optimization algorithm which aims to find the best performing feature subset. It repeatedly creates models and keeps aside the best or the worst performing feature at each iteration. It constructs the next model with the left features until all the features are exhausted. It then ranks the features based on the order of their elimination.**
5.
17. [**Relief**](https://medium.com/@yashdagli98/feature-selection-using-relief-algorithms-with-python-example-3c2006e18f83) **-** [**GIT**](https://github.com/GrantRVD/ReliefF) [**git2**](https://pypi.org/project/ReliefF/#description) **a new family of feature selection trying to optimize the distance of two samples from the selected one, one which should be closer the other farther.**
**“The weight updation of attributes works on a simple idea (line 6). That if instance Rᵢ and H have different value (i.e the diff value is large), that means that attribute separates two instance with the same class which is not desirable, thus we reduce the attributes weight. On the other hand, if the instance Rᵢ and M have different value, that means the attribute separates the two instance with different class, which is desirable.”**
1. [**Scikit-feature (includes relief)**](https://github.com/chappers/scikit-feature) **forked from** [**this**](https://github.com/jundongl/scikit-feature/tree/master/skfeature) [**(docs)**](http://featureselection.asu.edu/algorithms.php)
2. [**Scikit-rebate (based on relief)**](https://github.com/EpistasisLab/scikit-rebate)
[**Feature selection using entropy, information gain, mutual information and … in sklearn.**](https://gist.github.com/GaelVaroquaux/ead9898bd3c973c40429)
[**Entropy, mutual information and KL Divergence by AurelienGeron**](https://www.techleer.com/articles/496-a-short-introduction-to-entropy-cross-entropy-and-kl-divergence-aurelien-geron/)
### **FEATURE ENGINEERING**
1. [**Vidhya on FE, anomalies, engineering, imputing**](https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/?utm_source=outlierdetectionpyod\&utm_medium=blog)
2. [**Many types of FE**](https://towardsdatascience.com/understanding-feature-engineering-part-1-continuous-numeric-data-da4e47099a7b)**, including log and box cox transform - a very useful explanation.**
3. [**Categorical Data**](https://towardsdatascience.com/understanding-feature-engineering-part-2-categorical-data-f54324193e63)
4. [**Dummy variables and feature hashing**](https://towardsdatascience.com/understanding-feature-engineering-part-2-categorical-data-f54324193e63) **- hashing is really cool.**
5. [**Text data**](https://towardsdatascience.com/understanding-feature-engineering-part-3-traditional-methods-for-text-data-f6f7d70acd41) **- unigrams, bag of words, N-grams (2,3,..), tfidf matrix, cosine\_similarity(tfidf) ontop of a tfidf matrix, unsupervised hierarchical clustering with similarity measures on top of (cosine\_similarity), LDA for topic modelling in sklearn - pretty awesome, Kmeans(lda),.**
6. [**Deep learning data for FE**](https://towardsdatascience.com/understanding-feature-engineering-part-4-deep-learning-methods-for-text-data-96c44370bbfa) **-** [**Word embedding using keras, continuous BOW - CBOW, SKIPGRAM, word2vec - really good.**](https://towardsdatascience.com/understanding-feature-engineering-part-4-deep-learning-methods-for-text-data-96c44370bbfa)
7. [**Topic Modelling**](http://chdoig.github.io/pygotham-topic-modeling/#/) **- a fantastic slide show about topic modelling using LDA etc.**
8. **Dipanjan on feature engineering** [**1**](https://towardsdatascience.com/understanding-feature-engineering-part-1-continuous-numeric-data-da4e47099a7b) **- cont numeric** [ **2 -**](https://towardsdatascience.com/understanding-feature-engineering-part-2-categorical-data-f54324193e63) **categorical** [**3**](https://towardsdatascience.com/understanding-feature-engineering-part-3-traditional-methods-for-text-data-f6f7d70acd41) **- traditional methods**
9. [**Target encoding git**](https://pypi.org/project/target_encoding/)
10. [**Category encoding git**](https://pypi.org/project/category-encoders/)
### **REPRESENTATION LEARNING**
1. [**paper**](https://arxiv.org/abs/1807.03748?utm_campaign=The%20Batch\&utm_source=hs_email\&utm_medium=email\&utm_content=83602348&_hsenc=p2ANqtz-8DcgUdDF--k3tWOmhM51lm28wHerZxlXKoGNU6hIu2P4Fj-RuEuciKtbWZZdWmvBg7KGeI44FWUrmpHdZIbAM-pUicgg&_hsmi=83602348)
### **TFIDF**
1. [Max\_features in tf idf](https://stackoverflow.com/questions/46118910/scikit-learn-vectorizer-max-features) -Sometimes it is not effective to transform the whole vocabulary, as the data may have some exceptionally rare words, which, if passed to TfidfVectorizer().fit(), will add unwanted dimensions to inputs in the future. One of the appropriate techniques in this case, for instance, would be to print out word frequences accross documents and then set a certain threshold for them. Imagine you have set a threshold of 50, and your data corpus consists of 100 words. After looking at the word frequences 20 words occur less than 50 times. Thus, you set max\_features=80 and you are good to go. If max\_features is set to None, then the whole corpus is considered during the TF-IDFtransformation. Otherwise, if you pass, say, 5 to max\_features, that would mean creating a feature matrix out of the most 5 frequent words accross text documents.
2. [Understanding Term based retrieval ](https://towardsdatascience.com/understanding-term-based-retrieval-methods-in-information-retrieval-2be5eb3dde9f#:~:text=In%20summary%2C%20TF%2DIDF%20rewards,length%20and%20term%20frequency%20saturation.) - TFIDF Bm25
3. [understanding TFIDF and BM25](https://kmwllc.com/index.php/2020/03/20/understanding-tf-idf-and-bm-25/)
### **SIMILARITY**
1. [**Cosine similarity tutorial**](http://blog.christianperone.com/2013/09/machine-learning-cosine-similarity-for-vector-space-models-part-iii/)
1. [**Cosine vs dot product**](https://datascience.stackexchange.com/questions/744/cosine-similarity-versus-dot-product-as-distance-metrics)
2. [**Cosine vs dot product 2**](https://blog.christianperone.com/2013/09/machine-learning-cosine-similarity-for-vector-space-models-part-iii/)
3. [**Fast cosine similarity**](https://stackoverflow.com/questions/51425300/python-fast-cosine-distance-with-cython) **implementation**
2. **Edit distance similarity**
3. [**Diff lib similarity and soundex**](https://datascience.stackexchange.com/questions/12575/similarity-between-two-words)
4. [**Soft cosine and cosine**](https://www.machinelearningplus.com/nlp/gensim-tutorial/)
5. [**Pearson also used to detect similar vectors**](https://machinelearningmastery.com/how-to-use-correlation-to-understand-the-relationship-between-variables/)
### Distance
1. [Mastery on distance formulas](https://machinelearningmastery.com/distance-measures-for-machine-learning/)
1. Role of Distance Measures
2. Hamming Distance
3. Euclidean Distance
4. Manhattan Distance (Taxicab or City Block)
5. Minkowski Distance
2. Cosine distance = 1 - cosine similarity
3. [Haversine](https://kanoki.org/2019/12/27/how-to-calculate-distance-in-python-and-pandas-using-scipy-spatial-and-distance-functions/) distance
#### Distance Tools
1. [GeoPandas](https://geopandas.org/en/stable/index.html)
### **FEATURE IMPORTANCE**
**Note: point 2, about lime is used for explainability, please also check that topic, down below.**
1. [**Using RF and other methods, really good**](https://towardsdatascience.com/explaining-feature-importance-by-example-of-a-random-forest-d9166011959e)
2. [**Non parametric feature impact and importance**](https://arxiv.org/abs/2006.04750) **- while there are nonparametric feature selection algorithms, they typically provide feature rankings, rather than measures of impact or importance.In this paper, we give mathematical definitions of feature impact and importance, derived from partial dependence curves, that operate directly on the data.**
3. [**Paper**](https://arxiv.org/abs/1602.04938) **(**[**pdf**](https://arxiv.org/pdf/1602.04938.pdf)**,** [**blog post**](https://www.oreilly.com/learning/introduction-to-local-interpretable-model-agnostic-explanations-lime)**): (**[**GITHUB**](https://github.com/marcotcr/lime/blob/master/README.md)**) how to "explain the predictions of any classifier in an interpretable and faithful manner, by learning an interpretable model locally around the prediction."**\
\
**they want to understand the reasons behind the predictions, it’s a new field that says that many 'feature importance' measures shouldn’t be used. i.e., in a linear regression model, a feature can have an importance rank of 50 (for example), in a comparative model where you duplicate that feature 50 times, each one will have 1/50 importance and won’t be selected for the top K, but it will still be one of the most important features. so new methods needs to be developed to understand feature importance. this one has git code as well.**
**Several github notebook examples:** [**binary case**](https://marcotcr.github.io/lime/tutorials/Lime%20-%20basic%20usage%2C%20two%20class%20case.html)**,** [**multi class**](https://marcotcr.github.io/lime/tutorials/Lime%20-%20multiclass.html)**,** [**cont and cat features**](https://marcotcr.github.io/lime/tutorials/Tutorial%20-%20continuous%20and%20categorical%20features.html)**, there are many more for images in the github link.**
**“Intuitively, an explanation is a local linear approximation of the model's behaviour. While the model may be very complex globally, it is easier to approximate it around the vicinity of a particular instance. While treating the model as a black box, we perturb the instance we want to explain and learn a sparse linear model around it, as an explanation. The figure below illustrates the intuition for this procedure. The model's decision function is represented by the blue/pink background, and is clearly nonlinear. The bright red cross is the instance being explained (let's call it X). We sample instances around X, and weight them according to their proximity to X (weight here is indicated by size). We then learn a linear model (dashed line) that approximates the model well in the vicinity of X, but not necessarily globally. For more information, read our paper, or take a look at this blog post.”**\
\
### **FEATURE IMPUTING**
1. [**Vidhya on FE, anomalies, engineering, imputing**](https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/?utm_source=outlierdetectionpyod\&utm_medium=blog)
2. [**Fancy impute**](https://pypi.org/project/fancyimpute/)
###
### **FEATURE STORE**
1. **The importance of having one -** [**medium**](https://towardsdatascience.com/the-importance-of-having-a-feature-store-e2a9cfa5619f)
2. [**What is?**](https://feast.dev/blog/what-is-a-feature-store/)
3. [**Feature store vs data warehouse**](https://www.kdnuggets.com/2020/12/feature-store-vs-data-warehouse.html)
4. [**Why feature store is not enough**](https://towardsdatascience.com/effective-ai-infrastructure-or-why-feature-store-is-not-enough-43bc2d803401)
5. [**Feast**](https://docs.feast.dev/) [**what is 1**](https://feast.dev/blog/what-is-a-feature-store/) [**what is 2**](https://neptune.ai/blog/feature-stores-components-of-a-data-science-factory-guide)
1. [**Tecton.ai**](https://www.tecton.ai/) **(managed feature store)**\
2. [**Iguazio feature store**](https://www.iguazio.com/feature-store/)\
\
3.
# Evaluation Metrics
## **A metric learning reality check**
1. [**Medium**](https://medium.com/@tkm45/updates-to-a-metric-learning-reality-check-730b6914dfe7)
2. [**Git**](https://github.com/KevinMusgrave/pytorch-metric-learning?fbclid=IwAR3PmPTDgYFok4p095WmkRWLfWhixyjFXkZgFJzeYXs5Y92pofoNZL_lGTg)
3. [**Website**](https://kevinmusgrave.github.io/powerful-benchmarker/papers/mlrc/?fbclid=IwAR3jK3-qFphFsO7ocmjeN-zPLkcaQkTAcC78cFUNFVe1BgXzM-iE5PLh3bU)
## **SUPERVISED**
### Accuracy
1. accuracy
### Perplexity
1. [perplexity and accuracy in classification](https://medium.com/unpackai/perplexity-and-accuracy-in-classification-114b57bd820d)
2.
### **Precision \ Recall \ ROC \ AUC**
**-** [**Performance Measures**](http://machinelearningmastery.com/classification-accuracy-is-not-enough-more-performance-measures-you-can-use/)**:**
**A balanced confusion matrix is better than one that is either one row of numbers and one of zeros, or a column of numbers and a column of zeros. Therefore an algorithm that outputs a lower classification accuracy but has a better confusion matrix wins.**
**# of Positive predictions divided by the total number of positive class values predicted.**
**Precision = True Positives / (True Positives + False Positives)**
**Low can be thought of many false positives.**
**# of positive predictions divided by the number of positive class values in the test data**
**Recall (sensitivity) = True Positives / (True Positives + False Negatives)**
**Low can be thought of many false negatives.**
#### **F1 Harmonic Mean Score**
**F1\_Score = 2 \* ((Precision \* Recall) / (Precision + Recall))**
**F1 helps select a model based on a balance between precision and recall.**
**In a multi-class problem, there are many methods to calculate F1, some are more appropriate for balanced data, others are not.**
1. [**The best link yet**](https://simonhessner.de/why-are-precision-recall-and-f1-score-equal-when-using-micro-averaging-in-a-multi-class-problem/) **- micro macro weighted (macro balanced, micro imbalanced, weighted imbalanced)**
2. [**Micro vs macro** ](https://datascience.stackexchange.com/questions/15989/micro-average-vs-macro-average-performance-in-a-multiclass-classification-settin/16001)
3. [**Micro vs weighted (not a good link**](https://stats.stackexchange.com/questions/169439/micro-vs-weighted-f1-score)**)**
4. [**What is weighted**](https://stats.stackexchange.com/questions/283961/where-does-sklearns-weighted-f1-score-come-from)
5. [**Micro is accuracy**](https://stackoverflow.com/questions/37358496/is-f1-micro-the-same-as-accuracy) **in multi class**
**------------------------------------**
* **Accuracy = (1 – Error) = (TP + TN)/(PP + NP) = Pr(C), the probability of a correct classification.**
* **Sensitivity (recall) = TP/(TP + FN) = TP/PP = the ability of the test to detect disease in a population of diseased individuals.**
* **Specificity = TN/(TN + FP) = TN / NP = the ability of the test to correctly rule out the disease in a disease-free population.**
**(**[**What are ?)**](http://machinelearningmastery.com/assessing-comparing-classifier-performance-roc-curves-2/) **Sensitivity and specificity against ROC and AUC.**
[**ROC curve and AUC in weka**](https://www.youtube.com/watch?v=j97h_-b0gvw\&list=PLJbE6j2EG1pZnBhOg3_Rb63WLCprtyJag) **- explains how the curve should look like for the negative or positive predictions, against what is actually plotted.**
**Mean F1?** [**How**](https://datascience.stackexchange.com/questions/16179/what-is-the-correct-way-to-compute-mean-f1-score) **do we calculate** [**it**](http://rushdishams.blogspot.com/2011/08/micro-and-macro-average-of-precision.html)**.**
[**Multiclass Precision / Recall**](https://towardsdatascience.com/multi-class-metrics-made-simple-part-ii-the-f1-score-ebe8b2c2ca1)**,** [**part 1**](https://towardsdatascience.com/multi-class-metrics-made-simple-part-i-precision-and-recall-9250280bddc2)
[**Precision at K**](https://medium.com/@m_n_malaeb/recall-and-precision-at-k-for-recommender-systems-618483226c54)**,** [**formulas, examples**](https://surprise.readthedocs.io/en/latest/FAQ.html#how-to-compute-precision-k-and-recall-k)**,** [**git 1**](https://github.com/scikit-learn/scikit-learn/pull/14859)**,** [**git 2**](https://gist.github.com/mblondel/7337391)**,** [**git 3**](https://github.com/scikit-learn/scikit-learn/issues/7343) **(suggestive, recommendation application)**
#### **RECALL, PRECISION AND F1**
1. [**Medium on Controling the decision threshold using the probabilities any model gives, code, samples, tutorial**](https://towardsdatascience.com/fine-tuning-a-classifier-in-scikit-learn-66e048c21e65)
2. [**Another good medium explanation on precision / recall / fpr/ tpr etc**](https://towardsdatascience.com/beyond-accuracy-precision-and-recall-3da06bea9f6c)
3. [**Scikit lego on choosing the threshold using grid search**](https://scikit-lego.readthedocs.io/en/latest/meta.html#Thresholder)
4. [**Best explanation ever** ](https://www.quora.com/What-is-the-best-way-to-understand-the-terms-precision-and-recall)
**Recall**
* **one day, your girlfriend asks you: ‘Sweetie, do you remember all birthday surprises from me?’**
* **This simple question makes your life in danger. To extend your life, you need to recall all 10 surprising events from your memory.**
* **So, recall is the ratio of a number of events you can correctly recall to a number of all correct events. If you can recall all 10 events correctly, then, your recall ratio is 1.0 (100%). If you can recall 7 events correctly, your recall ratio is 0.7 (70%).**
**Precision**
* **For example, you answers 15 times, 10 events are correct and 5 events are wrong. This means you can recall all events but it’s not so precise.**
* **So, precision is the ratio of a number of events you can correctly recall to a number all events you recall (mix of correct and wrong recalls). In other words, it is how precise of your recall.**
* **From the previous example (10 real events, 15 answers: 10 correct answers, 5 wrong answers), you get 100% recall but your precision is only 66.67% (10 / 15).**
[**Confusion matrix wise**](http://www.kdnuggets.com/faq/precision-recall.html)**: bottom line is recall (% correct out of positive cases), right column is precision (% of POS predictions) & % accuracy in diagonal**
[**F1 score**](http://machinelearningmastery.com/classification-accuracy-is-not-enough-more-performance-measures-you-can-use/)**:**
* **conveys the balance between the precision and the recall**
* **2\*((precision\*recall)/(precision+recall)**
[**Yet another(pretty good) source**](http://blog.exsilio.com/all/accuracy-precision-recall-f1-score-interpretation-of-performance-measures/)
[**Another (bad) source**](https://chrisalbon.com/machine-learning/precision_recall_and_F1_scores.html) **for explaining, precision, recall, accuracy, true positive rate etc.**
**(How to use precision and recall?) answer by aurelien geron:**
* **In a binary classifier, the decision function is the function that produces a score for the positive class.**
* **In a logistic regression classifier, that decision function is simply a linear combination of the input features.**
* **If that score is greater than some threshold that you choose, then the classifier "predicts" the positive class, or else it predicts the negative class.**
* **If you want your model to have high precision (at the cost of a low recall), then you must set the threshold pretty high. This way, the model will only predict the positive class when it is absolutely certain. For example, you may want this if the classifier is selecting videos that are safe for kids: it's better to err on the safe side.**
* **Conversely, if you want high recall (at the cost of a low precision) then you must use a low threshold. For example, if the classifier is used to detect intruders in a nuclear plant, then you probably want to detect all actual intruders, even if it means getting a lot of false alarms (called "false positives").**
* **If you make a few assumptions about the distribution of the data (i.e., the positive and negative class are separated by a linear boundary plus Gaussian noise), then computing the logistic of the score gives you the probability that the instance belongs to the positive class. A score of 0 corresponds to the 50% probability. So by default, a LogisticClassifier predicts the positive class if it estimates the probability to be greater than 50%. In general, this sounds like a reasonable default threshold, but really it all depends on what you want to do with the classifier.**
* **If the assumptions I mentioned above were perfect, then if the Logistic Classifier outputs a probability of X% for an instance, it means there is exactly X% chance that it's positive. But in practice, the assumptions are imperfect, so I try to always make it clear that we are talking about an "estimated probability", not an actual probability.**
#### **ROC CURVES**
1. [**Diff between precision recall to roc curve**](https://www.quora.com/What-is-the-difference-between-a-ROC-curve-and-a-precision-recall-curve-When-should-I-use-each)
2. [**What is ROC AUC and PR AUC and when to use then (i.e for imbalanced data use PRAUC)**](http://www.chioka.in/differences-between-roc-auc-and-pr-auc/)
3. [**What is AUC (AUROC)**](https://stats.stackexchange.com/questions/132777/what-does-auc-stand-for-and-what-is-it)
[**(RMSE - what is?)**](https://stats.stackexchange.com/questions/56302/what-are-good-rmse-values) **- it is important to recall that RMSE has the same unit as the dependent variable (DV). It means that there is no absolute good or bad threshold, however you can define it based on your DV. For a datum which ranges from 0 to 1000, an RMSE of 0.7 is small, but if the range goes from 0 to 1, it is not that small anymore. However, although the smaller the RMSE, the better,**
[**(R^2 vs RMSE)**](https://stats.stackexchange.com/questions/142248/difference-between-r-square-and-rmse-in-linear-regression) **- R-squared is conveniently scaled between 0 and 1, whereas RMSE is not scaled to any particular values. This can be good or bad; obviously R-squared can be more easily interpreted, but with RMSE we explicitly know how much our predictions deviate, on average, from the actual values in the dataset. So in a way, RMSE tells you more.**
**I also found this** [**video**](https://www.youtube.com/watch?v=aq8VU5KLmkY) **really helpful.**
[**Kappa**](https://stats.stackexchange.com/questions/82162/cohens-kappa-in-plain-english) **- measures accuracy while considering imbalanced datasets**
**References:**
1. [**A Survey on Deep Learning in Medical Image Analysis**](https://arxiv.org/abs/1702.05747)
### **UNSUPERVISED**
1. [**Silhouette Analysis vs Elbow Method vs Davies-Bouldin Index: Selecting the optimal number of clusters for KMeans clustering**](https://gdcoder.com/silhouette-analysis-vs-elbow-method-vs-davies-bouldin-index-selecting-the-optimal-number-of-clusters-for-kmeans-clustering/)
2.
##
# Datasets
### Structured / Unstructured data
1. [Unstructured ](https://www.webopedia.com/TERM/U/unstructured_data.html)
2. [Structured](https://www.webopedia.com/TERM/S/structured_data.html)
### BIAS / VARIANCE
1. [Various Bias types](https://queue.acm.org/detail.cfm?id=3466134) by queue.acm
1. [Overfitting your test set, a statistican view point, a great article](https://lukeoakdenrayner.wordpress.com/2019/09/19/ai-competitions-dont-produce-useful-models/?fbclid=IwAR1WM5U7imq-2LFPifyCoTPp-MFwPoGROMLr2TZWAp41qgVeLdT-_2bkLyk\&blogsub=confirming#subscribe-blog), bottom line use bonferroni correction.
2. Understanding what is the next stage in DL (& ML) algorithm development: basic approach - [Andrew NG](https://www.youtube.com/watch?v=F1ka6a13S9I) on youtube
Terms: training, validation, test.
Split: training & validation 70%, test 30%
Procedure: cross fold training and validation, or further split 70% to training and validation.
BIAS - Situation 1 - doing much worse than human:
Human expert: 1% error
Training set error: 5% error (test on train)
Validation set error: 6% error (test on validation or CFV)
Conclusion: there is a BIAS between human expert and training set
Solution: 1. Train deeper or bigger\larger networks, 2. train longer, 3. May needs more data to get to the human expert level, Or 4. New model architecture.
VARIANCE - Situation 2 - validation set not close to training set error:
Human expert: 1% error
Training set error: 2% error
Validation set error: 6% error
Conclusion: there is a VARIANCE problem, i.e. OVERFITTING, between training and validation.
Solution: 1. Early stopping, 2. Regularization or 3. get more data, or 4. New model architecture.
Situation 3 - both:Human expert: 1% error
Training set error: 5% error
Validation set error: 10% error
Conclusion: both problems occur, i.e., BIAS as and VARIANCE.
Solution: do it al
* Underfitting = Get more data
* Overfitting = Early stop, regularization, reason: models detail & noise.
* Happens more in non parametric (and non linear) algorithms such as decision trees.
* Bottom line, bigger model or more data will solve most issues.
* In practice advice with [regularized linear regression.](http://www.holehouse.org/mlclass/10_Advice_for_applying_machine_learning.html)
IMPORTANT! For Test Train efficiency when the data is from different distributions:
E.g: TRAIN: 50K hours of voice chatter as the train set for a DLN, TEST: 10H for specific voice-based problem, i.e, taxi chatter.
Best practice: better to divide the validation & test from the same distribution, i.e. the 10H set.
Reason: improving scores on validation which is from a diff distribution will not be the same quality as improving scores on a validation set originated from the actual distribution of the problem’s data, i.e., 10H.
NOTE: Unlike the usual supervised learning, where all the data is from the same distribution, where we split the training to train and validation (cfv).
[Situation 4](https://youtu.be/F1ka6a13S9I?t=47m26s): However, when there are 2 distributions it’s possible to extend the division of the training set to validation\_training and training, and the test to validation and test.
Split: Train, Valid\_Train = 48K\2K & Valid, Test, 5K & 5K.
So situation 1 stays the same,
Situation 2 is Valid\_Train error (train\_dev)
Situation 3 is Valid\_Test error - need more data, data synthesis - tweak test to be similar to train data, new architecture as a solution
Situation 4 is now Test set error - get more data\
### SPARSE DATASETS
[Sparse matrices](https://machinelearningmastery.com/sparse-matrices-for-machine-learning/) in ML - one hot/tfidf, dictionary/list of lists/ coordinate list.
###
### TRAINING METHODOLOGIES
1. Train test split
2. Cross validation
3. Transfer learning - using a pre existing classifier similar to your domain, usually trained on millions of samples. fine-tuned on new data in order to create a new classifier that utilizes that information in the new domain. Examples such as w2v or classic resnet fine-tuning.
4. Bootstrapping training- using a similar dataset, such as yelp, with 5 stars to create a pos/neg sentiment classifier based on 1 star and 5 stars. Finally using that to label or sample select from an unlabelled dataset, in order to create a new classifier or just to sample for annotation etc.
5. [Student-teacher paradigm](https://developers.facebook.com/videos/2019/from-visual-recognition-to-reasoning/) (facebook), using a big labelled dataset to train a teacher classifier, predicting on unlabelled data, choosing the best classified examples based on probability, using those to train a new student model, finally fine-tune on the labeled dataset to create a more robust model, which is expected to know the unlabelled dataset and the labelled dataset with higher accuracy. With respect to the fully supervised teacher model / baseline.
1. Yoav’s method for transfer learning for languages - train a classifier on labelled data from english and spanish, fine tune using left out spanish data, stop before overfitting. This can be generalized to other domains.
#### TRANSFER LEARNING
1. [In deep learning](https://towardsdatascience.com/a-comprehensive-hands-on-guide-to-transfer-learning-with-real-world-applications-in-deep-learning-212bf3b2f27a)
2. 
###
### TRAIN / TEST / CROSS VALIDATION
[Scikit-lego on group-based splitting and transformation](https://scikit-lego.readthedocs.io/en/latest/meta.html#Grouped-Prediction)
[Images from here](https://www.kdnuggets.com/2017/08/dataiku-predictive-model-holdout-cross-validation.html).
[Train Test methodology](http://machinelearningmastery.com/how-to-choose-the-right-test-options-when-evaluating-machine-learning-algorithms/) -
“[The training](https://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set) set is used to fit the models; the validation set is used to estimate prediction error for model selection; the test set is used for assessment of the generalization error of the final chosen model. Ideally, the test set should be kept in a “vault,” and be brought out only at the end of the data analysis”
* Random Split tests 66\33 - problem: variance each time we rerun.
* Multiple times random split tests - problem: samples may not be included in train\test or selected multiple times.
* Cross validation - pretty good, diff random seed results in diff mean accuracy, variance due to randomness
* Multiple cross validation - accounts for the randomness of the CV
* Statistical significance ( t-test) on multi CV - are two samples drawn from the same population? (no difference). If “yes”, not significant, even if the mean and std deviations differ.
Finally, When in doubt, use k-fold cross validation (k=10) and use multiple runs of k-fold cross validation with statistical significance tests.
[Out of fold](https://machinelearningmastery.com/out-of-fold-predictions-in-machine-learning/) - leave unseen data, do cross fold on that. Good for ensembles.
### VARIOUS DATASETS
1. [26 of them](https://www.analyticsvidhya.com/blog/2018/05/24-ultimate-data-science-projects-to-boost-your-knowledge-and-skills/?utm_source=facebook.com\&utm_medium=social)
2. [24](https://lionbridge.ai/datasets/25-best-parallel-text-datasets-for-machine-translation-training/)
3. [Eu-](https://datarepository.wolframcloud.com/resources/Europarl-English-Spanish-Machine-Translation-Dataset-V7)es, [2](https://data.europa.eu/euodp/en/data/dataset/elrc_339)
4. 50K - [ModelDepot](https://modeldepot.io/) alone has over 50,000 freely accessible pre-trained models with search functionality to
5.
### IMBALANCED DATASETS
1. ([the BEST resource and a great api for python)](http://contrib.scikit-learn.org/imbalanced-learn/stable/over_sampling.html) with visual samples - it actually works well on clustering.
2. [Mastery on](https://machinelearningmastery.com/cost-sensitive-learning-for-imbalanced-classification/?fbclid=IwAR0_DeIydTAAkutypcMBfrnC4QyuyqVxDu_uej5t48AvQKShcRUqfMm8Rqo) cost sensitive sampling
3. [Smote for imbalance](https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/?fbclid=IwAR3W59c54ohoaIHnHLQFCcZZanFXI4QzIzuWiUtaUC851JFkwlevCAgvpbM)
[Systematic Investigation of imbalance effects in CNN’s](https://arxiv.org/abs/1710.05381), with several observations. This is crucial when training networks, because in real life you don’t always get a balanced DS.
They recommend the following:
1. (i) the effect of class imbalance on classification performance is detrimental;
2. (ii) the method of addressing class imbalance that emerged as dominant in almost all analyzed scenarios was oversampling;
3. (iii) oversampling should be applied to the level that totally eliminates the imbalance, whereas undersampling can perform better when the imbalance is only removed to some extent;
4. (iv) as opposed to some classical machine learning models, oversampling does not necessarily cause overfitting of CNNs;
5. (v) thresholding should be applied to compensate for prior class probabilities when overall number of properly classified cases is of interest.
General Rules:
1. Many samples - undersampling
2. Few samples - over sampling
3. Consider random and non-random schemes
4. Different sample rations, instead of 1:1 (proof? papers?)
Balancing data sets ([wiki](https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis), [scikit learn](https://github.com/scikit-learn-contrib/imbalanced-learn) & [examples in SKLEARN](http://contrib.scikit-learn.org/imbalanced-learn/auto_examples/index.html)):
1. Oversampling the minority class
1. (Random) duplication of samples
2. SMOTE [(in weka + needs to be installed](http://www.jair.org/media/953/live-953-2037-jair.pdf) & [paper)](http://www.jair.org/media/953/live-953-2037-jair.pdf) - find k nearest neighbours,
New\_Sample = (random num in \[0,1] ) \* vec(ki,current\_sample)
* (in weka) The nearestNeighbors parameter says how many nearest neighbor instances (surrounding the currently considered instance) are used to build an in between synthetic instance. The default value is 5. Thus the attributes of 5 nearest neighbors of a real existing instance are used to compute a new synthetic one.
* (in weka) The percentage parameter says how many synthetic instances are created based on the number of the class with less instances (by default - you can also use the majority class by setting the -Coption). The default value is 100. This means if you have 25 instances in your minority class, again 25 instances are created synthetically from these (using their nearest neighbours' values). With 200% 50 synthetic instances are created and so on.
1. ADASYN - shifts the classification boundary to the minority class, synthetic data generated for majority class.
2. Undersampling the majority class
1. Remove samples
2. Cluster centroids - replaces a cluster of samples (k-means) with a centroid.
3. Tomek links - cleans overlapping samples between classes in the majority class.
4. Penalizing the majority class during training
3. Combined over and under (hybrid) - i.e., SMOTE and tomek/ENN
4. Ensemble sampling
1. EasyEnsemble
2. BalanceCascade
5. Dont balance, try algorithms that perform well with unbalanced DS
1. Decision trees - C4.5\5\CART\Random Forest
2. SVM
6. Penalize Models -
1. added costs for misclassification on the minority class during training such as penalized-SVM
2. a [CostSensitiveClassifier](http://weka.sourceforge.net/doc.dev/weka/classifiers/meta/CostSensitiveClassifier.html) meta classifier in Weka that wraps classifiers and applies a custom penalty matrix for miss classification.
3. complex
##
### SAMPLE SELECTION
1. [How to choose your sample size from a population based on confidence interval](https://www.checkmarket.com/blog/how-to-estimate-your-population-and-survey-sample-size/)
1. [Data advice, should we get more data? How much](https://machinelearningmastery.com/much-training-data-required-machine-learning/)
[Gibbs sampling](https://wiseodd.github.io/techblog/2015/10/09/gibbs-sampling/): - Gibbs Sampling is a MCMC method to draw samples from a potentially really really complicated, high dimensional distribution, where analytically, it’s hard to draw samples from it. The usual suspect would be those nasty integrals when computing the normalizing constant of the distribution, especially in Bayesian inference. Now Gibbs Sampler can draw samples from any distribution, provided you can provide all of the conditional distributions of the joint distribution analytically.
###
### LEARNING CURVES
1. [Git examples](https://gist.github.com/orico/260097cb1a2926c6b6ca6f71c37c135b)
2. [Sklearn examples](https://stats.stackexchange.com/questions/283738/sklearn-learning-curve-example)
3. [Understanding bias variance via learning curves](http://digitheadslabnotebook.blogspot.com/2011/12/practical-advice-for-applying-machine.html)
4. [Unread - learning curve sampling applied to model based clustering](http://www.jmlr.org/papers/volume2/meek02a/meek02a.pdf) - seemed like active learning, i.e., sample using EM/cluster to achieve nearly as accurate on all data
5. Predicting sample size required for training
6. [Advice on many things, including learning curves](https://blog.acolyer.org/2018/03/28/deep-learning-scaling-is-predictable-empirically/amp/?fbclid=IwAR0V1X1vuCZYmeku12YHJI7wwK7RCKEyE2Q7aRDDT58hjRPzAOrHfvo98WY)
This is a really wonderful study with far-reaching implications that could even impact company strategies in some cases. It starts with a simple question: “how can we improve the state of the art in deep learning?” We have three main lines of attack:
1. We can search for improved model architectures.
2. We can scale computation
3. We can create larger training data sets.
### DISTILLING DATA
1. [Medium on](https://towardsdatascience.com/data-maps-datasets-can-be-distilled-too-1991c3c260d6) this [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://arxiv.org/abs/2009.10795). What I found interesting about this paper is that it challenges the common approach of “the more the merrier” when it comes to training data, and shifts the focus from the quantity of the data to the quality of the data.
### DATASET SELECTION
1. [Medium](https://medium.com/@amielmeiseles/how-to-choose-the-best-source-model-for-transfer-learning-41d5c91c1338)
2. 
# Dataset Confidence
[**Dataset cartography mapping and diagnosing datasets with training dynamics -**](https://arxiv.org/abs/2009.10795) **Large datasets have become commonplace in NLP research. However, the increased emphasis on data quantity has made it challenging to assess the quality of data. We introduce Data Maps---a model-based tool to characterize and diagnose datasets. We leverage a largely ignored source of information: the behavior of the model on individual instances during training (training dynamics) for building data maps. This yields two intuitive measures for each example---the model's confidence in the true class, and the variability of this confidence across epochs---obtained in a single run of training. Experiments across four datasets show that these model-dependent measures reveal three distinct regions in the data map, each with pronounced characteristics. First, our data maps show the presence of "ambiguous" regions with respect to the model, which contribute the most towards out-of-distribution generalization. Second, the most populous regions in the data are "easy to learn" for the model, and play an important role in model optimization. Finally, data maps uncover a region with instances that the model finds "hard to learn"; these often correspond to labeling errors. Our results indicate that a shift in focus from quantity to quality of data could lead to robust models and improved out-of-distribution generalization.**
# Hyper Parameter Optimization
1. [ **Using HyperOpt**](http://hyperopt.github.io/hyperopt/)
**Random Search**\
**Tree of Parzen Estimators (TPE)**\
\
**Hyperopt has been designed to accommodate Bayesian optimization algorithms based on Gaussian processes and regression trees, but these are not currently implemented.**\
\
**All algorithms can be run either serially, or in parallel by communicating via** [**MongoDB**](http://www.mongodb.org/)**.**\
\
[**Mlflow, Hyperparameterhunter,hyperopt, concept drift, unit tests.**](https://towardsdatascience.com/putting-ml-in-production-ii-logging-and-monitoring-algorithms-91f174044e4e)\
[**Hyperopt**](http://hyperopt.github.io/hyperopt/) **for hyperparameter search**
2. [**HyperparameterHunter**](https://github.com/HunterMcGushion/hyperparameter_hunter) \
**provides a wrapper for machine learning algorithms that saves all the important data. Simplify the experimentation and hyperparameter tuning process by letting HyperparameterHunter do the hard work of recording, organizing, and learning from your tests — all while using the same libraries you already do. Don't let any of your experiments go to waste, and start doing hyperparameter optimization the way it was meant to be.**
3. [**Implementation and comparison**](https://towardsdatascience.com/putting-ml-in-production-ii-logging-and-monitoring-algorithms-91f174044e4e) **- HH slower than HO due to usage of skopt.**
4. [**HumpDay**](https://github.com/microprediction/humpday) **- a package that compares optimization algorithms and ranks them**
# Training Strategies
**(amazing)** [**Framework for a successful training strategy**](https://towardsdatascience.com/framework-for-a-successful-continuous-training-strategy-8c83d17bb9dc)**, periodic, performance based, driven by data changes, dynamic window size, dynamic data selection, what to retrain and the level**
# Calibration
[**Why do we need to calibrate models, or in other words, dont trust predict\_proba to give you probabilities**](https://towardsdatascience.com/pythons-predict-proba-doesn-t-actually-predict-probabilities-and-how-to-fix-it-f582c21d63fc)
## **Classic Model Calibration**
1. **How do we do isotonic and sigmoid calibration - read** [**this**](http://tullo.ch/articles/speeding-up-isotonic-regression/)**, then** [**this**](http://fastml.com/classifier-calibration-with-platts-scaling-and-isotonic-regression/)**,** [**how to use in sklearn**](https://stats.stackexchange.com/questions/263393/scikit-correct-way-to-calibrate-classifiers-with-calibratedclassifiercv)
2. [**How to speed up isotonic regression for sklearn**](http://tullo.ch/articles/speeding-up-isotonic-regression/)
3. **TODO: how to calibrate a DNN (except sklearn wrapper for keras)**
4. **Allows us to use the probability as confidence. I.e, Well calibrated classifiers are probabilistic classifiers for which the output of the predict\_proba method can be directly interpreted as a confidence level**
5. **(good)** [**Probability Calibration Essentials (with code)**](https://medium.com/analytics-vidhya/probability-calibration-essentials-with-code-6c446db74265)
6. **The** [**Brier score**](https://en.wikipedia.org/wiki/Brier_score) **is a** [**proper score function**](https://en.wikipedia.org/wiki/Scoring_rule#ProperScoringRules) **that measures the accuracy of probabilistic predictions.**
7. [**Sk learn**](http://scikit-learn.org/stable/auto_examples/calibration/plot_calibration_curve.html#sphx-glr-auto-examples-calibration-plot-calibration-curve-py) **example**
8. [**‘calibrated classifier cv in sklearn**](http://scikit-learn.org/stable/modules/generated/sklearn.calibration.CalibratedClassifierCV.html#sklearn.calibration.CalibratedClassifierCV) **- The method to use for calibration. Can be ‘sigmoid’ which corresponds to Platt’s method or ‘isotonic’ which is a non-parametric approach. It is not advised to use isotonic calibration with too few calibration samples (<<1000) since it tends to overfit. Use sigmoids (Platt’s calibration) in this case.**\
**However, not all classifiers provide well-calibrated probabilities, some being over-confident while others being under-confident. Thus, a separate calibration of predicted probabilities is often desirable as a postprocessing. This example illustrates two different methods for this calibration and evaluates the quality of the returned probabilities using Brier’s score**
9. **Example** [**1**](http://scikit-learn.org/stable/auto_examples/calibration/plot_calibration.html#sphx-glr-auto-examples-calibration-plot-calibration-py) **- binary class below,** [**2**](http://scikit-learn.org/stable/auto_examples/calibration/plot_calibration_multiclass.html#sphx-glr-auto-examples-calibration-plot-calibration-multiclass-py) **- 3 class moving prob vectors to a well defined location,** [**3**](http://scikit-learn.org/stable/auto_examples/calibration/plot_compare_calibration.html#sphx-glr-auto-examples-calibration-plot-compare-calibration-py) **- comparison of non calibrated models, only logreg is calibrated naturally**
1. [**Mastery on why we need calibration**](https://machinelearningmastery.com/calibrated-classification-model-in-scikit-learn/)
2. [**Why softmax is not good as an uncertainty measure for DNN**](https://stats.stackexchange.com/questions/309642/why-is-softmax-output-not-a-good-uncertainty-measure-for-deep-learning-models)
3. [**If a model doesn't have probabilities use the decision function**](http://scikit-learn.org/stable/auto_examples/calibration/plot_calibration_curve.html#sphx-glr-auto-examples-calibration-plot-calibration-curve-py)
**y\_pred = clf.predict(X\_test)**\
**if hasattr(clf, "predict\_proba"):**\
**prob\_pos = clf.predict\_proba(X\_test)\[:, 1]**\
**else: # use decision function**\
**prob\_pos = clf.decision\_function(X\_test)**\
**prob\_pos = \\**\
**(prob\_pos - prob\_pos.min()) / (prob\_pos.max() - prob\_pos.min())**
## **Neural Net Calibration**
1. [Paper: Calibration of modern NN](https://arxiv.org/pdf/1706.04599.pdf)
2. [Calibration post](http://geoffpleiss.com/nn_calibration)
### Temperature
1. (great) [Softmax temperature](https://medium.com/mlearning-ai/softmax-temperature-5492e4007f71) by Harshit
2. [Interactive demo](https://lukesalamone.github.io/posts/what-is-temperature/)
3. [lower level explanation](http://www.kasimte.com/2020/02/14/how-does-temperature-affect-softmax-in-machine-learning.html#:~:text=What%20is%20softmax%20with%20temperature,makes%20the%20model%20less%20confident.) by kasim
4. [short explanation](https://medium.com/@majid.ghafouri/why-should-we-use-temperature-in-softmax-3709f4e0161) by Majid
5. [Change temperature in Keras](https://stackoverflow.com/questions/37246030/how-to-change-the-temperature-of-a-softmax-output-in-keras)
6. Calibration can also come in a different flavor, you want to make your algorithm certain, one trick is to use dropout layers when inferring/predicting/classifying, do it 100 times and average the results in some capacity , [see this chapter on BNN](https://docs.google.com/document/d/1dXELAcJn9KCPSRMDvZoumUyHx8K8Yn7wfFxesSpbNCM/edit#heading=h.slqfz2k65bd2)
[How Can We Know When Language Models Know? This paper is about calibration. ](http://phontron.com/paper/jiang20lmcalibration.pdf)“Recent works have shown that language models (LM) capture different types of knowledge regarding facts or common sense. However, because no model is perfect, they still fail to provide appropriate answers in many cases. In this paper, we ask the question “how can we know when language models know, with confidence, the answer to a particular query?” We examine this question from the point of view of calibration, the property of a probabilistic model’s predicted probabilities actually being well correlated with the probability of correctness. We first examine a state-ofthe-art generative QA model, T5, and examine whether its probabilities are well calibrated, finding the answer is a relatively emphatic no. We then examine methods to calibrate such models to make their confidence scores correlate better with the likelihood of correctness through fine-tuning, post-hoc probability modification, or adjustment of the predicted outputs or inputs. Experiments on a diverse range of datasets demonstrate the effectiveness of our methods. We also perform analysis to study the strengths and limitations of these methods, shedding light on further improvements that may be made in methods for calibrating LMs.”
# Datasets Reliability & Correctness
Or why we shouldn't trust models
1. [Clever Hans effect](https://thegradient.pub/nlps-clever-hans-moment-has-arrived/?fbclid=IwAR3vSx9EjXcSPhXU3Jyf7aWpTqhbVDARnh3qGpSw0rysv9rLeGyZFFCPnJA) - in relations to cues left in the dataset that models find, instead of actually solving the defined task!
1. Ablating, i.e. removing, part of a model and observing the impact this has on performance is a common method for verifying that the part in question is useful. If performance doesn't go down, then the part is useless and should be removed. Carrying this method over to datasets, it should become common practice to perform dataset ablations, as well, for example:
2. Provide only incomplete input (as done in the reviewed paper): This verifies that the complete input is required. If not, the dataset contains cues that allow taking shortcuts.
3. Shuffle the input: This verifies the importance of word (or sentence) order. If a bag-of-words/sentences gives similar results, even though the task requires sequential reasoning, then the model has not learned sequential reasoning and the dataset contains cues that allow the model to "solve" the task without it.
4. Assign random labels: How much does performance drop if ten percent of instances are relabeled randomly? How much with all random labels? If scores don't change much, the model probably didn't learning anything interesting about the task.
5. Randomly replace content words: How much does performance drop if all noun phrases and/or verb phrases are replaced with random noun phrases and verbs? If not much, the dataset may provide unintended non-content cues, such as sentence length or distribution of function words.
6. Datasets need more love
7. Datasets ablation and public beta
8. Inter-prediction agreement
2. [Paper](https://arxiv.org/abs/1908.05267?fbclid=IwAR1xOHxCF3gewyijMYAfZJSysu88Y8lgRIT2OiG-jWQav4zcbPqGYSoFFkk)
3. Behavioral testing and CHECKLIST
1. [Blog](https://amitness.com/2020/07/checklist/), [Youtube](https://www.youtube.com/watch?v=L3gaWctPg6E), [paper](https://arxiv.org/pdf/2005.04118.pdf), [git](https://github.com/marcotcr/checklist)
2. [Yonatan hadar on the subject in hebrew](https://www.facebook.com/groups/MDLI1/permalink/1627671704063538/)
# Data & Model Tests
## Model Testing
1. A great :P [unit test and logging](https://towardsdatascience.com/unit-testing-and-logging-for-data-science-d7fb8fd5d217?fbclid=IwAR3pze0DtV-2Q4L4ysPyjrInk7LB89mdiodxlEUTv4rv37ZoDzl_2I4ZbgA) post on medium - it’s actually mine :)
2. A mind blowing [lecture](https://www.youtube.com/watch?v=1fHGXOfiDO0\&feature=youtu.be\&fbclid=IwAR1bKByLgdYBDoBEr-e6Pw0Un5o0wvOg1yp4C-q4AoWZ1QuBEopTFFn0Gdw) about unit testing your data using Voluptuous & engrade & TDDA lecture
3. [Unit tests in python](https://jeffknupp.com/blog/2013/12/09/improve-your-python-understanding-unit-testing/)
4. [Unit tests in python - youtube](https://www.youtube.com/watch?v=6tNS--WetLI)
5. [Unit tests asserts](https://docs.python.org/3/library/unittest.html#unittest.TestCase.debug)
6. [Auger - automatic unit tests, has a blog post inside](https://github.com/laffra/auger), doesn't work with py 3+
7. [A rather naive unit tests article aimed for DS](https://medium.com/@danielhen/unit-tests-for-data-science-the-main-use-cases-1928d9e7a4d4)
8. A good pytest [tutorial](https://www.tutorialspoint.com/pytest/index.htm)
9. [Mock](https://medium.com/@yasufumy/python-mock-basics-674c33de1ced), [mock 2](https://medium.com/python-pandemonium/python-mocking-you-are-a-tricksy-beast-6c4a1f8d19b2)
## Data Testing
1. [Great expectations](https://greatexpectations.io/), [article](https://github.blog/2020-10-01-keeping-your-data-pipelines-healthy-with-the-great-expectations-github-action/), “TDDA” for Unit tests and CI, [Youtube](https://www.youtube.com/watch?v=uM9DB2ca8T8)
2. [DataProfiler git](https://github.com/capitalone/DataProfiler)
# Fairness, Accountability, and Transparency
### REGULATION FOR AI
1. [Preparing for EU regulations](https://towardsdatascience.com/how-ai-leaders-should-prepare-for-the-looming-eu-regulations-99e9d4f4c039) by MonaLabs
2. [EU regulation DOC](https://drive.google.com/file/d/1ZaBPsfor_aHKNeeyXxk9uJfTru747EOn/view)
3. [EIOPA](https://www.eiopa.europa.eu/content/eiopa-publishes-report-artificial-intelligence-governance-principles_en) - regulation for insurance companies.
4. Ethics and regulations in Israel
1. [First Report by the intelligence committee](https://www.globes.co.il/news/article.aspx?did=1001307714) headed by prof. Itzik ben israel and prof. evyatar matanya
2. [Second report by AI and data science committee ](https://innovationisrael.org.il/sites/default/files/%D7%93%D7%95%D7%97%20%D7%A1%D7%95%D7%A4%D7%99%20%D7%A1%D7%99%D7%9B%D7%95%D7%9D%20%D7%95%D7%95%D7%A2%D7%93%D7%AA%20%D7%AA%D7%9C%D7%9D%20%D7%9C%D7%AA%D7%9B%D7%A0%D7%99%D7%AA%20%D7%9E%D7%95%D7%A4%20%D7%9C%D7%90%D7%95%D7%9E%D7%99%D7%AA%20%D7%91%D7%91%D7%99%D7%A0%D7%94%20%D7%9E%D7%9C%D7%90%D7%9B%D7%95%D7%AA%D7%99%D7%AA%20-.pdf)
3. Third by meizam leumi for AI systems in [ethics and regulation in israel](https://machinelearning.co.il/4330/israelaiethicsreport/#more-4330), [lecture](https://machinelearning.co.il/3349/googleai/)
### FAIRNESS, ACCOUNTABILITY & TRANSPARENCY
1. FATML [website](https://www.fatml.org/) - The past few years have seen growing recognition that machine learning raises novel challenges for ensuring non-discrimination, due process, and understandability in decision-making. In particular, policymakers, regulators, and advocates have expressed fears about the potentially discriminatory impact of machine learning, with many calling for further technical research into the dangers of inadvertently encoding bias into automated decisions.
At the same time, there is increasing alarm that the complexity of machine learning may reduce the justification for consequential decisions to “the algorithm made me do it.”
1. [Principles and best practices](https://www.fatml.org/resources/principles-and-best-practices), [projects](https://www.fatml.org/resources/relevant-projects)
2. [FAccT](https://facctconference.org/) - A computer science conference with a cross-disciplinary focus that brings together researchers and practitioners interested in fairness, accountability, and transparency in socio-technical systems.
3. [Paper - there is no fairness, enforcing fairness can improve accuracy](https://openreview.net/forum?id=wXoHN-Zoel\&fbclid=IwAR1MZArpfpu8L8ildamF0ngnUbKgD8-9NFBCXVo0JKwS6yP9g-2BJmWUv68)
4. [Google on responsible ai practices](https://ai.google/responsibilities/responsible-ai-practices/) see also PAIR
5. [Bengio on ai](https://www.wired.com/story/ai-pioneer-algorithms-understand-why/?fbclid=IwAR03uWEmVSjrOmP4dp77v_mdjPAXOsKPams_xsUOKameKbuzY8JN4brGC9o)
6. [Poisoning attacks on fairness](https://arxiv.org/pdf/2004.07401.pdf) - Research in adversarial machine learning has shown how the performance of machine learning models can be seriously compromised by injecting even a small fraction of poisoning points into the training data. We empirically show that our attack is effective not only in the white-box setting, in which the attacker has full access to the target model, but also in a more challenging black-box scenario in which the attacks are optimized against a substitute model and then transferred to the target model
7. A [series of articles](https://jonathan-hui.medium.com/ai-bias-fairness-series-ce21ebf7b2e9) about Bias & Fairness by Johnathan Hui
1. [In Clinical research](https://jonathan-hui.medium.com/bias-in-clinical-research-data-science-machine-learning-deep-learning-40a8786a5046) - Selection , Sample , Time , Attrition , Survivorship, reporting, funding, citation, Volunteer , self-selection , non-response, pre-screening , healthy person, membership, ascertainment, performance, berkson admission, neyman, measurement, observer, expectation, response, self reporting, social desirability, recall, acquiescence agreement, leading, courtesy, attention verification, lead time, immortal time, misclassification, chronological, detection, spectrum, cofounder, susceptibility, collider, simpson, ommited, allocation, channeling.
2. [AI](https://jonathan-hui.medium.com/ai-bias-b85c86bbca90) - known cases in Vision, NLP - sentiment, embedding, language models, historical, compass, recommender, datasets.
3. [address AI Bias with Fairness criteria and tools](https://jonathan-hui.medium.com/address-ai-bias-with-fairness-criteria-tools-9af1ab8e4289) - per population, predictive parity, calibration by group
4. [Caveats and limitations of AI Fairness Approaches](https://jonathan-hui.medium.com/caveats-limitations-on-ai-fairness-approaches-8628e6a992fd) - sample bias, label bias, miscalibration outcome test, redlining, etc.
5. [AI Fairness Approaches](https://jonathan-hui.medium.com/ai-fairness-approaches-mathematical-definitions-49cc418feebd) - statistical fairness, equalizing acceptance rate, error rate, etc.
#### FAIRNESS TOOLS
1. [PII tools, by gensim](https://pii-tools.com/)
2. [Fair-learn](https://github.com/fairlearn/fairlearn) A Python package to assess and improve fairness of machine learning models.\
3. [Sk-lego](https://scikit-lego.readthedocs.io/en/latest/fairness.html)
1. Regression\
2. classification
3. 
4. 
5. 
6. information filter\
M. Zafar et al. (2017), Fairness Constraints: Mechanisms for Fair Classification
M. Hardt, E. Price and N. Srebro (2016), Equality of Opportunity in Supervised Learning
### BIAS
1. arize.ai on [model bias](https://arize.com/understanding-bias-in-ml-models/#MLMonitoring).
### DEBIASING MODELS
1. [Adversarial removal of demographic features](https://arxiv.org/abs/1808.06640) - “We show that demographic information of authors is encoded in -- and can be recovered from -- the intermediate representations learned by text-based neural classifiers. The implication is that decisions of classifiers trained on textual data are not agnostic to -- and likely condition on -- demographic attributes. “\
“we explore several techniques to improve the effectiveness of the adversarial component. Our main conclusion is a cautionary one: do not rely on the adversarial training to achieve invariant representation to sensitive features.”\
2. [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://arxiv.org/abs/2004.07667) (paper) , [github](https://github.com/shauli-ravfogel/nullspace_projection), [presentation](https://docs.google.com/presentation/d/1Xi5HLpvvRE8BqcNBZMyPS4gBa0i0lqZvRebz-AZxAPA/edit) by Shauli et al. - removing biased information such as gender from an embedding space using nullspace projection.\
The objective is this: give a representation of text, for example BERT embeddings of many resumes/CVs, we want to achieve a state where a certain quality, for example a gender representation of the person who wrote this resume is not encoded in X. they used the light version definition for “not encoded”, i.e., you cant predict the quality from the representation with a higher than random score, using a linear model. I.e., every linear model you will train, will not be able to predict the person’s gender out of the embedding space and will reach a 50% accuracy.\
This is done by an iterative process that includes. 1. Linear model training to predict the quality of the concept from the representation. 2. Performing ‘projection to null space’ for the linear classifier, this is an acceptable linear algebra calculation that has a meaning of zeroing the representation from the projection on the separation place that the linear model is representing, making the model useless. I.e., it will always predict the zero vector. This is done iteratively on the neutralized output, i.e., in the second iteration we look for an alternative way to predict the gender out of X, until we reach 50% accuracy (or some other metric you want to measure) at this point we have neutralized all the linear directions in the embedding space, that were predictive to the gender of the author.
For a matrix W, the null space is a sub-space of all X such that WX=0, i.e., W maps X to the zero vector, this is a linear projection of the zero vector into a subspace. For example you can take a 3d vectors and calculate its projection on XY.
1. Can we extinct predictive samples? Its an open question, Maybe we can use influence functions?
[Understanding Black-box Predictions via Influence Functions](https://arxiv.org/pdf/1703.04730.pdf) - How can we explain the predictions of a blackbox model? In this paper, we use influence functions — a classic technique from robust statistics — to trace a model’s prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction.
We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually indistinguishable training-set attacks.
1. [Removing ‘gender bias using pair mean pca](https://stackoverflow.com/questions/48019843/pca-on-word2vec-embeddings)
2. [Bias detector by intuit](https://github.com/intuit/bias-detector) - Based on first and last name/zip code the package analyzes the probability of the user belonging to different genders/races. Then, the model predictions per gender/race are compared using various bias metrics.
### PRIVACY
1. [Privacy in DataScience](http://www.unsupervised-podcast.xyz/ab55d406) podcast
2. [Fairness in AI](http://www.unsupervised-podcast.xyz/5d7fc118)
### DIFFERENTIAL PRIVACY
1. [Differential privacy](https://georgianpartners.com/what-is-differential-privacy/) has emerged as a major area of research in the effort to prevent the identification of individuals and private data. It is a mathematical definition for the privacy loss that results to individuals when their private information is used to create AI products. It works by injecting noise into a dataset, during a machine learning training process, or into the output of a machine learning model, without introducing significant adverse effects on data analysis or model performance. It achieves this by calibrating the noise level to the sensitivity of the algorithm. The result is a differentially private dataset or model that cannot be reverse engineered by an attacker, while still providing useful information. Uses BOTLON & EPSILON
2. [youtube](https://www.youtube.com/watch?v=gI0wk1CXlsQ\&feature=emb_title)
### ANONYMIZATION
1. [Using NER (omri mendels)](https://towardsdatascience.com/nlp-approaches-to-data-anonymization-1fb5bde6b929)
### DE-ANONYMIZATION
1. GPT2 - [Of language datasets ](https://arxiv.org/pdf/2012.07805.pdf)
# Interpretable & Explainable AI (XAI)
## XAI
1. [A series of videos about XAI.](https://www.youtube.com/watch?v=OZJ1IgSgP9E\&list=PLV8yxwGOxvvovp-j6ztxhF3QcKXT6vORU\&index=4)
2. [A curated document about XAI research resources. ](https://docs.google.com/spreadsheets/d/1uQy6a3BfxOXI8Nh3ECH0bqqSc95zpy4eIp_9JAMBkKg/edit?usp=sharing)
3. Interpretability and Explainability in Machine Learning [course](https://interpretable-ml-class.github.io/) / slides. Understanding, evaluating, rule based, prototype based, risk scores, generalized additive models, explaining black box, visualizing, feature importance, actionable explanations, casual models, human in the loop, connection with debugging.
4. [Explainable Machine Learning: Understanding the Limits & Pushing the Boundaries](https://drive.google.com/file/d/1xn2dCDAeEEhB_rex202KxMPqIPj31fZ4/view) a tutorial by Hima Lakkaraju (tutorial [VIDEO](https://www.chilconference.org/tutorial_T04.html), [youtube](https://www.youtube.com/watch?v=K6-ujR_67eY), [twitter](https://twitter.com/hima_lakkaraju/status/1390759698224271361))
5. [Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead](https://arxiv.org/pdf/1811.10154.pdf) by Cinthia rudin
1. A great[ talk](https://www.youtube.com/watch?app=desktop\&v=FEAk-U0dT8Y) on the topic by Shir Meir Lador
6. [explainML tutorial](https://explainml-tutorial.github.io/neurips20)
7. [When not to trust explanations :)](https://docs.google.com/presentation/d/10a0PNKwoV3a1XChzvY-T1mWudtzUIZi3sCMzVwGSYfM/edit#slide=id.p)
8. From the above image: [Paper: Principles and practice of explainable models](https://arxiv.org/abs/2009.11698) - a really good review for everything XAI - “a survey to help industry practitioners (but also data scientists more broadly) understand the field of explainable machine learning better and apply the right tools. Our latter sections build a narrative around a putative data scientist, and discuss how she might go about explaining her models by asking the right questions. From an organization viewpoint, after motivating the area broadly, we discuss the main developments, including the principles that allow us to study transparent models vs opaque models, as well as model-specific or model-agnostic post-hoc explainability approaches. We also briefly reflect on deep learning models, and conclude with a discussion about future research directions.”
9. [Book: interpretable machine learning](https://christophm.github.io/interpretable-ml-book/agnostic.html), [christoph mulner](https://christophm.github.io/)
10. (great) [Interpretability overview,](https://thegradient.pub/interpretability-in-ml-a-broad-overview/?fbclid=IwAR2ltYQWbS5jixIJzAnFg8dz1A-9y9eGIMxQfpB_Pp5x9knP1Y4JhQg3xgI) transparent (simultability, decomposability, algorithmic transparency) post-hoc interpretability (text explanation, visual local, explanation by example,), evaluation, utility.
11. [Medium: the great debate](https://medium.com/swlh/the-great-ai-debate-interpretability-1d139167b55)
12. [Paper: pitfalls to avoid when interpreting ML models](https://arxiv.org/abs/2007.04131) “A growing number of techniques provide model interpretations, but can lead to wrong conclusions if applied incorrectly. We illustrate pitfalls of ML model interpretation such as bad model generalization, dependent features, feature interactions or unjustified causal interpretations. Our paper addresses ML practitioners by raising awareness of pitfalls and pointing out solutions for correct model interpretation, as well as ML researchers by discussing open issues for further research.” - mulner et al.
13. \*\*\* [whitening a black box.](https://francescopochetti.com/whitening-a-black-box-how-to-interpret-a-ml-model/) This is very good, includes eli5, lime, shap, many others.
14. Book: [exploratory model analysis](https://pbiecek.github.io/ema/)
15. [Alibi-explain](https://github.com/SeldonIO/alibi) - White-box and black-box ML model explanation library. [Alibi](https://docs.seldon.io/projects/alibi) is an open source Python library aimed at machine learning model inspection and interpretation. The focus of the library is to provide high-quality implementations of black-box, white-box, local and global explanation methods for classification and regression models.
16. [Hands on explainable ai](https://www.youtube.com/watch?v=1mNhPoab9JI\&fbclid=IwAR1cV__3zBClI-mq3XpJfgn691xB7EM5gdZpejJ86wnrsVoiGmQFY9P5Uho) youtube, [git](https://github.com/PacktPublishing/Hands-On-Explainable-AI-XAI-with-Python?fbclid=IwAR012IQFa4ce3camoD13iIRyCfQlWPi3HwQs8VDjIGgFnGdcm3xkq7zir-U)
17. [Explainable methods](https://towardsdatascience.com/interpretable-machine-learning-with-xgboost-9ec80d148d27) are not always consistent and do not agree with each other, this article has a make-sense explanation and flow for using shap and its many plots.
Keras-vis for cnns, 3 methods, activation maximization, saliency and class activation maps
18. [The notebook!](https://github.com/FraPochetti/KagglePlaygrounds/blob/master/InterpretableML.ipynb) [Blog](https://francescopochetti.com/whitening-a-black-box-how-to-interpret-a-ml-model/)
19. [More resources!](https://docs.google.com/spreadsheets/d/1uQy6a3BfxOXI8Nh3ECH0bqqSc95zpy4eIp_9JAMBkKg/edit#gid=0)
20. [Visualizing the impact of feature attribution baseline](https://distill.pub/2020/attribution-baselines/) - Path attribution methods are a gradient-based way of explaining deep models. These methods require choosing a hyperparameter known as the baseline input. What does this hyperparameter mean, and how important is it? In this article, we investigate these questions using image classification networks as a case study. We discuss several different ways to choose a baseline input and the assumptions that are implicit in each baseline. Although we focus here on path attribution methods, our discussion of baselines is closely connected with the concept of missingness in the feature space - a concept that is critical to interpretability research.
21. WHAT IF TOOL - GOOGLE, [notebook](https://colab.research.google.com/github/PAIR-code/what-if-tool/blob/master/WIT_Smile_Detector.ipynb), [walkthrough](https://pair-code.github.io/what-if-tool/learn/tutorials/walkthrough/)
22. [Language interpretability tool (LIT) -](https://pair-code.github.io/lit/) The Language Interpretability Tool (LIT) is an open-source platform for visualization and understanding of NLP models.
23. [Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead](https://arxiv.org/abs/1811.10154) - “trying to \textit{explain} black box models, rather than creating models that are \textit{interpretable} in the first place, is likely to perpetuate bad practices and can potentially cause catastrophic harm to society. There is a way forward -- it is to design models that are inherently interpretable. This manuscript clarifies the chasm between explaining black boxes and using inherently interpretable models, outlines several key reasons why explainable black boxes should be avoided in high-stakes decisions, identifies challenges to interpretable machine learning, and provides several example applications where interpretable models could potentially replace black box models in criminal justice, healthcare, and computer vision.”
24. [Using genetic algorithms](https://towardsdatascience.com/interpreting-black-box-machine-learning-models-with-genetic-algorithms-a803bfd134cb)
25. [ Google’s what-if tool](https://pair-code.github.io/what-if-tool/demos/image.html) from [PAIR](https://pair.withgoogle.com/)
26. [Boruta](https://github.com/scikit-learn-contrib/boruta_py) ([medium](https://towardsdatascience.com/boruta-explained-the-way-i-wish-someone-explained-it-to-me-4489d70e154a)) was designed to automatically perform feature selection on a dataset using randomized features, i.e., measuring valid features against their shadow/noisy counterparts.
27. [InterpretML](https://interpret.ml/) by Microsoft, [git](https://github.com/interpretml/interpret).
28. [Connecting Interpretability and Robustness in Decision Trees through Separation](https://icml.cc/virtual/2021/poster/10107?fbclid=IwAR06qMwbn1cRgWLWtVHf_fAHEbasc0TNrWCdGiSGsIiv4kmQY1TMeTonC6I), [git](https://github.com/yangarbiter/interpretable-robust-trees?fbclid=IwAR3wqCFzuSPQgv30RVdCLi8FGjajErSvuGQd1Zq1VrkpC_bNNMgR4r_nd5w)
29. [Interpret Transformers](https://github.com/cdpierse/transformers-interpret) - explain transformers with 2 lines of code.
## Lime
1. [\*\*\* how lime works behind the scenes](https://medium.com/analytics-vidhya/explain-your-model-with-lime-5a1a5867b423)
2. [LIME to interpret models](https://www.oreilly.com/learning/introduction-to-local-interpretable-model-agnostic-explanations-lime) NLP and IMAGE, [github](https://github.com/marcotcr/lime)- In the experiments in [our research paper](http://arxiv.org/abs/1602.04938), we demonstrate that both machine learning experts and lay users greatly benefit from explanations similar to Figures 5 and 6 and are able to choose which models generalize better, improve models by changing them, and get crucial insights into the models' behavior.
## Anchor
1. [Anchor from the authors of Lime,](https://github.com/marcotcr/anchor) - An anchor explanation is a rule that sufficiently “anchors” the prediction locally – such that changes to the rest of the feature values of the instance do not matter. In other words, for instances on which the anchor holds, the prediction is (almost) always the same.
## Shap
1. Theory:
1. How Shap values are calculated - [youtube](https://www.youtube.com/watch?v=u7Om2joZWYs).
2. Cooporative game theory & Shapely values, [Medium](https://p17anshikap.medium.com/corporative-game-theory-and-shapley-values-b96dc7284701), [youtube](https://www.youtube.com/watch?v=w9O0fkfMkx0)
3. [Calculating a Taxi fare using Shap](https://www.youtube.com/watch?v=aThG4YAFErw)
4. [Shap explained](https://towardsdatascience.com/shap-explained-the-way-i-wish-someone-explained-it-to-me-ab81cc69ef30)
2. Intro to shap and lime, [part 1](https://blog.dominodatalab.com/shap-lime-python-libraries-part-1-great-explainers-pros-cons/), [part 2](https://blog.dominodatalab.com/shap-lime-python-libraries-part-2-using-shap-lime/)
3. A series on Shap, Lime.
1. Part I: [Explain Your Model with the SHAP Values](https://towardsdatascience.com/explain-your-model-with-the-shap-values-bc36aac4de3d)
2. Part II: [The SHAP with More Elegant Charts](https://dataman-ai.medium.com/the-shap-with-more-elegant-charts-bc3e73fa1c0c)
3. Part III: [How Is the Partial Dependent Plot Calculated?](https://dataman-ai.medium.com/how-is-the-partial-dependent-plot-computed-8d2001a0e556)
4. Part VI: [An Explanation for eXplainable AI](https://medium.com/analytics-vidhya/an-explanation-for-explainable-ai-xai-d56ae3dacd13)
5. Part V: [Explain Any Models with the SHAP Values — Use the KernelExplainer](https://towardsdatascience.com/explain-any-models-with-the-shap-values-use-the-kernelexplainer-79de9464897a)
6. Part VI: [The SHAP Values with H2O Models](https://medium.com/dataman-in-ai/the-shap-values-with-h2o-models-773a203b75e3)
7. Part VII: [Explain Your Model with LIME](https://medium.com/@Dataman.ai/explain-your-model-with-lime-5a1a5867b423)
8. Part VIII: [Explain Your Model with Microsoft’s InterpretML](https://medium.com/@Dataman.ai/explain-your-model-with-microsofts-interpretml-5daab1d693b4)
4. Medium [Intro to lime and shap](https://towardsdatascience.com/explain-nlp-models-with-lime-shap-5c5a9f84d59b)
5. \*\*\*\* In depth [SHAP](https://towardsdatascience.com/introducing-shap-decision-plots-52ed3b4a1cba)
6. [Github](https://github.com/slundberg/shap)
7. [Country happiness using shap](https://sararobinson.dev/2019/03/24/preventing-bias-machine-learning.html)
8. [Stackoverflow example, predicting tags, pandas keras etc](https://stackoverflow.blog/2019/05/06/predicting-stack-overflow-tags-with-googles-cloud-ai/)
9. [Intro to shapely and shap](https://towardsdatascience.com/a-new-perspective-on-shapley-values-an-intro-to-shapley-and-shap-6f1c70161e8d?)
10. [Fiddler on shap](https://medium.com/fiddlerlabs/case-study-explaining-credit-modeling-predictions-with-shap-2a7b3f86ec12)
11. Shapash
1. [shapash git - ](https://github.com/MAIF/shapash)[a web app](https://github.com/MAIF/shapash) (lime and shap)[. ](https://github.com/MAIF/shapash)
2. [making models understandable by everyone](https://pub.towardsai.net/shapash-making-ml-models-understandable-by-everyone-8f96ad469eb3) - Yann Golhen
3. [using shapash for confidence on XAI.](https://towardsdatascience.com/building-confidence-on-explainability-methods-66b9ee575514) - francesco marini\
using 3 new metrics
1. Consistency - *do different explainability methods give, on average, similar explanations?*
2. Stability - *for similar instances, are the explanations similar?*
3. Compacity - do fewer features drive the model?
12. Partial Shap
1. [Which Of Your Features Are Overfitting](https://towardsdatascience.com/which-of-your-features-are-overfitting-c46d0762e769)? by Samuele Mazzanti - "Discover “ParShap”: an advanced method to detect which columns make your model underperform on new data" implemented in [pingouin](https://pingouin-stats.org/)-stats.
13. Shap residuals
1. [medium](https://towardsdatascience.com/shapley-residuals-measuring-the-limitations-of-shapley-values-for-explainability-d9cdc3582522)
14. SHAP advanced
1. [Official shap tutorial on their plots, you can never read this too many times.](https://slundberg.github.io/shap/notebooks/plots/decision_plot.html)
2. [What are shap values on kaggle](https://www.kaggle.com/dansbecker/shap-values) - whatever you do start with this
3. [Shap values on kaggle #2](https://www.kaggle.com/dansbecker/advanced-uses-of-shap-values) - continue with this
4. How to calculate Shap values per class based on this graph
15. [A thorough post about the many ways of explaining a model, from regression, to bayes, to trees, forests, lime, beta, feature selection/elimination](https://lilianweng.github.io/lil-log/2017/08/01/how-to-explain-the-prediction-of-a-machine-learning-model.html#interpretable-models)
16. [Trusting models](https://arxiv.org/pdf/1602.04938.pdf)
17. [Interpret using uncertainty](https://becominghuman.ai/using-uncertainty-to-interpret-your-model-67a97c28fea5)
18. [Shap in Python](https://towardsdatascience.com/introduction-to-shap-with-python-d27edc23c454)
# Federated Learning
1. an awesome [guide](https://www.v7labs.com/blog/federated-learning-guide) by v7labs - centralized, decentralized, etc.
# Algorithms 101
1. [**Amortization**](https://medium.com/@aleksandrasays/amortised-analysis-in-the-nutshell-7b056277ab9b) **- Amortised analysis in the nutshell, Worst-Case vs Average-Case, and then where amortized is the average performance (over time) of each operation in the worst-case.**
# Meta Learning (AutoML)
[**What is?**](https://www.automl.org/) **Automated Machine Learning provides methods and processes to make Machine Learning available for non-Machine Learning experts, to improve efficiency of Machine Learning and to accelerate research on Machine Learning.**
**Personal note: automl algorithms in this field will bridge the gap and automate several key processes, but it will not allow a practitioner to do serious research or solve business or product problems easily. The importance of this field is to advance each subfield, whether HPO, NAS, etc. these selective novelties can help us solve specific issues, i.e, lets take HPO, we can use it to save time and money on redundant parameter searches, especially when it comes to resource heavy algorithms such as Deep learning (think GPU costs).**
**Personal thoughts on optimizations: be advised that optimizing problems will not guarantee a good result, you may over fit your problem in ways you are not aware of, beyond traditional overfitting and better accuracy doesn't guarantee a better result (for example if your dataset is unbalanced, needs imputing, cleaning, etc.).** \
**Always examine the data and results in order to see if they are correct.**
[**Automl.org’s github - it has a backup for the following projects.**](https://github.com/automl)
[**Automl.org**](https://www.automl.org/) **is a joint effort between two universitie, freiburg and hannover, their website curates information regarding:**
1. **HPO - hyper parameter optimization**
2. **NAS - neural architecture search**
3. **Meta Learning - learning across datasets, warmstarting of HPO and NAS etc.**
**Automl aims to automate these processes:**
* **Preprocess and clean the data.**
* **Select and construct appropriate features.**
* **Select an appropriate model family.**
* **Optimize model hyperparameters.**
* **Postprocess machine learning models.**
* **Critically analyze the results obtained.**
**Historically, AFAIK AutoML’s birth started with several methods to optimize each one of the previous processes in ml. IINM,** [**weka’s paper (2012**](https://arxiv.org/abs/1208.3719)**) was the first step in aggregating these ideas into a first public working solution.**
**The following is referenced from AutoML.org:**
### **ML Systems**
* [**AutoWEKA**](http://www.cs.ubc.ca/labs/beta/Projects/autoweka/) **is an approach for the simultaneous selection of a machine learning algorithm and its hyperparameters; combined with the** [**WEKA**](http://www.cs.waikato.ac.nz/ml/weka/) **package it automatically yields good models for a wide variety of data sets.**
* [**Auto-sklearn**](http://automl.github.io/auto-sklearn/stable/) **is an extension of AutoWEKA using the Python library** [**scikit-learn**](http://scikit-learn.org/stable/) **which is a drop-in replacement for regular scikit-learn classifiers and regressors.**
* [**TPOT**](http://epistasislab.github.io/tpot/) **is a data-science assistant which optimizes machine learning pipelines using genetic programming.**
* **(google)** [**H2O AutoML**](http://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html) **provides automated model selection and ensembling for the** [**H2O machine learning and data analytics platform**](http://docs.h2o.ai/h2o/latest-stable/h2o-docs/welcome.html)**. (**[**git**](https://github.com/google/automl)**)**
* [**TransmogrifAI**](https://github.com/salesforce/TransmogrifAI) **is an AutoML library running on top of Spark.**
* [**MLBoX**](https://github.com/AxeldeRomblay/MLBox) **is an AutoML library with three components: preprocessing, optimisation and prediction**
* [**MLJar**](https://mljar.com/) **(**[**git**](https://github.com/mljar/mljar-supervised)**)** [**medium**](https://medium.com/@MLJARofficial/mljar-supervised-automl-with-explanations-and-markdown-reports-36d5104e117)**,** [**2**](https://towardsdatascience.com/automating-eda-machine-learning-6ddb76c1eb4d) **- Automated Machine Learning for tabular data mljar builds a complete Machine Learning Pipeline. Perform exploratory analysis, search for a signal in the data, and discover relationships between features in your data with AutoML. Train top ML models with advanced feature engineering, many algorithms, hyper-parameters tuning, Ensembling, and Stacking. Stay ahead of competitors and predict the future with advanced ML. Deploy your models in the cloud or use them locally**
* **+ advanced feature engineering**
* **+ algorithms selection and tuning**
* **+ automatic documentation**
* **+ ML explanations**\
### **Hyper param optimization**
* [**Hyperopt**](http://jaberg.github.io/hyperopt/)**, including the TPE algorithm**
* [**Sequential Model-based Algorithm Configuration (SMAC)**](http://aclib.net/SMAC/)
* [**Spearmint**](https://github.com/JasperSnoek/spearmint)
* [**BOHB**](https://www.automl.org/automl/bohb/)**: Bayesian Optimization combined with HyperBand**
* [**RoBO – Robust Bayesian Optimization framework**](http://www.automl.org/automl/robo/)
* [**SMAC3**](https://github.com/automl/SMAC3) **– a python re-implementation of the SMAC algorithm**
### **Architecture Search**
* [**Auto-PyTorch**](https://github.com/automl/Auto-PyTorch)
* [**AutoKeras**](https://autokeras.com/)
* [**DEvol**](https://github.com/joeddav/devol)
* [**HyperAS**](https://github.com/maxpumperla/hyperas)**: a combination of Keras and Hyperopt**
* [**talos**](https://github.com/autonomio/talos)**: Hyperparameter Scanning and Optimization for Keras**
### **Auto Feature Engineering**
1. [automated feature engineering on medium by ](https://towardsdatascience.com/automated-feature-engineering-in-python-99baf11cc219)will koehrsen
# Probabilistic, Regression
### **PROBABILISTIC ALGORITHMS**
#### **NAIVE BAYES**
1. [**Vidhya on NB**](https://towardsdatascience.com/my-secret-sauce-to-be-in-top-2-of-a-kaggle-competition-57cff0677d3c?fbclid=IwAR3Iei5OmwswIMbbqcz2dNr5rLsWS-iuuaAuOjmhCELTTEBTPmSM85mTw7U)
2. [**Baysian tree**](https://github.com/UBS-IB/bayesian_tree)
3. [**NB, GNB, multi nominal NB**](https://jakevdp.github.io/PythonDataScienceHandbook/05.05-naive-bayes.html)
#### **BAYES, BAYESIAN BELIEF NETWORKS**
1. [**Mastery on bayes theorem**](https://machinelearningmastery.com/bayes-theorem-for-machine-learning/?fbclid=IwAR3txPR1zRLXhmArXsGZFSphhnXyLEamLyyqbAK8zBBSZ7TM3e6b3c3U49E)
2. [**Introduction To BBS**](https://codesachin.wordpress.com/2017/03/10/an-introduction-to-bayesian-belief-networks/) **- a very good blog post**
3. **A** [**complementing SLIDE presentation**](https://www.slideshare.net/GiladBarkan/bayesian-belief-networks-for-dummies) **that shows how to build the network’s tables**
4. **A** [**very nice presentation**](http://chem-eng.utoronto.ca/~datamining/Presentations/Bayesian_Belief_Network.pdf) **regarding BBS**
5.
6. [**Maximum Likelihood**](http://mathworld.wolfram.com/MaximumLikelihood.html) **(log likelihood) - proofs for bernoulli, normal, poisson.**
7. [**Another example**](https://codesachin.wordpress.com/2017/03/10/an-introduction-to-bayesian-belief-networks/)
#### **MARKOV MODELS**
**Random vs Stochastic (**[**here**](https://math.stackexchange.com/questions/114373/whats-the-difference-between-stochastic-and-random) **and** [**here**](https://math.stackexchange.com/questions/569951/what-is-the-difference-between-a-random-vector-and-a-stochastic-process)**):**
* **A variable is 'random'.**
* **A process is 'stochastic'.**
**Apart from this difference the two words are synonyms**
**In other words:**
* **A random vector is a generalization of a single random variables to many.**
* **A stochastic process is a sequence of random variables, or a sequence of random vectors (and then you have a vector-stochastic process).**
**(**[**What is a Markov Model?)**](http://cecas.clemson.edu/~ahoover/ece854/refs/Ramos-Intro-HMM.pdf) **A Markov Model is a stochastic(random) model which models temporal or sequential data, i.e., data that are ordered.**
* **It provides a way to model the dependencies of current information (e.g. weather) with previous information.**
* **It is composed of states, transition scheme between states, and emission of outputs (discrete or continuous).**
* **Several goals can be accomplished by using Markov models:**
* **Learn statistics of sequential data.**
* **Do prediction or estimation.**
* **Recognize patterns.**
**(**[**sunny cloudy explanation**](http://techeffigytutorials.blogspot.co.il/2015/01/markov-chains-explained.html)**) Markov Chains is a probabilistic process, that relies on the current state to predict the next state.**
* **to be effective the current state has to be dependent on the previous state in some way**
* **if it looks cloudy outside, the next state we expect is rain.**
* **If the rain starts to subside into cloudiness, the next state will most likely be sunny.**
* **Not every process has the Markov Property, such as the Lottery, this weeks winning numbers have no dependence to the previous weeks winning numbers.**
1. **They show how to build an order 1 markov table of probabilities, predicting the next state given the current.**
2. **Then it shows the state diagram built from this table.**
3. **Then how to build a transition matrix from the 3 states, i.e., from the probabilities in the table**
4. **Then how to calculate the next state using the “current state vector” doing vec\*matrix multiplications.**
5. **Then it talks about the setting always into the rain prediction, and the solution is using two last states in a bigger table of order 2. He is not really telling us why the probabilities don't change if we add more states, it stays the same as in order 1, just repeating.**
#### **MARKOV MODELS / HIDDEN MARKOV MODEL**
**HMM tutorials**
1. **HMM tutorial**
1. **Part** [**1**](http://gekkoquant.com/2014/05/18/hidden-markov-models-model-description-part-1-of-4/)**,** [**2**](http://gekkoquant.com/2014/05/26/hidden-markov-models-forward-viterbi-algorithm-part-2-of-4/)**,** [**3**](http://gekkoquant.com/2014/09/07/hidden-markov-models-examples-in-r-part-3-of-4/)**,** [**4**](http://gekkoquant.com/2015/02/01/hidden-markov-models-trend-following-sharpe-ratio-3-1-part-4-of-4/)
2. **Medium**
1. [**Intro to HMM**](https://towardsdatascience.com/introduction-to-hidden-markov-models-cd2c93e6b781) **/ MM**
2. [**Paper like example**](https://medium.com/@kangeugine/hidden-markov-model-7681c22f5b9)
3. [**HMM with sklearn and networkx**](http://www.blackarbs.com/blog/introduction-hidden-markov-models-python-networkx-sklearn/2/9/2017)
**HMM variants**
1. [**Stack exchange on hmm**](https://datascience.stackexchange.com/questions/8460/python-library-to-implement-hidden-markov-models)
2. [**HMM LEARN**](https://github.com/hmmlearn/hmmlearn) **(sklearn, still being developed)**
3. [**Pomegranate**](https://pomegranate.readthedocs.io/en/latest/) **(this is good)**
1. **General mixture models**
2. **Hmm**
3. **Basyes classifiers and naive bayes**
4. **Markov changes**
5. **Bayesian networks**
6. **Markov networks**
7. **Factor graphs**
4. [**GHMM with python wrappers**](http://ghmm.org/)**,**
5. [**Hmms**](https://github.com/lopatovsky/HMMs) **(old)**
**HMM (**[**what is? And why HIDDEN?)**](https://youtu.be/jY2E6ExLxaw?t=27m38s) **- the idea is that there are things that you CAN OBSERVE and there are things that you CAN'T OBSERVE. From the things you OBSERVE you want to INFER the things you CAN'T OBSERVE (HIDDEN). I.e., you play against someone else in a game, you don't see their choice of action, but you see the result.**
1. **Python** [**code**](https://github.com/hmmlearn/hmmlearn)**, previously part of** [**sklearn** ](http://scikit-learn.sourceforge.net/stable/modules/hmm.html)
2. **Python** [**seqLearn**](http://larsmans.github.io/seqlearn/reference.html) **- supervised multinomial HMM**
**This youtube video** [**part1**](https://www.youtube.com/watch?v=TPRoLreU9lA) **- explains about the hidden markov model. It shows the visual representation of the model and how we go from that the formula:** 
**It breaks down the formula to:**
* **transition probability formula - the probability of going from Zk to Zk+1**
* **emission probability formula - the probability of going from Zk to Xk**
* **(Pi) Initial distribution - the probability of Z1=i for i=1..m**
**In** [**part2**](https://www.youtube.com/watch?v=M_IIW0VYMEA) **of the video:**
**\*** [**HMM in weka, with github, working on 7.3, not on 9.1**](http://www.doc.gold.ac.uk/~mas02mg/software/hmmweka/index.html)
1. **Probably the simplest explanation of Markov Models and HMM as a “game” -** [**link**](http://www.fejes.ca/EasyHMM.html)
2. **This** [**video**](https://www.youtube.com/watch?v=jY2E6ExLxaw) **explains that building blocks of the needed knowledge in HMM, starting probabilities P0, transitions and emissions (state probabilities)**
3. **This** [**post**](https://www.quora.com/What-is-a-simple-explanation-of-the-Hidden-Markov-Model-algorithm)**, explains HMM and ties our understanding.**
[**A cute explanation on quora**](https://www.quora.com/What-is-a-simple-explanation-of-the-Hidden-Markov-Model-algorithm)**:**
**This is the iconic image of a Hidden Markov Model. There is some state (x) that changes with time (markov). And you want to estimate or track it. Unfortunately, you cannot directly observe this state (hidden). That's the hidden part. But, you can observe something correlated with the state (y).**
**OBSERVED DATA -> INFER -> what you CANT OBSERVE (HIDDEN).**
**Considering this model:**
* **where P(X0) is the initial state for happy or sad**
* **Where P(Xt | X t-1) is the transition model from time-1 to time**
* **Where P(Yt | Xt) is the observation model for happy and sad (X) in 4 situations (w, sad, crying, facebook)**
#### **INPUT OUTPUT HMM (IOHMM)**
1. [**Incomplete python code**](https://github.com/Mogeng/IOHMM) **for unsupervised / semi-supervised / supervised IOHMM - training is there, prediction is missing.**
2. [**Machine learning - a probabilistic approach, david barber.**](https://pdfs.semanticscholar.org/a632/9a41ee67fae978ccac1e37370f074497a4fe.pdf)
#### **CONDITIONAL RANDOM FIELDS (CRF)**
1. [**Make sense intro to CRF, comparison against HMM** ](https://medium.com/ml2vec/overview-of-conditional-random-fields-68a2a20fa541)
2. [**HMM, CRF, MEMM**](https://medium.com/@Alibaba_Cloud/hmm-memm-and-crf-a-comparative-analysis-of-statistical-modeling-methods-49fc32a73586)
3. [**Another crf article**](https://medium.com/@phylypo/nlp-text-segmentation-using-conditional-random-fields-e8ff1d2b6060)
4. **Neural network CRF** [**NNCRF**](https://medium.com/@Akhilesh_k_r/neural-networks-conditional-random-field-crf-973712a0fd30)
5. [**Another one**](https://towardsdatascience.com/conditional-random-fields-explained-e5b8256da776)
6. [**scikit-learn inspired API for CRFsuite**](https://github.com/TeamHG-Memex/sklearn-crfsuite)
7. [**Sklearn wrapper**](https://github.com/supercoderhawk/sklearn-crfsuite)
8. [**Python crfsuite**](https://github.com/scrapinghub/python-crfsuite) **wrapper**
9. [**Pycrf suite vidahya**](https://www.analyticsvidhya.com/blog/2018/08/nlp-guide-conditional-random-fields-text-classification/)
### **REGRESSION ALGORITHMS**
1. [**Sk-lego**](https://scikit-lego.readthedocs.io/en/latest/preprocessing.html#Interval-Encoders) **to fit with intervals a linear regressor on top of non linear data**
1. **Sk-lego monotonic**
1. [**Lightning**](https://github.com/scikit-learn-contrib/lightning) **- lightning is a library for large-scale linear classification, regression and ranking in Python.**\
2. **Linear regression TBC**
3. **CART -** [**classification and regression tree**](http://www.simafore.com/blog/bid/62482/2-main-differences-between-classification-and-regression-trees)**, basically the diff between classification and regression trees - instead of IG we use sum squared error**
4. **SVR - regression based svm, with kernel only.**
5. [**NNR**](https://deeplearning4j.org/linear-regression)**- regression based NN, one output node**
6. [**LOGREG**](http://www.statisticssolutions.com/what-is-logistic-regression/) **- Logistic regression - is used as a classification algo to describe data and to explain the relationship between one dependent binary variable and one or more nominal, ordinal, interval or ratio-level independent variables. Output is BINARY. I.e., If the likelihood of killing the bug is > 0.5 it is assumed dead, if it is < 0.5 it is assumed alive.**
* **Assumes binary outcome**
* **Assumes no outliers**
* **Assumes no intercorrelations among predictors (inputs?)**
**Regression Measurements:**
1. **R^2 -** [**several reasons it can be too high.**](http://blog.minitab.com/blog/adventures-in-statistics-2/five-reasons-why-your-r-squared-can-be-too-high)
1. **Too many variables**
2. **Overfitting**
3. **Time series - seasonality trends can cause this**
2. [**RMSE vs MAE**](https://medium.com/human-in-a-machine-world/mae-and-rmse-which-metric-is-better-e60ac3bde13d)
#### **KERNEL REGRESSION**
[**Gaussian Kernel Regression**](http://mccormickml.com/2014/02/26/kernel-regression/) **does–it takes a weighted average of the surrounding points**
* **variance, sigma^2. Informally, this parameter will control the smoothness of your approximated function.**
* **Smaller values of sigma will cause the function to overfit the data points, while larger values will cause it to underfit**
* **There is a proposed method to find sigma in the post!**
* **Gaussian Kernel Regression is equivalent to creating an RBF Network with the following properties: - described in the post**
### 
#### **DIMENSIONALITY REDUCTION**
**PRINCIPAL COMPONENT REGRESSION (PCR) / PARTIAL LEAST SQUARES (PLS)**
[**Principal component regression (PCR) Partial least squares and (PLS)**](https://www.kdnuggets.com/2017/11/10-statistical-techniques-data-scientists-need-master.html/2) **- basically PCA and linear regression , however PLS makes use of the response variable in order to identify the new features.**
**One can describe Principal Components Regression as an approach for deriving a low-dimensional set of features from a large set of variables. The first principal component direction of the data is along which the observations vary the most. In other words, the first PC is a line that fits as close as possible to the data. One can fit p distinct principal components. The second PC is a linear combination of the variables that is uncorrelated with the first PC, and has the largest variance subject to this constraint. The idea is that the principal components capture the most variance in the data using linear combinations of the data in subsequently orthogonal directions. In this way, we can also combine the effects of correlated variables to get more information out of the available data, whereas in regular least squares we would have to discard one of the correlated variables.**
**The PCR method that we described above involves identifying linear combinations of X that best represent the predictors. These combinations (directions) are identified in an unsupervised way, since the response Y is not used to help determine the principal component directions. That is, the response Y does not supervise the identification of the principal components, thus there is no guarantee that the directions that best explain the predictors also are the best for predicting the response (even though that is often assumed). Partial least squares (PLS) are a supervised alternative to PCR. Like PCR, PLS is a dimension reduction method, which first identifies a new smaller set of features that are linear combinations of the original features, then fits a linear model via least squares to the new M features. Yet, unlike PCR, PLS makes use of the response variable in order to identify the new features.**
# Data Mining
### **ASSOCIATION RULES**
1. [**Association rules slides**](https://www.slideshare.net/wanaezwani/apriori-and-eclat-algorithm-in-association-rule-mining) **- apriori, eclat, fp growth - pretty complete**
2. [**Terms**](https://www.kdnuggets.com/2016/04/association-rules-apriori-algorithm-tutorial.html) **- lift, confidence**
3. [**Paper - basic concepts and algo**](https://www-users.cs.umn.edu/~kumar001/dmbook/ch5_association_analysis.pdf)
**Knoldus**
1. [**Apriori**](https://blog.knoldus.com/machinex-why-no-one-uses-apriori-algorithm-for-association-rule-learning/)
2. [**Association rules**](https://blog.knoldus.com/machinex-two-parts-of-association-rule-learning/)
3. [**Fp-growth**](https://blog.knoldus.com/machinex-frequent-itemset-generation-with-the-fp-growth-algorithm/)
4. [**Fp-tree construction**](https://blog.knoldus.com/machinex-understanding-fp-tree-construction/)
**APRIORI**
1. [**Apyori tut**](https://stackabuse.com/association-rule-mining-via-apriori-algorithm-in-python/) [**git**](https://github.com/ymoch/apyori)
2. [**Efficient apriori**](https://github.com/tommyod/Efficient-Apriori)
3. [**One of the best known association rules algorithm**](https://machinelearningmastery.com/market-basket-analysis-with-association-rule-learning/) **- apriori in weka**
4. [**A very good visual example of a transaction DB with the apriori algorithm step by step**](http://www.lessons2all.com/Apriori.php)
5. [**Python 3.0 code**](http://adataanalyst.com/machine-learning/apriori-algorithm-python-3-0/)
6. [**Mlxtnd**](http://rasbt.github.io/mlxtend/api_subpackages/mlxtend.frequent_patterns/) [**tutorial**](https://www.geeksforgeeks.org/implementing-apriori-algorithm-in-python/)
1. **Apriori**
2. **Rules**
3. **pgrowth**
4. **fpmax**
**FP Growth**
1. [**How to construct the fp-tree**](https://www.youtube.com/watch?v=gq6nKbye648)
2. **The same example, but with a graph that shows that lower support cost less for fp-growth in terms of calc time.**
3. [**Coursera video**](https://www.coursera.org/learn/data-patterns/lecture/ugqCs/2-5-fpgrowth-a-pattern-growth-approach)
4. **Another clip video**
5. [**How to validate these algorithms**](https://stackoverflow.com/questions/32843093/how-to-validate-association-rules) **- probably the best way is confidence/support/lift**
**It depends on your task. But usually you want all** [**three to be high.**](https://stats.stackexchange.com/questions/229523/association-rules-support-confidence-and-lift)
* **high support: should apply to a large amount of cases**
* **high confidence: should be correct often**
* **high lift: indicates it is not just a coincidence**
1. [**Difference between apriori and fp-growth**](https://www.quora.com/What-is-the-difference-between-FPgrowth-and-Apriori-algorithms-in-terms-of-results)
# Process Mining
Processes were usually manual, giving trust in people following them, e.g. Process that was defined by the company, and no automation. Process mining identifies the process from the logs (**process discovery**).
1. Modeling done with [BPMN](https://www.bpmn.org/) business process model notation language (UML) (DAG), i.e., static, boxes & arrows vs , i.e., dynamic, token based, which allows simulations.
2. Conformance checking - a comparison the real process and the discovered.
1. if you do not do certain parts in the process you are not compliant.
2. for example to find out whether people taking shortcuts? optimizing the process without knowing.
3. can be used for offline vs real time process bug alerting
4. XES
### Tutorials
1.
2.
3.
4.
5.
6. YouTube
1.
### Tools
1. services need to be process-aware, i.e. send standardized logs - IBM process mining
2. Algorithms - can deal with parallelism
1. [Alpha miner](http://mlwiki.org/index.php/Alpha_Algorithm)
2. Inductive miner
3.
4.
5.
6.
# Label Algorithms
## Unbalanced labels
1. [imbalance learn](https://imbalanced-learn.org/stable/auto_examples/over-sampling/plot_comparison_over_sampling.html#sphx-glr-auto-examples-over-sampling-plot-comparison-over-sampling-py) - is an open-source, MIT-licensed library that provides tools when dealing with classification with imbalanced classes.
2. [Classifying Job Titles With Noisy Labels Using REINFORCE ](https://medium.com/@ziprecruiter.engineering/classifying-job-titles-with-noisy-labels-using-reinforce-ce1a4bde05e2)this article has a very nice trick in adding a reward component to the loss function in order to mitigate for unbalanced class label problem, instead of the usual balancing.
## Label Propagation / Spreading
Note: very much related to weakly and semi supervision, i.e., we have small amounts of labels and we want to generalize the labels to other samples, see also weak supervision methods.
1. Step 1: [build a laplacian](https://en.wikipedia.org/wiki/Laplacian_matrix) graph using KNN, distance metric is minkowski with p=2, i.e. euclidean distance.
2. [Step by step tutorial](https://medium.com/@graphml/introduction-to-label-propagation-with-networkx-part-1-abcbe954a2e8), [part 2](https://medium.com/@graphml/introduction-to-label-propagation-with-networkx-part-2-cd041fa44e1)
3. [Spreading](https://scikit-learn.org/stable/modules/generated/sklearn.semi_supervised.LabelSpreading.html) (propagation upgrade), Essentially a community graph algorithm, however it resembles KNN in its nature, using semi supervised data set (i.e., labeled and unlabeled data) to spread or propagate labels to unlabeled data, with small incrementations in the algorithm, using KNN-like methodology, each unlabeled sample will be given a label based on its 1st order friends, if there is a tie, a random label is chosen. Nodes are connected by using a euclidean distance.
4. [Difference](https://www.researchgate.net/post/What_is_the_difference_between_Label_propagation_and_Label_spreading_in_semi-supervised_learning_context) between propagation and spreading is a laplacian matrix, vs normalized LM
5. [Laplacian matrix on youtube, videos 30-33](https://www.youtube.com/watch?v=siCPjpUtE0A\&list=PLLssT5z_DsK9JDLcT8T62VtzwyW9LNepV\&index=33)
6. [Really good example notebook](https://github.com/DavidBrear/sklearn-cookbook/blob/master/Chapter%204/4.1.1%20Label%20Propagation%20with%20Semi-Supervised%20Learning.ipynb)
7. [Spreading vs propagation](https://www.researchgate.net/post/What_is_the_difference_between_Label_propagation_and_Label_spreading_in_semi-supervised_learning_context)
8.
9. Youtube [1](https://www.youtube.com/watch?v=UWf8hxeehOg), [2](https://www.youtube.com/watch?v=hmashUPJwSQ), [3](https://www.youtube.com/watch?v=F4f247IyOTs),
10. [Medium](https://medium.com/@graphml/introduction-to-label-propagation-with-networkx-part-1-abcbe954a2e8),
11. [Sklearn](https://scikit-learn.org/stable/modules/label_propagation.html), [1](https://scikit-learn.org/stable/modules/generated/sklearn.semi_supervised.LabelPropagation.html), [2](https://scikit-learn.org/stable/modules/generated/sklearn.semi_supervised.LabelPropagation.html), [3](https://scikit-learn.org/stable/auto_examples/semi_supervised/plot_label_propagation_digits.html), [4](https://plot.ly/scikit-learn/plot-label-propagation-structure/), 5,
1. [Git](https://github.com/benedekrozemberczki/LabelPropagation), [incremental LP](https://github.com/johny-c/incremental-label-propagation)
2. [Git2](https://github.com/yamaguchiyuto/label_propagation%5C)
1. Harmonic Function (HMN) \[Zhu+, ICML03]
2. Local and Global Consistency (LGC) \[Zhou+, NIPS04]
3. Partially Absorbing Random Walk (PARW) \[Wu+, NIPS12]
4. OMNI-Prop (OMNIProp) \[Yamaguchi+, AAAI15]
5. Confidence-Aware Modulated Label Propagation (CAMLP) \[Yamaguchi+, SDM16]
3.
1. Presentation [1](http://www.leonidzhukov.net/hse/2015/networks/lectures/lecture17.pdf),[2 ](https://www.slideshare.net/dav009/label-propagation-semisupervised-learning-with-applications-to-nlp)
Neo4j [1](https://dzone.com/articles/graph-algorithms-in-neo4j-label-propagation), 2, 3,
## Label Noise
1. [clean lab](https://github.com/cleanlab/cleanlab) - "`cleanlab` is the data-centric ML ops package for machine learning with noisy labels. `cleanlab` `clean`s `lab`els and supports finding, quantifying, and learning with label errors in datasets. See datasets cleaned with `cleanlab` at [labelerrors.com](https://labelerrors.com/). Check out the: [cleanlab code documentation](https://cleanlab.readthedocs.io/). `cleanlab` is powered by confident learning, published in this [paper](https://jair.org/index.php/jair/article/view/12125) | [blog](https://l7.curtisnorthcutt.com/confident-learning)."
1. Reference 1: [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://openreview.net/forum?id=XccDXrDNLek) by [Curtis G Northcutt](https://openreview.net/profile?id=~Curtis_G_Northcutt1), [Anish Athalye](https://openreview.net/profile?id=~Anish_Athalye1), [Jonas Mueller](https://openreview.net/profile?id=~Jonas_Mueller1)
2. Reference 2: [Confident Learning: Estimating Uncertainty in Dataset Labels](https://www.jair.org/index.php/jair/article/view/12125)
3. can be used for positive unlabeled learning
2. [PULearn](https://github.com/pulearn/pulearn) - "Positive-unlabeled learning (aka PU-learning) is a machine learning scenario for binary classification where the training set consists of a set of positively-labeled examples and an additional unlabeled set that contains positive and negative examples in unknown proportions (so no training example is explicitly labeled as negative). Positive-unlabeled learning methods aim to incorporate the unique structure of this scenario into the learning process, in a way that improves generalization of the learned notion of the positive class, when compared to simply treating all unlabeled examples as negative examples, or alternatively discarding them and training a one-class classifier over only the positive samples."
3. [PUMML](https://github.com/ncfrey/pumml), [Medium](https://towardsdatascience.com/positive-and-unlabeled-materials-machine-learning-8b216edea899) - "Positive and Unlabeled Materials Machine Learning (pumml) is a code that uses semi-supervised machine learning to classify materials from only positive and unlabeled examples."
# Clustering Algorithms
1. [Vidhya on clustering and methods](https://www.analyticsvidhya.com/blog/2016/11/an-introduction-to-clustering-and-different-methods-of-clustering/?utm_source=facebook.com)
2. [KNN](https://www.youtube.com/watch?v=4ObVzTuFivY) [intuition 2](https://www.youtube.com/watch?v=UqYde-LULfs), [thorough explanation 3](https://towardsdatascience.com/introduction-to-k-nearest-neighbors-3b534bb11d26) - classify a new sample by looking at the majority vote of its K-nearest neighbours. k=1 special case. Even amount of classes needs an odd K that is not a multiple of the amount of classes in order to break ties.
3. [Determinging the number of clusters, a comparison of several methods, elbow, silhouette etc](https://www.datanovia.com/en/lessons/determining-the-optimal-number-of-clusters-3-must-know-methods/)
4. [A good visual example of kmeans / gmm](https://medium.com/sfu-cspmp/distilling-gaussian-mixture-models-701fa9546d9)
5. [Kmeans with DTW, probably fixed length vectors, using tslearn](https://towardsdatascience.com/how-to-apply-k-means-clustering-to-time-series-data-28d04a8f7da3)
6. [Kmeans for variable length](https://medium.com/@iliazaitsev/how-to-classify-a-dataset-with-observations-of-various-length-96fab8e95baf), [notebook](https://github.com/devforfu/Blog/blob/master/trees/scikit_learn.py)
TOOLS
1. [pyClustering ](https://pyclustering.github.io/docs/0.10.1/html/index.html)
###
### Block Modeling - Distance Matrices
1. [Biclustering and spectral co clustering](https://scikit-learn.org/stable/modules/biclustering.html)
2. [Clustering correlation, or distance matrices.](https://stats.stackexchange.com/questions/138325/clustering-a-correlation-matrix)
1. Any of the “precomputed” algorithms in sklearn, just remember to [do 1-distanceMatrix](https://github.com/scikit-learn/scikit-learn/issues/6787). I.e., using dbscan/hdbscan/optics, you need a dissimilarity matrix.
2.
### [Kmeans](https://github.com/jakevdp/sklearn_pycon2015/blob/master/notebooks/04.2-Clustering-KMeans.ipynb)
1. Sensitive to outliers, can skew results (because we rely on the mean)
### [K-mediods](https://en.wikipedia.org/wiki/K-medoids)
\- basically k-means with a most center object rather than a center virtual point that was based on mean distance from all points, we keep choosing medoids samples based on minimised SSE
* k-medoid is a classical partitioning technique of clustering that clusters the data set of n objects into k clusters known a priori.
* It is more robust to noise and outliers as compared to [k-means](https://en.wikipedia.org/wiki/K-means) because it minimizes a sum of pairwise dissimilarities instead of a sum of squared Euclidean distances.
* A [medoid](https://en.wikipedia.org/wiki/Medoid) can be defined as the object of a cluster whose average dissimilarity to all the objects in the cluster is minimal. i.e. it is a most centrally located point in the cluster.
* Does Not scale to many samples, its O(K\*n-K)^2
* Randomized resampling can assure efficiency and quality.
[From youtube (okay video)](https://www.youtube.com/watch?v=OWpRBCrx5-M)
### K-modes
1. "Python implementations of the k-modes and k-prototypes clustering algorithms, for clustering categorical data" - [git](https://github.com/nicodv/kmodes#huang97)
2. [a guide to clustering mixed types](https://bpostance.github.io/posts/clustering-mixed-data/), i.e., numerics, categoricals
### X-means
X-means([paper](https://www.cs.cmu.edu/~dpelleg/download/xmeans.pdf)):
1. [Theory](https://stats.stackexchange.com/questions/13103/x-mean-algorithm-bic-calculation-question) behind bic calculation with a formula.
2. Code: [Calculate bic in k-means](https://stats.stackexchange.com/questions/90769/using-bic-to-estimate-the-number-of-k-in-kmeans?rq=1)
### G-means
G-means [Improves on X-means](https://papers.nips.cc/paper/2526-learning-the-k-in-k-means.pdf) in the paper: The G-means algorithm starts with a small number of k-means centers, and grows the number of centers. Each iteration of the algorithm splits into two those centers whose data appear not to come from a Gaussian distribution using the Anderson Darling test. Between each round of splitting, we run k-means on the entire dataset and all the centers to refine the current solution. We can initialize with just k = 1, or we can choose some larger value of k if we have some prior knowledge about the range of k. G-means repeatedly makes decisions based on a statistical test for the data assigned to each enter. If the data currently assigned to a k-means center appear to be Gaussian, then we want to represent that data with only one center.
### GMM - Gaussian Mixture Models
?- [What is GMM](https://datascience.stackexchange.com/questions/14435/how-to-get-the-probability-of-belonging-to-clusters-for-k-means) in short its knn with mean/variance centroids, a sample can be in several centroids with a certain probability.
Let us briefly talk about a probabilistic generalisation of k-means: the [Gaussian Mixture Model](https://en.wikipedia.org/wiki/Mixture_model)(GMM).
In k-means, you carry out the following procedure:
\- specify k centroids, initialising their coordinates randomly
\- calculate the distance of each data point to each centroid
\- assign each data point to its nearest centroid
\- update the coordinates of the centroid to the mean of all points assigned to it
\- iterate until convergence.
In a GMM, you carry out the following procedure:
\- specify k multivariate Gaussians (termed components), initialising their mean and variance randomly
\- calculate the probability of each data point being produced by each component (sometimes termed the responsibility each component takes for the data point)
\- assign each data point to the component it belongs to with the highest probability
\- update the mean and variance of the component to the mean and variance of all data points assigned to it
\- iterate until convergence
You may notice the similarity between these two procedures. In fact, k-means is a GMM with fixed-variance components. Under a GMM, the probabilities (I think) you're looking for are the responsibilities each component takes for each data point.
1. [Gmm code on sklearn](https://scikit-learn.org/stable/auto_examples/mixture/plot_gmm.html#sphx-glr-auto-examples-mixture-plot-gmm-py) using ellipsoids
2. [How to select the K using bic](https://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_selection.html#sphx-glr-auto-examples-mixture-plot-gmm-selection-py)
3. [Density estimation for gmm - nice graph](https://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_pdf.html#sphx-glr-auto-examples-mixture-plot-gmm-pdf-py)
### KMEANS++ / Kernel Kmeans
1. [A comparison of kmeans++ vs kernel kmeans](https://sandipanweb.wordpress.com/2016/08/29/kernel-k-means-and-cluster-evaluation/)
2. [Kernel Kmeans is part of TSLearn ](http://tslearn.readthedocs.io/en/latest/gen_modules/clustering/tslearn.clustering.GlobalAlignmentKernelKMeans.html)
3. [Elbow method](https://blog.cambridgespark.com/how-to-determine-the-optimal-number-of-clusters-for-k-means-clustering-14f27070048f),
4. [elbow and mean silhouette](https://www.datanovia.com/en/lessons/determining-the-optimal-number-of-clusters-3-must-know-methods/#elbow-method),
5. [elbow on medium using mean distance per cluster from the center](https://towardsdatascience.com/what-is-k-ddf36926a752)
6. [Kneed a library to find the knee in a curve](https://github.com/arvkevi/kneed)
1. [how to?](https://stackoverflow.com/questions/47623915/how-to-detect-in-real-time-a-knee-elbow-maximal-curvature-in-a-curve)
### KNN
1. [Nearpy](https://github.com/pixelogik/NearPy), knn in scale! On github
2. [finding the optimal K](https://towardsdatascience.com/how-to-find-the-optimal-value-of-k-in-knn-35d936e554eb)
3. [Benchmark of nearest neighbours libraries](https://github.com/erikbern/ann-benchmarks/)
4. [billion scale aprox nearest neighbour search](https://big-ann-benchmarks.com/)
### DBSCAN
1. [How to use effectively](https://towardsdatascience.com/how-to-use-dbscan-effectively-ed212c02e62)
2. [a DBSCAN visualization - very good!](https://www.naftaliharris.com/blog/visualizing-dbscan-clustering/)
3. [DBSCAN for GPS.](https://geoffboeing.com/2014/08/clustering-to-reduce-spatial-data-set-size/)
4. [A practical guide to dbscan - pretty good](https://towardsdatascience.com/a-practical-guide-to-dbscan-method-d4ec5ab2bc99)
5. [Custom DBSCAN “predict”](https://stackoverflow.com/questions/27822752/scikit-learn-predicting-new-points-with-dbscan)
6. [Haversine distances for](https://kanoki.org/2019/12/27/how-to-calculate-distance-in-python-and-pandas-using-scipy-spatial-and-distance-functions/) dbscan
7. Optimized dbscans:
1. [muDBSCAN](https://githubmemory.com/repo/AdityaAS/MuDBSCAN), [paper](https://adityaas.github.io/) - A fast, exact, and scalable algorithm for DBSCAN clustering. This repository contains the implementation for the distributed spatial clustering algorithm proposed in the paper μDBSCAN: An Exact Scalable DBSCAN Algorithm for Big Data Exploiting Spatial Locality
2. [Dbscan multiplex](https://github.com/GGiecold/DBSCAN_multiplex) - A fast and memory-efficient implementation of DBSCAN (Density-Based Spatial Clustering of Applications with Noise).
3. [Fast dbscan](https://github.com/harmslab/fast_dbscan) - A lightweight, fast dbscan implementation for use on peptide strings. It uses pure C for the distance calculations and clustering. This code is then wrapped in python.
4. [Faster dbscan paper](https://arxiv.org/pdf/1702.08607.pdf)
### ST-DBSCAN
1. [Paper - st-dbscan an algo for clustering spatio temporal data](https://www.sciencedirect.com/science/article/pii/S0169023X06000218)
2. [Popular git](https://github.com/eubr-bigsea/py-st-dbscan)
3. [git](https://github.com/gitAtila/ST-DBSCAN)
### HDBSCAN\*
(what is?) HDBSCAN is a clustering algorithm developed by [Campello, Moulavi, and Sander](http://link.springer.com/chapter/10.1007%2F978-3-642-37456-2_14). It extends DBSCAN by converting it into a hierarchical clustering algorithm, and then using a technique to extract a flat clustering based in the stability of clusters.
* [Github code](https://github.com/scikit-learn-contrib/hdbscan)
* (great) [Documentation](http://hdbscan.readthedocs.io/en/latest/basic_hdbscan.html) with examples, for clustering, outlier detection, comparison, benchmarking and analysis!
* ([jupytr example](http://nbviewer.jupyter.org/github/scikit-learn-contrib/hdbscan/blob/master/notebooks/How%20HDBSCAN%20Works.ipynb)) - take a look and see how to use it, usage examples are also in the docs and github
What are the algorithm’s [steps](http://nbviewer.jupyter.org/github/scikit-learn-contrib/hdbscan/blob/master/notebooks/How%20HDBSCAN%20Works.ipynb):
1. Transform the space according to the density/sparsity.
2. Build the minimum spanning tree of the distance weighted graph.
3. Construct a cluster hierarchy of connected components.
4. Condense the cluster hierarchy based on minimum cluster size.
5. Extract the stable clusters from the condensed tree.
### OPTICS
([What is?](https://en.wikipedia.org/wiki/OPTICS_algorithm)) Ordering points to identify the clustering structure (OPTICS) is an algorithm for finding density-based[\[1\]](https://en.wikipedia.org/wiki/OPTICS_algorithm#cite_note-1) [clusters](https://en.wikipedia.org/wiki/Cluster_analysis) in spatial data
* Its basic idea is similar to [DBSCAN](https://en.wikipedia.org/wiki/DBSCAN),[\[3\]](https://en.wikipedia.org/wiki/OPTICS_algorithm#cite_note-3)
* it addresses one of DBSCAN's major weaknesses: the problem of detecting meaningful clusters in data of varying density.
* (How?) the points of the database are (linearly) ordered such that points which are spatially closest become neighbors in the ordering.
* a special distance is stored for each point that represents the density that needs to be accepted for a cluster in order to have both points belong to the same cluster. (This is represented as a [dendrogram](https://en.wikipedia.org/wiki/Dendrogram).)
### SVM CLUSTERING
[Paper](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2099486/#:~:text=An%20SVM%2Dbased%20clustering%20algorithm,until%20an%20initial%20convergence%20occurs.)
An SVM-based clustering algorithm is introduced that clusters data with no a priori knowledge of input classes.
1. The algorithm initializes by first running a binary SVM classifier against a data set with each vector in the set randomly labelled, this is repeated until an initial convergence occurs.
2. Once this initialization step is complete, the SVM confidence parameters for classification on each of the training instances can be accessed.
3. The lowest confidence data (e.g., the worst of the mislabelled data) then has its' labels switched to the other class label.
4. The SVM is then re-run on the data set (with partly re-labelled data) and is guaranteed to converge in this situation since it converged previously, and now it has fewer data points to carry with mislabelling penalties.
5. This approach appears to limit exposure to the local minima traps that can occur with other approaches. Thus, the algorithm then improves on its weakly convergent result by SVM re-training after each re-labeling on the worst of the misclassified vectors – i.e., those feature vectors with confidence factor values beyond some threshold.
6. The repetition of the above process improves the accuracy, here a measure of separability, until there are no misclassifications. Variations on this type of clustering approach are shown.
### COP-CLUSTERING
Constrained K-means algorithm, [git](https://github.com/Behrouz-Babaki/COP-Kmeans), [paper](https://web.cse.msu.edu/~cse802/notes/ConstrainedKmeans.pdf), is a semi-supervised algorithm.
Clustering is traditionally viewed as an unsupervised method for data analysis. However, in some cases information about the problem domain is available in addition to the data instances themselves. In this paper, we demonstrate how the popular k-means clustering algorithm can be profitably modified to make use of this information. In experiments with artificial constraints on six data sets, we observe improvements in clustering accuracy. We also apply this method to the real-world problem of automatically detecting road lanes from GPS data and observe dramatic increases in performance.\
In the context of partitioning algorithms, instance level constraints are a useful way to express a priori knowledge about which instances should or should not be grouped together. Consequently, we consider two types of pairwise constraints:\
• Must-link constraints specify that two instances have to be in the same cluster.\
• Cannot-link constraints specify that two instances must not be placed in the same cluster.
# Anomaly Detection
**“whether a new observation belongs to the same distribution as existing observations (it is an inlier), or should be considered as different (it is an outlier).**
**=> Often, this ability is used to clean real data sets**
**Two important distinctions must be made:**
| **novelty detection:** | |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| | **The training data is not polluted by outliers, and we are interested in detecting anomalies in new observations.** |
| **outlier detection:** | |
| |
The training data contains outliers, and we need to fit the central mode of the training data, ignoring the deviant observations
|
1. [Using IQR for AD and why IQR difference is 2.7 sigma](https://towardsdatascience.com/why-1-5-in-iqr-method-of-outlier-detection-5d07fdc82097)
2. [**Medium**](https://towardsdatascience.com/anomaly-detection-for-dummies-15f148e559c1#:~:text=K%20%2D%20Nearest%20Neighbors%20\(KNN\),algorithms%20were%20not%20very%20different.) **- good**
3. [**kdnuggets**](https://www.kdnuggets.com/2017/04/datascience-introduction-anomaly-detection.html)
4. **Index for** [**Z-score and other moving averages.** ](https://turi.com/learn/userguide/anomaly_detection/moving_zscore.html)
5. [**A survey**](https://d1wqtxts1xzle7.cloudfront.net/49916547/Mohiuddin_Survey_financial_2015.pdf?1477591055=\&response-content-disposition=inline%3B+filename%3DA_survey_of_anomaly_detection_techniques.pdf\&Expires=1594649751\&Signature=U~N32meGWYyIIQz1zRYC4s2tCb7e5ut28GIBC3GSG4250UjhgTMQwEIB63zwPKtS5JyKew7RWVog8gytIhc3GSSfTwsRM7lqyghuDgbds-QMp3mNyVw2bYNztnoOWncHG8rhtkwUK1EbWcYeLKvqARnJoAS177C8r1GAhfKp14GgJzHpmnsoSkB6AowJ68nauf2VyA1b~w1m~UfSNoWtjbL59clAqHn7nfqw5PGBuLHSSSxCa5PX09mADy4VzuOySzYjIviRwOlgT1eQrART0KqozqVSiGKM3SeapuI3K5tSERVPPSTnpupp--WJyYCNzzvPrdjB121P2XU7fq73wQ__\&Key-Pair-Id=APKAJLOHF5GGSLRBV4ZA)
6. [**A great tutorial**](https://www.analyticsvidhya.com/blog/2019/02/outlier-detection-python-pyod/?utm_source=facebook.com\&utm_medium=social\&fbclid=IwAR33KDnGMf5zp491WmhTsCFtinBDUp5RaVnoC4Cfxcc5rfo2yHreMo3M_M4) **about AD using 20 algos in a** [**single python package**](https://github.com/yzhao062/pyod)**.**
7. [**Mastery on classifying rare events using lstm-autoencoder**](https://machinelearningmastery.com/lstm-model-architecture-for-rare-event-time-series-forecasting/)
8. **A** [**comparison**](http://scikit-learn.org/stable/modules/outlier_detection.html#outlier-detection) **of One-class SVM versus Elliptic Envelope versus Isolation Forest versus LOF in sklearn. (The examples below illustrate how the performance of the** [**covariance.EllipticEnvelope**](http://scikit-learn.org/stable/modules/generated/sklearn.covariance.EllipticEnvelope.html#sklearn.covariance.EllipticEnvelope) **degrades as the data is less and less unimodal. The** [**svm.OneClassSVM**](http://scikit-learn.org/stable/modules/generated/sklearn.svm.OneClassSVM.html#sklearn.svm.OneClassSVM) **works better on data with multiple modes and** [**ensemble.IsolationForest**](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html#sklearn.ensemble.IsolationForest) **and**[**neighbors.LocalOutlierFactor**](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.LocalOutlierFactor.html#sklearn.neighbors.LocalOutlierFactor) **perform well in every cases.)**
9. [**Using Autoencoders**](https://shiring.github.io/machine_learning/2017/05/01/fraud) **- the information is there, but its all over the place.**
10. **Twitter anomaly -**
11. **Microsoft anomaly - a well documented black box, i cant find a description of the algorithm, just hints to what they sort of did**
1. [**up/down trend, dynamic range, tips and dips**](https://blogs.technet.microsoft.com/machinelearning/2014/11/05/anomaly-detection-using-machine-learning-to-detect-abnormalities-in-time-series-data/)
2. [**Api here**](https://docs.microsoft.com/en-us/azure/machine-learning/team-data-science-process/apps-anomaly-detection-api)
12. **STL and** [**LSTM for anomaly prediction**](https://github.com/omri374/moda/blob/master/moda/example/lstm/LSTM_AD.ipynb) **by microsoft**
1. [**Medium on AD**](https://towardsdatascience.com/machine-learning-for-anomaly-detection-and-condition-monitoring-d4614e7de770)
2. [**Medium on AD using mahalanobis, AE and**](https://towardsdatascience.com/how-to-use-machine-learning-for-anomaly-detection-and-condition-monitoring-6742f82900d7)
### **OUTLIER DETECTION**
1. [**Alibi Detect**](https://github.com/SeldonIO/alibi-detect) **is an open source Python library focused on outlier, adversarial and drift detection. The package aims to cover both online and offline detectors for tabular data, text, images and time series. The outlier detection methods should allow the user to identify global, contextual and collective outliers.**\
\
2. [**Pyod**](https://pyod.readthedocs.io/en/latest/pyod.html)
1. [**Anomaly detection resources**](https://github.com/yzhao062/anomaly-detection-resources) **(great)**
2. [**Novelty and outlier detection inm sklearn**](https://scikit-learn.org/stable/modules/outlier_detection.html)
3. [**SUOD**](https://github.com/yzhao062/suod) **(Scalable Unsupervised Outlier Detection) is an acceleration framework for large-scale unsupervised outlier detector training and prediction. Notably, anomaly detection is often formulated as an unsupervised problem since the ground truth is expensive to acquire. To compensate for the unstable nature of unsupervised algorithms, practitioners often build a large number of models for further combination and analysis, e.g., taking the average or majority vote. However, this poses scalability challenges in high-dimensional, large datasets, especially for proximity-base models operating in Euclidean space.**
**SUOD is therefore proposed to address the challenge at three complementary levels: random projection (data level), pseudo-supervised approximation (model level), and balanced parallel scheduling (system level). As mentioned, the key focus is to accelerate the training and prediction when a large number of anomaly detectors are presented, while preserving the prediction capacity. Since its inception in Jan 2019, SUOD has been successfully used in various academic researches and industry applications, include PyOD** [**\[2\]**](https://github.com/yzhao062/suod#zhao2019pyod) **and** [**IQVIA**](https://www.iqvia.com/) **medical claim analysis. It could be especially useful for outlier ensembles that rely on a large number of base estimators.**
1. [**Skyline**](https://github.com/earthgecko/skyline)
2. [**Scikit-lego**](https://scikit-lego.readthedocs.io/en/latest/outliers.html)
1. 
2. 
1. 
###
### **ISOLATION FOREST**
[**The best resource to explain isolation forest**](http://blog.easysol.net/using-isolation-forests-anamoly-detection/) **- the basic idea is that for an anomaly (in the example) only 4 partitions are needed, for a regular point in the middle of a distribution, you need many many more.**
[**Isolation Forest**](http://scikit-learn.org/stable/auto_examples/ensemble/plot_isolation_forest.html) **-Isolating observations:**
* **randomly selecting a feature**
* **randomly selecting a split value between the maximum and minimum values of the selected feature.**
**Recursive partitioning can be represented by a tree structure, the number of splittings required to isolate a sample is equivalent to the path length from the root node to the terminating node.**
**This path length, averaged over a forest of such random trees, is a measure of normality and our decision function.**
**Random partitioning produces noticeable shorter paths for anomalies.**
**=> when a forest of random trees collectively produce shorter path lengths for particular samples, they are highly likely to be anomalies.**
[**the paper is pretty good too -**](https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/icdm08b.pdf) **In the training stage, iTrees are constructed by recursively partitioning the given training set until instances are isolated or a specific tree height is reached of which results a partial model.**
**Note that the tree height limit l is automatically set by the sub-sampling size ψ: l = ceiling(log2 ψ), which is approximately the average tree height \[7].**
**The rationale of growing trees up to the average tree height is that we are only interested in data points that have shorter-than average path lengths, as those points are more likely to be anomalies**\
### **LOCAL OUTLIER FACTOR**
* [**LOF**](http://scikit-learn.org/stable/modules/outlier_detection.html#local-outlier-factor) **computes a score (called local outlier factor) reflecting the degree of abnormality of the observations.**
* **It measures the local density deviation of a given data point with respect to its neighbors. The idea is to detect the samples that have a substantially lower density than their neighbors.**
* **In practice the local density is obtained from the k-nearest neighbors.**
* **The LOF score of an observation is equal to the ratio of the average local density of his k-nearest neighbors, and its own local density:**
* **a normal instance is expected to have a local density similar to that of its neighbors,**
* **while abnormal data are expected to have much smaller local density.**
### **ELLIPTIC ENVELOPE**
1. **We assume that the regular data come from a known distribution (e.g. data are Gaussian distributed).**
2. **From this assumption, we generally try to define the “shape” of the data,**
3. **And can define outlying observations as observations which stand far enough from the fit shape.**
### **ONE CLASS SVM**
1. [**A nice article about ocs, with github code, two methods are described.**](http://rvlasveld.github.io/blog/2013/07/12/introduction-to-one-class-support-vector-machines/)
2. [**Resources for ocsvm**](https://www.quora.com/What-is-a-good-resource-for-understanding-One-Class-SVM-for-distribution-esitmation)
3. **It looks like there are** [**two such methods**](http://rvlasveld.github.io/blog/2013/07/12/introduction-to-one-class-support-vector-machines/)**, - The 2nd one: The algorithm obtains a spherical boundary, in feature space, around the data. The volume of this hypersphere is minimized, to minimize the effect of incorporating outliers in the solution.**
**The resulting hypersphere is characterized by a center and a radius R>0 as distance from the center to (any support vector on) the boundary, of which the volume R2 will be minimized.**\
### **CLUSTERING METRICS**
**For community detection, text clusters, etc.**
[**Google search for convenience**](https://www.google.com/search?biw=1600\&bih=912\&sxsrf=ALeKk00NbB52pfM6J1N42ieEddIOirBmcQ%3A1597514997743\&ei=9SQ4X-_uLLLhkgWJsIbADw\&q=word+embedding+silhouette+score\&oq=word+embedding+silhouette+score\&gs_lcp=CgZwc3ktYWIQAzoECAAQRzoECCMQJzoHCCMQsAIQJ1DVd1jqjQFgm5ABaARwAXgBgAGMAogBrQ2SAQUwLjkuMpgBAKABAaoBB2d3cy13aXrAAQE\&sclient=psy-ab\&ved=0ahUKEwivvdqP553rAhWysKQKHQmYAfg4ChDh1QMIDA\&uact=5)
**Silhouette:**
1. [**TFIDF, PCA, SILHOUETTE**](https://towardsdatascience.com/mmmm-foodporn-a-clustering-and-classification-study-using-natural-language-processing-e2eae8ddefe1) **for deciding how many clusters to use, the knee/elbow method.**
2. [**Embedding based silhouette community detection**](https://link.springer.com/article/10.1007/s10994-020-05882-8#Sec10)
3. [**A notebook**](https://rlbarter.github.io/superheat-examples/word2vec/)**, using the SuperHeat package, clustering w2v cosine similarity matrix, measuring using silhouette score.**
4. [**Topic modelling clustering, cant access this document on github**](https://github.com/danielwilentz/Cuisine-Classifier/blob/master/topic_modeling/clustering.ipynb)
# Decision Trees
* [**Using hellinger distance to split supervised datasets, instead of gini and entropy. Claims better results.**](https://medium.com/@evgeni.dubov/classifying-imbalanced-data-using-hellinger-distance-f6a4330d6f9a)
* **Visualize decision** [**trees**](https://towardsdatascience.com/interactive-visualization-of-decision-trees-with-jupyter-widgets-ca15dd312084)**,** [**forests**](https://towardsdatascience.com/how-to-visualize-a-decision-tree-from-a-random-forest-in-python-using-scikit-learn-38ad2d75f21c)
### [**CART TREES**](http://machinelearningmastery.com/classification-and-regression-trees-for-machine-learning/)
**explains about the similarities and how to measure. which is the best split? based on SSE and GINI (good info about gini here).**
* **For classification the Gini cost function is used which provides an indication of how “pure” the leaf nodes are (how mixed the training data assigned to each node is).**
**Gini = sum(pk \* (1 – pk))**
* **Early stop - 1 sample per node is overfitting, 5-10 are good**
* **Pruning - evaluate what happens if the lead nodes are removed, if there is a big drop, we need it.**
### **KDTREE**
1. [**Similar to a binary search tree, just by using the median and selecting a feature randomly for each level.**](https://www.youtube.com/watch?v=TLxWtXEbtFE)
2. [**Used to find nearest neighbours.**](https://www.youtube.com/watch?v=Y4ZgLlDfKDg)
3. [**Many applications of using KD tree, reduce color space, Database key search, etc**](https://www.quora.com/What-is-a-kd-tree-and-what-is-it-used-for)
### **RANDOM FOREST**
[**Using an ensemble of trees to create a high dimensional and sparse representation of the data and classifying using a linear classifier**](http://scikit-learn.org/stable/auto_examples/ensemble/plot_feature_transformation.html#sphx-glr-auto-examples-ensemble-plot-feature-transformation-py)
[**How do deal with imbalanced data in Random-forest**](http://statistics.berkeley.edu/sites/default/files/tech-reports/666.pdf) **-**
1. **One is based on cost sensitive learning.**
2. **Other is based on a sampling technique**
### **EXTRA TREES**
1. [**A comparison between random forest and extra trees** ](https://www.thekerneltrip.com/statistics/random-forest-vs-extra-tree/)**Fig. 1: Comparison of random forests and extra trees in presence of irrelevant predictors. In blue are presented the results from the random forest and red for the extra trees. The results are quite striking: Extra Trees perform consistently better when there are a few relevant predictors and many noisy ones**
2. [**Difference between RF and ET**](https://stats.stackexchange.com/questions/175523/difference-between-random-forest-and-extremely-randomized-trees)
3. [**Differences #2**](https://stackoverflow.com/questions/22409855/randomforestclassifier-vs-extratreesclassifier-in-scikit-learn)
# Active Learning Algorithms
### **PASSIVE AGGRESSIVE CLASSIFIER**
1. [**The Passive Aggressive**](https://www.quora.com/Classification-machine-learning-What-is-an-intuitive-explanation-of-the-Passive-Aggressive-classifier) **(PA) algorithm is perfect for classifying massive streams of data (e.g. Twitter). It's easy to implement and very fast, but does not provide global guarantees like the support-vector machine (SVM).**
2. [**Youtube, seems like active learning in stream..?**](https://www.youtube.com/watch?v=TJU8NfDdqNQ)
# Linear Separator Algorithms
### **SEQUENTIAL MINIMAL OPTIMIZATION (SMO)**
[**What is the SMO (SVM) classifier?**](https://www.microsoft.com/en-us/research/publication/sequential-minimal-optimization-a-fast-algorithm-for-training-support-vector-machines/?from=http%3A%2F%2Fresearch.microsoft.com%2Fpubs%2F69644%2Ftr-98-14.pdf) **- Sequential Minimal Optimization, or SMO. Training a support vector machine requires the solution of a very large quadratic programming (QP) optimization problem. SMO breaks this large QP problem into a series of the smallest possible QP problems. These small QP problems are solved analytically, which avoids using a time-consuming numerical QP optimization as an inner loop. The amount of memory required for SMO is linear in the training set size, which allows SMO to handle very large training sets. Because matrix computation is avoided, SMO scales somewhere between linear and quadratic in the training set size for various test problems, while the standard chunking SVM algorithm scales somewhere between linear and cubic in the training set size. SMO’s computation time is dominated by SVM evaluation, hence SMO is fastest for linear SVMs and sparse data sets. On real-world sparse data sets, SMO can be more than 1000 times faster than the chunking algorithm.**
[**Differences between libsvm and liblinear**](https://stackoverflow.com/questions/11508788/whats-the-difference-between-libsvm-and-liblinear) **&** [**smo vs libsvm**](https://stackoverflow.com/questions/23674411/weka-smo-vs-libsvm)
### **SUPPORT VECTOR MACHINES (SVM)**
**-** [**Definition**](http://docs.opencv.org/3.0-beta/modules/ml/doc/support_vector_machines.html)**,** [**tutorial**](https://jakevdp.github.io/PythonDataScienceHandbook/05.07-support-vector-machines.html)**\*\*\*:**
* **For Optimal 2-class classifier.**
* **Extended for regression and clustering problems (1 class).**
* **Kernel-based**
* **maps feature vectors into a higher-dimensional space using a kernel function**
* **builds an optimal linear discriminating function in this space (linear?) or an optimal hyper-plane (RBF?) that fits the training data**
* **In case of SVM, the kernel is not defined explicitly.**
* **A distance needs to be defined between any 2 points in the hyper-space.**
* **The solution is optimal, the margin is maximal. between the separating hyper-plane and the nearest feature vectors**
* **The feature vectors that are the closest to the hyper-plane are called support vectors, which means that the position of other vectors does not affect the hyper-plane (the decision function).**
* **The model produced by support vector classification (as described above) depends only on a subset of the training data, because the cost function for building the model does not care about training points that lie beyond the margin.**
### [**MULTI CLASS SVM**](https://www.csie.ntu.edu.tw/~cjlin/papers/multisvm.pdf)
[ ](https://www.csie.ntu.edu.tw/~cjlin/papers/multisvm.pdf)**- one against all, one against one, and Direct Acyclic Graph SVM (one against one with DAG). bottom line One Against One in LIBSVM.**
[**A few good explanation about SVM, formulas, figures, C, gamma, etc.**](https://www.quora.com/What-are-C-and-gamma-with-regards-to-a-support-vector-machine)
**Math of SVM on youtube:**
* [**very good number-based example #1**](https://www.youtube.com/watch?v=1NxnPkZM9bc)
* [**Very good but lengthy and chatty example with make-sense math #2**](https://www.youtube.com/watch?v=mU_N3nmv0Go\&list=PLAwxTw4SYaPlkESDcHD-0oqVx5sAIgz7O\&index=4) **- udacity**
* **Linear - maximize the margin, optimal solution, only a few close points are really needed the others are zeroes by the alphas (alpha says “pay attention to this variable”) in the quadratic programming equation. XtX is a similarity function (pairs of points that relate to each other in output labels and how similar to one another, Xi’s point in the same direction) y1y2 are the labels. Therefore further points are not needed. But the similarity is important here(?)**
* **Non-linear - e.g. circle inside a circle, needs to map to a higher plane, a measure of similarity as XtX is important. We use this similarity idea to map into a higher plane, but we choose the higher plane for the purpose of a final function that behaves likes a known function, such as (A+B)^2. It turns out that (q1,q2,root(2)q1q2) is engineered with that root(2) thing for the purpose of making the multiplication of X^tY, which turns out to be (X^tY)^2. We can substitute this formula (X^tY)^2 instead of the X^tX in the quadratic equation to do that for us.This is the kernel trick that maps the inner class to one side and the outer circle class to the other and passes a plane in between them.**
* **Similarity is defined intuitively as all the points in one class vs the other.. I think**
* **A general kernel K=(X^tY + C)^p is a polynomial kernel that can define the above function and others.**
* **Quadratic eq with possible kernels including the polynomial.**
* **Most importantly the kernel function is our domain knowledge. (?) IMO we should choose a kernel that fits our feature data.**
* **The output of K is a number(?)**
* **Infinite dimensions - possible as well.**
* **Mercer condition - it acts like a distance\similar so that is the “rule” of which a kernel needs to follow.**
* [**Super good lecture on MIT OPEN COURSE WARE**](https://www.youtube.com/watch?v=_PwhiWxHK8o) **- expands on the quadratic equations that were introduced in the previous course above.**
### **Regularization and influence**
**- (basically punishment for overfitting and raising the non- linear class points higher and lower)**
* [**How does regularization look like in SVM**](https://datascience.stackexchange.com/questions/4943/intuition-for-the-regularization-parameter-in-svm) **- controlling ‘C’**
* [**The best explanation about Gamma (and C) in SVM!**](https://www.quora.com/What-are-C-and-gamma-with-regards-to-a-support-vector-machine)
### **SUPPORT VECTOR REGRESSION (SVR)**
**-** [**Definition Support Vector Regression**](http://scikit-learn.org/stable/modules/svm.html#svm-implementation-details)**.:**
* **The method of SVM can be extended to solve regression problems.**
* **Similar to SVM, the model produced by Support Vector Regression depends only on a subset of the training data, because the cost function for building the model ignores any training data close to the model prediction.**
### [**LibSVM vs LibLinear**](https://stackoverflow.com/questions/11508788/whats-the-difference-between-libsvm-and-liblinear)
**- using many kernel transforms to turn a non-linear problem into a linear problem beforehand.**
**From the link above, it seems like liblinear is very much the same thing, without those kernel transforms. So, as they say, in cases where the kernel transforms are not needed (they mention document classification), it will be faster.**
* **libsvm (SMO) implementation**
* **kernel (n^2)**
* **Linear SVM (n^3)**
* **liblinear - optimized to deal with linear classification without kernels**
* **Complexity O(n)**
* **does not support kernel SVMs.**
* **Scores higher**
**n is the number of samples in the training dataset.**
**Conclusion: In practice libsvm becomes painfully slow at 10k samples. Hence for medium to large scale datasets use liblinear and forget about libsvm (or maybe have a look at approximate kernel SVM solvers such as** [**LaSVM**](http://leon.bottou.org/projects/lasvm)**, which saves training time and memory usage for large scale datasets).**
### **Support vector clustering (SVC)**
[**paper**](http://www.jmlr.org/papers/volume2/horn01a/horn01a.pdf)**,** [**short explanation**](https://www.quora.com/Is-it-possible-to-use-SVMs-for-unsupervised-learning-density-estimation)
### **KERNELS**
[**What are kernels in SVM**](https://www.quora.com/What-are-Kernels-in-Machine-Learning-and-SVM) **- intuition and example**
* **allows us to do certain calculations faster which otherwise would involve computations in higher dimensional space.**
* **K(x, y) = \. Here K is the kernel function, x, y are n dimensional inputs. f is a map from n-dimension to m-dimension space. < x,y> denotes the dot product. usually m is much larger than n.**
* **normally calculating \ requires us to calculate f(x), f(y) first, and then do the dot product. These two computation steps can be quite expensive as they involve manipulations in m dimensional space, where m can be a large number.**
* **Result is ONLY a scalar, i..e., 1-dim space.**
* **We don’t need to do that calc if we use a clever kernel.**
**Example:**
**Simple Example: x = (x1, x2, x3); y = (y1, y2, y3). Then for the function f(x) = (x1x1, x1x2, x1x3, x2x1, x2x2, x2x3, x3x1, x3x2, x3x3), the kernel is K(x, y ) = (\)^2.**
**Let's plug in some numbers to make this more intuitive: suppose x = (1, 2, 3); y = (4, 5, 6). Then:**
**f(x) = (1, 2, 3, 2, 4, 6, 3, 6, 9) and f(y) = (16, 20, 24, 20, 25, 30, 24, 30, 36)**
**\ = 16 + 40 + 72 + 40 + 100+ 180 + 72 + 180 + 324 = 1024 i.e., 1\*16+2\*20+\*3\*24..**
**A lot of algebra. Mainly because f is a mapping from 3-dimensional to 9 dimensional space.**
**With a kernel its faster.**
**K(x, y) = (4 + 10 + 18 ) ^2 = 32^2 = 1024**
**A kernel is a magical shortcut to calculate even infinite dimensions!**
[**Relation to SVM**](https://www.quora.com/What-are-Kernels-in-Machine-Learning-and-SVM)**?:**
* **The idea of SVM is that y = w phi(x) +b, where w is the weight, phi is the feature vector, and b is the bias.**
* **if y> 0, then we classify datum to class 1, else to class 0.**
* **We want to find a set of weight and bias such that the margin is maximized.**
* **Previous answers mention that kernel makes data linearly separable for SVM. I think a more precise way to put this is, kernels do not make the the data linearly separable.**
* **The feature vector phi(x) makes the data linearly separable. Kernel is to make the calculation process faster and easier, especially when the feature vector phi is of very high dimension (for example, x1, x2, x3, ..., x\_D^n, x1^2, x2^2, ...., x\_D^2).**
* **Why it can also be understood as a measure of similarity: if we put the definition of kernel above, \, in the context of SVM and feature vectors, it becomes \. The inner product means the projection of phi(x) onto phi(y). or colloquially, how much overlap do x and y have in their feature space. In other words, how similar they are.**
[**Kernels**](http://docs.opencv.org/3.0-beta/modules/ml/doc/support_vector_machines.html)**:**
* **SVM::LINEAR Linear kernel. No mapping is done, linear discrimination (or regression) is done in the original feature space. It is the fastest option.** **.**
* **SVM::RBF Radial basis function (RBF), a good choice in most cases.** **.**
### [**Intuition for regularization in SVM**](https://datascience.stackexchange.com/questions/4943/intuition-for-the-regularization-parameter-in-svm)
[**Grid search for SVM Hyper parameters**](http://docs.opencv.org/3.0-beta/modules/ml/doc/support_vector_machines.html) **- in openCV.** [**Example in log space**](https://stackoverflow.com/questions/29128074/choosing-the-best-svm-kernel-type-and-parameters-using-opencv-on-python)
* [**I.e., (for example**](https://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf)**, C = 2^-5 , 2 ^-3 , . . . , 2^15 , γ = 2^-15 , 2 ^-13 , . . . , 2^3 ).**
* **There are heuristic methods that skip some search options**
* **However, no need for heuristics, computation-time is small, grid can be paralleled and we dont skip parameters.**
* **Controlling search complexity using two tier grid, coarse grid and then fine tune.**
### [**Overfitting advice for SVM:** ](https://stats.stackexchange.com/questions/35276/svm-overfitting-curse-of-dimensionality)
* **Regularization parameter C -** [**penalty example**](http://scikit-learn.org/stable/auto_examples/svm/plot_rbf_parameters.html)
* **In non linear kernels:**
* **Kernel choice**
* **Kernel parameters**
* [**RBF**](http://scikit-learn.org/stable/auto_examples/svm/plot_rbf_parameters.html) **- gamma, low and high values are far and near influence**
* **Great** [**Tutorial at LIBSVM**](https://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf)
* **Reasonable first choice**
* **when the relation between class labels and attributes is nonlinear.**
* **Special case of C can make this similar to linear kernel (only! After finding C and gamma)**
* **Certain parameters makes it behave like the sigmoid kernel.**
* **Less hyperparameters than RBF kernel.**
* **0 \> number of features. Usually high dimensional mapping using non linear kernel. If we insist on liblinear, -s 2 leads to faster training.**
[**Kdnuggets: When to use DL over SVM and other algorithms. Computationally expensive for a very small boost in accuracy.**](http://www.kdnuggets.com/2016/04/deep-learning-vs-svm-random-forest.html)
# Regression
## Metrics:
1. [R2](https://en.wikipedia.org/wiki/Coefficient_of_determination)
2. Medium [1](https://towardsdatascience.com/regression-an-explanation-of-regression-metrics-and-what-can-go-wrong-a39a9793d914), [2](https://medium.com/@george.drakos62/how-to-select-the-right-evaluation-metric-for-machine-learning-models-part-1-regrression-metrics-3606e25beae0), [3](https://medium.com/@george.drakos62/how-to-select-the-right-evaluation-metric-for-machine-learning-models-part-2-regression-metrics-d4a1a9ba3d74), [4](https://medium.com/usf-msds/choosing-the-right-metric-for-machine-learning-models-part-1-a99d7d7414e4),
3. [Tutorial](https://www.dataquest.io/blog/understanding-regression-error-metrics/)
###
# Ensembles
1. (good) [review on voting, bagging, boosting stacking, cascading methodologies](https://www.toptal.com/machine-learning/ensemble-methods-kaggle-machine-learn)
2. [How to combine several sklearn algorithms into a voting ensemble](https://www.youtube.com/watch?v=vlTQLb_a564\&list=PLQVvvaa0QuDf2JswnfiGkliBInZnIC4HL\&index=16)
3. [Stacking api, MLXTEND](http://rasbt.github.io/mlxtend/user_guide/classifier/StackingClassifier/)
4. Machine learning Mastery on
1. [stacking neural nets - really good](https://machinelearningmastery.com/stacking-ensemble-for-deep-learning-neural-networks/)
1. Stacked Generalization Ensemble
2. Multi-Class Classification Problem
3. Multilayer Perceptron Model
4. Train and Save Sub-Models
5. Separate Stacking Model
6. Integrated Stacking Model
2. [How to Combine Predictions for Ensemble Learning](https://machinelearningmastery.com/combine-predictions-for-ensemble-learning/?fbclid=IwAR3sEAjoqP1KNScXrKV1HdiG98PZC-_gfB7ngDEwL_NMMSngRNqcwxABejQ)
1. Plurality Voting.
2. Majority Voting.
3. Unanimous Voting.
4. Weighted Voting.
3. [Essence of Stacking Ensembles for Machine Learning](https://machinelearningmastery.com/essence-of-stacking-ensembles-for-machine-learning/?fbclid=IwAR18Tm_CzyxufVpFjjd-n_VvpFNZRj3TuMBNd02EXmNhYWdG80KVyBjzmfo)
1. Voting Ensembles
2. Weighted Average
3. Blending Ensemble
4. Super Learner Ensemble
4. [Dynamic Ensemble Selection (DES) for Classification in Python](https://machinelearningmastery.com/dynamic-ensemble-selection-in-python/?fbclid=IwAR2cFTJY3bXiCkPKFIGM7X5HDsTjZEehINfA40wyqPWw8KAIOpXCdblu3eM) - Dynamic Ensemble Selection algorithms operate much like DCS algorithms, except predictions are made using votes from multiple classifier models instead of a single best model. In effect, each region of the input feature space is owned by a subset of models that perform best in that region.
1. k-Nearest Neighbor Oracle (KNORA) With Scikit-Learn
1. KNORA-Eliminate (KNORA-E)
2. KNORA-Union (KNORA-U)
2. Hyperparameter Tuning for KNORA
1. Explore k in k-Nearest Neighbor
2. Explore Algorithms for Classifier Pool
5. [A Gentle Introduction to Mixture of Experts Ensembles](https://machinelearningmastery.com/mixture-of-experts/?fbclid=IwAR3Y9K-QOmF6H06vZOYQH8phv5C0a2rhV-4FfNffCb2XKmvDsL-d8bMOuLM)
1. Mixture of Experts
1. Subtasks
2. Expert Models
3. Gating Model
4. Pooling Method
2. Relationship With Other Techniques
1. Mixture of Experts and Decision Trees
2. Mixture of Experts and Stacking
6. [Strong Learners vs. Weak Learners in Ensemble Learning](https://machinelearningmastery.com/strong-learners-vs-weak-learners-for-ensemble-learning/?fbclid=IwAR0yQzfYq0JGZu7xYErX2W42jtm949pOYSbKN8jClQCPMwgEUDNyv6uuXFU) - Weak learners are models that perform slightly better than random guessing. Strong learners are models that have arbitrarily good accuracy.\
Weak and strong learners are tools from computational learning theory and provide the basis for the development of the boosting class of ensemble methods.
5. [Vidhya on trees, bagging boosting, gbm, xgb](https://www.analyticsvidhya.com/blog/2016/04/complete-tutorial-tree-based-modeling-scratch-in-python/?utm_source=facebook.com\&utm_medium=social\&fbclid=IwAR1Fji6N01Zc3rhLCJiIq76CX5aC8W0dWmw0hpyceYwMr9Z3QPCbnPu0a2A#three)
6. [Parallel grad boost treest](http://zhanpengfang.github.io/418home.html)
7. [A comprehensive guide to ensembles read!](https://www.analyticsvidhya.com/blog/2018/06/comprehensive-guide-for-ensemble-models/) (samuel jefroykin)
1. Basic Ensemble Techniques
2. 2.1 Max Voting
3. 2.2 Averaging
4. 2.3 Weighted Average
5. Advanced Ensemble Techniques
6. 3.1 Stacking
7. 3.2 Blending
8. 3.3 Bagging
9. 3.4 Boosting
10. Algorithms based on Bagging and Boosting
11. 4.1 Bagging meta-estimator
12. 4.2 Random Forest
13. 4.3 AdaBoost
14. 4.4 GBM
15. 4.5 XGB
16. 4.6 Light GBM
17. 4.7 CatBoost
8. [Kaggler guide to stacking](http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/)
9. [Blending vs stacking](https://www.quora.com/What-are-examples-of-blending-and-stacking-in-Machine-Learning)
10. [Kaggle ensemble guide](https://mlwave.com/kaggle-ensembling-guide/)
### [Ensembles in WEKA](http://machinelearningmastery.com/use-ensemble-machine-learning-algorithms-weka/)
\- bagging (random sample selection, multi classifier training), random forest (random feature selection for each tree, multi tree training), boosting(creating stumps, each new stump tries to fix the previous error, at last combining results using new data, each model is assigned a skill weight and accounted for in the end), voting(majority vote, any set of algorithms within weka, results combined via mean or some other way), stacking(same as voting but combining predictions using a meta model is used).
### BAGGING - bootstrap aggregating
[Bagging](https://www.youtube.com/watch?v=2Mg8QD0F1dQ\&list=PLAwxTw4SYaPnIRwl6rad_mYwEk4Gmj7Mx\&index=192) - best example so far, create m bags, put n’\
**Adaboost: similar to bagging, create a system that chooses from samples that were modelled poorly before.**
1. **create bag\_1 with n’ features \
[**Special case of random forest using XGBOOST**](https://github.com/dmlc/xgboost/blob/master/R-package/vignettes/discoverYourData.Rmd#special-note-what-about-random-forests)**:**
**#Random Forest™ - 1000 trees**\
**bst <- xgboost(data = train$data, label = train$label, max\_depth = 4, num\_parallel\_tree = 1000, subsample = 0.5, colsample\_bytree =0.5, nrounds = 1, objective = "binary:logistic")**\
\
**#Boosting - 3 rounds**\
**bst <- xgboost(data = train$data, label = train$label, max\_depth = 4, nrounds = 3, objective = "binary:logistic")**
**RF1000: - max\_depth = 4, num\_parallel\_tree = 1000, subsample = 0.5, colsample\_bytree =0.5, nrounds = 1, nthread = 2**
**XG: nrounds = 10, max\_depth = 4, eta = 0.5, nthread = 2**
### **Gradient Boosting Classifier**
1. [**Loss functions and GBC vs XGB**](https://stats.stackexchange.com/questions/202858/loss-function-approximation-with-taylor-expansion)
2. [**Why is XGB faster than SK GBC** ](https://datascience.stackexchange.com/questions/10943/why-is-xgboost-so-much-faster-than-sklearn-gradientboostingclassifier)
3. [**Good XGB vs GBC**](https://towardsdatascience.com/boosting-algorithm-xgboost-4d9ec0207d) **tutorial**
4. [**XGB vs GBC**](https://stats.stackexchange.com/questions/282459/xgboost-vs-python-sklearn-gradient-boosted-trees)
## CatBoost
1. (great) [what is so special?](https://hanishrohit.medium.com/whats-so-special-about-catboost-335d64d754ae)
2. [the fastest algo](https://medium.com/almabetter/catboost-the-fastest-algorithm-c21d44f8b990)
3. [a new game in ML](https://affine.medium.com/catboost-a-new-game-of-machine-learning-72a7dcea0ac4)
4. [use it here is why](https://towardsdatascience.com/you-should-use-catboost-heres-why-72f124dcdad7)
# Reinforcement Learning
## Introduction
1. [Reinforcement Learning: An Introduction second edition WIP](https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf) & completed [book](http://incompleteideas.net/book/RLbook2020.pdf)
2. Vidhya on [Getting ready for AI based gaming agents – Overview of Open Source Reinforcement Learning Platforms](https://www.analyticsvidhya.com/blog/2016/12/getting-ready-for-ai-based-gaming-agents-overview-of-open-source-reinforcement-learning-platforms/)
3. Vidhya on [Simple Beginner’s guide to Reinforcement Learning & its implementation](https://www.analyticsvidhya.com/blog/2017/01/introduction-to-reinforcement-learning-implementation/)
4. ZipRecruiter on [Classifying Job Titles With Noisy Labels Using REINFORCE ](https://medium.com/@ziprecruiter.engineering/classifying-job-titles-with-noisy-labels-using-reinforce-ce1a4bde05e2)- Fine-grained job title classification with noisy labels using the REINFORCE algorithm and multi-task learning
-> this article has a very nice trick in adding a reward component to the loss function in order to mitigate for unbalanced class label problem, instead of the usual balancing.
5. David Silver - [Home Page](https://www.davidsilver.uk/teaching/) - [1](https://www.youtube.com/watch?v=2pWv7GOvuf0) [2](https://www.youtube.com/watch?v=lfHX2hHRMVQ) [3](https://www.youtube.com/watch?v=Nd1-UUMVfz4) [4](https://www.youtube.com/watch?v=PnHCvfgC_ZA) [5](https://www.youtube.com/watch?v=0g4j2k_Ggc4) [6](https://www.youtube.com/watch?v=UoPei5o4fps) [7](https://www.youtube.com/watch?v=KHZVXao4qXs) [8](https://www.youtube.com/watch?v=ItMutbeOHtc) [9](https://www.youtube.com/watch?v=sGuiWX07sKw) [10](https://www.youtube.com/watch?v=kZ_AUmFcZtk)\
6. [Sequential Decision Analytics and Modeling](https://castle.princeton.edu/sdamodeling/) book
### **Q-LEARN**
* **Markov chain problem, (state, action, new state, reward)**
* **Lots of Exploration in the beginning, then exploitation**
* **Returns optimal policy.**
* **Refer to youtube** [**here**](https://www.youtube.com/watch?v=9m_6q_KECTk)
### **Deep Learning**
1. [A review paper about RL in DL](https://arxiv.org/pdf/1701.07274.pdf)
2. [deep Q-learning](https://www.analyticsvidhya.com/blog/2019/04/introduction-deep-q-learning-python/)
3. Pytorch
1. [DQN](https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html)
2. [PPO](https://pytorch.org/tutorials/intermediate/reinforcement_ppo.html)
3. [Mario example](https://pytorch.org/tutorials/intermediate/mario_rl_tutorial.html)
## RLHF
1. [illustrated RLHF by Huggingface](https://huggingface.co/blog/rlhf)
# Incremental Learning
**(wiki) In computer science, incremental learning is a method of machine learning in which input data is continuously used to extend the existing model's knowledge i.e. to further train the model.**\
*
* **Using incremental learning for time series, i.e., sent**[**iment from twitter api streat, learning from smiley in weka.**](https://www.youtube.com/watch?v=jScSxkuSei8\&list=PLm4W7_iX_v4Msh-7lDOpSFWHRYU_6H5Kx\&index=12)
* **And** [**time series in weka**](https://www.youtube.com/watch?v=9R0mz_gfhBs\&list=PLm4W7_iX_v4Msh-7lDOpSFWHRYU_6H5Kx\&index=3)**, with some ideas about dealing with time series data, especially date and time.**
### **HOEFFDING TREE**
* **IS STATE OF THE ART**
# Dimensionality Reduction Methods
* **A series on DR for dummies on medium part** [**1**](https://towardsdatascience.com/https-medium-com-abdullatif-h-dimensionality-reduction-for-dummies-part-1-a8c9ec7b7e79?_branch_match_id=584170448791192656) [**2**](https://towardsdatascience.com/dimensionality-reduction-for-dummies-part-2-3b1e3490bdc9) [**3**](https://towardsdatascience.com/dimensionality-reduction-for-dummies-part-3-f25729f74c0a)
* [**A small blog post about PCA, AE & TSNE**](https://towardsdatascience.com/reducing-dimensionality-from-dimensionality-reduction-techniques-f658aec24dfe) **in tensorflow**
* [**Visualizing PCA/TSNE using plots**](https://towardsdatascience.com/visualising-high-dimensional-datasets-using-pca-and-t-sne-in-python-8ef87e7915b)
* [**Parallex by uber for tsne \ pca visualization**](https://github.com/uber-research/parallax)
* [**About tsne / ae / pca**](https://towardsdatascience.com/reducing-dimensionality-from-dimensionality-reduction-techniques-f658aec24dfe)
* [**Does dim-reduction loses information - yes and no, in pca yes only if you use less than the entire matrix**](https://stats.stackexchange.com/questions/66060/does-dimension-reduction-always-lose-some-information)
* [**Performance comparison between dim-reduction implementations, tsne etc.**](https://umap-learn.readthedocs.io/en/latest/benchmarking.html)
###
### **TSNE**
1. [**Stat quest**](https://www.youtube.com/watch?v=NEaUSP4YerM\&list=PLblh5JKOoLUICTaGLRoHQDuF_7q2GfuJF\&index=30) **- the jist of it is that we assume a t- distribution on distances and remove those that are farther.normalized for density. T-dist used so that clusters are not clamped in the middle.**
**Iteratively moving from the left to the right**
1. [**TSNE algorithm**](https://www.oreilly.com/learning/an-illustrated-introduction-to-the-t-sne-algorithm)
2. [**Are there cases where PCA more suitable than TSNE**](https://stats.stackexchange.com/questions/238538/are-there-cases-where-pca-is-more-suitable-than-t-sne?rq=1)
3. [**PCA preserving pairwise distances over tSNE?**](https://stats.stackexchange.com/questions/176672/what-is-meant-by-pca-preserving-only-large-pairwise-distances) **How why, all here.**
4. [**Another advice about using tsne and the possible misinterpetations**](https://towardsdatascience.com/why-you-are-using-t-sne-wrong-502412aab0c0)
### **PCA**
1. **Machine learning mastery:**
1. [**Expected value, variance, covariance** ](https://machinelearningmastery.com/introduction-to-expected-value-variance-and-covariance)
2. [**PCA**](https://machinelearningmastery.com/calculate-principal-component-analysis-scratch-python/) **(remove the mean from A, calculate cov(A), calculate eig(cov), A\*eigK = PCA)**
3. [**EigenDecomposition**](https://machinelearningmastery.com/introduction-to-eigendecomposition-eigenvalues-and-eigenvectors/) **- what is an eigen vector - simply put its a vector that satisfies A\*v = lambda\*v, how to use eig() and how to confirm an eigenvector/eigenvalue and reconstruct the original A matrix.**
4. [**SVD**](https://machinelearningmastery.com/singular-value-decomposition-for-machine-learning)
5. **What is missing is how the EigenDecomposition is calculated.**
2. [**PCA on large matrices!**](https://amedee.me/post/pca-large-matrices/)
1. **Randomized svd**
2. **Incremental svd**
3. [**PCA on Iris**](http://sebastianraschka.com/Articles/2015_pca_in_3_steps.html)
4. **(did not read)** [**What is PCA?**](https://stats.stackexchange.com/questions/222/what-are-principal-component-scores)
5. **(did not read)** [**What is a covariance matrix?**](https://en.wikipedia.org/wiki/Covariance_matrix)
6. **(did not read)** [**Variance covariance matrix**](http://stattrek.com/matrix-algebra/covariance-matrix.aspx)
7. [**Visualization of the first PCA vectors**](https://medium.com/@rtjeannier/using-pca-to-visualize-high-dimensional-data-6ff028c911c5)**, it is unclear what he is trying to show.**
8. [**A very nice introductory tutorial on how to use PCA**](https://www.analyticsvidhya.com/blog/2016/03/practical-guide-principal-component-analysis-python/)
9. **\*\*** [**An in-depth tutorial on PCA (paper)**](https://www.cs.princeton.edu/picasso/mats/PCA-Tutorial-Intuition_jp.pdf)
10. **\*\*** [**yet another tutorial paper on PCA (looks good)**](https://www.researchgate.net/publication/309165405_Principal_component_analysis_-_a_tutorial)
11. [**How to use PCA in Cross validation and for train\test split**](https://stats.stackexchange.com/questions/114560/pca-on-train-and-test-datasets-do-i-need-to-merge-them)**. (bottom line, do it on the train only.)**
12. [**Another tutorial paper - looks decent**](https://www.researchgate.net/publication/309165405_Principal_component_analysis_-_a_tutorial)
13. [**PCA whitening**](http://mccormickml.com/2014/06/03/deep-learning-tutorial-pca-and-whitening/)**,** [**Stanford tutorial**](http://ufldl.stanford.edu/tutorial/unsupervised/PCAWhitening/) **(pca/zca whitening),** [**Stackoverflow (really good)**](https://stats.stackexchange.com/questions/117427/what-is-the-difference-between-zca-whitening-and-pca-whitening/117459) **,**
**There are two things we are trying to accomplish with whitening:**
1. **Make the features less correlated with one another.**
2. **Give all of the features the same variance.**
**Whitening has two simple steps:**
1. **Project the dataset onto the eigenvectors. This rotates the dataset so that there is no correlation between the components.**
2. **Normalize the the dataset to have a variance of 1 for all components. This is done by simply dividing each component by the square root of its eigenvalue.**
### **SVD**
1. [**An explanation about SVD’s formulas.**](https://blog.statsbot.co/singular-value-decomposition-tutorial-52c695315254)
### **KPCA**
1. [**First they say that**](https://web.cs.hacettepe.edu.tr/~aykut/classes/fall2016/bbm406/slides/l25-kernel_pca.pdf) **Autoencoder is PCA based on their equation, i.e. minimize the reconstruction error formula.**
2. **Then they say that PCA cant separate certain non-linear situations (circle within a circle), therefore they introduce kernel based PCA (using the kernel trick - like svm) which mapps the space to another linearly separable space, and performs PCA on it,**
1. **Finally, showing results how KPCA works well on noisy images, compared to PCA.**
### **LDA - Linear discriminant analysis**
[**A comparison / tutorial with code on pca vs lda - read!**](http://rstudio-pubs-static.s3.amazonaws.com/84669_cd15214061d44e1493ffee69c5d55925.html)
[**A comprehensive tutorial on LDA - read!**](https://iksinc.online/2018/11/12/dimensionality-reduction-via-linear-discriminant-analysis/?fbclid=IwAR3d0ja_HP0DkamiL3W4QxzjcsIfoySB_G7LetTNf0cE0ed_MVduXfi6bv0)
[**Dim reduction with LDA - nice examples**](https://iksinc.online/2018/11/12/dimensionality-reduction-via-linear-discriminant-analysis/?fbclid=IwAR3d0ja_HP0DkamiL3W4QxzjcsIfoySB_G7LetTNf0cE0ed_MVduXfi6bv0)
**(**[**Not to be confused with the other LDA**](http://sebastianraschka.com/Articles/2014_python_lda.html)**) - Linear Discriminant Analysis (LDA) is most commonly used as dimensionality reduction technique in the pre-processing step for pattern-classification and machine learning applications. The goal is to project a dataset onto a lower-dimensional space with good class-separability in order avoid overfitting (“curse of dimensionality”) and also reduce computational costs.**
**PCA vs LDA:**
**Both Linear Discriminant Analysis (LDA) and Principal Component Analysis (PCA) are linear transformation techniques used for dimensionality reduction.**
* **PCA can be described as an “unsupervised” algorithm, since it “ignores” class labels and its goal is to find the directions (the so-called principal components) that maximize the variance in a dataset.**
* **In contrast to PCA, LDA is “supervised” and computes the directions (“linear discriminants”) that will represent the axes that maximize the separation between multiple classes.**
**Although it might sound intuitive that LDA is superior to PCA for a multi-class classification task where the class labels are known, this might not always the case.**
**For example, comparisons between classification accuracies for image recognition after using PCA or LDA show that :**
* **PCA tends to outperform LDA if the number of samples per class is relatively small (**[**PCA vs. LDA**](http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=908974)**, A.M. Martinez et al., 2001).**
* **In practice, it is also not uncommon to use both LDA and PCA in combination:**
**Best Practice: PCA for dimensionality reduction can be followed by an LDA. But before we skip to the results of the respective linear transformations, let us quickly recapitulate the purposes of PCA and LDA: PCA finds the axes with maximum variance for the whole data set where LDA tries to find the axes for best class separability. In practice, often a LDA is done followed by a PCA for dimensionality reduction.**
**\*\* To fully understand the details please follow the LDA link to the original and very informative article**
**\*\*\* TODO: need some benchmarking for PCA\LDA\LSA\ETC..**
###
### **KDA - KERNEL DISCRIMINANT ANALYSIS**
1. [**pyDML package**](https://pydml.readthedocs.io/en/latest/dml.html#dml.kda.KDA) **- has KDA - This package provides the classic algorithms of supervised distance metric learning, together with some of the newest proposals.**
### **LSA**
[**LSA**](http://mccormickml.com/2016/03/25/lsa-for-text-classification-tutorial/) **is quite simple, you just use SVD to perform dimensionality reduction on the tf-idf vectors–that’s really all there is to it! And** [**LSA CLUSTERING**](http://mccormickml.com/2015/08/05/document-clustering-example-in-scikit-learn/)
**Here is a very nice** [**tutorial about LSA,**](https://technowiki.wordpress.com/2011/08/27/latent-semantic-analysis-lsa-tutorial/) **with code, explaining what are the three matrices, word clustering, sentence clustering and vector importance. They say that for sentence space we need to remove the first vector as it is correlated with sentence length.**
**\*how to** [**interpret LSA vectors**](http://mccormickml.com/2016/03/25/lsa-for-text-classification-tutorial/)
**PCA vs LSA: (**[**intuition1**](https://stats.stackexchange.com/questions/65699/lsa-vs-pca-document-clustering)**,** [**intuition2**](https://math.stackexchange.com/questions/3869/what-is-the-intuitive-relationship-between-svd-and-pca)**)**
* **reduction of the dimensionality**
* **noise reduction**
* **incorporating relations between terms into the representation.**
* **SVD and PCA and "total least-squares" (and several other names) are the same thing. It computes the orthogonal transform that decorrelates the variables and keeps the ones with the largest variance. There are two numerical approaches: one by SVD of the (centered) data matrix, and one by Eigen decomposition of this matrix "squared" (covariance).**
[**LSA vs W2V**](https://arxiv.org/pdf/1610.01520.pdf)
### **ICA**
1. **While PCA is global, it finds global variables (with images we get eigen faces, good for reconstruction) that maximizes variance in orthogonal directions, and is not influenced by the TRANSPOSE of the data matrix.**
2. **On the other hand, ICA is local and finds local variables (with images we get eyes ears, mouth, basically edges!, etc), ICA will result differently on TRANSPOSED matrices, unlike PCA, its also “directional” - consider the “cocktail party” problem. On documents, ICA gives topics.**
3. **It helps, similarly to PCA, to help us analyze our data.**
**Sparse** [**info on ICA with security returns.**](https://www.quantopian.com/posts/an-experiment-with-independent-component-analysis)
### **MANIFOLD**
1. [**The best tutorial that explains manifold (high to low dim projection/mapping/visuzation)**](https://jhui.github.io/2017/01/15/Machine-learning-Multi-dimensional-scaling-and-visualization/) **(pca, sammon, isomap, tsne)**
2. [**Many manifold methods used to visualize high dimensional data.** ](http://scikit-learn.org/stable/modules/manifold.html#t-sne)
3. [**Comparing manifold methods**](http://scikit-learn.org/stable/auto_examples/manifold/plot_compare_methods.html#sphx-glr-auto-examples-manifold-plot-compare-methods-py)
#### **T-SNE**
1. [**Code and in-depth tutorial on TSNE, mapping probabilities to distributions**](https://towardsdatascience.com/t-sne-python-example-1ded9953f26)**\*\*\*\***
2. [**A great example of using PCA and then TSNE to see clusters that arent visible with PCA only.**](https://towardsdatascience.com/dimensionality-reduction-by-stacking-pca-and-t-sne-420d9fcfab54)
3. [**Misreading T-SNE**](https://distill.pub/2016/misread-tsne/)**, this is a very important read.**
4. **In contrary to what it says on sklearn’s website, TSNE is not suited ONLY for visualization, you** [**can also use it for data reduction**](https://lvdmaaten.github.io/tsne/)
5. **“t-Distributed Stochastic Neighbor Embedding (t-SNE) is a (**[**prize-winning**](http://blog.kaggle.com/2012/11/02/t-distributed-stochastic-neighbor-embedding-wins-merck-viz-challenge/)**) technique for dimensionality reduction that is particularly well suited for the visualization of high-dimensional datasets.”**
6. [**Comparing PCA and TSNE, then pushing PCA to TSNE and seeing what happens (as recommended in SKLEARN**](https://medium.com/@luckylwk/visualising-high-dimensional-datasets-using-pca-and-t-sne-in-python-8ef87e7915b)
7. [**TSNE + AUTOENCODER example**](https://towardsdatascience.com/reducing-dimensionality-from-dimensionality-reduction-techniques-f658aec24dfe)
#### **Sammons embedding mapping**
1. [**In tensorflow**](https://datawarrior.wordpress.com/2017/06/01/sammon-embedding-with-tensorflow/)
#### **IVIS**
1. [**Paper:** ](https://www.nature.com/articles/s41598-019-45301-0)
2. [**Git**](https://github.com/beringresearch/ivis)**,** [**docs**](https://bering-ivis.readthedocs.io/en/latest/)
3. [**Ivis animate**](https://github.com/beringresearch/ivis-animate)
4. [**Ivis explain**](https://github.com/beringresearch/ivis-explain)
# Genetic Algorithms & Genetic Programming
[**What is the difference?**](https://stackoverflow.com/questions/3819977/what-are-the-differences-between-genetic-algorithms-and-genetic-programming) **Genetic programming and genetic algorithms are very similar. They are both used to evolve the answer to a problem, by comparing the fitness of each candidate in a population of potential candidates over many generations.**
**Each generation, new candidates are found by randomly changing (mutation) or swapping parts (crossover) of other candidates. The least 'fit' candidates are removed from the population. - peterjwest**
**Genetic algorithms (GA) are search algorithms that mimic the process of natural evolution, where each individual is a candidate solution: individuals are generally "raw data" (in whatever encoding format has been defined).**
**Genetic programming (GP) is considered a special case of GA, where each individual is a computer program (not just "raw data"). GP explore the algorithmic search space and evolve computer programs to perform a defined task.**
**johnIdol**
# Learning Classifier Systems
### **LCS**
**(WIKI)** [**Learning classifier systems**](https://en.wikipedia.org/wiki/Learning_classifier_system)**, or LCS, are a paradigm of** [**rule-based machine learning**](https://en.wikipedia.org/wiki/Rule-based_machine_learning) **methods that combine a discovery component (e.g. typically a** [**genetic algorithm**](https://en.wikipedia.org/wiki/Genetic_algorithm)**) with a learning component (performing either** [**supervised learning**](https://en.wikipedia.org/wiki/Supervised_learning)**,** [**reinforcement learning**](https://en.wikipedia.org/wiki/Reinforcement_learning)**, or** [**unsupervised learning**](https://en.wikipedia.org/wiki/Unsupervised_learning)**).**
### **XCS**
[**XCS**](http://hosford42.github.io/xcs/) **is a type of** [**Learning Classifier System (LCS)**](http://en.wikipedia.org/wiki/Learning_classifier_system)**, a** [**machine learning**](http://en.wikipedia.org/wiki/Machine_learning) **algorithm that utilizes a** [**genetic algorithm**](http://en.wikipedia.org/wiki/Genetic_algorithm) **acting on a rule-based system, to solve a** [**reinforcement learning**](http://en.wikipedia.org/wiki/Reinforcement_learning) **problem.**
[**Scikit-xcs**](https://github.com/UrbsLab/scikit-XCS) 
[**Tutorial**](https://pythonhosted.org/xcs/)
# Recommender Systems
1. [Beginner guide](https://www.analyticsvidhya.com/blog/2015/08/beginners-guide-learn-content-based-recommender-systems/) vidhya
2. [Real python on CF](https://realpython.com/build-recommendation-engine-collaborative-filtering/#steps-involved-in-collaborative-filtering)
3. [Intro to, using item-item or user-item](https://www.ethanrosenthal.com/2015/11/02/intro-to-collaborative-filtering/), validating using imdb data, git
4. [Tfidf cosine similarity](https://towardsdatascience.com/recommender-engine-under-the-hood-7869d5eab072), [countvec cosine](https://www.datacamp.com/community/tutorials/recommender-systems-python)
5. [Various implementations of CF](https://towardsdatascience.com/various-implementations-of-collaborative-filtering-100385c6dfe0), a serious review of algorithms
6. [Collaborative filtering, SVD](https://hackernoon.com/introduction-to-recommender-system-part-1-collaborative-filtering-singular-value-decomposition-44c9659c5e75)
7. [Part1,](https://hackernoon.com/introduction-to-recommender-system-part-1-collaborative-filtering-singular-value-decomposition-44c9659c5e75) [Spotlight, item2vec, Neural nets for Recommender systems](https://towardsdatascience.com/introduction-to-recommender-system-part-2-adoption-of-neural-network-831972c4cbf7)
8. [A general tutorial, has a nice intro](https://www.datacamp.com/community/tutorials/recommender-systems-python)
9. Medium on Movies
1. Part 1 [matrix factorization in movies, users vs movies.](https://towardsdatascience.com/fast-ai-season-1-episode-5-1-movie-recommendation-using-fastai-a53ed8e41269)[ ](https://towardsdatascience.com/fast-ai-season-1-episode-5-2-collaborative-filtering-from-scratch-1877640f514a)
2. [Part 2 using collaborative filtering](https://towardsdatascience.com/fast-ai-season-1-episode-5-2-collaborative-filtering-from-scratch-1877640f514a) using open ai
3. [Part 3 using col-filtering with neural nets](https://towardsdatascience.com/fast-ai-season-1-episode-5-3-collaborative-filtering-using-neural-network-48e49d7f9b36)
10. Medium series on collaborative filtering and embeddings [Part 1](https://towardsdatascience.com/collaborative-filtering-and-embeddings-part-1-63b00b9739ce), [part 2](https://towardsdatascience.com/collaborative-filtering-and-embeddings-part-2-919da17ecefb), [git](https://github.com/shik3519/collaborative-filtering)
11. [Movie recommender systems](https://www.kaggle.com/rounakbanik/movie-recommender-systems) on kaggle
1. [On git](https://github.com/jaypatel00174/Movie-Recommendation)
12. [Matrix factorization ](https://towardsdatascience.com/paper-summary-matrix-factorization-techniques-for-recommender-systems-82d1a7ace74)
13. [Collaborative filtering with binary countvec data, item-item, didnt work well on another domain](https://medium.com/radon-dev/item-item-collaborative-filtering-with-binary-or-unary-data-e8f0b465b2c3)
14. [Netflix competition, matrix factorization over classical algorithms, a survey paper](https://towardsdatascience.com/paper-summary-matrix-factorization-techniques-for-recommender-systems-82d1a7ace74)
15. [Movie similarity based on genre ](https://towardsdatascience.com/content-based-recommender-systems-28a1dbd858f5)
16. [Similar entities, matrix multiplication](https://medium.com/wbaa/https-medium-com-ingwbaa-boosting-selection-of-the-most-similar-entities-in-large-scale-datasets-450b3242e618) high sparsity
17. [Euclidean distance with high sparse data](https://stats.stackexchange.com/questions/117354/euclidean-distance-with-sparse-and-high-dimension-data)
18. Excel & fastai, [git](https://github.com/shik3519/collaborative-filtering/blob/master/cf-scratch-movielens/collaborative%20filtering%20from%20scratch.ipynb)
19. [CF for movie recommendation](https://medium.com/@wwwbbb8510/python-implementation-of-baseline-item-based-collaborative-filtering-2ba7c8960590)
20. [Comparison item vs user cf](https://medium.com/@wwwbbb8510/comparison-of-user-based-and-item-based-collaborative-filtering-f58a1c8a3f1d)
21. [build a recommendation engine with collaborative filtering](https://realpython.com/build-recommendation-engine-collaborative-filtering/)
## Evaluating Recommender Systems
1. [An exhaustive list of methods to evaluate](https://towardsdatascience.com/an-exhaustive-list-of-methods-to-evaluate-recommender-systems-a70c05e121de)
2. [Choosing the best for your business](https://medium.com/recombee-blog/evaluating-recommender-systems-choosing-the-best-one-for-your-business-c688ab781a35)
3. [Evaluating](https://medium.com/the-owl/evaluating-recommender-systems-749570354976)
4. [survey of accuracy eval metrics for RS by Microsoft](https://www.jmlr.org/papers/volume10/gunawardana09a/gunawardana09a.pdf)
5. [Building a validation framework](https://medium.com/moosend-engineering-data-science/building-a-validation-framework-for-recommender-systems-a-quest-ec173a24b56f)
6. [Evaluation Metrics for RS](https://towardsdatascience.com/evaluation-metrics-for-recommender-systems-df56c6611093)
7. [offline vs online validation](https://www.quora.com/How-do-I-validate-my-recommendation-system-without-prior-user-interaction-data)
8. [Evaluating RS](https://tzin.bgu.ac.il/~shanigu/Publications/EvaluationMetrics.17.pdf)
## **TOOLS**
1. [**Surprise**](https://github.com/NicolasHug/Surprise)**,** [**docs**](https://surprise.readthedocs.io/en/stable/FAQ.html#how-to-get-the-top-n-recommendations-for-each-user)**,**
2. [**Grover prince**](https://github.com/groverpr/Machine-Learning) **,** [**related article**](https://towardsdatascience.com/various-implementations-of-collaborative-filtering-100385c6dfe0)
3. [**Recsys**](https://github.com/ocelma/python-recsys) **git**
# Timeseries
1. [**Random walk**](https://machinelearningmastery.com/gentle-introduction-random-walk-times-series-forecasting-python/) **- what is?**
1. [**Time series decomposition book**](https://otexts.com/fpp2/forecasting-decomposition.html) **- stl x11 seats**
2. [**Mastery on ts decomposition**](https://machinelearningmastery.com/decompose-time-series-data-trend-seasonality/)
### **TOOLS**
1. **SKtime - is a sk-based api,** [**medium**](https://towardsdatascience.com/sktime-a-unified-python-library-for-time-series-machine-learning-3c103c139a55)**, integrates algos from tsfresh and tslearn**
2. **(really good)** [**A LightGBM Autoregressor — Using Sktime**](https://towardsdatascience.com/a-lightgbm-autoregressor-using-sktime-6402726e0e7b)**, explains about the basics in time series prediction, splitting, next step, delayed step, multi step, deseason.**
3. [**SKtime-DL - using keras and DL**](https://github.com/sktime/sktime-dl)
4. [**TSFresh**](http://tsfresh.readthedocs.io) **- extracts 1200 features, filters them using FDR for time series classification etc**
5. [**TSlearn**](http://tslearn.readthedocs.io) **- DTW, shapes, shapelets (keras layer), time series kmeans/clustering/svm/svr/KNN/bary centers/PAA/SAX** 
6. [**DTAIDistance**](https://dtaidistance.readthedocs.io/en/latest/index.html) **- Library for time series distances (e.g. Dynamic Time Warping) used in the** [**DTAI Research Group**](https://dtai.cs.kuleuven.be/)**. The library offers a pure Python implementation and a faster implementation in C. The C implementation has only Cython as a dependency. It is compatible with Numpy and Pandas and implemented to avoid unnecessary data copy operations**\
[**dtaidistance.clustering.hierarchical**](https://dtaidistance.readthedocs.io/en/latest/modules/clustering/hierarchical.html)
7. [**Darts**](https://unit8co.github.io/darts/) is a Python library for user-friendly forecasting and anomaly detection on time series. [Forecasting models](https://unit8co.github.io/darts/#forecasting-models) & [Examples](https://unit8co.github.io/darts/#example-usage)
[**Ddtaidistance.clustering.kmeans**](https://dtaidistance.readthedocs.io/en/latest/modules/clustering/kmeans.html)
[**Dtaidistance.clustering.medoids**](https://dtaidistance.readthedocs.io/en/latest/modules/clustering/medoids.html)
**\* Identify anomalies, outliers or abnormal behaviour (see for example the** [**anomatools package**](https://github.com/Vincent-Vercruyssen/anomatools)**).**
1. **Semi supervised with DTAIDistance - Active semi-supervised clustering**
**The recommended method for perform active semi-supervised clustering using DTAIDistance is to use the COBRAS for time series clustering:** [**https://github.com/ML-KULeuven/cobras**](https://github.com/ML-KULeuven/cobras)**. COBRAS is a library for semi-supervised time series clustering using pairwise constraints, which natively supports both dtaidistance.dtw and kshape.**
1. [**Affine warp**](https://github.com/ahwillia/affinewarp)**, a neural net with time warping - as part of the following manuscript, which focuses on analysis of large-scale neural recordings (though this code can be also be applied to many other data types)**
2. [**Neural warp**](https://github.com/josifgrabocka/neuralwarp) **-** [**NeuralWarp**](https://arxiv.org/pdf/1812.08306.pdf)**: Time-Series Similarity with Warping Networks**
3.
[**A great introduction into time series**](https://medium.com/making-sense-of-data/time-series-next-value-prediction-using-regression-over-a-rolling-window-228f0acae363) **- “The approach is to come up with a list of features that captures the temporal aspects so that the auto correlation information is not lost.” basically tells us to take sequence features and create (auto)-correlated new variables using a time window, i.e., “Time series forecasts as regression that factor in autocorrelation as well.”. we can transform raw features into other type of features that explain the relationship in time between features. we measure success using loss functions, MAE RMSE MAPE RMSEP AC-ERROR-RATE**
[**Interesting idea**](http://blog.kaggle.com/2016/02/03/rossmann-store-sales-winners-interview-2nd-place-nima-shahbazi/) **on how to define ‘time series’ dummy variables that utilize beginning\end of certain holiday events, including important information on what NOT to filter even if it seems insignificant, such as zero sales that may indicate some relationship to many sales the following day.**
[**Time series patterns:** ](https://www.otexts.org/fpp/2/1)
* **A trend (a,b,c) exists when there is a long-term increase or decrease in the data.**
* **A seasonal (a - big waves) pattern occurs when a time series is affected by seasonal factors such as the time of the year or the day of the week. The monthly sales induced by the change in cost at the end of the calendar year.**
* **A cycle (a) occurs when the data exhibit rises and falls that are not of a fixed period - sometimes years.**
[**Some statistical measures**](https://www.otexts.org/fpp/2/2) **(mean, median, percentiles, iqr, std dev, bivariate statistics - correlation between variables)**
**Bivariate Formula: this correlation measures the extent of a linear relationship between two variables. high number = high correlation between two variable. The value of r always lies between -1 and 1 with negative values indicating a negative relationship and positive values indicating a positive relationship. Negative = decreasing, positive = increasing.**
**But correlation can LIE, the following has 0.8 correlation for all of the graphs:**
**Autocorrelation measures the linear relationship between lagged values of a time series.**
**L8 is correlated, and has a high measure of 0.83**
* **White-noise has autocorrelation of 0.**
### [**Forecasting methods**](https://www.otexts.org/fpp/2/3)
* **Average: Forecasts of all future values are equal to the mean of the historical data.**
* **Naive: Forecasts are simply set to be the value of the last observation.**
* **Seasonal Naive: forecast to be equal to the last observed value from the same season of the year**
* **Drift: A variation on the naïve method is to allow the forecasts to increase or decrease over time, the drift is set to be the average change seen in the historical data.**
### [**Data Transformations**](https://www.otexts.org/fpp/2/4)
* **Log**
* **Box cox**
* **Back transform**
* **Calendrical adjustments**
* **Inflation adjustment**
[**Transforming time series data to tabular (in order to use tabular based approach)**](https://towardsdatascience.com/approaching-time-series-with-a-tree-based-model-87c6d1fb6603)
### **SPLITTING TIME SERIES DATA**
1. **SK-lego** [**With a gap**](https://scikit-lego.readthedocs.io/en/latest/timegapsplit.html) **-** [**now with even timeseries split by group**](https://towardsdatascience.com/a-highly-anticipated-time-series-cross-validator-is-finally-here-7dc99f672736)
### [**Evaluate forecast accuracy**](https://www.otexts.org/fpp/2/5)
* **Dummy variables: sunday, monday, tues,wed,thurs, friday. NO SATURDAY!**
* **notice that only six dummy variables are needed to code seven categories. That is because the seventh category (in this case Sunday) is specified when the dummy variables are all set to zero. Many beginners will try to add a seventh dummy variable for the seventh category. This is known as the "dummy variable trap" because it will cause the regression to fail.**
* **Outliers: If there is an outlier in the data, rather than omit it, you can use a dummy variable to remove its effect. In this case, the dummy variable takes value one for that observation and zero everywhere else.**
* **Public holidays: For daily data, the effect of public holidays can be accounted for by including a dummy variable predictor taking value one on public holidays and zero elsewhere.**
* **Easter: is different from most holidays because it is not held on the same date each year and the effect can last for several days. In this case, a dummy variable can be used with value one where any part of the holiday falls in the particular time period and zero otherwise.**
* **Trading days: The number of trading days in a month can vary considerably and can have a substantial effect on sales data. To allow for this, the number of trading days in each month can be included as a predictor. An alternative that allows for the effects of different days of the week has the following predictors. # Mondays in month;# Tuesdays in month;# Sundays in month.**
* **Advertising: $advertising for previous month;$advertising for two months previously**
### [**Rolling window analysis**](https://link.springer.com/chapter/10.1007%2F978-0-387-32348-0_9)
**“compute parameter estimates over a rolling window of a fixed size through the sample. If the parameters are truly constant over the entire sample, then the estimates over the rolling windows should not be too different. If the parameters change at some point during the sample, then the rolling estimates should capture this instability”**
### [**Moving average window**](https://www.otexts.org/fpp/6/2)
**estimate the trend cycle**
* **3-5-7-9? If its too large its going to flatten the curve, too low its going to be similar to the actual curve.**
* **two tier moving average, first 4 then 2 on the resulted moving average.**
[**Visual example**](https://www.youtube.com/watch?v=_YXoRTQQI3U) **of ARIMA algorithm - captures the time series trend or forecast.**
### **Decomposition**
1. [**Creating**](https://scikit-lego.readthedocs.io/en/latest/preprocessing.html#Repeating-Basis-Function-Transformer) **curves to explain a complex seasonal fit.**
2. 
3. 
### **Weighted “window”**
[**1, scikit-lego with a decay estimator**](https://scikit-lego.readthedocs.io/en/latest/meta.html#Decayed-Estimation)
### [**Time Series Components**](http://machinelearningmastery.com/time-series-forecasting/)
1. **Level. The baseline value for the series if it were a straight line.**
2. **Trend. The optional and often linear increasing or decreasing behavior of the series over time.**
3. **Seasonality. The optional repeating patterns or cycles of behavior over time.**
4. **Noise. The optional variability in the observations that cannot be explained by the model.**
**All time series have a level, most have noise, and the trend and seasonality are optional.**
**One step forecast using a window of “1” and a typical sample “time, measure1, measure2”:**
* **linear/nonlinear classifiers: predict a single output value - using the t-1 previous line, i.e., “measure1 t, measure 2 t, measure 1 t+1, measure 2 t+1 (as the class)”**
* **Neural networks: predict multiple output values, i.e., “measure1 t, measure 2 t, measure 1 t+1(class1), measure 2 t+1(class2)”**
**One-Step Forecast: This is where the next time step (t+1) is predicted.**
**Multi-Step Forecast: This is where two or more future time steps are to be predicted.**
**Multi-step forecast using a window of “1” and a typical sample “time, measure1”, i.e., using the current value input we label it as the two future input labels:**
* **“measure1 t, measure1 t+1(class) , measure1 t+2(class1)”**
[**This article explains**](http://web.engr.oregonstate.edu/~tgd/publications/mlsd-ssspr.pdf) **about ML Methods for Sequential Supervised Learning - Six methods that have been applied to solve sequential supervised learning problems:**
1. **sliding-window methods - converts a sequential supervised problem into a classical supervised problem**
2. **recurrent sliding windows**
3. **hidden Markov models**
4. **maximum entropy Markov models**
5. **input-output Markov models**
6. **conditional random fields**
7. **graph transformer networks**
##
### **STATIONARY TIME SERIES**
[**What is?**](https://machinelearningmastery.com/time-series-data-stationary-python/) **A time series without a trend or seasonality, in other words non-stationary has a trend or seasonality**
**There are ways to** [**remove the trend and seasonality**](https://machinelearningmastery.com/difference-time-series-dataset-python/)**, i.e., take the difference between time points.**
1. **T+1 - T**
2. **Bigger lag to support seasonal changes**
3. **pandas.diff()**
4. **Plot a histogram, plot a log(X) as well.**
5. **Test for the unit root null hypothesis - i.e., use the Augmented dickey fuller test to determine if two samples originate in a stationary or a non-stationary (seasonal/trend) time series**
[**Shay on stationary time series, AR, ARMA**](https://towardsdatascience.com/stationarity-in-time-series-analysis-90c94f27322)
**(amazing)** [**STL**](https://otexts.com/fpp2/stl.html) **and more.**
### **SHORT TIME SERIES**
1. [**Short time series**](https://robjhyndman.com/hyndsight/short-time-series/)
2. [**PDarima -**](https://towardsdatascience.com/efficient-time-series-using-pythons-pmdarima-library-f6825407b7f0) **Pmdarima‘s auto\_arima function is extremely useful when building an ARIMA model as it helps us identify the most optimal p,d,q parameters and return a fitted ARIMA model.**
3. [**Min sample size for short seasonal time series**](https://robjhyndman.com/papers/shortseasonal.pdf)
4. [**More mastery on short time series.**](https://machinelearningmastery.com/time-series-forecasting-methods-in-python-cheat-sheet/?fbclid=IwAR0iU9B-wsRaOPOY13F4xesGWUMevRBuPck5I9jTNlV5zmPFCX1NoG05_jI)
1. **Autoregression (AR)**
2. **Moving Average (MA)**
3. **Autoregressive Moving Average (ARMA)**
4. **Autoregressive Integrated Moving Average (ARIMA)**
5. **Seasonal Autoregressive Integrated Moving-Average (SARIMA)**
6. **Seasonal Autoregressive Integrated Moving-Average with Exogenous Regressors (SARIMAX)**
7. **Vector Autoregression (VAR)**
8. **Vector Autoregression Moving-Average (VARMA)**
9. **Vector Autoregression Moving-Average with Exogenous Regressors (VARMAX)**
10. **Simple Exponential Smoothing (SES)**
11. **Holt Winter’s Exponential Smoothing (HWES)**
**Predicting actual Values of time series using observations**
1. [**Using kalman filters**](https://www.youtube.com/watch?v=CaCcOwJPytQ) **- explains the concept etc, 1 out of 55 videos.**
### [**Kalman filters in matlab**](https://www.youtube.com/watch?v=4OerJmPpkRg)
### [**LTSM for time series**](http://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/)
**There are three types of gates within a unit:**
* **Forget Gate: conditionally decides what information to throw away from the block.**
* **Input Gate: conditionally decides which values from the input to update the memory state.**
* **Output Gate: conditionally decides what to output based on input and the memory of the block.**
**Using lstm to predict sun spots, has some autocorrelation usage**
* [**Part 1**](https://www.business-science.io/timeseries-analysis/2018/04/18/keras-lstm-sunspots-time-series-prediction.html)
* [**Part 2**](https://www.business-science.io/timeseries-analysis/2018/07/01/keras-lstm-sunspots-part2.html)
### **CLASSIFICATION**
1. [**Stackexchange**](https://stats.stackexchange.com/questions/131281/dynamic-time-warping-clustering/131284) **- Yes, you can use DTW approach for classification and clustering of time series. I've compiled the following resources, which are focused on this very topic (I've recently answered a similar question, but not on this site, so I'm copying the contents here for everybody's convenience):**
* **UCR Time Series Classification/Clustering:** [**main page**](http://www.cs.ucr.edu/~eamonn/time_series_data)**,** [**software page**](http://www.cs.ucr.edu/~eamonn/UCRsuite.html) **and** [**corresponding paper**](http://www.cs.ucr.edu/~eamonn/SIGKDD_trillion.pdf)
* **Time Series Classification and Clustering with Python:** [**a blog post**](http://alexminnaar.com/2014/04/16/Time-Series-Classification-and-Clustering-with-Python.html)
* **Capital Bikeshare: Time Series Clustering:** [**another blog post**](http://ofdataandscience.blogspot.com/2013/03/capital-bikeshare-time-series-clustering.html)
* **Time Series Classification and Clustering:** [**ipython notebook**](http://nbviewer.ipython.org/github/alexminnaar/time-series-classification-and-clustering/blob/master/Time%20Series%20Classification%20and%20Clustering.ipynb)
* **Dynamic Time Warping using rpy and Python:** [**another blog post**](https://nipunbatra.wordpress.com/2013/06/09/dynamic-time-warping-using-rpy-and-python)
* **Mining Time-series with Trillions of Points: Dynamic Time Warping at Scale:** [**another blog post**](http://practicalquant.blogspot.com/2012/10/mining-time-series-with-trillions-of.html)
* **Time Series Analysis and Mining in R (to add R to the mix):** [**yet another blog post**](http://rdatamining.wordpress.com/2011/08/23/time-series-analysis-and-mining-with-r)
* **And, finally, two tools implementing/supporting DTW, to top it off:** [**R package**](http://rdatamining.wordpress.com/2011/08/23/time-series-analysis-and-mining-with-r) **and** [**Python module**](http://mlpy.sourceforge.net/)
### **CLUSTERING TS**
1. [**Clustering time series, subsequences with a rolling window, the pitfall.**](https://towardsdatascience.com/dont-make-this-mistake-when-clustering-time-series-data-d9403f39bbb2)
2. [**Clustering using tslearn**](https://tslearn.readthedocs.io/en/stable/user_guide/clustering.html)
3. [**Kmeans for variable length**](https://medium.com/@iliazaitsev/how-to-classify-a-dataset-with-observations-of-various-length-96fab8e95baf) **,** [**notebook**](https://github.com/devforfu/Blog/blob/master/trees/scikit_learn.py)
### **ANOMALY DETECTION TS**
1. [**What is stationary (process**](https://en.wikipedia.org/wiki/Stationary_process)**),** [**stationary time series analysis**](https://towardsdatascience.com/stationarity-in-time-series-analysis-90c94f27322) **(shay palachi),**
2. [**mastery on arimas**](https://machinelearningmastery.com/time-series-forecasting-methods-in-python-cheat-sheet/?fbclid=IwAR0iU9B-wsRaOPOY13F4xesGWUMevRBuPck5I9jTNlV5zmPFCX1NoG05_jI)
3. [**TS anomaly algos (stl, trees, arima)**](https://blog.statsbot.co/time-series-anomaly-detection-algorithms-1cef5519aef2)
4. [**AD techniques**](https://medium.com/dp6-us-blog/anomaly-detection-techniques-c3817e8e7b2f)**, part** [**2**](https://medium.com/dp6-us-blog/anomaly-detection-techniques-part-ii-9a08b6562619)**, part** [**3**](https://medium.com/dp6-us-blog/anomaly-detection-techniques-part-iii-d27e7b0d6c8a)
5. [**Z-score, modified z-score and iqr an intro why z-score is not robust**](http://colingorrie.github.io/outlier-detection.html)
6. [**Adtk**](https://adtk.readthedocs.io/en/stable/userguide.html) **a sklearn-like toolkit with an amazing intro, various algorithms for non seasonal and seasonal, transformers, ensembles.**
7. [**Awesome TS anomaly detection**](https://github.com/rob-med/awesome-TS-anomaly-detection) **on github**
8. [**Transfer learning toolkit**](https://github.com/FuzhenZhuang/Transfer-Learning-Toolkit)**,** [**paper and benchmarks**](https://arxiv.org/pdf/1911.08967.pdf)
9. [**Ransac is a good baseline**](https://medium.com/@iamhatesz/random-sample-consensus-bd2bb7b1be75) **- random sample consensus for outlier detection**
1. [**Ransac**](https://medium.com/@angel.manzur/got-outliers-ransac-them-f12b6b5f606e)**,** [**2**](https://medium.com/@saurabh.dasgupta1/outlier-detection-using-the-ransac-algorithm-de52670adb4a)**,** [**3**](https://towardsdatascience.com/detecting-the-fault-line-using-k-mean-clustering-and-ransac-9a74cb61bb96)**,** [**4**](http://www.cs.tau.ac.il/~turkel/imagepapers/RANSAC4Dummies.pdf)**, 5, 6**
2. **You can feed ransac with tsfresh/tslearn features.**
10. [**Anomaly detection for time series**](https://medium.com/@jetnew/anomaly-detection-of-time-series-data-e0cb6b382e33)**,**
11. **AD for TS, recommended by DTAIDistance,** [**anomatools**](https://github.com/Vincent-Vercruyssen/anomatools)
12. **STL:**
1. [**AD where anomalies coincide with seasonal peaks!!**](https://medium.com/@richa.mishr01/anomaly-detection-in-seasonal-time-series-where-anomalies-coincide-with-seasonal-peaks-9859a6a6b8ba)
2. [**AD challenges, stationary, seasonality, trend**](https://cloudfabrix.com/blog/aiops/anomaly-detection-time-series-data/)
3. [**Rt anomaly detection for time series pinterest**](https://medium.com/pinterest-engineering/building-a-real-time-anomaly-detection-system-for-time-series-at-pinterest-a833e6856ddd) **using stl decomposition**
4. [**AD**](https://medium.com/wwblog/anomaly-detection-using-stl-76099c9fd5a7)
13. **Sliding windows**
1. [**Solving sliding window problems**](https://medium.com/outco/how-to-solve-sliding-window-problems-28d67601a66)
2. [**Rolling window regression**](https://medium.com/making-sense-of-data/time-series-next-value-prediction-using-regression-over-a-rolling-window-228f0acae363)
14. **Forecasting using Arima** [**1**](https://towardsdatascience.com/time-series-forecasting-using-auto-arima-in-python-bb83e49210cd)**,** [**2**](http://alkaline-ml.com/pmdarima/)
15. **Auto arima** [**1**](https://towardsdatascience.com/time-series-forecasting-using-auto-arima-in-python-bb83e49210cd)**,** [**2**](https://stackoverflow.com/questions/22770352/auto-arima-equivalent-for-python)**,** [**3**](https://www.analyticsvidhya.com/blog/2018/08/auto-arima-time-series-modeling-python-r/)
16. [**Twitters ESD test**](https://medium.com/@elisha_12808/time-series-anomaly-detection-with-twitters-esd-test-50cce409ced1) **for outliers, using z-score and t test**
1. **Another esd test inside** [**here**](https://towardsdatascience.com/anomaly-detection-def662294a4e)
17. [**Minimal sample size for seasonal forecasting**](https://robjhyndman.com/papers/shortseasonal.pdf)
18. [**Golden signals**](https://www.usenix.org/conference/srecon19asia/presentation/chen-yu)**,** [**youtube**](https://www.youtube.com/watch?v=3T9ZzQQiPSo)
19. [**Graph-based Anomaly Detection and Description: A Survey**](https://arxiv.org/pdf/1404.4679.pdf)
20. [**Time2vec**](https://towardsdatascience.com/time2vec-for-time-series-features-encoding-a03a4f3f937e)**,** [**paper**](https://arxiv.org/pdf/1907.05321.pdf) **(for deep learning, as a layer)**
###
### **Dynamic Time Warping (DTW)**
**DTW, ie., how to compute a better distance for two time series.**
1. [**The three myths of using DTW**](http://alumni.cs.ucr.edu/~ratana/RatanamC.pdf)
**Myth 1: The ability of DTW to handle sequences of different lengths is a great advantage, and therefore the simple lower bound that requires different-length sequences to be reinterpolated to equal length is of limited utility \[10]\[19]\[21]. In fact, as we will show, there is no evidence in the literature to suggest this, and extensive empirical evidence presented here suggests that comparing sequences of different lengths and reinterpolating them to equal length produce no statistically significant difference in accuracy or precision/recall.**\
**Myth 2: Constraining the warping paths is a necessary evil that we inherited from the speech processing community to make DTW tractable, and that we should find ways to speed up DTW with no (or larger) constraints\[19]. In fact, the opposite is true. As we will show, the 10% constraint on warping inherited blindly from the speech processing community is actually too large for real world data mining.**\
**Myth 3: There is a need (and room) for improvements in the speed of DTW for data mining applications. In fact, as we will show here, if we use a simple lower bounding technique, DTW is essentially O(n) for data mining applications. At least for CPU time, we are almost certainly at the asymptotic limit for speeding up DTW.**
1. [**Youtube - explains everything**](https://www.youtube.com/watch?v=_K1OsqCicBY)
2. [**Python code**](https://github.com/alexminnaar/time-series-classification-and-clustering) **with a** [**good tutorial.**](http://nbviewer.ipython.org/github/alexminnaar/time-series-classification-and-clustering/blob/master/Time%20Series%20Classification%20and%20Clustering.ipynb)
3. **Another function for dtw distance in python**
4. [**Medium**](https://medium.com/datadriveninvestor/dynamic-time-warping-dtw-d51d1a1e4afc)**, mentions prunedDTW, sparseDTW and fastDTW**
5. [**DTW in TSLEARN**](https://tslearn.readthedocs.io/en/latest/user_guide/dtw.html#soft-dtw)
6. [**DynamicTimeWarping**](https://dynamictimewarping.github.io/py-api/html/api/dtw.dtw.html#dtw.dtw) **git**
1. **(duplicate above in classification)** [**Stackexchange**](https://stats.stackexchange.com/questions/131281/dynamic-time-warping-clustering/131284) **- Yes, you can use DTW approach for classification and clustering of time series. I've compiled the following resources, which are focused on this very topic (I've recently answered a similar question, but not on this site, so I'm copying the contents here for everybody's convenience):**
* **UCR Time Series Classification/Clustering:** [**main page**](http://www.cs.ucr.edu/~eamonn/time_series_data)**,** [**software page**](http://www.cs.ucr.edu/~eamonn/UCRsuite.html) **and** [**corresponding paper**](http://www.cs.ucr.edu/~eamonn/SIGKDD_trillion.pdf)
* **Time Series Classification and Clustering with Python:** [**a blog post**](http://alexminnaar.com/2014/04/16/Time-Series-Classification-and-Clustering-with-Python.html)
* **Capital Bikeshare: Time Series Clustering:** [**another blog post**](http://ofdataandscience.blogspot.com/2013/03/capital-bikeshare-time-series-clustering.html)
* **Time Series Classification and Clustering:** [**ipython notebook**](http://nbviewer.ipython.org/github/alexminnaar/time-series-classification-and-clustering/blob/master/Time%20Series%20Classification%20and%20Clustering.ipynb)
* **Dynamic Time Warping using rpy and Python:** [**another blog post**](https://nipunbatra.wordpress.com/2013/06/09/dynamic-time-warping-using-rpy-and-python)
* **Mining Time-series with Trillions of Points: Dynamic Time Warping at Scale:** [**another blog post**](http://practicalquant.blogspot.com/2012/10/mining-time-series-with-trillions-of.html)
* **Time Series Analysis and Mining in R (to add R to the mix):** [**yet another blog post**](http://rdatamining.wordpress.com/2011/08/23/time-series-analysis-and-mining-with-r)
* **And, finally, two tools implementing/supporting DTW, to top it off:** [**R package**](http://rdatamining.wordpress.com/2011/08/23/time-series-analysis-and-mining-with-r) **and** [**Python module**](http://mlpy.sourceforge.net/)
1. [**Time Series Hierarchical Clustering using Dynamic Time Warping in Python**](https://towardsdatascience.com/time-series-hierarchical-clustering-using-dynamic-time-warping-in-python-c8c9edf2fda5) **-** [**notebook**](https://github.com/avchauzov/_articles/blob/master/1.1.trajectoriesClustering.ipynb)
2. [**K-Means with DTW, probably fixed length vectors, using tslearn**](https://towardsdatascience.com/how-to-apply-k-means-clustering-to-time-series-data-28d04a8f7da3)
3. **(nice)** [**With time series**](https://medium.com/@shachiakyaagba_41915/dynamic-time-warping-with-time-series-1f5c05fb8950)
# Fourier Transform
## **Fourier Transform**
**is a technique to decompose signals into sine and cosine waves**
1. [**Deconstructing time series using FT**](https://medium.com/@khairulomar/deconstructing-time-series-using-fourier-transform-e52dd535a44e)
2. **Medium series, mostly on the math, parts** [**1**](https://medium.com/sho-jp/fourier-transform-101-part-1-b69ea3cb4837) [**2**](https://medium.com/sho-jp/fourier-transform-101-part-2-complex-fourier-series-934a885b3921) [**3**](https://medium.com/sho-jp/fourier-transform-101-part-3-fourier-transform-6def0bd2ca9b) [**4**](https://medium.com/sho-jp/fourier-transform-101-part-4-discrete-fourier-transform-8fc3fbb763f3) [**5**](https://medium.com/sho-jp/fourier-transform-101-part-5-fast-fourier-transform-fft-38c22e05ead3)
## Wavelets
1. Medium on [What is a wavelet and how do we use it for DS](https://towardsdatascience.com/what-is-wavelet-and-how-we-use-it-for-data-science-d19427699cef)
2. [Medium on the wavelet transform ](https://towardsdatascience.com/the-wavelet-transform-e9cfa85d7b34)
3. Medium [multiple time series classification using continuous wavelet transformation and scalograms](https://towardsdatascience.com/multiple-time-series-classification-by-using-continuous-wavelet-transformation-d29df97c0442)
4. [pywavelets](https://pywavelets.readthedocs.io/en/latest/) - PyWavelets is open source wavelet transform software for [Python](http://python.org/). It combines a simple high level interface with low level C and Cython performance.
# Digital Signal Processing (DSP)
1. [**Scipy signal processing**](https://docs.scipy.org/doc/scipy/reference/signal.html)
2. [**Script find peaks**](https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.find_peaks.html)
3. **Beat detection**
1. [**Real time bpm beat det**](https://github.com/shunfu/python-beat-detector)
2. [**Librosa**](https://librosa.org/doc/latest/core.html#time-domain-processing)**:** [**Beat detection (and temp)**](https://librosa.org/doc/latest/beat.html#beat)
4. [**Mastery on Human activity recognition, smartphones**](https://machinelearningmastery.com/cnn-models-for-human-activity-recognition-time-series-classification/)
5. **(Out of place) -** [**using self-attention for sound signal processing**](https://medium.com/ai%C2%B3-theory-practice-business/toward-interpretable-music-tagging-with-self-attention-67a8136048d0)
##
# Propensity Score Matching
1. [psmpy](https://pypi.org/project/psmpy/): [propensity score matching intro](https://towardsdatascience.com/psmpy-propensity-score-matching-in-python-a3e0cd4d2631)
2. [Fit multiple propensity models and choose the best performing one for implementing a profit optimisation by Diego usai](https://towardsdatascience.com/using-h2o-and-dalex-to-estimate-the-likelihood-to-purchase-a-financial-product-a53f891f0d1)
3. [a meta discussion about propensity](https://medium.com/the-official-integrate-ai-blog/heres-what-you-need-to-know-about-propensity-modeling-521ab660cb43)
4. [types of propensity and use cases](https://www.altexsoft.com/blog/propensity-model/)
5. [log reg example](https://datatonic.com/insights/propensity-scoring-free-report/)
6.
# Diffusion models
1. [Lilian Weng on diffusion](https://lilianweng.github.io/posts/2021-07-11-diffusion-models/)
2. [Diffusion models made easy](https://towardsdatascience.com/diffusion-models-made-easy-8414298ce4da) - J. Rafid.
# Graph Theory
## Graph/GNN courses
1. [machine learning with graphs by Stanford](http://web.stanford.edu/class/cs224w/?fbclid=IwAR0nQR4lhyKCoTchsGQrcZ5E8EPBt2Bi4d8K8MYX-UN0ygQSxQ5bMoohhis), from ML to GNN.
2. [Graph deep learning course](https://geometricdeeplearning.com/lectures/) - graphs, sets, groups, GNNs. [youtube](https://www.youtube.com/watch?app=desktop\&v=w6Pw4MOzMuo)
## Graph Topics
1. [**General purpose and community detection GIT**](https://github.com/benedekrozemberczki/karateclub) **karate club bene**
2. **Connectivity**
3. **Min-cut:** [**1**](https://github.com/gsw73/min-cut/blob/master/karger_min_cut.py)**,** [**2**](https://github.com/ChuntaoLu/Algorithms-Design-and-Analysis/blob/master/week3%20Karger%20min%20cut/min_cut.py)**,** [**3**](https://github.com/WithaK16/kargerMinCut/blob/master/kargerMinCut.py)**, 4, 5, 6**
4. [**Louvain community**](https://github.com/taynaud/python-louvain/)
5. **Girwan newman** [**gist**](https://gist.github.com/chelsea1992/6c725a24d358763097bebe8223c2014a)**,** [**this worked**](https://github.com/ZwEin27/Community-Detection)**, t**[**his is potentially good too**](https://github.com/riteshkasat/Community-Detection-Algorithm)**,** [**another**](https://github.com/ServiceCutter/girvan-newman)**,** [**another**](https://github.com/ZwEin27/Community-Detection)**,** [**another**](https://github.com/kjahan/community)
6. [**Node2vec**](https://github.com/eliorc/Medium/blob/master/Nod2Vec-FIFA17-Example.ipynb)**,** [**paper**](https://arxiv.org/pdf/1607.00653.pdf)**,** [**medium1**](https://towardsdatascience.com/think-your-data-different-ddc435f70850)**,** [**medium 2**](https://towardsdatascience.com/node2vec-embeddings-for-graph-data-32a866340fef) **- tutorial -** [**code**](https://github.com/eliorc/node2vec)**,** [**git code**](https://github.com/eliorc/Medium/blob/master/Nod2Vec-FIFA17-Example.ipynb)**,** [**original py2 code**](https://github.com/aditya-grover/node2vec)**,** [**taboola code for their medium paper**](https://github.com/taboola/node2vec-example/blob/master/node2vec.ipynb)
7. [**Evaluation metrics for community detection**](https://stackoverflow.com/questions/28952104/evaluation-metrics-for-community-detection-algorithms)
8. [**Review for community detection algorithms**](https://arxiv.org/pdf/0906.0612.pdf) **-** [**paper**](https://arxiv.org/abs/0906.0612)
9. [**Term: community structure**](https://en.wikipedia.org/wiki/Community_structure#Algorithms_for_finding_communities)
10. [**Term: modularity of networks**](https://en.wikipedia.org/wiki/Modularity_%28networks%29)
11. [**Unread paper**](http://science.sciencemag.org/content/328/5980/876)
12. [**Unread comparison of community detection algos**](https://arxiv.org/abs/1406.2205)
13. [**Clustering adjacency matrices**](https://stats.stackexchange.com/questions/125295/the-best-way-for-clustering-an-adjacency-matrix)
14. [**Spectral-clustering**](https://calculatedcontent.com/2012/10/09/spectral-clustering/) **(is this suppose to be here?)**
15. [**Finding natural groups in undirected graphs**](https://stats.stackexchange.com/questions/142297/finding-natural-groups-clusters-in-an-undirected-graph-over-several-undirect)
16. [**Awesome community detection on github**](https://github.com/benedekrozemberczki/awesome-community-detection?fbclid=IwAR3Ab2oh_skVqwUP6xOh-3G_t715eyPESzGhHQIVRogRFHK0SZ6dzoublqE)
17. [**Various algorithms**](https://neo4j.com/docs/graph-algorithms/current/algorithms/closeness-centrality/)
[**5. Centrality algorithms**](https://neo4j.com/docs/graph-algorithms/current/algorithms/centrality/)
1. [**5.1. The PageRank algorithm**](https://neo4j.com/docs/graph-algorithms/current/algorithms/page-rank/)
2. [**5.2. The Betweenness Centrality algorithm**](https://neo4j.com/docs/graph-algorithms/current/algorithms/betweenness-centrality/)
3. [**5.3. The Closeness Centrality algorithm**](https://neo4j.com/docs/graph-algorithms/current/algorithms/closeness-centrality/)
4. [**5.4. The Degree Centrality algorithm**](https://neo4j.com/docs/graph-algorithms/current/algorithms/degree-centrality/)
[**6. Community detection algorithms**](https://neo4j.com/docs/graph-algorithms/current/algorithms/community/)
1. [**6.1. The Louvain algorithm**](https://neo4j.com/docs/graph-algorithms/current/algorithms/louvain/)
2. [**6.2. The Label Propagation algorithm**](https://neo4j.com/docs/graph-algorithms/current/algorithms/label-propagation/)
3. [**6.3. The Connected Components algorithm**](https://neo4j.com/docs/graph-algorithms/current/algorithms/connected-components/)
[**7. Experimental algorithms**](https://neo4j.com/docs/graph-algorithms/current/experimental-algorithms/)
1. [**7.1. Procedures**](https://neo4j.com/docs/graph-algorithms/current/experimental-procedures/)
2. [**7.2. Centrality algorithms**](https://neo4j.com/docs/graph-algorithms/current/experimental-algorithms/centrality/)
3. [**7.3. Community detection algorithms**](https://neo4j.com/docs/graph-algorithms/current/experimental-algorithms/community/)
4. [**7.4. Path finding algorithms**](https://neo4j.com/docs/graph-algorithms/current/experimental-algorithms/pathfinding/)
5. [**7.5. Similarity algorithms**](https://neo4j.com/docs/graph-algorithms/current/experimental-algorithms/similarity/)
6. [**7.6. Link Prediction algorithms**](https://neo4j.com/docs/graph-algorithms/current/experimental-algorithms/linkprediction/)
7. [**7.7. Preprocessing functions and**](https://neo4j.com/docs/graph-algorithms/current/experimental-algorithms/preprocessing/)
## Graph Tools
1. [Graph-tool](https://graph-tool.skewed.de/) is an efficient Python module for manipulation and statistical analysis of graphs
2. [NetworkX](https://github.com/networkx/networkx) is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.
# Social Network Analysis
1. [**Wiki**](https://en.wikipedia.org/wiki/Social_network)
2. [**Paper: algorithmic approach to social networks**](http://www.cs.carleton.edu/faculty/dlibenno/papers/thesis/thesis.pdf)
3. [**Steve borgatti**](https://sites.google.com/site/steveborgatti/home)
4. [**Intro to SNA**](http://www.orgnet.com/sna.html)
1. **Centrality**
2. **Betweenness centrality**
3. **Network centralization**
4. **Network reach**
5. **Network integration**
6. **Boundary spanners**
7. **Peripheral players**
5. [**Social Network Analysis: Can Quantity Compensate for Quality?**](https://33bits.wordpress.com/2009/02/15/social-network-analysis-can-quantity-substitute-for-quality/)
6. [**Nicholas Christakis**](http://www.wjh.harvard.edu/soc/faculty/christakis/) **of Harvard and** [**James Fowler**](http://jhfowler.ucsd.edu/) **of UC San Diego have produced a series of ground-breaking papers analyzing the spread of various traits in social networks:** [**obesity**](http://content.nejm.org/cgi/content/full/357/4/370)**,** [**smoking**](http://content.nejm.org/cgi/content/full/358/21/2249)**,** [**happiness**](http://www.bmj.com/cgi/content/full/337/dec04_2/a2338)**, and most recently, in collaboration with John Cacioppo,** [**loneliness**](http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1319108)**. The Christakis-Fowler collaboration has now become** [**well-known**](http://jhfowler.ucsd.edu/science_friendship_as_a_health_factor.pdf)**, but from a technical perspective, what was special about their work?**\
\
**It turns out that they found a way to distinguish between the three reasons why people who are related in a social network are similar to each other.**\
\
**Homophily is the tendency of people to seek others who are alike. For example, most of us restrict our dates to smokers or non-smokers, mirroring our own behavior.**\
\
**Confounding is the phenomenon of related individuals developing a trait because of a (shared) environmental circumstance. For example, people living right next to a McDonald’s might all gradually become obese.**\
\
**Induction is the process of one individual passing a trait or behavior on to their friends, whether by active encouragement or by setting an example**
7. [**Networkx**](https://networkx.org/documentation//networkx-1.10/reference/algorithms.html) **- Centrality is just a fraction of the algorithms contained in networkx.**
8. [**Social Network analysis from theory to applications**](https://towardsdatascience.com/social-network-analysis-from-theory-to-applications-with-python-d12e9a34c2c7) **-** [**dima goldenberg**](https://www.linkedin.com/in/dimgold/)
# Deep Neural Nets Basics
## Perceptron
1. [perceptron](https://towardsdatascience.com/perceptrons-logical-functions-and-the-xor-problem-37ca5025790a) - logical functions and XOR
2. The chain rule
1. [mastery on the chain rule for multi and univariate functions](https://machinelearningmastery.com/the-chain-rule-of-calculus-for-univariate-and-multivariate-functions/)
2. [derivative of a sigmoid](https://towardsdatascience.com/understanding-the-derivative-of-the-sigmoid-function-cbfd46fb3716)
3. [derivative for ML people](https://towardsdatascience.com/a-quick-introduction-to-derivatives-for-machine-learning-people-3cd913c5cf33)
3. [Step by step backpropagation example](https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/)
4. [understanding backprop](https://towardsdatascience.com/understanding-backpropagation-algorithm-7bb3aa2f95fd)
## DNN
* [**Deep learning notes from Andrew NG’s course.**](https://www.slideshare.net/TessFerrandez/notes-from-coursera-deep-learning-courses-by-andrew-ng)
* **Jay Alammar on NN** [**Part 1**](http://jalammar.github.io/visual-interactive-guide-basics-neural-networks/)**,** [**Part 2**](http://jalammar.github.io/feedforward-neural-networks-visual-interactive/)
* [**NN in general**](http://briandolhansky.com/blog/?tag=neural+network#show-archive) **- 5 introductions tutorials.**
* [**Segmentation examples**](https://meetshah1995.github.io/semantic-segmentation/deep-learning/pytorch/visdom/2017/06/01/semantic-segmentation-over-the-years.html)
**MLP: fully connected, input, hidden layers, output. Gradient on the backprop takes a lot of time to calculate. Has vanishing gradient problem, because of multiplications when it reaches the first layers the loss correction is very small (0.1\*0.1\*01 = 0.001), therefore the early layers train slower than the last ones, and the early ones capture the basics structures so they are the more important ones.**
**AutoEncoder - unsupervised, drives the input through fully connected layers, sometime reducing their neurons amount, then does the reverse and expands the layer’s size to get to the input (images are multiplied by the transpose matrix, many times over), Comparing the predicted output to the input, correcting the cost using gradient descent and redoing it, until the networks learns the output.**
* **Convolutional auto encoder**
* **Denoiser auto encoder - masking areas in order to create an encoder that understands noisy images**
* **Variational autoencoder - doesnt rely on distance between pixels, rather it maps them to a function (gaussian), eventually the DS should be explained by this mapping, uses 2 new layers added to the network. Gaussian will create blurry images, but similar. Please note that it also works with CNN.**
**What are** [**logits**](https://stackoverflow.com/questions/41455101/what-is-the-meaning-of-the-word-logits-in-tensorflow) **in neural net - the vector of raw (non-normalized) predictions that a classification model generates, which is ordinarily then passed to a normalization function. If the model is solving a multi-class classification problem, logits typically become an input to the softmax function. The softmax function then generates a vector of (normalized) probabilities with one value for each possible class.**
[**WORD2VEC**](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/) **- based on autoencode, we keep only the hidden layer ,** [**Part 2**](http://mccormickml.com/2017/01/11/word2vec-tutorial-part-2-negative-sampling/)
**RBM- restricted (no 2 nodes share a connection) boltzman machine**
**An Autoencoder of features, tries to encode its own structure.**
**Works best on pics, video, voice, sensor data. 2 layers, visible and hidden, error and bias calculated via KL Divergence.**
* **Also known as a shallow network.**
* **Two layers, input and output, goes back and forth until it learns its output.**
**DBN - deep belief networks, similar structure to multi layer perceptron. fully connected, input, hidden(s), output layers. Can be thought of as stacks of RBM. training using GPU optimization, accurate and needs smaller labelled data set to complete the training.**
**Solves the ‘vanishing gradient’ problem, imagine a fully connected network, advancing each 2 layers step by step until each boltzman network (2 layers) learns the output, keeps advancing until finished.. Each layer learns the entire input.**
**Next step is to fine tune using a labelled test set, improves performance and alters the net. So basically using labeled samples we fine tune and associate features and pattern with a name. Weights and biases are altered slightly and there is also an increase in performance. Unlike CNN which learns features then high level features.**
**Accurate and reasonable in time, unlike fully connected that has the vanishing gradient problem.**
**Transfer Learning = like Inception in Tensor flow, use a prebuilt network to solve many problems that “work” similarly to the original network.**
* [**CS course definition**](http://cs231n.github.io/transfer-learning/) **- also very good explanation of the common use cases:**
* **Feature extraction from the CNN part (removing the fully connected layer)**
* **Fine-tuning, everything or partial selection of the hidden layers, mainly good to keep low level neurons that know what edges and color blobs are, but not dog breeds or something not as general.**
* [**CNN checkpoints**](https://github.com/BVLC/caffe/wiki/Model-Zoo#cascaded-fully-convolutional-networks-for-biomedical-image-segmentation) **for many problems with transfer learning. Has several relevant references**
* **Such as this “**[**How transferable are features in deep neural networks?**](http://arxiv.org/abs/1411.1792) **“**
* **(the indian guy on facebook)** [**IMDB transfer learning using cnn vgg and word2vec**](https://spandan-madan.github.io/DeepLearningProject/)**, the word2vec is interesting, the cnn part is very informative. With python code, keras.**
**CNN, Convolutional Neural Net (**[**this link explains CNN quite well**](https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/)**,** [**2nd tutorial**](https://hackernoon.com/deep-learning-cnns-in-tensorflow-with-gpus-cba6efe0acc2) **- both explain about convolution, padding, relu - sparsity, max and avg pooling):**
* **Common Layers: input->convolution->relu activation->pooling to reduce dimensionality \*\*\*\* ->fully connected layer**
* **\*\*\*\*repeat several times over as this discover patterns but needs another layer -> fully connected layer**
* **Then we connect at the end a fully connected layer (fcl) to classify data samples.**
* **Good for face detection, images etc.**
* **Requires lots of data, not always possible in a real world situation**
* **Relu is quite resistant to vanishing gradient & allows for deactivating neurons and for sparsity.**
**RNN - what is RNN by Andrej Karpathy -** [**The Unreasonable Effectiveness of Recurrent Neural Networks**](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)**, basically a lot of information about RNNs and their usage cases**
* **basic NN node with a loop, previous output is merged with current input. for the purpose of remembering history, for time series, to predict the next X based on the previous Y.**
* **1 to N = frame captioning**
* **N to 1 = classification**
* **N to N = predict frames in a movie**
* **N\2 with time delay to N\2 = predict supply and demand**
* **Vanishing gradient is 100 times worse.**
* **Gate networks like LSTM solves vanishing gradient.**
[**SNN**](https://medium.com/@eliorcohen/selu-make-fnns-great-again-snn-8d61526802a9) **- SELU activation function is inside not outside, results converge better.**
**Probably useful for feedforward networks**
[**DEEP REINFORCEMENT LEARNING COURSE**](https://www.youtube.com/watch?v=QDzM8r3WgBw\&t=2958s) **(for motion planning)or**\
[**DEEP RL COURSE**](https://www.youtube.com/watch?v=PtAIh9KSnjo) **(Q-LEARNING?) - using unlabeled data, reward, and probably a CNN to solve games beyond human level.**
**A** [**brief survey of DL for Reinforcement learning**](https://arxiv.org/abs/1708.05866)
[**WIKI**](https://en.wikipedia.org/wiki/Recurrent_neural_network#Long_short-term_memory) **has many types of RNN networks (unread)**
**Unread and potentially good tutorials:**
1. [**deep learning python**](https://www.datacamp.com/community/tutorials/deep-learning-python)
**EXAMPLES of Using NN on images:**
[**Deep image prior / denoiser/ high res/ remove artifacts/ etc..**](https://dmitryulyanov.github.io/deep_image_prior)
## **GRADIENT DESCENT**
**(**[**What are**](http://machinelearningmastery.com/gentle-introduction-mini-batch-gradient-descent-configure-batch-size/)**?) batch, stochastic, and mini-batch gradient descent are and the benefits and limitations of each method.**
[**What is gradient descent, how to use it, local minima okay to use, compared to global. Saddle points, learning rate strategies and research points**](https://blog.paperspace.com/intro-to-optimization-in-deep-learning-gradient-descent/)
1. **Gradient descent is an optimization algorithm often used for finding the weights or coefficients of machine learning algorithms, such as artificial neural networks and logistic regression.**
2. **the model makes predictions on training data, then use the error on the predictions to update the model to reduce the error.**
3. **The goal of the algorithm is to find model parameters (e.g. coefficients or weights) that minimize the error of the model on the training dataset. It does this by making changes to the model that move it along a gradient or slope of errors down toward a minimum error value. This gives the algorithm its name of “gradient descent.”**
### **Stochastic**
* **calculate error and updates the model after every training sample**
### **Batch**
* **calculates the error for each example in the training dataset, but only updates the model after all training examples have been evaluated.**
### **Mini batch (most common)**
* **splits the training dataset into small batches, used to calculate model error and update model coefficients.**
* **Implementations may choose to sum the gradient over the mini-batch or take the average of the gradient (reduces variance of gradient) (unclear?)**
**+ Tips on how to choose and train using mini batch in the link above**
[**Dont decay the learning rate, increase batchsize - paper**](https://arxiv.org/abs/1711.00489) **(optimization of a network)**
* [**Big batches are not the cause for the ‘generalization gap’ between mini and big batches, it is not advisable to use large batches because of the low update rate, however if you change that, authors claim its okay**](https://arxiv.org/abs/1705.08741)**.**
* [**So what is a batch size in NN (another source)**](https://stats.stackexchange.com/questions/153531/what-is-batch-size-in-neural-network) **- and how to find the “right” number. In general terms a good mini bach between 1 and all samples is a good idea. Figure it out empirically.**
* **one epoch = one forward pass and one backward pass of all the training examples**
* **batch size = the number of training examples in one forward/backward pass. The higher the batch size, the more memory space you'll need.**
* **number of iterations = number of passes, each pass using \[batch size] number of examples. To be clear, one pass = one forward pass + one backward pass (we do not count the forward pass and backward pass as two different passes).**
**Example: if you have 1000 training examples, and your batch size is 500, then it will take 2 iterations to complete 1 epoch.**
* [**How to balance and what is the tradeoff between batch size and the number of iterations.**](https://stats.stackexchange.com/questions/164876/tradeoff-batch-size-vs-number-of-iterations-to-train-a-neural-network)
[**GD with Momentum**](https://towardsdatascience.com/stochastic-gradient-descent-with-momentum-a84097641a5d) **- explain**
## **Batch size**
**(**[**a good read)**](https://machinelearningmastery.com/use-different-batch-sizes-training-predicting-python-keras/) **about batch sizes in keras, specifically LSTM, read this first!**
**A sequence prediction problem makes a good case for a varied batch size as you may want to have a batch size equal to the training dataset size (batch learning) during training and a batch size of 1 when making predictions for one-step outputs.**
**power of 2: have some advantages with regards to vectorized operations in certain packages, so if it's close it might be faster to keep your batch\_size in a power of 2.**
**(**[**pushing batches of samples to memory in order to train)**](https://stats.stackexchange.com/questions/153531/what-is-batch-size-in-neural-network) **-**
**Batch size defines number of samples that going to be propagated through the network.**
**For instance, let's say you have 1050 training samples and you want to set up batch\_size equal to 100. Algorithm takes first 100 samples (from 1st to 100th) from the training dataset and trains network. Next it takes second 100 samples (from 101st to 200th) and train network again. We can keep doing this procedure until we will propagate through the networks all samples. The problem usually happens with the last set of samples. In our example we've used 1050 which is not divisible by 100 without remainder. The simplest solution is just to get final 50 samples and train the network.**
**Advantages:**
* **It requires less memory. Since you train network using less number of samples the overall training procedure requires less memory. It's especially important in case if you are not able to fit dataset in memory.**
* **Typically networks trains faster with mini-batches. That's because we update weights after each propagation. In our example we've propagated 11 batches (10 of them had 100 samples and 1 had 50 samples) and after each of them we've updated network's parameters. If we used all samples during propagation we would make only 1 update for the network's parameter.**
**Disadvantages:**
* **The smaller the batch the less accurate estimate of the gradient. In the figure below you can see that mini-batch (green color) gradient's direction fluctuates compare to the full batch (blue color).**
[**Small batch size has an effect on validation accuracy.**](http://forums.fast.ai/t/batch-size-effect-on-validation-accuracy/413)
**IMPORTANT: batch size in ‘.prediction’ is needed for some models,** [**only for technical reasons as seen here**](https://github.com/fchollet/keras/issues/3027)**, in keras.**
1. **(**[**unread**](https://www.quora.com/Intuitively-how-does-mini-batch-size-affect-the-performance-of-stochastic-gradient-descent)**) about mini batches and performance.**
2. **(**[**unread**](https://stats.stackexchange.com/questions/164876/tradeoff-batch-size-vs-number-of-iterations-to-train-a-neural-network)**) tradeoff between bath size and number of iterations**
[**Another observation, probably empirical**](https://stackoverflow.com/questions/35050753/how-big-should-batch-size-and-number-of-epochs-be-when-fitting-a-model-in-keras) **- to answer your questions on Batch Size and Epochs:**
**In general: Larger batch sizes result in faster progress in training, but don't always converge as fast. Smaller batch sizes train slower, but can converge faster. It's definitely problem dependent.**
**In general, the models improve with more epochs of training, to a point. They'll start to plateau in accuracy as they converge. Try something like 50 and plot number of epochs (x axis) vs. accuracy (y axis). You'll see where it levels out.**
## **BIAS**
[**The role of bias in NN**](https://stackoverflow.com/questions/2480650/role-of-bias-in-neural-networks) **- similarly to the ‘b’ in linear regression.**
## **BATCH NORMALIZATION**
1. **The** [**best explanation**](https://blog.paperspace.com/busting-the-myths-about-batch-normalization/) **to what is BN and why to use it, including busting the myth that it solves internal covariance shift - shifting input distribution, and saying that it should come after activations as it makes more sense (it does),also a nice quote on where a layer ends is really good - it can end at the activation (or not). How to use BN in the test, hint: use a moving window. Bn allows us to use 2 parameters to control the input distribution instead of controlling all the weights.**
2. [**Medium on BN**](https://towardsdatascience.com/an-alternative-to-batch-normalization-2cee9051e8bc)
3. [**Medium on BN**](https://towardsdatascience.com/batch-normalization-theory-and-how-to-use-it-with-tensorflow-1892ca0173ad)
4. [**Ian goodfellow on BN**](https://www.youtube.com/watch?v=Xogn6veSyxA\&feature=youtu.be\&t=325)
5. [**Medium #2 - a better one on BN, and adding to VGG**](https://towardsdatascience.com/batch-normalization-in-neural-networks-1ac91516821c)
6. [**Reddit on BN, mainly on the paper saying to use it before, but best practice is to use after**](https://www.reddit.com/r/MachineLearning/comments/67gonq/d_batch_normalization_before_or_after_relu/)
7. [**Diff between batch and norm (weak explanation)**](https://www.quora.com/What-are-the-practical-differences-between-batch-normalization-and-layer-normalization-in-deep-neural-networks)
8. [**Weight normalization for keras and TF**](http://krasserm.github.io/2018/11/10/weightnorm-implementation-options/)
9. [**Layer normalization keras**](https://pypi.org/project/keras-layer-normalization/)
10. [**Instance normalization keras**](https://github.com/keras-team/keras-contrib/blob/master/keras_contrib/layers/normalization/instancenormalization.py)
11. [**batch/layer/instance in TF with code**](https://towardsdatascience.com/implementing-spatial-batch-instance-layer-normalization-in-tensorflow-manual-back-prop-in-tf-77faa8d2c362)
12. **Layer** [**norm for rnn’s or whatever name it is in this post**](https://twimlai.com/new-layer-normalization-technique-speeds-rnn-training/) **with** [**code**](https://gist.github.com/udibr/7f46e790c9e342d75dcbd9b1deb9d940) **for GRU**
[**What is the diff between batch/layer/recurrent batch and back rnn normalization**](https://datascience.stackexchange.com/questions/12956/paper-whats-the-difference-between-layer-normalization-recurrent-batch-normal)
* **Layer normalization (Ba 2016): Does not use batch statistics. Normalize using the statistics collected from all units within a layer of the current sample. Does not work well with ConvNets.**
* **Recurrent Batch Normalization (BN) (Cooijmans, 2016; also proposed concurrently by Qianli Liao & Tomaso Poggio, but tested on Recurrent ConvNets, instead of RNN/LSTM): Same as batch normalization. Use different normalization statistics for each time step. You need to store a set of mean and standard deviation for each time step.**
* **Batch Normalized Recurrent Neural Networks (Laurent, 2015): batch normalization is only applied between the input and hidden state, but not between hidden states. i.e., normalization is not applied over time.**
* **Streaming Normalization (Liao et al. 2016) : it summarizes existing normalizations and overcomes most issues mentioned above. It works well with ConvNets, recurrent learning and online learning (i.e., small mini-batch or one sample at a time):**
* **Weight Normalization (Salimans and Kingma 2016): whenever a weight is used, it is divided by its L2 norm first, such that the resulting weight has L2 norm 1. That is, output y=x∗(w/|w|), where x and w denote the input and weight respectively. A scalar scaling factor g is then multiplied to the output y=y∗g. But in my experience g seems not essential for performance (also downstream learnable layers can learn this anyway).**
* **Cosine Normalization (Luo et al. 2017): weight normalization is very similar to cosine normalization, where the same L2 normalization is applied to both weight and input: y=(x/|x|)∗(w/|w|). Again, manual or automatic differentiation can compute appropriate gradients of x and w.**
* **Note that both Weight and Cosine Normalization have been extensively used (called normalized dot product) in the 2000s in a class of ConvNets called HMAX (Riesenhuber 1999) to model biological vision. You may find them interesting.**
[**More about Batch/layer/instance/group norm are different methods for normalizing the inputs to the layers of deep neural networks**](https://nealjean.com/ml/neural-network-normalization/)
1. **Layer normalization solves the rnn case that batch couldnt - Is done per feature within the layer and normalized features are replaced**
2. **Instance does it for (cnn?) using per channel normalization**
3. **Group does it for group of channels**
4. 
[**Part1: intuitive explanation to batch normalization**](http://mlexplained.com/2018/01/10/an-intuitive-explanation-of-why-batch-normalization-really-works-normalization-in-deep-learning-part-1/)
**Part2:** [**batch/layer/weight normalization**](http://mlexplained.com/2018/01/13/weight-normalization-and-layer-normalization-explained-normalization-in-deep-learning-part-2/) **- This is a good resource for advantages for every layer**
* **Layer, per feature in a batch,**
* **weight - divided by the norm**
## **HYPER PARAM GRID SEARCHES**
1. [**A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay**](https://arxiv.org/abs/1803.09820)
## **LOSS**
[**Very Basic advice**](https://stats.stackexchange.com/questions/232754/reference-to-learn-how-to-interpret-learning-curves-of-deep-convolutional-neural)**: You should probably switch train/validation repartition to something like 80% training and 20% validation. In most cases it will improve the classifier performance overall (more training data = better performance)**
**+If Training error and test error are too close (your system is unable to overfit on your training data), this means that your model is too simple. Solution: more layers or more neurons per layer.**
**Early stopping**
**If you have never heard about "early-stopping" you should look it up, it's an important concept in the neural network domain :** [**https://en.wikipedia.org/wiki/Early\_stopping**](https://en.wikipedia.org/wiki/Early_stopping) **. To summarize, the idea behind early-stopping is to stop the training once the validation loss starts plateauing. Indeed, when this happens it almost always mean you are starting to overfitt your classifier. The training loss value in itself is not something you should trust, beacause it will continue to increase event when you are overfitting your classifier.**
**With** [**cross entropy**](https://www.quora.com/Loss-cross-entropy-is-decreasing-but-accuracy-remains-the-same-while-training-convolutional-neural-networks-How-can-it-happen) **there can be an issue where the accuracy is the same for two cases, one where the loss is decreasing and the other when the loss is not changing much.**
[**How to read LOSS graphs (and accuracy on top)**](https://github.com/fchollet/keras/issues/3755)
 
**This indicates that the model is overfitting. It continues to get better and better at fitting the data that it sees (training data) while getting worse and worse at fitting the data that it does not see (validation data).**
[**This is a very good example of a train/test loss and an accuracy behavior.**](https://machinelearningmastery.com/display-deep-learning-model-training-history-in-keras/)
 
[**Cross entropy formula with soft labels (probability) rather than classes.**](https://stats.stackexchange.com/questions/206925/is-it-okay-to-use-cross-entropy-loss-function-with-soft-labels)
[**Mastery on cross entropy, brier, roc auc, how to ‘game’ them and calibrate them**](https://machinelearningmastery.com/how-to-score-probability-predictions-in-python/)
[**Game changer paper - a general adaptive loss search in nn**](https://www.reddit.com/r/computervision/comments/bsd82j/a_general_and_adaptive_robust_loss_function/?utm_medium=android_app\&utm_source=share)
## **LEARNING RATE REDUCTION**
[**Intro to Learning Rate methods**](https://medium.com/@chengweizhang2012/quick-notes-on-how-to-choose-optimizer-in-keras-9d3d12d09039) **- what they are doing and what they are fixing in other algos.**
[**Callbacks**](https://keras.io/callbacks/)**, especially ReduceLROnPlateau - this callback monitors a quantity and if no improvement is seen for a 'patience' number of epochs, the learning rate is reduced.**
[**Cs123**](http://cs231n.github.io/neural-networks-3/) **(very good): explains about many things related to CNN, but also about LR and adaptive methods.**
[**An excellent comparison of several learning rate schedule methods and adaptive methods:**](https://medium.com/towards-data-science/learning-rate-schedules-and-adaptive-learning-rate-methods-for-deep-learning-2c8f433990d1) **(**[**same here but not as good**](https://machinelearningmastery.com/using-learning-rate-schedules-deep-learning-models-python-keras/)**)**
**Adaptive gradient descent algorithms such as** [**Adagrad**](https://en.wikipedia.org/wiki/Stochastic_gradient_descent#AdaGrad)**, Adadelta,** [**RMSprop**](https://en.wikipedia.org/wiki/Stochastic_gradient_descent#RMSProp)**,** [**Adam**](https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Adam)**, provide an alternative to classical SGD.**
**These per-parameter learning rate methods provide heuristic approach without requiring expensive work in tuning hyperparameters for the learning rate schedule manually.**
1. **Adagrad performs larger updates for more sparse parameters and smaller updates for less sparse parameter. It has good performance with sparse data and training large-scale neural network. However, its monotonic learning rate usually proves too aggressive and stops learning too early when training deep neural networks.**
2. **Adadelta is an extension of Adagrad that seeks to reduce its aggressive, monotonically decreasing learning rate.**
3. **RMSprop adjusts the Adagrad method in a very simple way in an attempt to reduce its aggressive, monotonically decreasing learning rate.**
4. [**Adam**](https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/) **is an update to the RMSProp optimizer which is like RMSprop with momentum.**
**adaptive learning rate methods demonstrate better performance than learning rate schedules, and they require much less effort in hyperparamater settings**
[**Recommended paper**](https://arxiv.org/pdf/1206.5533v2.pdf)**: practical recommendation for gradient based DNN**
**Another great comparison -** [**pdf paper**](https://arxiv.org/abs/1609.04747) **and** [**webpage link**](http://ruder.io/optimizing-gradient-descent/) **-**
* **if your input data is sparse, then you likely achieve the best results using one of the adaptive learning-rate methods.**
* **An additional benefit is that you will not need to tune the learning rate but will likely achieve the best results with the default value.**
* **In summary, RMSprop is an extension of Adagrad that deals with its radically diminishing learning rates. It is identical to Adadelta, except that Adadelta uses the RMS of parameter updates in the numerator update rule. Adam, finally, adds bias-correction and momentum to RMSprop. Insofar, RMSprop, Adadelta, and Adam are very similar algorithms that do well in similar circumstances. Kingma et al. \[10] show that its bias-correction helps Adam slightly outperform RMSprop towards the end of optimization as gradients become sparser. Insofar, Adam might be the best overall choice**
## **TRAIN / VAL accuracy in NN**
**The second important quantity to track while training a classifier is the validation/training accuracy. This plot can give you valuable insights into the amount of overfitting in your model:**
* **The gap between the training and validation accuracy indicates the amount of overfitting.**
* **Two possible cases are shown in the diagram on the left. The blue validation error curve shows very small validation accuracy compared to the training accuracy, indicating strong overfitting (note, it's possible for the validation accuracy to even start to go down after some point).**
* **NOTE: When you see this in practice you probably want to increase regularization:**
* **stronger L2 weight penalty**
* **Dropout**
* **collect more data.**
* **The other possible case is when the validation accuracy tracks the training accuracy fairly well. This case indicates that your model capacity is not high enough: make the model larger by increasing the number of parameters.**
## **INITIALIZERS**
**XAVIER GLOROT:**
[**Why’s Xavier initialization important?**](http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization)
**In short, it helps signals reach deep into the network.**
* **If the weights in a network start too small, then the signal shrinks as it passes through each layer until it’s too tiny to be useful.**
* **If the weights in a network start too large, then the signal grows as it passes through each layer until it’s too massive to be useful.**
**Xavier initialization makes sure the weights are ‘just right’, keeping the signal in a reasonable range of values through many layers.**
**To go any further than this, you’re going to need a small amount of statistics - specifically you need to know about random distributions and their variance.**
[**When to use glorot uniform-over-normal initialization?**](https://datascience.stackexchange.com/questions/13061/when-to-use-he-or-glorot-normal-initialization-over-uniform-init-and-what-are)
**However, i am still not seeing anything empirical that says that glorot surpesses everything else under certain conditions (**[**except the glorot paper**](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)**), most importantly, does it really help in LSTM where the vanishing gradient is \~no longer an issue?**
[**He-et-al Initialization**](https://towardsdatascience.com/random-initialization-for-neural-networks-a-thing-of-the-past-bfcdd806bf9e)
**This method of initializing became famous through a paper submitted in 2015 by He et al, and is similar to Xavier initialization, with the factor multiplied by two. In this method, the weights are initialized keeping in mind the size of the previous layer which helps in attaining a global minimum of the cost function faster and more efficiently.**
**w=np.random.randn(layer\_size\[l],layer\_size\[l-1])\*np.sqrt(2/layer\_size\[l-1])**
## **ACTIVATION FUNCTIONS**
1. [**a bunch of observations, seems like a personal list**](http://sentiment-mining.blogspot.co.il/2015/08/the-difference-of-activation-function.html) **-**
1. **Output layer - linear for regression, softmax for classification**
2. **Hidden layers - hyperbolic tangent for shallow networks (less than 3 hidden layers), and ReLU for deep networks**
2. **ReLU - The purpose of ReLU is to introduce non-linearity, since most of the real-world data we would want our network to learn would be nonlinear (e.g. convolution is a linear operation – element wise matrix multiplication and addition, so we account for nonlinearity by introducing a nonlinear function like ReLU, e.g** [**here**](https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/) **- search for ReLU).**
1. **Relu is quite resistant to vanishing gradient & allows for deactivating neurons and for sparsity.**
2. **Other nonlinear functions such as tanh or sigmoid can also be used instead of ReLU, but ReLU has been found to perform better in most situations.**
3. [**Visual + description of activation functions**](https://medium.com/the-theory-of-everything/understanding-activation-functions-in-neural-networks-9491262884e0)
4. [**A very good explanation + figures about activations functions**](https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6)
5. [**Selu**](https://towardsdatascience.com/selu-make-fnns-great-again-snn-8d61526802a9) **- better than RELU? Possibly.**
6. [**Mish**](https://github.com/digantamisra98/Mish)**: A Self Regularized Non-Monotonic Neural Activation Function,** [**yam peleg’s code**](https://gist.github.com/ypeleg/3af35d07d7f659f387952c9843849772?fbclid=IwAR2x_Hzlg79_mo_zQMJGFbQWORbpdydnllnHoA_RmUlCLpqKdGwClBuJy8g)
7. [**Mish, Medium, Keras Code, with benchmarks, computationally expensive.**](https://towardsdatascience.com/mish-8283934a72df)
8. [Gelu](https://paperswithcode.com/method/gelu#:~:text=The%20Gaussian%20Error%20Linear%20Unit,x%201%20x%20%3E%200%20\).) (Used by OpenAI
9. [Deep Learning 101: Transformer Activation Functions Explainer - Sigmoid, ReLU, GELU, Swish](https://www.saltdatalabs.com/blog/deep-learning-101-transformer-activation-functions-explainer-relu-leaky-relu-gelu-elu-selu-softmax-and-more#:~:text=ReLU%20vs%20GelU,easy%20to%20optimize%20during%20training.)
## **OPTIMIZERS**
**There are several optimizers, each had his 15 minutes of fame, some optimizers are recommended for CNN, Time Series, etc..**
**There are also what I call ‘experimental’ optimizers, it seems like these pop every now and then, with or without a formal proof. It is recommended to follow the literature and see what are the ‘supposedly’ state of the art optimizers atm.**
[**Adamod**](https://medium.com/@lessw/meet-adamod-a-new-deep-learning-optimizer-with-memory-f01e831b80bd) **deeplearning optimizer with memory**
[**Backstitch**](http://www.danielpovey.com/files/2017_nips_backstitch.pdf) **- September 17 - supposedly an improvement over SGD for speech recognition using DNN. Note: it wasnt tested with other datasets or other network types.**
**(how does it work?) take a negative step back, then a positive step forward. I.e., When processing a minibatch, instead of taking a single SGD step, we first take a step with −α times the current learning rate, for α > 0 (e.g. α = 0.3), and then a step with 1 + α times the learning rate, with the same minibatch (and a recomputed gradient). So we are taking a small negative step, and then a larger positive step. This resulted in quite large improvements – around 10% relative improvement \[37] – for our best speech recognition DNNs. The recommended hyper parameters are in the paper.**
**Drawbacks: takes twice to train, momentum not implemented or tested, dropout is mandatory for improvement, slow starter.**
[**Documentation about optimizers**](https://keras.io/optimizers/) **in keras**
* **SGD can be fine tuned**
* **For others Leave most parameters as they were**
[**Best description on optimizers with momentum etc, from sgd to nadam, formulas and intuition**](https://towardsdatascience.com/10-gradient-descent-optimisation-algorithms-86989510b5e9)
## **DROPOUT LAYERS IN KERAS AND GENERAL**
[**A very influential paper about dropout and how beneficial it is - bottom line always use it.**](http://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf)
**OPEN QUESTIONs:**
1. **does a dropout layer improve performance even if an lstm layer has dropout or recurrent dropout.**
2. **What is the diff between a separate layer and inside the lstm layer.**
3. **What is the diff in practice and intuitively between drop and recurrentdrop**
[**Dropout layers in keras, or dropout regularization:**](https://machinelearningmastery.com/dropout-regularization-deep-learning-models-keras/)
* **Dropout is a technique where randomly selected neurons are ignored RANDOMLY during training.**
* **contribution to the activation of downstream neurons is temporally removed on the forward pass and any weight updates are not applied to the neuron on the backward pass.**
* **As a neural network learns, neuron weights settle into their context within the network.**
* **Weights of neurons are tuned for specific features providing some specialization. Neighboring neurons become to rely on this specialization, which if taken too far can result in a fragile model too specialized to the training data. (overfitting)**
* **This reliant on context for a neuron during training is referred to complex co-adaptations.**
* **After dropout, other neurons will have to step in and handle the representation required to make predictions for the missing neurons, which is believed to result in multiple independent internal representations being learned by the network.**
* **Thus, the effect of dropout is that the network becomes less sensitive to the specific weights of neurons.**
* **This in turn leads to a network with better generalization capability and less likely to overfit the training data.**
[**Another great answer about drop out**](https://www.quora.com/In-Keras-what-is-a-dense-and-a-dropout-layer) **-**
* **as a consequence of the 50% dropout, the neural network will learn different, redundant representations; the network can’t rely on the particular neurons and the combination (or interaction) of these to be present.**
* **Another nice side effect is that training will be faster.**
* **Rules:**
* **Dropout is only applied during training,**
* **Need to rescale the remaining neuron activations. E.g., if you set 50% of the activations in a given layer to zero, you need to scale up the remaining ones by a factor of 2.**
* **if the training has finished, you’d use the complete network for testing (or in other words, you set the dropout probability to 0).**
[**Implementation of drop out in keras**](https://datascience.stackexchange.com/questions/18088/convolutional-layer-dropout-layer-in-keras/18098) **is “inverse dropout” - n the Keras implementation, the output values are corrected during training (by dividing, in addition to randomly dropping out the values) instead of during testing (by multiplying). This is called "inverted dropout".**
**Inverted dropout is functionally equivalent to original dropout (as per your link to Srivastava's paper), with a nice feature that the network does not use dropout layers at all during test and prediction. This is explained a little in this** [**Keras issue**](https://github.com/fchollet/keras/issues/3305)**.**
[**Dropout notes and rules of thumb aka “best practice” -**](http://blog.mrtanke.com/2016/10/09/Keras-Study-Notes-3-Dropout-Regularization-for-Deep-Networks/)
* **dropout value of 20%-50% of neurons with 20% providing a good starting point. (A probability too low has minimal effect and a value too high results in underlearning by the network.)**
* **Use a large network for better performance, i.e., when dropout is used on a larger network, giving the model more of an opportunity to learn independent representations.**
* **Use dropout on VISIBLE AND HIDDEN. Application of dropout at each layer of the network has shown good results.**
* **Unclear ? Use a large learning rate with decay and a large momentum. Increase your learning rate by a factor of 10 to 100 and use a high momentum value of 0.9 or 0.99.**
* **Unclear ? Constrain the size of network weights. A large learning rate can result in very large network weights. Imposing a constraint on the size of network weights such as max-norm regularization with a size of 4 or 5 has been shown to improve results.**
[**Difference between LSTM ‘dropout’ and ‘recurrent\_dropout’**](https://stackoverflow.com/questions/44924690/keras-the-difference-between-lstm-dropout-and-lstm-recurrent-dropout) **- vertical vs horizontal.**
**I suggest taking a look at (the first part of)** [**this paper**](https://arxiv.org/pdf/1512.05287.pdf)**. Regular dropout is applied on the inputs and/or the outputs, meaning the vertical arrows from x\_t and to h\_t. In you add it as an argument to your layer, it will mask the inputs; you can add a Dropout layer after your recurrent layer to mask the outputs as well. Recurrent dropout masks (or "drops") the connections between the recurrent units; that would be the horizontal arrows in your picture.**
**This picture is taken from the paper above. On the left, regular dropout on inputs and outputs. On the right, regular dropout PLUS recurrent dropout:**
## **NEURAL NETWORK OPTIMIZATION TECHNIQUES**
**Basically do these after you have a working network**
1. [**Dont decay the learning rate, increase batchsize - paper**](https://arxiv.org/abs/1711.00489) **(optimization of a network)**
2. [**Add one neuron with skip connection, or to every layer in a binary classification network to get global minimum**](https://arxiv.org/abs/1805.08671)**.**
3. \*\*\*\*[**RESNET, DENSENET UNET**](https://medium.com/swlh/resnets-densenets-unets-6bbdbcfdf010) **- the trick behind them, concatenating both f(x) = x**
4. \*\*\*\*[**skip connections**](https://www.analyticsvidhya.com/blog/2021/08/all-you-need-to-know-about-skip-connections/) \*\*\*\* by Siravam / Vidhya- \*\*"\*\*Skip Connections (or Shortcut Connections) as the name suggests skips some of the layers in the neural network and feeds the output of one layer as the input to the next layers.
Skip Connections were introduced to solve different problems in different architectures. In the case of ResNets, skip connections solved the *degradation problem* that we addressed earlier whereas, in the case of DenseNets, it ensured **feature reusability**. We’ll discuss them in detail in the following sections.
Skip connections were introduced in literature even before residual networks. For example, [**Highway Networks**](https://arxiv.org/abs/1505.00387) (Srivastava et al.) had skip connections with gates that controlled and learned the flow of information to deeper layers. This concept is similar to the gating mechanism in LSTM. Although ResNets is actually a special case of Highway networks, the performance isn’t up to the mark comparing to ResNets. This suggests that it’s better to keep the gradient highways clear than to go for any gates – simplicity wins here!"
## **Fine tuning**
1. [**3 methods to fine tune, cut softmax layer, smaller learning rate, freeze layers**](https://flyyufelix.github.io/2016/10/03/fine-tuning-in-keras-part1.html)
2. [**Fine tuning on a sunset of data**](https://stats.stackexchange.com/questions/289036/fine-tuning-with-a-subset-of-the-same-data)
## **Deep Learning for NLP**
* **(did not fully read)** [**Yoav Goldberg’s course**](https://docs.google.com/document/d/1Xf_dqjf7mWmSoYX0HTKnml2mssP5BjrKUs-4E17CbNo/edit) **syllabus with lots of relevant topics on DL4NLP, including bidirectional RNNS and tree RNNs.**
* **(did not fully read)** [**CS224d**](http://cs224d.stanford.edu/index.html)**: Deep Learning for Natural Language Processing, with** [**slides etc.**](http://cs224d.stanford.edu/syllabus.html)
[**Deep Learning using Linear Support Vector Machines**](http://deeplearning.net/wp-content/uploads/2013/03/dlsvm.pdf) **- 1-3% decrease in error by replacing the softmax layer with a linear support vector machine**
## **MULTI LABEL/OUTPUT**
1. **A machine learning framework for** [**multi-output/multi-label**](https://github.com/scikit-multiflow/scikit-multiflow) **and stream data. Inspired by MOA and MEKA, following scikit-learn's philosophy.** [**https://scikit-multiflow.github.io/**](https://scikit-multiflow.github.io/)
2. [**Medium on MO, sklearn and keras**](https://towardsdatascience.com/what-data-scientists-should-know-about-multi-output-and-multi-label-training-b9d4be620e11)
3. [**MO in keras, see functional API on how.**](https://www.pyimagesearch.com/2018/06/04/keras-multiple-outputs-and-multiple-losses//)
### **FUZZY MULTI LABEL**
1. [**Ie., probabilities or soft values instead of hard labels**](https://datascience.stackexchange.com/questions/48111/multilabel-classifcation-in-sklearn-with-soft-fuzzy-labels)
## **SIAMESE NETWORKS**
1. [**Siamese for conveyor belt fault prediction**](https://towardsdatascience.com/predictive-maintenance-with-lstm-siamese-network-51ee7df29767)
2. [**Burlow**](https://arxiv.org/abs/2103.03230)**,** [**fb post**](https://www.facebook.com/yann.lecun/posts/10157682573642143) **- Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks. A successful approach to SSL is to learn representations which are invariant to distortions of the input sample. However, a recurring issue with this approach is the existence of trivial constant solutions. Most current methods avoid such solutions by careful implementation details. We propose an objective function that naturally avoids such collapse by measuring the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of a sample, and making it as close to the identity matrix as possible. This causes the representation vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors.**
## **Gated Multi-Layer Perceptron (GMLP)**
1. \*\*\*\*[**paper**](https://arxiv.org/abs/2105.08050)**,** [**git1**](https://github.com/jaketae/g-mlp)**,** [**git2**](https://github.com/lucidrains/g-mlp-pytorch) \*\*- "\*\*a simple network architecture, gMLP, based on MLPs with gating, and show that it can perform as well as Transformers in key language and vision applications. Our comparisons show that self-attention is not critical for Vision Transformers, as gMLP can achieve the same accuracy."
# Deep Neural Frameworks
## **PYTORCH**
1. **Deep learning with pytorch -** [**The book**](https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf)
2. [**Pytorch DL course**](https://atcold.github.io/pytorch-Deep-Learning/)**,** [**git**](https://github.com/Atcold/pytorch-Deep-Learning) **- yann lecun**
3. **Pytorch Official**
1. [**Tutorials**](https://pytorch.org/tutorials/)\
2. [**Learning with examples**](https://pytorch.org/tutorials/beginner/pytorch_with_examples.html)
3. [Learn the Basics](https://pytorch.org/tutorials/beginner/basics/intro.html) || [Quickstart](https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html) || [Tensors](https://pytorch.org/tutorials/beginner/basics/tensorqs_tutorial.html) || [Datasets & DataLoaders](https://pytorch.org/tutorials/beginner/basics/data_tutorial.html) || [Transforms](https://pytorch.org/tutorials/beginner/basics/transforms_tutorial.html) || [Build Model](https://pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html) || [Autograd](https://pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html) || [Optimization](https://pytorch.org/tutorials/beginner/basics/optimization_tutorial.html) || [Save & Load Model](https://pytorch.org/tutorials/beginner/basics/saveloadrun_tutorial.html)
4. [60 minute blitz](https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html)
5. (good) - [youtube series](https://pytorch.org/tutorials/beginner/introyt.html)
## **FAST.AI**
1. [**git**](https://github.com/fastai/fastai)
## **KERAS**
[**A make sense introduction into keras**](https://www.youtube.com/playlist?list=PLFxrZqbLojdKuK7Lm6uamegEFGW2wki6P)**, has several videos on the topic, going through many network types, creating custom activation functions, going through examples.**
**+ Two extra videos from the same author,** [**examples**](https://www.youtube.com/watch?v=6RdflAr66-E) **and** [**examples-2**](https://www.youtube.com/watch?v=fDKdITMBAGk)
**Didn’t read:**
1. [**Keras cheatsheet**](https://www.datacamp.com/community/blog/keras-cheat-sheet)
2. [**Seq2Seq RNN**](https://stackoverflow.com/questions/41933958/how-to-code-a-sequence-to-sequence-rnn-in-keras)
3. [**Stateful LSTM**](https://github.com/fchollet/keras/blob/master/examples/stateful_lstm.py) **- Example script showing how to use stateful RNNs to model long sequences efficiently.**
4. [**CONV LSTM**](https://github.com/fchollet/keras/blob/master/examples/conv_lstm.py) **- this script demonstrate the use of a conv LSTM network, used to predict the next frame of an artificially generated move which contains moving squares.**
[**How to force keras to use tensorflow**](https://github.com/ContinuumIO/anaconda-issues/issues/1735) **and not teano (set the .bat file)**
[**Callbacks - how to create an AUC ROC score callback with keras**](https://keunwoochoi.wordpress.com/2016/07/16/keras-callbacks/) **- with code example.**
[**Batch size vs. Iteratio**](https://stats.stackexchange.com/questions/164876/tradeoff-batch-size-vs-number-of-iterations-to-train-a-neural-network)**ns in NN Keras.**
[**Keras metrics**](https://machinelearningmastery.com/custom-metrics-deep-learning-keras-python/) **- classification regression and custom metrics**
[**Keras Metrics 2**](https://machinelearningmastery.com/metrics-evaluate-machine-learning-algorithms-python/) **- accuracy, ROC, AUC, classification, regression r^2.**
[**Introduction to regression models in Keras,**](https://machinelearningmastery.com/regression-tutorial-keras-deep-learning-library-python/) **using MSE, comparing baseline vs wide vs deep networks.**
[**How does Keras calculate accuracy**](https://datascience.stackexchange.com/questions/14415/how-does-keras-calculate-accuracy)**? Formula and explanation**
**Compares label with the rounded predicted float, i.e. bigger than 0.5 = 1, smaller than = 0**
**For categorical we take the argmax for the label and the prediction and compare their location.**
**In both cases, we average the results.**
[**Custom metrics (precision recall) in keras**](https://stackoverflow.com/questions/41458859/keras-custom-metric-for-single-class-accuracy)**. Which are taken from** [**here**](https://github.com/autonomio/talos/tree/master/talos/metrics)**, including entropy and f1**
### **KERAS MULTI GPU**
1. [**When using SGD only batches between 32-512 are adequate, more can lead to lower performance, less will lead to slow training times.**](https://arxiv.org/pdf/1609.04836.pdf)
2. **Note: probably doesn't reflect on adam, is there a reference?**
3. [**Parallel gpu-code for keras. Its a one liner, but remember to scale batches by the amount of GPU used in order to see a (non linear) scaability in training time.**](https://datascience.stackexchange.com/questions/23895/multi-gpu-in-keras)
4. [**Pitfalls in GPU training, this is a very important post, be aware that you can corrupt your weights using the wrong combination of batches-to-input-size**](http://blog.datumbox.com/5-tips-for-multi-gpu-training-with-keras/)**, in keras-tensorflow. When you do multi-GPU training, it is important to feed all the GPUs with data. It can happen that the very last batch of your epoch has less data than defined (because the size of your dataset can not be divided exactly by the size of your batch). This might cause some GPUs not to receive any data during the last step. Unfortunately some Keras Layers, most notably the Batch Normalization Layer, can’t cope with that leading to nan values appearing in the weights (the running mean and variance in the BN layer).**
5. [**5 things to be aware of for multi gpu using keras, crucial to look at before doing anything**](http://blog.datumbox.com/5-tips-for-multi-gpu-training-with-keras/)
**KERAS FUNCTIONAL API**
[**What is and how to use?**](https://machinelearningmastery.com/keras-functional-api-deep-learning/) **A flexible way to declare layers in parallel, i.e. parallel ways to deal with input, feature extraction, models and outputs as seen in the following images.**\
### **KERAS EMBEDDING LAYER**
1. [**Injecting glove to keras embedding layer and using it for classification + what is and how to use the embedding layer in keras.**](https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/)
2. [**Keras blog - using GLOVE for pretrained embedding layers.**](https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html)
3. [**Word embedding using keras, continuous BOW - CBOW, SKIPGRAM, word2vec - really good.**](https://towardsdatascience.com/understanding-feature-engineering-part-4-deep-learning-methods-for-text-data-96c44370bbfa)
4. [**Fasttext - comparison of key feature against word2vec**](https://www.quora.com/What-is-the-main-difference-between-word2vec-and-fastText)
5. [**Multiclass classification using word2vec/glove + code**](https://github.com/dennybritz/cnn-text-classification-tf/issues/69)
6. [**word2vec/doc2vec/tfidf code in python for text classification**](https://github.com/davidsbatista/text-classification/blob/master/train_classifiers.py)
7. [**Lda & word2vec**](https://www.kaggle.com/vukglisovic/classification-combining-lda-and-word2vec)
8. [**Text classification with word2vec**](http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/)
9. [**Gensim word2vec**](https://radimrehurek.com/gensim/models/word2vec.html)**, and** [**another one**](http://kavita-ganesan.com/gensim-word2vec-tutorial-starter-code/)
10. [**Fasttext paper**](https://arxiv.org/abs/1607.01759)
### **Keras: Predict vs Evaluate**
[**here:**](https://www.quora.com/What-is-the-difference-between-keras-evaluate-and-keras-predict)
**.predict() generates output predictions based on the input you pass it (for example, the predicted characters in the** [**MNIST example**](https://github.com/fchollet/keras/blob/master/examples/mnist_mlp.py)**)**
**.evaluate() computes the loss based on the input you pass it, along with any other metrics that you requested in the metrics param when you compiled your model (such as accuracy in the** [**MNIST example**](https://github.com/fchollet/keras/blob/master/examples/mnist_mlp.py)**)**
**Keras metrics**
[**For classification methods - how does keras calculate accuracy, all functions.**](https://www.quora.com/How-does-Keras-calculate-accuracy)
### **LOSS IN KERAS**
[**Why is the training loss much higher than the testing loss?**](https://keras.io/getting-started/faq/#why-is-the-training-loss-much-higher-than-the-testing-loss) **A Keras model has two modes: training and testing. Regularization mechanisms, such as Dropout and L1/L2 weight regularization, are turned off at testing time.**
**The training loss is the average of the losses over each batch of training data. Because your model is changing over time, the loss over the first batches of an epoch is generally higher than over the last batches. On the other hand, the testing loss for an epoch is computed using the model as it is at the end of the epoch, resulting in a lower loss.**
# Embedding
## **Intro**
**(amazing)** [**embeddings from the ground up singlelunch**](https://www.singlelunch.com/2020/02/16/embeddings-from-the-ground-up/)
## **VECTOR SIMILARITY SEARCH**
1. [**Faiss**](https://github.com/facebookresearch/faiss) **- a library for efficient similarity search**
2. [**Benchmarking**](https://github.com/erikbern/ann-benchmarks) **- complete with almost everything imaginable**
3. [**Singlestore**](https://www.singlestore.com/solutions/predictive-ml-ai/)
4. **Elastic search -** [**dense vector**](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/query-dsl-script-score-query.html#vector-functions)
5. **Google cloud vertex matching engine** [**NN search**](https://cloud.google.com/blog/products/ai-machine-learning/vertex-matching-engine-blazing-fast-and-massively-scalable-nearest-neighbor-search)
1. **search**
1. **Recommendation engines**
2. **Search engines**
3. **Ad targeting systems**
4. **Image classification or image search**
5. **Text classification**
6. **Question answering**
7. **Chat bots**
2. **Features**
1. **Low latency**
2. **High recall**
3. **managed**
4. **Filtering**
5. **scale**
6. **Pinecone - managed** [**vector similarity search**](https://www.pinecone.io/) **- Pinecone is a fully managed vector database that makes it easy to add vector search to production applications. No more hassles of benchmarking and tuning algorithms or building and maintaining infrastructure for vector search.**
7. [**Nmslib**](https://github.com/nmslib/nmslib) **(**[**benchmarked**](https://github.com/erikbern/ann-benchmarks) **- Benchmarks of approximate nearest neighbor libraries in Python) is a Non-Metric Space Library (NMSLIB): An efficient similarity search library and a toolkit for evaluation of k-NN methods for generic non-metric spaces.**
8. **scann,**
9. [**Vespa.ai**](https://vespa.ai/) **- Make AI-driven decisions using your data, in real time. At any scale, with unbeatable performance**
10. [**Weaviate**](https://www.semi.technology/developers/weaviate/current/) **- Weaviate is an** [**open source**](https://github.com/semi-technologies/weaviate) **vector search engine and vector database. Weaviate uses machine learning to vectorize and store data, and to find answers to natural language queries, or any other media type.**
11. [**Neural Search with BERT and Solr**](https://dmitry-kan.medium.com/list/vector-search-e9b564d14274) **- Indexing BERT vector data in Solr and searching with full traversal**
12. [**Fun With Apache Lucene and BERT Embeddings**](https://medium.com/swlh/fun-with-apache-lucene-and-bert-embeddings-c2c496baa559) **- This post goes much deeper -- to the similarity search algorithm on Apache Lucene level. It upgrades the code from 6.6 to 8.0**
13. [**Speeding up BERT Search in Elasticsearch**](https://towardsdatascience.com/speeding-up-bert-search-in-elasticsearch-750f1f34f455) **- Neural Search in Elasticsearch: from vanilla to KNN to hardware acceleration**
14. [**Ask Me Anything about Vector Search**](https://towardsdatascience.com/ask-me-anything-about-vector-search-4252a01f3889) **- In the Ask Me Anything: Vector Search! session Max Irwin and Dmitry Kan discussed major topics of vector search, ranging from its areas of applicability to comparing it to good ol’ sparse search (TF-IDF/BM25), to its readiness for prime time and what specific engineering elements need further tuning before offering this to users.**
15. [**Search with BERT vectors in Solr and Elasticsearch**](https://github.com/DmitryKey/bert-solr-search) **- GitHub repository used for experiments with Solr and Elasticsearch using DBPedia abstracts comparing Solr, vanilla Elasticsearch, elastiknn enhanced Elasticsearch, OpenSearch, and GSI APU**
16. [**Not All Vector Databases Are Made Equal**](https://towardsdatascience.com/milvus-pinecone-vespa-weaviate-vald-gsi-what-unites-these-buzz-words-and-what-makes-each-9c65a3bd0696) **- A detailed comparison of Milvus, Pinecone, Vespa, Weaviate, Vald, GSI and Qdrant**
17. [**Vector Podcast**](https://dmitry-kan.medium.com/vector-podcast-e27d83ecd0be) **- Podcast hosted by Dmitry Kan, interviewing the makers in the Vector / Neural Search industry. Available on YouTube, Spotify, Apple Podcasts and RSS**
18. [**Players in Vector Search: Video**](https://dmitry-kan.medium.com/players-in-vector-search-video-2fd390d00d6) **-Video recording and slides of the talk presented on London IR Meetup on the topic of players, algorithms, software and use cases in Vector Search**
19. **(paper)** [**Hybrid retrieval using search and semantic search**](https://arxiv.org/abs/2210.11934)
## **TOOLS**
### **FLAIR**
1. **Name-Entity Recognition (NER): It can recognise whether a word represents a person, location or names in the text.**
2. **Parts-of-Speech Tagging (PoS): Tags all the words in the given text as to which “part of speech” they belong to.**
3. **Text Classification: Classifying text based on the criteria (labels)**
4. **Training Custom Models: Making our own custom models.**
5. **It comprises of popular and state-of-the-art word embeddings, such as GloVe, BERT, ELMo, Character Embeddings, etc. There are very easy to use thanks to the Flair API**
6. **Flair’s interface allows us to combine different word embeddings and use them to embed documents. This in turn leads to a significant uptick in results**
7. **‘Flair Embedding’ is the signature embedding provided within the Flair library. It is powered by contextual string embeddings. We’ll understand this concept in detail in the next section**
8. **Flair supports a number of languages – and is always looking to add new ones**
### **HUGGING FACE**
1. [**Git**](https://github.com/huggingface/transformers)
2. 1. [**Hugging face pytorch transformers**](https://github.com/huggingface/pytorch-transformers)
3. [**Hugging face nlp pretrained**](https://huggingface.co/models?search=Helsinki-NLP%2Fopus-mt\&fbclid=IwAR0YN7qn9uTlCeBOZw4jzWgq9IXq_9ju1ww_rVL-f1fa9EjlSP50q05QcmU)
4. [**hugging face on emotions**](https://medium.com/huggingface/understanding-emotions-from-keras-to-pytorch-3ccb61d5a983)
1. **how to make a custom pyTorch LSTM with custom activation functions,**
2. **how the PackedSequence object works and is built,**
3. **how to convert an attention layer from Keras to pyTorch,**
4. **how to load your data in pyTorch: DataSets and smart Batching,**
5. **how to reproduce Keras weights initialization in pyTorch.**
5. **A** [**thorough tutorial on bert**](http://mccormickml.com/2019/07/22/BERT-fine-tuning/)**, fine tuning using hugging face transformers package.** [**Code**](https://colab.research.google.com/drive/1Y4o3jh3ZH70tl6mCd76vz_IxX23biCPP)
**Youtube** [**ep1**](https://www.youtube.com/watch?v=FKlPCK1uFrc)**,** [**2**](https://www.youtube.com/watch?v=zJW57aCBCTk)**,** [**3**](https://www.youtube.com/watch?v=x66kkDnbzi4)**,** [**3b**](https://www.youtube.com/watch?v=Hnvb9b7a_Ps)**,**
## **LANGUAGE EMBEDDINGS**
### **History**
1. [**Google’s intro to transformers and multi-head self attention**](https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html)
2. [**How self attention and relative positioning work**](https://medium.com/@_init_/how-self-attention-with-relative-position-representations-works-28173b8c245a) **(great!)**
1. **Rnns are sequential, same word in diff position will have diff encoding due to the input from the previous word, which is inherently different.**
2. **Attention without positional! Will have distinct (Same) encoding.**
3. **Relative look at a window around each word and adds a distance vector in terms of how many words are before and after, which fixes the problem.**
4. 
5. 
6. **The authors hypothesized that precise relative position information is not useful beyond a certain distance.**
7. **Clipping the maximum distance enables the model to generalize to sequence lengths not seen during training.**
3. [**From bert to albert**](https://medium.com/@hamdan.hussam/from-bert-to-albert-pre-trained-langaug-models-5865aa5c3762)
4. [**All the latest buzz algos**](https://www.topbots.com/most-important-ai-nlp-research/#ai-nlp-paper-2018-12)
5. **A** [**Summary of them**](https://www.topbots.com/ai-nlp-research-pretrained-language-models/?utm_source=facebook\&utm_medium=group_post\&utm_campaign=pretrained\&fbclid=IwAR0smqf8qanfMayo4fRH2hFuc5LYA8-Bn5oEp-xedKcRR43QsqXIelIAzEE)
6. [**8 pretrained language embeddings**](https://www.analyticsvidhya.com/blog/2019/03/pretrained-models-get-started-nlp/)
7. [**Hugging face pytorch transformers**](https://github.com/huggingface/pytorch-transformers)
8. [**Hugging face nlp pretrained**](https://huggingface.co/models?search=Helsinki-NLP%2Fopus-mt\&fbclid=IwAR0YN7qn9uTlCeBOZw4jzWgq9IXq_9ju1ww_rVL-f1fa9EjlSP50q05QcmU)
### **Embedding Foundation Knowledge**
1. [**Medium on Introduction into word embeddings, sentence embeddings, trends in the field.**](https://towardsdatascience.com/deep-transfer-learning-for-natural-language-processing-text-classification-with-universal-1a2c69e5baa9) **The Indian guy,** [**git**](https://nbviewer.jupyter.org/github/dipanjanS/data_science_for_all/blob/master/tds_deep_transfer_learning_nlp_classification/Deep%20Transfer%20Learning%20for%20NLP%20-%20Text%20Classification%20with%20Universal%20Embeddings.ipynb) **notebook,** [**his git**](https://github.com/dipanjanS)**,**
1. **Baseline Averaged Sentence Embeddings**
2. **Doc2Vec**
3. **Neural-Net Language Models (Hands-on Demo!)**
4. **Skip-Thought Vectors**
5. **Quick-Thought Vectors**
6. **InferSent**
7. **Universal Sentence Encoder**
2. [**Shay palachy on word embedding covering everything from bow to word/doc/sent/phrase.**](https://medium.com/@shay.palachy/document-embedding-techniques-fed3e7a6a25d)
3. [**Another intro, not as good as the one above**](https://medium.com/huggingface/universal-word-sentence-embeddings-ce48ddc8fc3a)
4. [**Using sklearn vectorizer to create custom ones, i.e. a vectorizer that does preprocessing and tfidf and other things.**](https://towardsdatascience.com/hacking-scikit-learns-vectorizers-9ef26a7170af)
5. [**TFIDF - n-gram based top weighted tfidf words**](https://stackoverflow.com/questions/25217510/how-to-see-top-n-entries-of-term-document-matrix-after-tfidf-in-scikit-learn)
6. [**Gensim bi-gram phraser/phrases analyser/converter**](https://radimrehurek.com/gensim/models/phrases.html)
7. [**Countvectorizer, stemmer, lemmatization code tutorial**](https://medium.com/@rnbrown/more-nlp-with-sklearns-countvectorizer-add577a0b8c8)
8. [**Current 2018 best universal word and sentence embeddings -> elmo**](https://medium.com/huggingface/universal-word-sentence-embeddings-ce48ddc8fc3a)
9. [**5-part series on word embeddings**](http://ruder.io/word-embeddings-1/)**,** [**part 2**](http://ruder.io/word-embeddings-softmax/index.html)**,** [**3**](http://ruder.io/secret-word2vec/index.html)**,** [**4 - cross lingual review**](http://ruder.io/cross-lingual-embeddings/index.html)**,** [**5-future trends**](http://ruder.io/word-embeddings-2017/index.html)
10. [**Word embedding posts**](https://datawarrior.wordpress.com/2016/05/15/word-embedding-algorithms/)
11. [**Facebook github for embedings called starspace**](https://github.com/facebookresearch/StarSpace)
12. [**Medium on Fast text / elmo etc**](https://medium.com/huggingface/universal-word-sentence-embeddings-ce48ddc8fc3a)
### **Language modeling**
1. [**Ruder on language modelling as the next imagenet**](http://ruder.io/nlp-imagenet/) **- Language modelling, the last approach mentioned, has been shown to capture many facets of language relevant for downstream tasks, such as** [**long-term dependencies**](https://arxiv.org/abs/1611.01368) **,** [**hierarchical relations**](https://arxiv.org/abs/1803.11138) **, and** [**sentiment**](https://arxiv.org/abs/1704.01444) **. Compared to related unsupervised tasks such as skip-thoughts and autoencoding,** [**language modelling performs better on syntactic tasks even with less training data**](https://openreview.net/forum?id=BJeYYeaVJ7)**.**
2. **A** [**tutorial**](https://blog.myyellowroad.com/unsupervised-sentence-representation-with-deep-learning-104b90079a93) **about w2v skipthought - with code!, specifically language modelling here is important - Our second method is training a language model to represent our sentences. A language model describes the probability of a text existing in a language. For example, the sentence “I like eating bananas” would be more probable than “I like eating convolutions.” We train a language model by slicing windows of n words and predicting what the next word will be in the text**
3. [**Unread - universal language model fine tuning for text-classification**](https://arxiv.org/abs/1801.06146)
4. **ELMO -** [**medium**](https://towardsdatascience.com/beyond-word-embeddings-part-2-word-vectors-nlp-modeling-from-bow-to-bert-4ebd4711d0ec)
5. [**Bert**](https://arxiv.org/abs/1810.04805v1) **\*\*\[python git]\(****)**- We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT representations can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks.\*\* 
6. [**Open.ai on language modelling**](https://blog.openai.com/language-unsupervised/) **- We’ve obtained state-of-the-art results on a suite of diverse language tasks with a scalable, task-agnostic system, which we’re also releasing. Our approach is a combination of two existing ideas:** [**transformers**](https://arxiv.org/abs/1706.03762) **and** [**unsupervised pre-training**](https://arxiv.org/abs/1511.01432)**.** [**READ PAPER**](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf)**,** [**VIEW CODE**](https://github.com/openai/finetune-transformer-lm)**.**
7. **Scikit-learn inspired model finetuning for natural language processing.**
[**finetune**](https://finetune.indico.io/#module-finetune) **ships with a pre-trained language model from** [**“Improving Language Understanding by Generative Pre-Training”**](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf) **and builds off the** [**OpenAI/finetune-language-model repository**](https://github.com/openai/finetune-transformer-lm)**.**
1. **Did not read -** [**The annotated Transformer**](http://nlp.seas.harvard.edu/2018/04/03/attention.html?fbclid=IwAR2_ZOfUfXcto70apLdT_StObPwatYHNRPP4OlktcmGfj9uPLhgsZPsAXzE) **- jupyter on transformer with annotation**
2. **Medium on** [**Dissecting Bert**](https://medium.com/dissecting-bert/dissecting-bert-part-1-d3c3d495cdb3)**,** [**appendix**](https://medium.com/dissecting-bert/dissecting-bert-appendix-the-decoder-3b86f66b0e5f)
3. [**Medium on distilling 6 patterns from bert**](https://towardsdatascience.com/deconstructing-bert-distilling-6-patterns-from-100-million-parameters-b49113672f77)
### **Embedding spaces**
1. [**A good overview of sentence embedding methods**](http://mlexplained.com/2017/12/28/an-overview-of-sentence-embedding-methods/) **- w2v ft s2v skip, d2v**
2. [**A very good overview of word embeddings**](http://sanjaymeena.io/tech/word-embeddings/)
3. [**Intro to word embeddings - lots of images**](https://www.springboard.com/blog/introduction-word-embeddings/)
4. [**A very long and extensive thesis about embeddings**](http://ad-publications.informatik.uni-freiburg.de/theses/Bachelor_Jon_Ezeiza_2017.pdf)
5. [**Sent2vec by gensim**](https://rare-technologies.com/sent2vec-an-unsupervised-approach-towards-learning-sentence-embeddings/) **- sentence embedding is defined as the average of the source word embeddings of its constituent words. This model is furthermore augmented by also learning source embeddings for not only unigrams but also n-grams of words present in each sentence, and averaging the n-gram embeddings along with the words**
6. [**Sent2vec vs fasttext - with info about s2v parameters**](https://github.com/epfml/sent2vec/issues/19)
7. [**Wordrank vs fasttext vs w2v comparison**](https://en.wikipedia.org/wiki/Automatic_summarization#TextRank_and_LexRank) **- the better word similarity algorithm**
8. [**W2v vs glove vs sppmi vs svd by gensim**](https://rare-technologies.com/making-sense-of-word2vec/)
9. [**Medium on a gentle intro to d2v**](https://medium.com/scaleabout/a-gentle-introduction-to-doc2vec-db3e8c0cce5e)
10. [**Doc2vec tutorial by gensim**](https://rare-technologies.com/doc2vec-tutorial/) **- Doc2vec (aka paragraph2vec, aka sentence embeddings) modifies the word2vec algorithm to unsupervised learning of continuous representations for larger blocks of text, such as sentences, paragraphs or entire documents. - Most importantly this tutorial has crucial information about the implementation parameters that should be read before using it.**
11. [**Lbl2Vec**](https://github.com/sebischair/Lbl2Vec)**,** [medium](https://towardsdatascience.com/unsupervised-text-classification-with-lbl2vec-6c5e040354de), is an algorithm for unsupervised document classification and unsupervised document retrieval. It automatically generates jointly embedded label, document and word vectors and returns documents of categories modeled by manually predefined keywords.
12. [**Git for word embeddings - taken from mastery’s nlp course**](https://github.com/IshayTelavivi/nlp_crash_course)
13. [**Skip-thought -**](http://mlexplained.com/2017/12/28/an-overview-of-sentence-embedding-methods/) **\*\*\[git]\(****)**- Where word2vec attempts to predict surrounding words from certain words in a sentence, skip-thought vector extends this idea to sentences: it predicts surrounding sentences from a given sentence. NOTE: Unlike the other methods, skip-thought vectors require the sentences to be ordered in a semantically meaningful way. This makes this method difficult to use for domains such as social media text, where each snippet of text exists in isolation.\*\*
14. [**Fastsent**](http://mlexplained.com/2017/12/28/an-overview-of-sentence-embedding-methods/) **- Skip-thought vectors are slow to train. FastSent attempts to remedy this inefficiency while expanding on the core idea of skip-thought: that predicting surrounding sentences is a powerful way to obtain distributed representations. Formally, FastSent represents sentences as the simple sum of its word embeddings, making training efficient. The word embeddings are learned so that the inner product between the sentence embedding and the word embeddings of surrounding sentences is maximized. NOTE: FastSent sacrifices word order for the sake of efficiency, which can be a large disadvantage depending on the use-case.**
15. **Weighted sum of words - In this method, each word vector is weighted by the factor**  **where**  **is a hyperparameter and**  **is the (estimated) word frequency. This is similar to tf-idf weighting, where more frequent terms are weighted downNOTE: Word order and surrounding sentences are ignored as well, limiting the information that is encoded.**
16. [**Infersent by facebook**](https://github.com/facebookresearch/InferSent) **-** [**paper**](https://arxiv.org/abs/1705.02364) **InferSent is a sentence embeddings method that provides semantic representations for English sentences. It is trained on natural language inference data and generalizes well to many different tasks. ABSTRACT: we show how universal sentence representations trained using the supervised data of the Stanford Natural Language Inference datasets can consistently outperform unsupervised methods like SkipThought vectors on a wide range of transfer tasks. Much like how computer vision uses ImageNet to obtain features, which can then be transferred to other tasks, our work tends to indicate the suitability of natural language inference for transfer learning to other NLP tasks.**
17. [**Universal sentence encoder - google**](https://tfhub.dev/google/universal-sentence-encoder/1) **-** [**notebook**](https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/semantic_similarity_with_tf_hub_universal_encoder.ipynb#scrollTo=8OKy8WhnKRe_)**,** [**git**](https://github.com/tensorflow/hub/blob/master/examples/colab/semantic_similarity_with_tf_hub_universal_encoder.ipynb) **The Universal Sentence Encoder encodes text into high dimensional vectors that can be used for text classification, semantic similarity, clustering and other natural language tasks. The model is trained and optimized for greater-than-word length text, such as sentences, phrases or short paragraphs. It is trained on a variety of data sources and a variety of tasks with the aim of dynamically accommodating a wide variety of natural language understanding tasks. The input is variable length English text and the output is a 512 dimensional vector. We apply this model to the** [**STS benchmark**](http://ixa2.si.ehu.es/stswiki/index.php/STSbenchmark) **for semantic similarity, and the results can be seen in the** [**example notebook**](https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/semantic_similarity_with_tf_hub_universal_encoder.ipynb) **made available. The universal-sentence-encoder model is trained with a deep averaging network (DAN) encoder.**
18. [**Multi language universal sentence encoder**](https://ai.googleblog.com/2019/07/multilingual-universal-sentence-encoder.html?fbclid=IwAR2fubNOwrxWWxYous7IyQCJ3_bY0UAdAYO_yuWONMv-aV3o8hDckSS3FCE) **- no hebrew**
19. **Pair2vec -** [**paper**](https://arxiv.org/abs/1810.08854) **- paper proposes new methods for learning and using embeddings of word pairs that implicitly represent background knowledge about such relationships. I.e., using p2v information with existing models to increase performance. Experiments show that our pair embeddings can complement individual word embeddings, and that they are perhaps capturing information that eludes the traditional interpretation of the Distributional Hypothesis**
20. [**Fast text python tutorial**](http://ai.intelligentonlinetools.com/ml/fasttext-word-embeddings-text-classification-python-mlp/)
## Embedding Models
### **Cat2vec**
1. **Part1:** [**Label encoder/ ordinal, One hot, one hot with a rare bucket, hash**](https://blog.myyellowroad.com/using-categorical-data-in-machine-learning-with-python-from-dummy-variables-to-deep-category-66041f734512)
2. [**Part2: cat2vec using w2v**](https://blog.myyellowroad.com/using-categorical-data-in-machine-learning-with-python-from-dummy-variables-to-deep-category-42fd0a43b009)**, and entity embeddings for categorical data**
### **ENTITY EMBEDDINGS**
1. **Star -** [**General purpose embedding paper with code somewhere**](https://arxiv.org/pdf/1709.03856.pdf)
2. [**Using embeddings on tabular data, specifically categorical - introduction**](http://www.fast.ai/2018/04/29/categorical-embeddings/)**, using fastai without limiting ourselves to pytorch - the material from this post is covered in much more detail starting around 1:59:45 in** [**the Lesson 3 video**](http://course.fast.ai/lessons/lesson3.html) **and continuing in** [**Lesson 4**](http://course.fast.ai/lessons/lesson4.html) **of our free, online** [**Practical Deep Learning for Coders**](http://course.fast.ai/) **course. To see example code of how this approach can be used in practice, check out our** [**Lesson 3 jupyter notebook**](https://github.com/fastai/fastai/blob/master/courses/dl1/lesson3-rossman.ipynb)**. Perhaps Saturday and Sunday have similar behavior, and maybe Friday behaves like an average of a weekend and a weekday. Similarly, for zip codes, there may be patterns for zip codes that are geographically near each other, and for zip codes that are of similar socio-economic status. The jupyter notebook doesn't seem to have the embedding example they are talking about.**
3. [**Rossman on kaggle**](http://blog.kaggle.com/2016/01/22/rossmann-store-sales-winners-interview-3rd-place-cheng-gui/)**, used entity-embeddings,** [**here**](https://www.kaggle.com/c/rossmann-store-sales/discussion/17974)**,** [**github**](https://github.com/entron/entity-embedding-rossmann)**,** [**paper**](https://arxiv.org/abs/1604.06737)
4. [**Medium on rossman - good**](https://towardsdatascience.com/deep-learning-structured-data-8d6a278f3088)
5. [**Embedder**](https://github.com/dkn22/embedder) **- git code for a simplified entity embedding above.**
6. **Finally what they do is label encode each feature using labelEncoder into an int-based feature, then push each feature into its own embedding layer of size 1 with an embedding size defined by a rule of thumb (so it seems), merge all layers, train a synthetic regression/classification and grab the weights of the corresponding embedding layer.**
7. [**Entity2vec**](https://github.com/ot/entity2vec)
8. [**Categorical using keras**](https://medium.com/@satnalikamayank12/on-learning-embeddings-for-categorical-data-using-keras-165ff2773fc9)
### **ALL2VEC EMBEDDINGS**
1. [**ALL ???-2-VEC ideas**](https://github.com/MaxwellRebo/awesome-2vec)
2. **Fast.ai** [**post**](http://www.fast.ai/2018/04/29/categorical-embeddings/) **regarding embedding for tabular data, i.e., cont and categorical data**
3. [**Entity embedding for**](https://github.com/fastai/fastai/blob/master/courses/dl1/lesson3-rossman.ipynb) **categorical data +** [**notebook**](https://github.com/fastai/fastai/blob/master/courses/dl1/lesson3-rossman.ipynb)
4. [**Kaggle taxi competition + code**](http://blog.kaggle.com/2015/07/27/taxi-trajectory-winners-interview-1st-place-team-%F0%9F%9A%95/)
5. [**Ross man competition - entity embeddings, code missing**](http://blog.kaggle.com/2016/01/22/rossmann-store-sales-winners-interview-3rd-place-cheng-gui/) **+**[**alternative code**](https://github.com/entron/entity-embedding-rossmann)
6. [**CODE TO CREATE EMBEDDINGS straight away, based onthe ideas by cheng guo in keras**](https://github.com/dkn22/embedder)
7. [**PIN2VEC - pinterest embeddings using the same idea**](https://medium.com/the-graph/applying-deep-learning-to-related-pins-a6fee3c92f5e)
8. [**Tweet2Vec**](https://github.com/soroushv/Tweet2Vec) **- code in theano,** [**paper**](https://dl.acm.org/citation.cfm?doid=2911451.2914762)**.**
9. [**Clustering**](https://github.com/svakulenk0/tweet2vec_clustering) **of tweet2vec,** [**paper**](https://arxiv.org/abs/1703.05123)
10. **Paper:** [**Character neural embeddings for tweet clustering**](https://arxiv.org/pdf/1703.05123.pdf)
11. **Diff2vec - might be useful on social network graphs,** [**paper**](http://homepages.inf.ed.ac.uk/s1668259/papers/sequence.pdf)**,** [**code**](https://github.com/benedekrozemberczki/diff2vec)
12. **emoji 2vec (below)**
13. [**Char2vec**](https://hackernoon.com/chars2vec-character-based-language-model-for-handling-real-world-texts-with-spelling-errors-and-a3e4053a147d) **\*\*\[Git]\(****)**, similarity measure for words with types. **\[ \*\***]\()
**EMOJIS**
1. **1.** [**Deepmoji**](http://datadrivenjournalism.net/featured_projects/deepmoji_using_emojis_to_teach_ai_about_emotions)**,**
2. [**hugging face on emotions**](https://medium.com/huggingface/understanding-emotions-from-keras-to-pytorch-3ccb61d5a983)
1. **how to make a custom pyTorch LSTM with custom activation functions,**
2. **how the PackedSequence object works and is built,**
3. **how to convert an attention layer from Keras to pyTorch,**
4. **how to load your data in pyTorch: DataSets and smart Batching,**
5. **how to reproduce Keras weights initialization in pyTorch.**
3. [**Another great emoji paper, how to get vector representations from**](https://aclweb.org/anthology/S18-1039)
4. [**3. What can we learn from emojis (deep moji)**](https://www.media.mit.edu/posts/what-can-we-learn-from-emojis/)
5. [**Learning millions of**](https://arxiv.org/pdf/1708.00524.pdf) **for emoji, sentiment, sarcasm,** [**medium**](https://medium.com/@bjarkefelbo/what-can-we-learn-from-emojis-6beb165a5ea0)
6. [**EMOJI2VEC**](https://tech.instacart.com/deep-learning-with-emojis-not-math-660ba1ad6cdc) **- medium article with keras code, a**[**nother paper on classifying tweets using emojis**](https://arxiv.org/abs/1708.00524)
7. [**Group2vec**](https://github.com/cerlymarco/MEDIUM_NoteBook/tree/master/Group2Vec) **git and** [**medium**](https://towardsdatascience.com/group2vec-for-advance-categorical-encoding-54dfc7a08349)**, which is a multi input embedding network using a-f below. plus two other methods that involve groupby and applying entropy and join/countvec per class. Really interesting**
1. **Initialize embedding layers for each categorical input;**
2. **For each category, compute dot-products among other embedding representations. These are our ‘groups’ at the categorical level;**
3. **Summarize each ‘group’ adopting an average pooling;**
4. **Concatenate ‘group’ averages;**
5. **Apply regularization techniques such as BatchNormalization or Dropout;**
6. **Output probabilities.**
### **WORD2VEC**
1. **Monitor** [**train loss**](https://stackoverflow.com/questions/52038651/loss-does-not-decrease-during-training-word2vec-gensim) **using callbacks for word2vec**
2. **Cleaning datasets using weighted w2v sentence encoding, then pca and isolation forest to remove outlier sentences.**
3. [**Removing ‘gender bias using pair mean pca**](https://stackoverflow.com/questions/48019843/pca-on-word2vec-embeddings)
4. [**KPCA w2v approach on a very small dataset**](https://medium.com/@vishwanigupta/kpca-skip-gram-model-improving-word-embedding-a6a0cb7aad49)**,** [**similar git**](https://github.com/niitsuma/wordca) **for correspondence analysis,** [**paper**](https://arxiv.org/abs/1605.05087)
5. [**The best w2v/tfidf/bow/ embeddings post ever**](https://www.analyticsvidhya.com/blog/2017/06/word-embeddings-count-word2veec/)
6. [**Chris mccormick ml on w2v,**](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/) **\*\*\[post #2]\(****)** - negative sampling “Negative sampling addresses this by having each training sample only modify a small percentage of the weights, rather than all of them. With negative sampling, we are instead going to randomly select just a small number of “negative” words (let’s say 5) to update the weights for. (In this context, a “negative” word is one for which we want the network to output a 0 for). We will also still update the weights for our “positive” word (which is the word “quick” in our current example). The “negative samples” (that is, the 5 output words that we’ll train to output 0) are chosen using a “unigram distribution”. Essentially, the probability for selecting a word as a negative sample is related to its frequency, with more frequent words being more likely to be selected as negative samples.\*\*
7. [**Chris mccormick on negative sampling and hierarchical soft max**](https://www.youtube.com/watch?v=pzyIWCelt_E) **training, i.e., huffman binary tree for the vocabulary, learning internal tree nodes ie.,, the path as the probability vector instead of having len(vocabulary) neurons.**
8. [**Great W2V tutorial**](https://towardsdatascience.com/word2vec-skip-gram-model-part-1-intuition-78614e4d6e0b)
9. **Another** [**gensim-based w2v tutorial**](http://kavita-ganesan.com/gensim-word2vec-tutorial-starter-code/)**, with starter code and some usage examples of similarity**
10. [**Clustering using gensim word2vec**](http://ai.intelligentonlinetools.com/ml/k-means-clustering-example-word2vec/)
11. [**Yet another w2v medium explanation**](https://towardsdatascience.com/word-embeddings-exploration-explanation-and-exploitation-with-code-in-python-5dac99d5d795)
12. **Mean w2v**
13. **Sequential w2v embeddings.**
14. [**Negative sampling, why does it work in w2v - didnt read**](https://www.quora.com/How-does-negative-sampling-work-in-Word2vec-models)
15. [**Semantic contract using w2v/ft - he chose a good food category and selected words that worked best in order to find similar words to good bad etc. lior magen**](https://groups.google.com/forum/#!topic/gensim/wh7B00cc80w)
16. [**Semantic contract, syn-antonym DS, using w2v, a paper that i havent read**](http://anthology.aclweb.org/P16-2074) **yet but looks promising**
17. [**Amazing w2v most similar tutorial, examples for vectors, misspellings, semantic contrast and relations that may or may not be captured in the network.**](https://quomodocumque.wordpress.com/2016/01/15/messing-around-with-word2vec/)
18. [**Followup tutorial about genderfying words using ‘he’ ‘she’ similarity**](https://quomodocumque.wordpress.com/2016/01/15/gendercycle-a-dynamical-system-on-words/)
19. [**W2v Analogies using predefined anthologies of the**](https://gist.github.com/kylemcdonald/9bedafead69145875b8c) **form x:y:**:a:**b, plus code, plus insights of why it works and doesn't. presence : absence :: happy : unhappy absence : presence :: happy : proud abundant : scarce :: happy : glad refuse : accept :: happy : satisfied accurate : inaccurate :: happy : disappointed admit : deny :: happy : delighted never : always :: happy : Said\_Hirschbeck modern : ancient :: happy : ecstatic**
20. [**Nlpforhackers on bow, w2v embeddings with code on how to use**](https://nlpforhackers.io/word-embeddings/)
21. [**Hebrew word embeddings with w2v, ron shemesh, on wiki/twitter**](https://drive.google.com/drive/folders/1qBgdcXtGjse9Kq7k1wwMzD84HH_Z8aJt?fbclid=IwAR03PeUTGCgluILOQ6EaMR7AgkcRux5rs6Z8HEgWMRvFAwLGqb7-7bznbxM)
**GLOVE**
1. [**W2v vs glove vs fasttext, in terms of overfitting and what is the idea behind**](https://www.kaggle.com/sbongo/do-pretrained-embeddings-give-you-the-extra-edge)
2. [**W2v against glove performance**](http://dsnotes.com/post/glove-enwiki/) **comparison - glove wins in % and time.**
3. [**How glove and w2v work, but the following has a very good description**](https://geekyisawesome.blogspot.com/2017/03/word-embeddings-how-word2vec-and-glove.html) **- “GloVe takes a different approach. Instead of extracting the embeddings from a neural network that is designed to perform a surrogate task (predicting neighbouring words), the embeddings are optimized directly so that the dot product of two word vectors equals the log of the number of times the two words will occur near each other (within 5 words for example). For example if "dog" and "cat" occur near each other 10 times in a corpus, then vec(dog) dot vec(cat) = log(10). This forces the vectors to somehow encode the frequency distribution of which words occur near them.”**
4. [**Glove vs w2v, concise explanation**](https://www.quora.com/What-is-the-difference-between-fastText-and-GloVe/answer/Ajit-Rajasekharan)
### **FastText**
1. [**Fasttext - using fast text and upsampling/oversapmling on twitter data**](https://medium.com/@media_73863/fasttext-sentiment-analysis-for-tweets-a-straightforward-guide-9a8c070449a2)
2. [**A great youtube lecture 9m about ft, rarity, loss, class tree speedup**](https://www.youtube.com/watch?v=4l_At3oalzk) *\*\**
3. [**A thorough tutorial about what is FT and how to use it, performance, pros and cons.**](https://www.analyticsvidhya.com/blog/2017/07/word-representations-text-classification-using-fasttext-nlp-facebook/)
4. [**Docs**](https://fasttext.cc/blog/2016/08/18/blog-post.html)
5. [**Medium: word embeddings with w2v and fast text in gensim**](https://towardsdatascience.com/word-embedding-with-word2vec-and-fasttext-a209c1d3e12c) **, data cleaning and word similarity**
6. **Gensim -** [**fasttext docs**](https://radimrehurek.com/gensim/models/fasttext.html)**, similarity, analogies**
7. [**Alternative to gensim**](https://github.com/plasticityai/magnitude#benchmarks-and-features) **- promises speed and out of the box support for many embeddings.**
8. [**Comparison of usage w2v fasttext**](http://ai.intelligentonlinetools.com/ml/fasttext-word-embeddings-text-classification-python-mlp/)
9. [**Using gensim fast text - recommendation against using the fb version**](https://blog.manash.me/how-to-use-pre-trained-word-vectors-from-facebooks-fasttext-a71e6d55f27)
10. [**A comparison of w2v vs ft using gensim**](https://rare-technologies.com/fasttext-and-gensim-word-embeddings/) **- “Word2Vec embeddings seem to be slightly better than fastText embeddings at the semantic tasks, while the fastText embeddings do significantly better on the syntactic analogies. Makes sense, since fastText embeddings are trained for understanding morphological nuances, and most of the syntactic analogies are morphology based.**
1. [**Syntactic**](https://stackoverflow.com/questions/48356421/what-is-the-difference-between-syntactic-analogy-and-semantic-analogy) **means syntax, as in tasks that have to do with the structure of the sentence, these include tree parsing, POS tagging, usually they need less context and a shallower understanding of world knowledge**
2. [**Semantic**](https://stackoverflow.com/questions/48356421/what-is-the-difference-between-syntactic-analogy-and-semantic-analogy) **tasks mean meaning related, a higher level of the language tree, these also typically involve a higher level understanding of the text and might involve tasks s.a. question answering, sentiment analysis, etc...**
3. **As for analogies, he is referring to the mathematical operator like properties exhibited by word embedding, in this context a syntactic analogy would be related to plurals, tense or gender, those sort of things, and semantic analogy would be word meaning relationships s.a. man + queen = king, etc... See for instance** [**this article**](http://www.aclweb.org/anthology/W14-1618) **(and many others)**
11. [**Skip gram vs CBOW**](https://www.quora.com/What-are-the-continuous-bag-of-words-and-skip-gram-architectures)
1. [**Paper**](http://workshop.colips.org/dstc6/papers/track2_paper18_zhuang.pdf) **on fasttext vs glove vs w2v on a single DS, performance comparison. Ft wins by a small margin**
2. [**Medium on w2v/fast text ‘most similar’ words with code**](https://towardsdatascience.com/word-embedding-with-word2vec-and-fasttext-a209c1d3e12c)
3. [**keras/tf code for a fast text implementation**](http://debajyotidatta.github.io/nlp/deep/learning/word-embeddings/2016/09/28/fast-text-and-skip-gram/)
4. [**Medium on fast text and imbalance data**](https://medium.com/@yeyrama/fasttext-and-imbalanced-classification-1f9543f9e0ce)
5. **Medium on universal** [**Sentence encoder, w2v, Fast text for sentiment**](https://medium.com/@jatinmandav3/opinion-mining-sometimes-known-as-sentiment-analysis-or-emotion-ai-refers-to-the-use-of-natural-874f369194c0) **with code.**
### **SENTENCE EMBEDDING**
#### **Sense2vec**
1. [**Blog**](https://explosion.ai/blog/sense2vec-with-spacy)**,** [**github**](https://github.com/explosion/sense2vec)**: Using spacy or not, with w2v using POS/ENTITY TAGS to find similarities.based on reddit. “We follow Trask et al in adding part-of-speech tags and named entity labels to the tokens. Additionally, we merge named entities and base noun phrases into single tokens, so that they receive a single vector.”**
2. **>>> model.similarity('fair\_game|NOUN', 'game|NOUN') 0.034977455677555599 >>> model.similarity('multiplayer\_game|NOUN', 'game|NOUN') 0.54464530644393849**
#### **SENT2VEC aka “skip-thoughts”**
1. [**Gensim implementation of sent2vec**](https://rare-technologies.com/sent2vec-an-unsupervised-approach-towards-learning-sentence-embeddings/) **- usage examples, parallel training, a detailed comparison against gensim doc2vec**
2. [**Git implementation**](https://github.com/ryankiros/skip-thoughts)
3. [**Another git - worked**](https://github.com/epfml/sent2vec)
#### **USE - Universal sentence encoder**
1. [**Git notebook, usage and sentence similarity benchmark / visualization**](https://github.com/tensorflow/hub/blob/master/examples/colab/semantic_similarity_with_tf_hub_universal_encoder.ipynb)
#### **BERT+W2V**
1. [**Sentence similarity**](https://towardsdatascience.com/how-to-compute-sentence-similarity-using-bert-and-word2vec-ab0663a5d64)
### **PARAGRAPH2Vec**
1. [**Paragraph2VEC by stanford**](https://cs.stanford.edu/~quocle/paragraph_vector.pdf)
### **Doc2Vec**
1. [**Shuffle before training each**](https://groups.google.com/forum/#!topic/gensim/IVQBUF5n6aI) **epoch in d2v in order to fight overfitting**
# Deep Learning Models
## **AUTOENCODERS**
1. [**How to use AE for dimensionality reduction + code**](https://statcompute.wordpress.com/2017/01/15/autoencoder-for-dimensionality-reduction/) **- using keras’ functional API**
2. [**Keras.io blog post about AE’s**](https://blog.keras.io/building-autoencoders-in-keras.html) **- regular, deep, sparse, regularized, cnn, variational**
1. **A keras.io** [**replicate post**](https://towardsdatascience.com/deep-inside-autoencoders-7e41f319999f) **but explains AE quite nicely.**
3. [**Examples of vanilla, multi layer, CNN and sparse AE’s**](https://wiseodd.github.io/techblog/2016/12/03/autoencoders/)
4. [**Another example of CNN-AE**](https://hackernoon.com/autoencoders-deep-learning-bits-1-11731e200694)
5. [**Another AE tutorial**](https://towardsdatascience.com/how-to-reduce-image-noises-by-autoencoder-65d5e6de543)
6. [**Hinton’s coursera course**](https://www.coursera.org/learn/neural-networks/lecture/JiT1i/from-pca-to-autoencoders-5-mins) **on PCA vs AE, basically some info about what PCA does - maximizing variance and projecting and then what AE does and can do to achieve similar but non-linear dense representations**
7. [**A great tutorial on how does the clusters look like after applying PCA/ICA/AE**](https://www.kaggle.com/den3b81/2d-visualization-pca-ica-vs-autoencoders)
8. [**Another great presentation on PCA vs AE,**](https://web.cs.hacettepe.edu.tr/~aykut/classes/fall2016/bbm406/slides/l25-kernel_pca.pdf) **summarized in the KPCA section of this notebook. +**[**another one**](https://www.cs.toronto.edu/~urtasun/courses/CSC411/14_pca.pdf) **+**[**StackE**](https://stats.stackexchange.com/questions/261265/factor-analysis-vs-autoencoders)**xchange**
9. [**Autoencoder tutorial with python code and how to encode after**](https://ramhiser.com/post/2018-05-14-autoencoders-with-keras/)**,** [**mastery**](https://machinelearningmastery.com/encoder-decoder-attention-sequence-to-sequence-prediction-keras/)
10. [**Git code for low dimensional auto encoder**](https://github.com/Mylittlerapture/Low-Dimensional-Autoencoder)
11. [**Bart denoising AE**](https://arxiv.org/pdf/1910.13461.pdf)**, sequence to sequence pre training for NL generation translation and comprehension.**
12. [**Attention based seq to seq auto encoder**](https://wanasit.github.io/attention-based-sequence-to-sequence-in-keras.html)**,** [**git**](https://github.com/wanasit/katakana)
[**AE for anomaly detection, fraud detection**](https://medium.com/@curiousily/credit-card-fraud-detection-using-autoencoders-in-keras-tensorflow-for-hackers-part-vii-20e0c85301bd)
## **Variational AE**
1. **Unread -** [**Simple explanation**](https://medium.com/@dmonn/what-are-variational-autoencoders-a-simple-explanation-ea7dccafb0e3)
2. [**Pixel art VAE**](https://mlexplained.wordpress.com/category/generative-models/vae/)
3. [**Unread - another VAE**](https://towardsdatascience.com/teaching-a-variational-autoencoder-vae-to-draw-mnist-characters-978675c95776)
4. [**Pixel GAN VAE**](https://medium.com/@Synced/pixelgan-autoencoders-17496632b755)
5. [**Disentangled VAE**](https://www.youtube.com/watch?v=9zKuYvjFFS8) **- improves VAE**
6. **Optimus -** [**pretrained VAE**](https://github.com/ophiry/Optimus)**,** [**paper**](https://arxiv.org/abs/2004.04092)**,** [**Microsoft blog**](https://www.microsoft.com/en-us/research/blog/a-deep-generative-model-trifecta-three-advances-that-work-towards-harnessing-large-scale-power/)\*\*\*\*
## **SELF ORGANIZING MAPS (SOM)**
1. **Git**
1. [**Sompy**](https://github.com/sevamoo/SOMPY)**,**
2. **\*\*\***[**minisom!**](https://github.com/JustGlowing/minisom)
3. [**Many graph examples**](https://medium.com/@s.ganjoo96/self-organizing-maps-b2cf58b74fdb)**,** [**example**](https://github.com/lightsalsa251/Self-Organizing-Map)
2. [**Step by step with examples, calculations**](https://mc.ai/self-organizing-mapsom/)
3. [**Adds intuition regarding “magnetism”’**](https://towardsdatascience.com/self-organizing-maps-1b7d2a84e065)
4. [**Implementation and faces**](https://medium.com/@navdeepsingh_2336/self-organizing-maps-for-machine-learning-algorithms-ad256a395fc5)**, intuition towards each node and what it represents in a vision. I.e., each face resembles one of K clusters.**
5. [**Medium on kohonen networks, i.e., SOM**](https://towardsdatascience.com/kohonen-self-organizing-maps-a29040d688da)
6. [**Som on iris**](https://towardsdatascience.com/self-organizing-maps-ff5853a118d4)**, explains inference - averaging, and cons of the method.**
7. [**Simple explanation**](https://medium.com/@valentinerutto/selforganizingmaps-in-english-35574f95b0ac)
8. [**Algorithm, formulas**](https://towardsdatascience.com/kohonen-self-organizing-maps-a29040d688da)
## **NEURO EVOLUTION (GA/GP based)**
**NEAT**
[**NEAT**](http://www.cs.ucf.edu/~kstanley/neat.html) \*\*stands for NeuroEvolution of Augmenting Topologies. It is a method for evolving artificial neural networks with a genetic algorithm.
NEAT implements the idea that it is most effective to start evolution with small, simple networks and allow them to become increasingly complex over generations.\*\*
\*\*That way, just as organisms in nature increased in complexity since the first cell, so do neural networks in NEAT.
This process of continual elaboration allows finding highly sophisticated and complex neural networks.\*\*
[**A great article about NEAT**](http://hunterheidenreich.com/blog/neuroevolution-of-augmenting-topologies/)
**HYPER-NEAT**
[**HyperNEAT**](http://eplex.cs.ucf.edu/hyperNEATpage/) \*\*computes the connectivity of its neural networks as a function of their geometry.
HyperNEAT is based on a theory of representation that hypothesizes that a good representation for an artificial neural network should be able to describe its pattern of connectivity compactly.\*\*
**The encoding in HyperNEAT, called** [**compositional pattern producing networks**](http://en.wikipedia.org/wiki/Compositional_pattern-producing_network)\*\*, is designed to represent patterns with regularities such as symmetry, repetition, and repetition with variationץ
(WIKI) **\[Compositional pattern-producing networks]\(****)** (CPPNs) are a variation of artificial neural networks (ANNs) that have an architecture whose evolution is guided by genetic algorithms\*\*
[**A great HyperNeat tutorial on Medium.**](https://towardsdatascience.com/hyperneat-powerful-indirect-neural-network-evolution-fba5c7c43b7b)
## **Radial Basis Function Network (RBFN)**
**+** [**RBF layer in Keras.**](https://github.com/PetraVidnerova/rbf_keras/blob/master/test.py)
**The** [**RBFN**](http://mccormickml.com/2013/08/15/radial-basis-function-network-rbfn-tutorial/) **approach is more intuitive than the MLP.**
* **An RBFN performs classification by measuring the input’s similarity to examples from the training set.**
* **Each RBFN neuron stores a “prototype”, which is just one of the examples from the training set.**
* **When we want to classify a new input, each neuron computes the Euclidean distance between the input and its prototype.**
* **Roughly speaking, if the input more closely resembles the class A prototypes than the class B prototypes, it is classified as class A.**
## **Bayesian Neural Network (BNN)**
[**BNN**](https://eng.uber.com/neural-networks-uncertainty-estimation/) **- (what is?)** [**Bayesian neural network (BNN)**](http://edwardlib.org/tutorials/bayesian-neural-network) **according to Uber - architecture that more accurately forecasts time series predictions and uncertainty estimations at scale. “how Uber has successfully applied this model to large-scale time series anomaly detection, enabling better accommodate rider demand during high-traffic intervals.”**
**Under the BNN framework, prediction uncertainty can be categorized into three types:**
1. **Model uncertainty captures our ignorance of the model parameters and can be reduced as more samples are collected.**
2. **model misspecification**
3. **inherent noise captures the uncertainty in the data generation process and is irreducible.**
**Note: in a series of articles, uber explains about time series and leads to a BNN architecture.**
1. [**Neural networks**](https://eng.uber.com/neural-networks/) **- training on multi-signal raw data, training X and Y are window-based and the window size(lag) is determined in advance.**
**Vanilla LSTM did not work properly, therefore an architecture of**
**Regarding point 1: ‘run prediction with dropout 100 times’**
**\*\*\*** [**MEDIUM with code how to do it.**](https://medium.com/hal24k-techblog/how-to-generate-neural-network-confidence-intervals-with-keras-e4c0b78ebbdf)
[**Why do we need a confidence measure when we have a softmax probability layer?**](https://hjweide.github.io/quantifying-uncertainty-in-neural-networks) **The blog post explains, for example, that with a CNN of apples, oranges, cat and dogs, a non related example such as a frog image may influence the network to decide its an apple, therefore we can’t rely on the probability as a confidence measure. The ‘run prediction with dropout 100 times’ should give us a confidence measure because it draws each weight from a bernoulli distribution.**
**“By applying dropout to all the weight layers in a neural network, we are essentially drawing each weight from a** [**Bernoulli distribution**](https://en.wikipedia.org/wiki/Bernoulli_distribution)**. In practice, this mean that we can sample from the distribution by running several forward passes through the network. This is referred to as** [**Monte Carlo dropout**](http://arxiv.org/abs/1506.02158)**.”**
**Taken from Yarin Gal’s** [**blog post**](http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce.html) **. In this figure we see how sporadic is the signal from a forward pass (black line) compared to a much cleaner signal from 100 dropout passes.**
**Is it applicable for time series? In the figure below he tried to predict the missing signal between each two dotted lines, A is a bad estimation, but with a dropout layer we can see that in most cases the signal is better predicted.**
**Going back to uber, they are actually using this idea to predict time series with LSTM, using encoder decoder framework.**
**Note: this is probably applicable in other types of networks.**
[**Phd Thesis by Yarin**](http://mlg.eng.cam.ac.uk/yarin/blog_2248.html?fref=gc\&dti=999449923520287)**, he talks about uncertainty in Neural networks and using BNNs. he may have proved this thesis, but I did not read it. This blog post links to his full Phd.**
**Old note:** [**The idea behind uncertainty is (**](http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce.html)[**paper here**](https://arxiv.org/pdf/1506.02142.pdf)**) that in order to trust your network’s classification, you drop some of the neurons during prediction, you do this \~100 times and you average the results. Intuitively this will give you confidence in your classification and increase your classification accuracy, because only a partial part of your network participated in the classification, randomly, 100 times. Please note that Softmax doesn't give you certainty.**
[**Medium post on prediction with drop out**](https://towardsdatascience.com/is-your-algorithm-confident-enough-1b20dfe2db08)
**The** [**solution for keras**](https://github.com/keras-team/keras/issues/9412) **says to add trainable=true for every dropout layer and add another drop out at the end of the model. Thanks sam.**
**“import keras**
**inputs = keras.Input(shape=(10,))**
**x = keras.layers.Dense(3)(inputs)**
**outputs = keras.layers.Dropout(0.5)(x, training=True)**
**model = keras.Model(inputs, outputs)“**
## **CONVOLUTIONAL NEURAL NET**
**(**[**an excellent and thorough explanation about LeNet**](https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/)**) -**
* **Convolution Layer primary purpose is to extract features from the input image. Convolution preserves the spatial relationship between pixels by learning image features using small squares of input data.**
* **ReLU (more in the activation chapter) - The purpose of ReLU is to introduce non-linearity in our ConvNet**
* **Spatial Pooling (also called subsampling or downsampling) reduces the dimensionality of each feature map but retains the most important information. Spatial Pooling can be of different types: Max, Average, Sum etc.**
* **Dense / Fully Connected - a traditional Multi Layer Perceptron that uses a softmax activation function in the output layer to classify. The output from the convolutional and pooling layers represent high-level features of the input image. The purpose of the Fully Connected layer is to use these features for classifying the input image into various classes based on the training dataset.**
**The overall training process of the Convolutional Network may be summarized as below:**
* **Step1: We initialize all filters and parameters / weights with random values**
* **Step2: The network takes a single training image as input, goes through the forward propagation step (convolution, ReLU and pooling operations along with forward propagation in the Fully Connected layer) and finds the output probabilities for each class.**
* **Let's say the output probabilities for the boat image above are \[0.2, 0.4, 0.1, 0.3]**
* **Since weights are randomly assigned for the first training example, output probabilities are also random.**
* **Step3: Calculate the total error at the output layer (summation over all 4 classes)**
* **(L2) Total Error = ∑ ½ (target probability – output probability) ²**
* **Step4: Use Backpropagation to calculate the gradients of the error with respect to all weights in the network and use gradient descent to update all filter values / weights and parameter values to minimize the output error.**
* **The weights are adjusted in proportion to their contribution to the total error.**
* **When the same image is input again, output probabilities might now be \[0.1, 0.1, 0.7, 0.1], which is closer to the target vector \[0, 0, 1, 0].**
* **This means that the network has learnt to classify this particular image correctly by adjusting its weights / filters such that the output error is reduced.**
* **Parameters like number of filters, filter sizes, architecture of the network etc. have all been fixed before Step 1 and do not change during training process – only the values of the filter matrix and connection weights get updated.**
* **Step5: Repeat steps 2-4 with all images in the training set.**
**The above steps train the ConvNet – this essentially means that all the weights and parameters of the ConvNet have now been optimized to correctly classify images from the training set.**
**When a new (unseen) image is input into the ConvNet, the network would go through the forward propagation step and output a probability for each class (for a new image, the output probabilities are calculated using the weights which have been optimized to correctly classify all the previous training examples). If our training set is large enough, the network will (hopefully) generalize well to new images and classify them into correct categories.**
[**Illustrated 10 CNNS architectures**](https://towardsdatascience.com/illustrated-10-cnn-architectures-95d78ace614d)
[**A study that deals with class imbalance in CNN’s**](https://arxiv.org/pdf/1710.05381.pdf) **- we systematically investigate the impact of class imbalance on classification performance of convolutional neural networks (CNNs) and compare frequently used methods to address the issue**
1. **Over sampling**
2. **Undersampling**
3. **Thresholding probabilities (ROC?)**
4. **Cost sensitive classification -different cost to misclassification**
5. **One class - novelty detection. This is a concept learning technique that recognizes positive instances rather than discriminating between two classes**
**Using several imbalance scenarios, on several known data sets, such as MNIST**
**The results indication (loosely) that oversampling is usually better in most cases, and doesn't cause overfitting in CNNs.**
**CONV-1D**
1. [**How to setup a conv1d in keras, most importantly how to reshape your input vector**](https://stackoverflow.com/questions/43396572/dimension-of-shape-in-conv1d/43399308#43399308)
2. [**Mastery on Character ngram cnn for sentiment analysis**](https://machinelearningmastery.com/develop-n-gram-multichannel-convolutional-neural-network-sentiment-analysis/)
**1x1 CNN**
1. [**Mastery**](https://machinelearningmastery.com/introduction-to-1x1-convolutions-to-reduce-the-complexity-of-convolutional-neural-networks/) **on 1x1 cnn, for dim reduction, decreasing feature maps and other usages.**
1. **“This is the most common application of this type of filter and in this way, the layer is often called a feature map pooling layer.”**
2. **“In the paper, the authors propose the need for an MLP convolutional layer and the need for cross-channel pooling to promote learning across channels.”**
3. **“the 1×1 filter was used explicitly for dimensionality reduction and for increasing the dimensionality of feature maps after pooling in the design of the inception module, used in the GoogLeNet model”**
4. **“The 1×1 filter was used as a projection technique to match the number of filters of input to the output of residual modules in the design of the residual network “**
5.
**MASKED R-CNN**
[**1. Using mask rnn for object detection**](https://machinelearningmastery.com/how-to-perform-object-detection-in-photographs-with-mask-r-cnn-in-keras/)
**Invariance in CNN**
1. [**Making cnn shift invariance**](https://richzhang.github.io/antialiased-cnns/) **- “Small shifts -- even by a single pixel -- can drastically change the output of a deep network (bars on left). We identify the cause: aliasing during downsampling. We anti-alias modern deep networks with classic signal processing, stabilizing output classifications (bars on right). We even observe accuracy increases (see plot below).**
**MAX AVERAGE POOLING**
[**Intuitions to the differences between max and average pooling:**](https://stats.stackexchange.com/questions/291451/feature-extracted-by-max-pooling-vs-mean-pooling)
1. **A max-pool layer compressed by taking the maximum activation in a block. If you have a block with mostly small activation, but a small bit of large activation, you will loose the information on the low activations. I think of this as saying "this type of feature was detected in this general area".**
2. **A mean-pool layer compresses by taking the mean activation in a block. If large activations are balanced by negative activations, the overall compressed activations will look like no activation at all. On the other hand, you retain some information about low activations in the previous example.**
3. **MAX pooling In other words: Max pooling roughly means that only those features that are most strongly triggering outputs are used in the subsequent layers. You can look at it a little like focusing the network’s attention on what’s most characteristic for the image at hand.**
4. [**GLOBAL MAX pooling**](https://alexisbcook.github.io/2017/global-average-pooling-layers-for-object-localization/)**: In the last few years, experts have turned to global average pooling (GAP) layers to minimize overfitting by reducing the total number of parameters in the model. Similar to max pooling layers, GAP layers are used to reduce the spatial dimensions of a three-dimensional tensor. However, GAP layers perform a more extreme type of dimensionality reduction,**
5. [**Hinton’s controversy thoughts on pooling**](https://mirror2image.wordpress.com/2014/11/11/geoffrey-hinton-on-max-pooling-reddit-ama/)
**Dilated CNN**
1. [**For improved performance**](https://stackoverflow.com/questions/41178576/whats-the-use-of-dilated-convolutions)
2. \*\*\*\*[**RESNET, DENSENET UNET**](https://medium.com/swlh/resnets-densenets-unets-6bbdbcfdf010) **- the trick behind them, concatenating both f(x) = x**
## **Graph Convolutional Networks**
[**Explaination here, with some examples**](https://tkipf.github.io/graph-convolutional-networks/)
## **CAPSULE NEURAL NETS**
1. [**The solution to CNN’s shortcomings**](https://hackernoon.com/capsule-networks-are-shaking-up-ai-heres-how-to-use-them-c233a0971952)**, where features can be identified without relations to each other in an image, i.e. changing the location of body parts will not affect the classification, and changing the orientation of the image will. The promise of capsule nets is that these two issues are solved.**
2. [**Understanding capsule nets - part 2,**](https://medium.com/ai%C2%B3-theory-practice-business/understanding-hintons-capsule-networks-part-ii-how-capsules-work-153b6ade9f66) **there are more parts to the series**
## **Transfer Learning using CNN**
1. **To Add keras book chapter 5 (i think)**
2. [**Mastery**](https://machinelearningmastery.com/how-to-use-transfer-learning-when-developing-convolutional-neural-network-models/) **on TL using CNN**
1. **Classifier: The pre-trained model is used directly to classify new images.**
2. **Standalone Feature Extractor: The pre-trained model, or some portion of the model, is used to pre-process images and extract relevant features.**
3. **Integrated Feature Extractor: The pre-trained model, or some portion of the model, is integrated into a new model, but layers of the pre-trained model are frozen during training.**
4. **Weight Initialization: The pre-trained model, or some portion of the model, is integrated into a new model, and the layers of the pre-trained model are trained in concert with the new model.**
## **VISUALIZE CNN**
1. [**How to**](https://towardsdatascience.com/how-to-visualize-convolutional-features-in-40-lines-of-code-70b7d87b0030)
## **Recurrent Neural Net (RNN)**
### **RNN**
**a basic NN node with a loop, previous output is merged with current input (using tanh?), for the purpose of remembering history, for time series - to predict the next X based on the previous Y.**
**(What is RNN?) by Andrej Karpathy -** [**The Unreasonable Effectiveness of Recurrent Neural Networks**](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)**, basically a lot of information about RNNs and their usage cases 1 to N = frame captioning**
* **N to 1 = classification**
* **N to N = predict frames in a movie**
* **N\2 with time delay to N\2 = predict supply and demand**
* **Vanishing gradient is 100 times worse.**
* **Gate networks like LSTM solves vanishing gradient.**
**(how to initialize?)** [**Benchmarking RNN networks for text**](https://danijar.com/benchmarking-recurrent-networks-for-language-modeling) **- don't worry about initialization, use normalization and GRU for big networks.**
**\*\* Experimental improvements:**
[**Ref**](https://arxiv.org/abs/1709.02755) **- ”Simplified RNN, with pytorch implementation” - changing the underlying mechanism in RNNs for the purpose of parallelizing calculation, seems to work nicely in terms of speed, not sure about state of the art results.** [**Controversy regarding said work**](https://www.facebook.com/cho.k.hyun/posts/10208564563785149)**, author claims he already mentioned these ideas (QRNN)** [**first**](https://www.reddit.com/r/MachineLearning/comments/6zduh2/r_170902755_training_rnns_as_fast_as_cnns/dmv9gnh/)**, a year before, however it seems like his ideas have also been reviewed as** [**incremental**](https://openreview.net/forum?id=H1zJ-v5xl) **(PixelRNN). Its probably best to read all 3 papers in chronological order and use the most optimal solution.**
[**RNNCELLS - recurrent shop**](https://github.com/farizrahman4u/recurrentshop)**, enables you to build complex rnns with keras. Details on their significance are inside the link**
**Masking for RNNs - the ideas is simple, we want to use variable length inputs, although rnns do use that, they require a fixed size input. So masking of 1’s and 0’s will help it understand the real size or where the information is in the input. Motivation: Padded inputs are going to contribute to our loss and we dont want that.**
[**Source 1**](https://www.quora.com/What-is-masking-in-a-recurrent-neural-network-RNN)**,** [**source 2**](https://r2rt.com/recurrent-neural-networks-in-tensorflow-iii-variable-length-sequences.html)**,**
**Visual attention RNNS - Same idea as masking but on a window-based cnn.** [**Paper**](https://papers.nips.cc/paper/5542-recurrent-models-of-visual-attention.pdf)
### **LSTM**
* [**The best, hands down, lstm post out there**](https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21)
* **LSTM -** [**what is?**](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) **the first reference for LSTM on the web, but you should know the background before reading.**
* 
* [**Hidden state vs cell state**](https://www.quora.com/How-is-the-hidden-state-h-different-from-the-memory-c-in-an-LSTM-cell) **- you have to understand this concept before you dive in. i.e, Hidden state is overall state of what we have seen so far. Cell state is selective memory of the past. The hidden state (h) carries the information about what an RNN cell has seen over the time and supply it to the present time such that a loss function is not just dependent upon the data it is seeing in this time instant, but also, data it has seen historically.**
* [**Illustrated rnn lstm gru**](https://towardsdatascience.com/animated-rnn-lstm-and-gru-ef124d06cf45)
* [**Paper**](https://arxiv.org/pdf/1503.04069.pdf) **- a comparison of many LSTMs variants and they are pretty much the same performance wise**
* [**Paper**](https://arxiv.org/pdf/1503.04069.pdf) **- comparison of lstm variants, vanilla is mostly the best, forget and output gates are the most important in terms of performance. Other conclusions in the paper..**
* **Master on** [**unrolling RNN’s introductory post**](https://machinelearningmastery.com/rnn-unrolling/)
* **Mastery on** [**under/over fitting lstms**](https://machinelearningmastery.com/diagnose-overfitting-underfitting-lstm-models/) **- but makes sense for all types of networks**
* **Mastery on r**[**eturn\_sequence and return\_state in keras LSTM**](https://machinelearningmastery.com/return-sequences-and-return-states-for-lstms-in-keras/)
* **That return sequences return the hidden state output for each input time step.**
* **That return state returns the hidden state output and cell state for the last input time step.**
* **That return sequences and return state can be used at the same time.**
* **Mastery on** [**understanding stateful vs stateless**](https://machinelearningmastery.com/understanding-stateful-lstm-recurrent-neural-networks-python-keras/)**,** [**stateful stateless for time series**](https://machinelearningmastery.com/stateful-stateless-lstm-time-series-forecasting-python/)
* **Mastery on** [**timedistributed layer**](https://machinelearningmastery.com/timedistributed-layer-for-long-short-term-memory-networks-in-python/) **and seq2seq**
* **TimeDistributed Layer - used to connect 3d inputs from lstms to dense layers, in order to utilize the time element. Otherwise it gets flattened when the connection is direct, nulling the lstm purpose. Note: nice trick that doesn't increase the dense layer structure multiplied by the number of dense neurons. It loops for each time step! I.e., The TimeDistributed achieves this trick by applying the same Dense layer (same weights) to the LSTMs outputs for one time step at a time. In this way, the output layer only needs one connection to each LSTM unit (plus one bias).**
**For this reason, the number of training epochs needs to be increased to account for the smaller network capacity. I doubled it from 500 to 1000 to match the first one-to-one example**
* **Sequence Learning Problem**
* **One-to-One LSTM for Sequence Prediction**
* **Many-to-One LSTM for Sequence Prediction (without TimeDistributed)**
* **Many-to-Many LSTM for Sequence Prediction (with TimeDistributed)**
* * **Mastery on** [**wrapping cnn-lstm with time distributed**](https://machinelearningmastery.com/cnn-long-short-term-memory-networks/)**, as a whole model wrap, or on every layer in the model which is equivalent and preferred.**
* **Master on** [**visual examples**](https://machinelearningmastery.com/sequence-prediction/) **for sequence prediction**
* **Unread - sentiment classification of IMDB movies using** [**Keras and LSTM**](http://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/)
* [**Very important - how to interpret LSTM neurons in keras**](https://yerevann.github.io/2017/06/27/interpreting-neurons-in-an-LSTM-network/)
* [**LSTM for time-series**](http://www.jakob-aungiers.com/articles/a/LSTM-Neural-Network-for-Time-Series-Prediction) **- (jakob) single point prediction, sequence prediction and shifted-sequence prediction with code.**
**Stateful vs Stateless: crucial for understanding how to leverage LSTM networks:**
1. [**A good description on what it is and how to use it.**](https://groups.google.com/forum/#!topic/keras-users/l1RV_tthjoY)
2. [**ML mastery**](https://machinelearningmastery.com/stateful-stateless-lstm-time-series-forecasting-python/) *\*\**
3. [**Philippe remy**](http://philipperemy.github.io/keras-stateful-lstm/) **on stateful vs stateless, intuition mostly with code, but not 100% clear**
**Machine Learning mastery:**
[**A good tutorial on LSTM:**](https://machinelearningmastery.com/time-series-forecasting-long-short-term-memory-network-python/) **important notes:**
**1. Scale to -1,1, because the internal activation in the lstm cell is tanh.**
**2.**[**stateful**](https://machinelearningmastery.com/understanding-stateful-lstm-recurrent-neural-networks-python-keras/) **- True, needs to reset internal states, False =stateless. Great info & results** [**HERE**](https://machinelearningmastery.com/stateful-stateless-lstm-time-series-forecasting-python/)**, with seeding, with training resets (and not) and predicting resets (and not) - note: empirically matching the shampoo input, network config, etc.**
[**Another explanation/tutorial about stateful lstm, should be thorough.**](http://philipperemy.github.io/keras-stateful-lstm/)
**3.** [**what is return\_sequence, return\_states**](https://machinelearningmastery.com/return-sequences-and-return-states-for-lstms-in-keras/)**, and how to use each one and both at the same time.**
**Return\_sequence is needed for stacked LSTM layers.**
**4.**[**stacked LSTM**](https://machinelearningmastery.com/stacked-long-short-term-memory-networks/) **- each layer has represents a higher level of abstraction in TIME!**
[**Keras Input shape**](https://stackoverflow.com/questions/44747343/keras-input-explanation-input-shape-units-batch-size-dim-etc) **- a good explanation about differences between input\_shape, dim, and what is. Additionally about layer calculation of inputs and output based on input shape, and sequence model vs API model.**
**A** [**comparison**](https://danijar.com/language-modeling-with-layer-norm-and-gru/) **of LSTM/GRU/MGU with batch normalization and various initializations, GRu/Xavier/Batch are the best and recommended for RNN**
[**Benchmarking LSTM variants**](http://proceedings.mlr.press/v37/jozefowicz15.pdf)**: - it looks like LSTM and GRU are competitive to mutation (i believe its only in pytorch) adding a bias to LSTM works (a bias of 1 as recommended in the** [**paper**](https://pdfs.semanticscholar.org/1154/0131eae85b2e11d53df7f1360eeb6476e7f4.pdf)**), but generally speaking there is no conclusive empirical evidence that says one type of network is better than the other for all tests, but the mutated networks tend to win over lstm\gru variants.**
[**BIAS 1 in keras**](https://keras.io/layers/recurrent/#lstm) **- unit\_forget\_bias: Boolean. If True, add 1 to the bias of the forget gate at initializationSetting it to true will also force bias\_initializer="zeros". This is recommended in** [**Jozefowicz et al.**](http://www.jmlr.org/proceedings/papers/v37/jozefowicz15.pdf)
[**Validation\_split arg**](https://www.quora.com/What-is-the-importance-of-the-validation-split-variable-in-Keras) **- The validation split variable in Keras is a value between \[0..1]. Keras proportionally split your training set by the value of the variable. The first set is used for training and the 2nd set for validation after each epoch.**
**This is a nice helper add-on by Keras, and most other Keras examples you have seen the training and test set was passed into the fit method, after you have manually made the split. The value of having a validation set is significant and is a vital step to understand how well your model is training. Ideally on a curve you want your training accuracy to be close to your validation curve, and the moment your validation curve falls below your training curve the alarm bells should go off and your model is probably busy over-fitting.**
**Keras is a wonderful framework for deep learning, and there are many different ways of doing things with plenty of helpers.**
[**Return\_sequence**](https://stackoverflow.com/questions/42755820/how-to-use-return-sequences-option-and-timedistributed-layer-in-keras)**: unclear.**
[**Sequence.pad\_sequences**](https://stackoverflow.com/questions/42943291/what-does-keras-io-preprocessing-sequence-pad-sequences-do) **- using maxlength it will either pad with zero if smaller than, or truncate it if bigger.**
[**Using batch size for LSTM in Keras**](https://machinelearningmastery.com/use-different-batch-sizes-training-predicting-python-keras/)
**Imbalanced classes? Use** [**class\_weight**](https://stackoverflow.com/questions/43459317/keras-class-weight-vs-sample-weights-in-the-fit-generator)**s, another explanation** [**here**](https://stackoverflow.com/questions/43459317/keras-class-weight-vs-sample-weights-in-the-fit-generator) **about class\_weights and sample\_weights.**
**SKlearn Formula for balanced class weights and why it works,** [**example**](https://stackoverflow.com/questions/50152377/in-sklearn-logistic-regression-class-balanced-helps-run-the-model-with-imbala/50154388)
[**number of units in LSTM**](https://www.quora.com/What-is-the-meaning-of-%E2%80%9CThe-number-of-units-in-the-LSTM-cell)
[**Calculate how many params are in an LSTM layer?**](https://stackoverflow.com/questions/38080035/how-to-calculate-the-number-of-parameters-of-an-lstm-network)
[**Understanding timedistributed in Keras**](https://machinelearningmastery.com/timedistributed-layer-for-long-short-term-memory-networks-in-python/)**, but with focus on lstm one to one, one to many and many to many - here the timedistributed is applying a dense layer to each output neuron from the lstm, which returned\_sequence = true for that purpose.**
**This tutorial clearly shows how to manipulate input construction, lstm output neurons and the target layer for the purpose of those three problems (1:1, 1:m, m:m).**
**BIDIRECTIONAL LSTM**
**(what is?) Wiki - The basic idea of BRNNs is to connect two hidden layers of opposite directions to the same output. By this structure, the output layer can get information from past and future states.**
**BRNN are especially useful when the context of the input is needed. For example, in handwriting recognition, the performance can be enhanced by knowledge of the letters located before and after the current letter.**
[**Another**](https://machinelearningmastery.com/develop-bidirectional-lstm-sequence-classification-python-keras/) **explanation- It involves duplicating the first recurrent layer in the network so that there are now two layers side-by-side, then providing the input sequence as-is as input to the first layer and providing a reversed copy of the input sequence to the second.**
**.. It allows you to specify the merge mode, that is how the forward and backward outputs should be combined before being passed on to the next layer. The options are:**
* **‘sum‘: The outputs are added together.**
* **‘mul‘: The outputs are multiplied together.**
* **‘concat‘: The outputs are concatenated together (the default), providing double the number of outputs to the next layer.**
* **‘ave‘: The average of the outputs is taken.**
**The default mode is to concatenate, and this is the method often used in studies of bidirectional LSTMs.**
[**Another simplified example**](https://stackoverflow.com/questions/43035827/whats-the-difference-between-a-bidirectional-lstm-and-an-lstm)
### **BACK PROPAGATION**
[**A great Slide about back prop, on a simple 3 neuron network, with very easy to understand calculations.**](https://www.slideshare.net/AhmedGadFCIT/backpropagation-understanding-how-to-update-anns-weights-stepbystep)
### **UNSUPERVISED LSTM**
1. [**Paper**](ftp://ftp.idsia.ch/pub/juergen/icann2001unsup.pdf)**,** [**paper2**](https://arxiv.org/pdf/1502.04681.pdf)**,** [**paper3**](https://arxiv.org/abs/1709.02081)
2. [**In keras**](https://www.reddit.com/r/MachineLearning/comments/4adrie/unsupervised_lstm_using_keras/)
### **GRU**
[**A tutorial about GRU**](https://towardsdatascience.com/understanding-gru-networks-2ef37df6c9be) **- To solve the vanishing gradient problem of a standard RNN, GRU uses, so called, update gate and reset gate. Basically, these are two vectors which decide what information should be passed to the output. The special thing about them is that they can be trained to keep information from long ago, without washing it through time or remove information which is irrelevant to the prediction.**
1. **update gate helps the model to determine how much of the past information (from previous time steps) needs to be passed along to the future.**
2. **Reset gate essentially, this gate is used from the model to decide how much of the past information to forget.**
**RECURRENT WEIGHTED AVERAGE (RNN-WA)**
**What is? (a type of cell that converges to higher accuracy faster than LSTM.**
**it implements attention into the recurrent neural network:**
**1. the keras implementation is available at** [**https://github.com/keisuke-nakata/rwa**](https://github.com/keisuke-nakata/rwa) *\*\**
**2. the whitepaper is at** [**https://arxiv.org/pdf/1703.01253.pdf**](https://arxiv.org/pdf/1703.01253.pdf)
### **QRNN**
[**Potential competitor to the transformer**](https://towardsdatascience.com/qrnn-a-potential-competitor-to-the-transformer-86b5aef6c137)
## **GRAPH NEURAL NETWORKS (GNN)**
1. **(amazing)** [**Why i am luke warm about GNN’s**](https://www.singlelunch.com/2020/12/28/why-im-lukewarm-on-graph-neural-networks/) **- really good insight to what they do (compressing data, vs adjacy graphs, vs graphs, high dim relations, etc.)**
2. (amazing) [Graphical intro to GNNs](https://distill.pub/2021/gnn-intro/)
3. [**Learning on graphs youtube - uriel singer**](https://www.youtube.com/watch?v=snLsWos_1WU\&feature=youtu.be\&fbclid=IwAR0JlvF9aPgKMmeh2zGr3l3j_8AebOTjknVGyMsz0Y2EvgcqrS0MmLkBTMU)
4. [**Benchmarking GNN’s, methodology, git, the works.**](https://graphdeeplearning.github.io/post/benchmarking-gnns/)
5. [**Awesome graph classification on github**](https://github.com/benedekrozemberczki/awesome-graph-classification)
6. **Octavian in medium on graphs,** [**A really good intro to graph networks, too long too summarize**](https://medium.com/octavian-ai/deep-learning-with-knowledge-graphs-3df0b469a61a)**, clever, mcgraph, regression, classification, embedding on graphs.**
7. [**Application of graph networks**](https://towardsdatascience.com/https-medium-com-aishwaryajadhav-applications-of-graph-neural-networks-1420576be574) *\*\**
8. [**Recommender systems using GNN**](https://towardsdatascience.com/recommender-systems-applying-graph-and-nlp-techniques-619dbedd9ecc)**, w2v, pytorch w2v, networkx, sparse matrices, matrix factorization, dictionary optimization, part 1 here** [**(how to find product relations, important: creating negative samples)**](https://eugeneyan.com/2020/01/06/recommender-systems-beyond-the-user-item-matrix)
9. [**Transformers are GNN**](https://towardsdatascience.com/transformers-are-graph-neural-networks-bca9f75412aa)**, original:** [**Transformers are graphs, not the typical embedding on a graph, but a more holistic approach to understanding text as a graph.**](https://thegradient.pub/transformers-are-graph-neural-networks/)
10. [**Cnn for graphs**](https://towardsdatascience.com/how-to-do-deep-learning-on-graphs-with-graph-convolutional-networks-62acf5b143d0)
11. [**Staring with gnn**](https://medium.com/octavian-ai/how-to-get-started-with-machine-learning-on-graphs-7f0795c83763)
12. **Really good -** [**Basics deep walk and graphsage**](https://towardsdatascience.com/a-gentle-introduction-to-graph-neural-network-basics-deepwalk-and-graphsage-db5d540d50b3) *\*\**
13. [**Application of gnn**](https://towardsdatascience.com/https-medium-com-aishwaryajadhav-applications-of-graph-neural-networks-1420576be574)
14. **Michael Bronstein’s** [**Central page for Graph deep learning articles on Medium**](https://towardsdatascience.com/graph-deep-learning/home) **(worth reading)**
15. [**GAT graphi attention networks**](https://petar-v.com/GAT/)**, paper, examples - The graph attentional layer utilised throughout these networks is computationally efficient (does not require costly matrix operations, and is parallelizable across all nodes in the graph), allows for (implicitly) assigning different importances to different nodes within a neighborhood while dealing with different sized neighborhoods, and does not depend on knowing the entire graph structure upfront—thus addressing many of the theoretical issues with approaches.**
16. **Medium on** [**Intro, basics, deep walk, graph sage**](https://towardsdatascience.com/a-gentle-introduction-to-graph-neural-network-basics-deepwalk-and-graphsage-db5d540d50b3)
17. [Struc2vec](https://leoribeiro.github.io/struc2vec.html), [youtube](https://www.youtube.com/watch?v=lu0xMOO48Xo\&embeds_euri=https%3A%2F%2Fleoribeiro.github.io%2F\&source_ve_path=MjM4NTE\&feature=emb_title): Learning Node Representations from Structural Identity- The *struc2vec* algorithm learns continuous representations for nodes in any graph. struc2vec captures structural equivalence between nodes.
### GNN courses
1. [machine learning with graphs by Stanford](http://web.stanford.edu/class/cs224w/?fbclid=IwAR0nQR4lhyKCoTchsGQrcZ5E8EPBt2Bi4d8K8MYX-UN0ygQSxQ5bMoohhis), from ML to GNN.
2. [Graph deep learning course](https://geometricdeeplearning.com/lectures/) - graphs, sets, groups, GNNs. [youtube](https://www.youtube.com/watch?app=desktop\&v=w6Pw4MOzMuo)
### **Deep walk**
1. [**Git**](https://github.com/phanein/deepwalk)
2. [**Paper**](https://arxiv.org/abs/1403.6652)
3. [**Medium**](https://medium.com/@_init_/an-illustrated-explanation-of-using-skipgram-to-encode-the-structure-of-a-graph-deepwalk-6220e304d71b) **and medium on** [**W2v, deep walk, graph2vec, n2v**](https://towardsdatascience.com/graph-embeddings-the-summary-cc6075aba007)
### **Node2vec**
1. [**Git**](https://github.com/eliorc/node2vec)
2. [**Stanford**](https://snap.stanford.edu/node2vec/)
3. [**Elior on medium**](https://towardsdatascience.com/node2vec-embeddings-for-graph-data-32a866340fef)**,** [**youtube**](https://www.youtube.com/watch?v=828rZgV9t1g)
4. [**Paper**](https://cs.stanford.edu/~jure/pubs/node2vec-kdd16.pdf)
### **Graphsage**
1. [**medium**](https://towardsdatascience.com/a-gentle-introduction-to-graph-neural-network-basics-deepwalk-and-graphsage-db5d540d50b3)
### **SDNE - structural deep network embedding**
1. [**medium**](https://towardsdatascience.com/graph-embeddings-the-summary-cc6075aba007)
### **Diff2vec**
1. [**Git**](https://github.com/benedekrozemberczki/diff2vec)
2. 
### **Splitter**
**,** [**git**](https://github.com/benedekrozemberczki/Splitter)**,** [**paper**](http://epasto.org/papers/www2019splitter.pdf)**, “Is a Single Embedding Enough? Learning Node Representations that Capture Multiple Social Contexts”**
**Recent interest in graph embedding methods has focused on learning a single representation for each node in the graph. But can nodes really be best described by a single vector representation? In this work, we propose a method for learning multiple representations of the nodes in a graph (e.g., the users of a social network). Based on a principled decomposition of the ego-network, each representation encodes the role of the node in a different local community in which the nodes participate. These representations allow for improved reconstruction of the nuanced relationships that occur in the graph a phenomenon that we illustrate through state-of-the-art results on link prediction tasks on a variety of graphs, reducing the error by up to 90%. In addition, we show that these embeddings allow for effective visual analysis of the learned community structure.**
**16.** [**Self clustering graph embeddings**](https://github.com/benedekrozemberczki/GEMSEC)
**17.** [**Walklets**](https://github.com/benedekrozemberczki/walklets?fbclid=IwAR2ymD7lbgP_sUde5UvKGZp7TYYYmACMFJS6UGNjqW29ethONHy7ibmDL0Q)**, similar to deep walk with node skips. - lots of improvements, works in scale due to lower size representations, improves results, etc.**
**Nodevectors**
[**Git**](https://github.com/VHRanger/nodevectors)**, The fastest network node embeddings in the west**
## **SIGNAL PROCESSING NN (FFT, WAVELETS, SHAPELETS)**
1. [**Fourier Transform**](https://www.youtube.com/watch?v=spUNpyF58BY) **- decomposing frequencies**
2. [**WAVELETS On youtube (4 videos)**](https://www.youtube.com/watch?v=QX1-xGVFqmw)**:**
1. [**used for denoising**](https://www.youtube.com/watch?v=veCvP1mYpww)**, compression, detect edges, detect features with various orientation, analyse signal power, detect and localize transients, change points in time series data and detect optimal signal representation (peaks etc) of time freq analysis of images and data.**
2. **Can also be used to** [**reconstruct time and frequencies**](https://www.youtube.com/watch?v=veCvP1mYpww)**, analyse images in space, frequencies, orientation, identifying coherent time oscillation in time series**
3. **Analyse signal variability and correlation**
4.
## **HIERARCHICAL RNN**
1. [**githubcode**](https://github.com/keras-team/keras/blob/master/examples/mnist_hierarchical_rnn.py)
## **NN-Sequence Analysis**
**(did not read)** [**A causal framework for explaining the predictions of black-box sequence-to-sequence models**](http://people.csail.mit.edu/tommi/papers/AlvJaa_EMNLP2017.pdf) **- can this be applied to other time series prediction?**
## **SIAMESE NETWORKS (one shot)**
1. [**Siamese CNN, learns a similarity between images, not to classify**](https://medium.com/predict/face-recognition-from-scratch-using-siamese-networks-and-tensorflow-df03e32f8cd0)
2. [**Visual tracking, explains contrastive and triplet loss**](https://medium.com/intel-student-ambassadors/siamese-networks-for-visual-tracking-96262eaaba77)
3. [**One shot learning, very thorough, baseline vs siamese**](https://towardsdatascience.com/one-shot-learning-with-siamese-networks-using-keras-17f34e75bb3d)
4. [**What is triplet loss**](https://towardsdatascience.com/siamese-network-triplet-loss-b4ca82c1aec8)
5. **MULTI NETWORKS**
6. [**Google whitening black boxes using multi nets, segmentation and classification**](https://medium.com/health-ai/google-deepmind-might-have-just-solved-the-black-box-problem-in-medical-ai-3ed8bc21f636)
# Deep Network Optimization
## **PRUNING / KNOWLEDGE DISTILLATION / LOTTERY TICKET**
1. [**Awesome Knowledge distillation**](https://github.com/dkozlov/awesome-knowledge-distillation)
2. **Lottery ticket**
1. [**1**](https://towardsdatascience.com/breaking-down-the-lottery-ticket-hypothesis-ca1c053b3e58)**,** [**2**](https://arxiv.org/pdf/1803.03635.pdf)**-paper**
2. [**Uber on Lottery ticket, masking weights retraining**](https://eng.uber.com/deconstructing-lottery-tickets/?utm_campaign=the_algorithm.unpaid.engagement\&utm_source=hs_email\&utm_medium=email\&utm_content=72562707&_hsenc=p2ANqtz--3mi4IwIFWZsW8UaWeuiv2nCzXDXattjRENzdKT-7J6wc7ftReuDXbn39mxCnX5y18o3z7cXfxPXQgysBMJnVnfeYpHg&_hsmi=72562707)
3. [**Facebook article and paper**](https://ai.facebook.com/blog/understanding-the-generalization-of-lottery-tickets-in-neural-networks)
3. [**Knowledge distillation 1**](https://medium.com/neuralmachine/knowledge-distillation-dc241d7c2322)**,** [**2**](https://towardsdatascience.com/knowledge-distillation-a-technique-developed-for-compacting-and-accelerating-neural-nets-732098cde690)**,** [**3**](https://medium.com/neuralmachine/knowledge-distillation-dc241d7c2322)
4. [**Pruning 1**](https://towardsdatascience.com/scooping-into-model-pruning-in-deep-learning-da92217b84ac)**,** [**2**](https://towardsdatascience.com/pruning-deep-neural-network-56cae1ec5505)
5. [**Teacher-student knowledge distillation**](https://towardsdatascience.com/model-distillation-and-compression-for-recommender-systems-in-pytorch-5d81c0f2c0ec) **focusing on Knowledge & Ranking distillation**
1. [**Deep network compression using teacher student**](https://github.com/Zhengyu-Li/Deep-Network-Compression-based-on-Student-Teacher-Network-)
2. [**Lottery ticket on BERT**](https://thegradient.pub/when-bert-plays-the-lottery-all-tickets-are-winning/)**, magnitude vs structured pruning on a various metrics, i.e., LT works on bert. The classical Lottery Ticket Hypothesis was mostly tested with unstructured pruning, specifically magnitude pruning (m-pruning) where the weights with the lowest magnitude are pruned irrespective of their position in the model. We iteratively prune 10% of the least magnitude weights across the entire fine-tuned model (except the embeddings) and evaluate on dev set, for as long as the performance of the pruned subnetwork is above 90% of the full model.**
**We also experiment with structured pruning (s-pruning) of entire components of BERT architecture based on their importance scores: specifically, we 'remove' the least important self-attention heads and MLPs by applying a mask. In each iteration, we prune 10% of BERT heads and 1 MLP, for as long as the performance of the pruned subnetwork is above 90% of the full model. To determine which heads/MLPs to prune, we use a loss-based approximation: the importance scores proposed by** [**Michel, Levy and Neubig (2019)**](https://thegradient.pub/when-bert-plays-the-lottery-all-tickets-are-winning/#RefMichel) **for self-attention heads, which we extend to MLPs. Please see our paper and the original formulation for more details.**
1. **Troubleshooting Neural Nets**
**(**[**37 reasons**](https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607?fref=gc\&dti=543283492502370)**,** [**10 more**](http://theorangeduck.com/page/neural-network-not-working?utm_campaign=Revue%20newsletter\&utm_medium=Newsletter\&utm_source=The%20Wild%20Week%20in%20AI\&fref=gc\&dti=543283492502370)**) - copy pasted and rewritten here for convenience, it's pretty thorough, but long and extensive, you should have some sort of intuition and not go through all of these. The following list is has much more insight and information in the article itself.**
**The author of the original article suggests to turn everything off and then start building your network step by step, i.e., “a divide and conquer ‘debug’ method”.**
**Dataset Issues**
**1. Check your input data - for stupid mistakes**
**2. Try random input - if the error behaves the same on random data, there is a problem in the net. Debug layer by layer**
**3. Check the data loader - input data is possibly broken. Check the input layer.**
**4. Make sure input is connected to output - do samples have correct labels, even after shuffling?**
**5. Is the relationship between input and output too random? - the input are not sufficiently related to the output. Its pretty amorphic, just look at the data.**
**6. Is there too much noise in the dataset? - badly labelled datasets.**
**7. Shuffle the dataset - useful to counteract order in the DS, always shuffle input and labels together.**
**8. Reduce class imbalance - imbalance datasets may add a bias to class prediction. Balance your class, your loss, do something.**
**9. Do you have enough training examples? - training from scratch? \~1000 images per class, \~probably similar numbers for other types of samples.**
**10. Make sure your batches don’t contain a single label - this is probably something you wont notice and will waste a lot of time figuring out! In certain cases shuffle the DS to prevent batches from having the same label.**
**11. Reduce batch size -** [**This paper**](https://arxiv.org/abs/1609.04836) **points out that having a very large batch can reduce the generalization ability of the model. However, please note that I found other references that claim a too small batch will impact performance.**
**12. Test on well known Datasets**
**Data Normalization/Augmentation**
**12. Standardize the features - zero mean and unit variance, sounds like normalization.**
**13. Do you have too much data augmentation?**
**Augmentation has a regularizing effect. Too much of this combined with other forms of regularization (weight L2, dropout, etc.) can cause the net to underfit.**
**14. Check the preprocessing of your pretrained model - with a pretrained model make sure your input data is similar in range\[0, 1], \[-1, 1] or \[0, 255]?**
**15. Check the preprocessing for train/validation/test set - CS231n points out a** [**common pitfall**](http://cs231n.github.io/neural-networks-2/#datapre)**:**
**Any preprocessing should be computed ONLY on the training data, then applied to val/test**
**Implementation issues**
**16. Try solving a simpler version of the problem -divide and conquer prediction, i.e., class and box coordinates, just use one.**
**17. Look for correct loss “at chance” - calculat loss for chance level, i.e 10% baseline is -ln(0.1) = 2.3 Softmax loss is the negative log probability. Afterwards increase regularization strength which should increase the loss.**
**18. Check your custom loss function.**
**19. Verify loss input - parameter confusion.**
**20. Adjust loss weights -If your loss is composed of several smaller loss functions, make sure their magnitude relative to each is correct. This might involve testing different combinations of loss weights.**
**21. Monitor other metrics -like accuracy.**
**22. Test any custom layers, debugging them.**
**23. Check for “frozen” layers or variables - accidentally frozen?**
**24. Increase network size - more layers, more neurons.**
**25. Check for hidden dimension errors - confusion due to vectors ->(64, 64, 64)**
**26. Explore Gradient checking -does your backprop work for custon gradients?** [**1**](http://ufldl.stanford.edu/tutorial/supervised/DebuggingGradientChecking/) **\*\*\[2]\(****) \*\***[**3**](https://www.coursera.org/learn/machine-learning/lecture/Y3s6r/gradient-checking)**.**
**Training issues**
**27. Solve for a really small dataset - can you generalize on 2 samples?**
**28. Check weights initialization -** [**Xavier**](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf) **or** [**He**](http://www.cv-foundation.org/openaccess/content_iccv_2015/papers/He_Delving_Deep_into_ICCV_2015_paper.pdf) **or forget about it for networks such as RNN.**
**29. Change your hyperparameters - grid search**
**30. Reduce regularization - too much may underfit, try for dropout, batch norm, weight, bias , L2.**
**31. Give it more training time as long as the loss is decreasing.**
**32. Switch from Train to Test mode - not clear.**
**33. Visualize the training - activations, weights, layer updates, biases.** [**Tensorboard**](https://www.tensorflow.org/get_started/summaries_and_tensorboard) **and** [**Crayon**](https://github.com/torrvision/crayon)**. Tips on** [**Deeplearning4j**](https://deeplearning4j.org/visualization#usingui)**. Expect gaussian distribution for weights, biases start at 0 and end up almost gaussian. Keep an eye out for parameters that are diverging to +/- infinity. Keep an eye out for biases that become very large. This can sometimes occur in the output layer for classification if the distribution of classes is very imbalanced.**
**34. Try a different optimizer, Check this** [**excellent post**](http://ruder.io/optimizing-gradient-descent/) **about gradient descent optimizers.**
**35. Exploding / Vanishing gradients - Gradient clipping may help. Tips on:** [**Deeplearning4j**](https://deeplearning4j.org/visualization#usingui)**: “A good standard deviation for the activations is on the order of 0.5 to 2.0. Significantly outside of this range may indicate vanishing or exploding activations.”**
**36. Increase/Decrease Learning Rate, or use adaptive learning**
**37. Overcoming NaNs, big issue for RNN - decrease LR,** [**how to deal with NaNs**](http://russellsstewart.com/notes/0.html)**. evaluate layer by layer, why does it appear.**
# Attention
1. [**Illustrated attention-**](https://towardsdatascience.com/attn-illustrated-attention-5ec4ad276ee3) **AMAZING**
2. [**Illustrated self attention - great**](https://towardsdatascience.com/illustrated-self-attention-2d627e33b20a)
3. [**Jay alamar on attention, the first one is better.**](http://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/)
4. [**Attention is all you need (paper)**](https://arxiv.org/abs/1706.03762?fbclid=IwAR3-gxVldr_xW0D9m6QvwyIV5vhvl-crVOc2kEI6HZskodJP678ynJKj1-o)
5. [**The annotated transformer - reviewing the paper**](http://nlp.seas.harvard.edu/2018/04/03/attention.html?fbclid=IwAR2_ZOfUfXcto70apLdT_StObPwatYHNRPP4OlktcmGfj9uPLhgsZPsAXzE)
6. [**Lilian weng on attention**](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html)**, self, soft vs hard, global vs local, neural turing machines, pointer networks, transformers, snail, self attention GAN.**
7. [**Understanding attention in rnns**](https://medium.com/datadriveninvestor/attention-in-rnns-321fbcd64f05)
8. [**Another good intro with gifs to attention**](https://towardsdatascience.com/attn-illustrated-attention-5ec4ad276ee3)
9. [**Clear insight to what attention is, a must read**](http://webcache.googleusercontent.com/search?q=cache:http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/)**!**
10. [**Transformer NN by google**](https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html) **- faster, better, more accurate**
11. [**Intuitive explanation to attention**](https://towardsdatascience.com/an-intuitive-explanation-of-self-attention-4f72709638e1)
12. [**Attention by vidhya**](https://www.analyticsvidhya.com/blog/2019/11/comprehensive-guide-attention-mechanism-deep-learning/)
13. [**Augmented rnns**](https://distill.pub/2016/augmented-rnns/) **- including turing / attention / adaptive computation time etc. general overview, not as clear as the one below.** 
1. [**A really good REVIEW on attention and its many forms, historical changes, etc**](https://medium.com/@joealato/attention-in-nlp-734c6fa9d983)
2. [**Medium on comparing cnn / rnn / han**](https://medium.com/jatana/report-on-text-classification-using-cnn-rnn-han-f0e887214d5f) **- will change on other data, my impression is that the data is too good in this article**
3. **Mastery on** [**rnn vs attention vs global attention**](https://machinelearningmastery.com/global-attention-for-encoder-decoder-recurrent-neural-networks/) **- a really unclear intro**
4. **Mastery on** [**attention**](https://machinelearningmastery.com/how-does-attention-work-in-encoder-decoder-recurrent-neural-networks/) **- this makes the whole process clear, scoring encoder vs decoder input outputs, normalizing them using softmax (annotation weights), multiplying score and the weight summed on all (i.e., context vector), and then we decode the context vector.**
1. **Soft (above) and hard crisp attention**
2. **Dropping the hidden output - HAN or AB BiLSTM**
3. **Attention concat to input vec**
4. **Global vs local attention**
5. **Mastery on** [**attention with lstm encoding / decoding**](https://machinelearningmastery.com/implementation-patterns-encoder-decoder-rnn-architecture-attention/) **- a theoretical discussion about many attention architectures. This adds make-sense information to everything above.**
1. **Encoder: The encoder is responsible for stepping through the input time steps and encoding the entire sequence into a fixed length vector called a context vector.**
2. **Decoder: The decoder is responsible for stepping through the output time steps while reading from the context vector.**
3. **A problem with the architecture is that performance is poor on long input or output sequences. The reason is believed to be because of the fixed-sized internal representation used by the encoder.**
1. **Enc-decoder**
2. **Recursive**
3. **Enc-dev with recursive**
6. **Code on GIT:**
1. **HAN -** [**GIT**](https://github.com/richliao/textClassifier)**,** [**paper**](https://www.cs.cmu.edu/~diyiy/docs/naacl16.pdf)
2. [**Non penalized self attention**](https://github.com/uzaymacar/attention-mechanisms/blob/master/examples/sentiment_classification.py)
3. **LSTM,** [**BiLSTM attention**](https://github.com/gentaiscool/lstm-attention)**,** [**paper**](https://arxiv.org/pdf/1805.12307.pdf)
4. **Tushv89,** [**Keras layer attention implementation**](https://github.com/thushv89/attention_keras)
5. **Richliao, hierarchical** [**Attention code for document classification using keras**](https://github.com/richliao/textClassifier/blob/master/textClassifierHATT.py)**,** [**blog**](https://richliao.github.io/supervised/classification/2016/12/26/textclassifier-HATN/)**,** [**group chatter**](https://groups.google.com/forum/#!topic/keras-users/IWK9opMFavQ)
**note: word level then sentence level embeddings.**
**figure= >**
1. [**Self Attention pip for keras**](https://pypi.org/project/keras-self-attention/)**,** [**git**](https://github.com/CyberZHG/keras-self-attention)
2. [**Phillip remy on attention in keras, not a single layer, a few of them to make it.**](https://github.com/philipperemy/keras-attention-mechanism)
3. [**Self attention with relative positiion representations**](https://medium.com/@_init_/how-self-attention-with-relative-position-representations-works-28173b8c245a)
4. [**nMT - jointly learning to align and translate**](https://arxiv.org/abs/1409.0473) *\*\**
5. [**Medium on attention plus code, comparison keras and pytorch**](https://medium.com/huggingface/understanding-emotions-from-keras-to-pytorch-3ccb61d5a983)
**BERT/ROBERTA**
1. [**Do attention heads in bert roberta track syntactic dependencies?**](https://medium.com/@phu_pmh/do-attention-heads-in-bert-track-syntactic-dependencies-81c8a9be311a) **- tl;dr: The attention weights between tokens in BERT/RoBERTa bear similarity to some syntactic dependency relations, but the results are less conclusive than we’d like as they don’t significantly outperform linguistically uninformed baselines for all types of dependency relations. In the case of MAX, our results indicate that specific heads in the BERT models may correspond to certain dependency relations, whereas for MST, we find much less support “generalist” heads whose attention weights correspond to a full syntactic dependency structure.**
**In both cases, the metrics do not appear to be representative of the extent of linguistic knowledge learned by the BERT models, based on their strong performance on many NLP tasks. Hence, our takeaway is that while we can tease out some structure from the attention weights of BERT models using the above methods, studying the attention weights alone is unlikely to give us the full picture of BERT’s strength processing natural language.**
1. **TRANSFORMERS**
2. [**Jay alammar on transformers**](http://jalammar.github.io/illustrated-transformer/) **(amazing)**
3. [**J.A on Bert Elmo**](http://jalammar.github.io/illustrated-bert/) **(amazing)**
4. [**Jay alammar on a visual guide of bert for the first time**](http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/)
5. [**J.A on GPT2**](http://jalammar.github.io/illustrated-bert/)
6. [**Super fast transformers**](http://transformer)
7. [**A survey of long term context in transformers.**](https://www.pragmatic.ml/a-survey-of-methods-for-incorporating-long-term-context/)
8. [**Lilian Wang on the transformer family**](https://lilianweng.github.io/lil-log/2020/04/07/the-transformer-family.html) **(seems like it is constantly updated)**
9. 
10. **Hugging face,** [**encoders decoders in transformers for seq2seq**](https://medium.com/huggingface/encoder-decoders-in-transformers-a-hybrid-pre-trained-architecture-for-seq2seq-af4d7bf14bb8)
11. [**The annotated transformer**](http://nlp.seas.harvard.edu/2018/04/03/attention.html)
12. [**Large memory layers with product keys**](https://arxiv.org/abs/1907.05242) **- This memory layer allows us to tackle very large scale language modeling tasks. In our experiments we consider a dataset with up to 30 billion words, and we plug our memory layer in a state-of-the-art transformer-based architecture. In particular, we found that a memory augmented model with only 12 layers outperforms a baseline transformer model with 24 layers, while being twice faster at inference time.**
13. [**Adaptive sparse transformers**](https://arxiv.org/abs/1909.00015) **- This sparsity is accomplished by replacing softmax with**
**α-entmax: a differentiable generalization of softmax that allows low-scoring words to receive precisely zero weight. Moreover, we derive a method to automatically learn the**
**α parameter -- which controls the shape and sparsity of**
**α-entmax -- allowing attention heads to choose between focused or spread-out behavior. Our adaptively sparse Transformer improves interpretability and head diversity when compared to softmax Transformers on machine translation datasets.**
### **ELMO**
1. [**Short tutorial on elmo, pretrained, new data, incremental(finetune?)**](https://github.com/PrashantRanjan09/Elmo-Tutorial)**,** [**using elmo pretrained**](https://github.com/PrashantRanjan09/WordEmbeddings-Elmo-Fasttext-Word2Vec)
2. [**Why you cant use elmo to encode words (contextualized)**](https://github.com/allenai/allennlp/issues/1737)
3. [**Vidhya on elmo**](https://www.analyticsvidhya.com/blog/2019/03/learn-to-use-elmo-to-extract-features-from-text/) **- everything you want to know with code**
4. [**Sebastien ruder on language modeling embeddings for the purpose of transfer learning, ELMO, ULMFIT, open AI transformer, BILSTM,**](https://thegradient.pub/nlp-imagenet/)
5. [**Another good tutorial on elmo**](http://www.realworldnlpbook.com/blog/improving-sentiment-analyzer-using-elmo.html)**.**
6. [**ELMO**](https://allennlp.org/elmo)**,** [**tutorial**](https://allennlp.org/tutorials)**,** [**github**](https://allennlp.org/tutorials)
7. [**Elmo on google hub and code**](https://tfhub.dev/google/elmo/2)
8. [**How to use elmo embeddings, advice for word and sentence**](https://github.com/tensorflow/hub/issues/149)
9. [**Using elmo as a lambda embedding layer**](https://towardsdatascience.com/transfer-learning-using-elmo-embedding-c4a7e415103c)
10. [**Elmbo tutorial notebook**](https://github.com/sambit9238/Deep-Learning/blob/master/elmo_embedding_tfhub.ipynb)
11. [**Elmo code on git**](https://github.com/allenai/allennlp/blob/master/tutorials/how_to/elmo.md)
12. [**Elmo on keras using lambda**](https://towardsdatascience.com/elmo-helps-to-further-improve-your-word-embeddings-c6ed2c9df95f)
13. [**Elmo pretrained models for many languages**](https://github.com/HIT-SCIR/ELMoForManyLangs)**, for** [**russian**](http://docs.deeppavlov.ai/en/master/intro/pretrained_vectors.html) **too,** [**mean elmo**](https://stackoverflow.com/questions/53061423/how-to-represent-elmo-embeddings-as-a-1d-array/53088523)
14. [**Ari’s intro on word embeddings part 2, has elmo and some bert**](https://towardsdatascience.com/beyond-word-embeddings-part-2-word-vectors-nlp-modeling-from-bow-to-bert-4ebd4711d0ec)
15. [**Mean elmo**](https://www.analyticsvidhya.com/blog/2019/03/learn-to-use-elmo-to-extract-features-from-text/?utm_source=facebook.com\&utm_medium=social\&fbclid=IwAR24LwsmhUJshC7gk3P9RIIACCyYYcjlYMa_NbgdzcNBBhD7g38FM2KTA-Q)**, batches, with code and linear regression i**
16. [**Elmo projected using TSNE - grouping are not semantically similar**](https://towardsdatascience.com/elmo-contextual-language-embedding-335de2268604)
### **ULMFIT**
1. [**Tutorial and code by vidhya**](https://www.analyticsvidhya.com/blog/2018/11/tutorial-text-classification-ulmfit-fastai-library/)**,** [**medium**](https://medium.com/analytics-vidhya/tutorial-on-text-classification-nlp-using-ulmfit-and-fastai-library-in-python-2f15a2aac065)
2. [**Paper**](https://arxiv.org/abs/1801.06146)
3. [**Ruder on transfer learning**](http://ruder.io/nlp-imagenet/)
4. [**Medium on how - unclear**](https://blog.frame.ai/learning-more-with-less-1e618a5aa160)
5. [**Fast NLP on how**](http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html)
6. [**Paper: ulmfit**](https://arxiv.org/abs/1801.06146)
7. [**Fast.ai on ulmfit**](http://nlp.fast.ai/category/classification.html)**,** [**this too**](https://github.com/fastai/fastai/blob/c502f12fa0c766dda6c2740b2d3823e2deb363f9/nbs/examples/ulmfit.ipynb)
8. [**Vidhya on ulmfit using fastai**](https://www.analyticsvidhya.com/blog/2018/11/tutorial-text-classification-ulmfit-fastai-library/?utm_source=facebook.com\&fbclid=IwAR0ghBUHEphXrSRZZfkbEOklY1RtveC7XG3I48eH_LNAfCnRQzgraw-AZWs)
9. [**Medium on ulmfit**](https://towardsdatascience.com/explainable-data-efficient-text-classification-888cc7a1af05)
10. [**Building blocks of ulm fit**](https://medium.com/mlreview/understanding-building-blocks-of-ulmfit-818d3775325b)
11. [**Applying ulmfit on entity level sentiment analysis using business news artcles**](https://github.com/jannenev/ulmfit-language-model)
12. [**Understanding language modelling using Ulmfit, fine tuning etc**](https://towardsdatascience.com/understanding-language-modelling-nlp-part-1-ulmfit-b557a63a672b)
13. [**Vidhaya on ulmfit + colab**](https://www.analyticsvidhya.com/blog/2018/11/tutorial-text-classification-ulmfit-fastai-library/) **“The one cycle policy provides some form of regularisation”, if you wish to know more about one cycle policy, then feel free to refer to this excellent paper by Leslie Smith – “**[**A disciplined approach to neural network hyper-parameters: Part 1 — learning rate, batch size, momentum, and weight decay**](https://arxiv.org/abs/1803.09820)**”.**
### **BERT**
1. [**The BERT PAPER**](https://arxiv.org/pdf/1810.04805.pdf)
1. [**Prerequisite about transformers and attention - this is not enough**](http://nlp.seas.harvard.edu/2018/04/03/attention.html)
2. [**Embeddings using bert in python**](https://hackerstreak.com/word-embeddings-using-bert-in-python/?fbclid=IwAR1sQDbxgCekqsFZBjZ6VAHYDUk41ijgvwNu_oAXJpgAdWG0KrMAPhePEF4) **- using bert as a service to encode 1024 vectors and do cosine similarity**
3. [**Identifying the right meaning with bert**](https://towardsdatascience.com/identifying-the-right-meaning-of-the-words-using-bert-817eef2ac1f0) **- the idea is to classify the word duck into one of three meanings using bert embeddings, which promise contextualized embeddings. I.e., to duck, the Duck, etc**
4. [**Google neural machine translation (attention) - too long**](https://arxiv.org/pdf/1609.08144.pdf)
2. [**What is bert**](https://towardsdatascience.com/breaking-bert-down-430461f60efb)
3. **(amazing) Deconstructing bert**
1. **I found some fairly distinctive and surprisingly intuitive attention patterns. Below I identify six key patterns and for each one I show visualizations for a particular layer / head that exhibited the pattern.**
2. [**part 1**](https://towardsdatascience.com/deconstructing-bert-distilling-6-patterns-from-100-million-parameters-b49113672f77) **- attention to the next/previous/ identical/related (same and other sentences), other words predictive of a word, delimeters tokens**
3. **(good)** [**Deconstructing bert part 2**](https://towardsdatascience.com/deconstructing-bert-part-2-visualizing-the-inner-workings-of-attention-60a16d86b5c1) **- looking at the visualization and attention heads, focusing on Delimiter attention, bag of words attention, next word attention - patterns.**
4. [**Bert demystified**](https://medium.com/@_init_/why-bert-has-3-embedding-layers-and-their-implementation-details-9c261108e28a) **(read this first!)**
5. [**Read this after**](https://towardsdatascience.com/understanding-bert-is-it-a-game-changer-in-nlp-7cca943cf3ad)**, the most coherent explanation on bert, 15% masked word prediction and next sentence prediction. Roberta, xlm bert, albert, distilibert.**
6. **A** [**thorough tutorial on bert**](http://mccormickml.com/2019/07/22/BERT-fine-tuning/)**, fine tuning using hugging face transformers package.** [**Code**](https://colab.research.google.com/drive/1Y4o3jh3ZH70tl6mCd76vz_IxX23biCPP)
**Youtube** [**ep1**](https://www.youtube.com/watch?v=FKlPCK1uFrc)**,** [**2**](https://www.youtube.com/watch?v=zJW57aCBCTk)**,** [**3**](https://www.youtube.com/watch?v=x66kkDnbzi4)**,** [**3b**](https://www.youtube.com/watch?v=Hnvb9b7a_Ps)**,**
1. [**How to train bert**](https://medium.com/@vineet.mundhra/loading-bert-with-tensorflow-hub-7f5a1c722565) **from scratch using TF, with \[CLS] \[SEP] etc**
2. [**Extending a vocabulary for bert, another kind of transfer learning.**](https://towardsdatascience.com/3-ways-to-make-new-language-models-f3642e3a4816)
3. [**Bert tutorial**](http://mccormickml.com/2019/07/22/BERT-fine-tuning/?fbclid=IwAR3TBQSjq3lcWa2gH3gn2mpBcn3vLKCD-pvpHGue33Cs59RQAz34dPHaXys)**, on fine tuning, some talk on from scratch and probably not discussed about using embeddings as input**
4. [**Bert for summarization thread**](https://github.com/google-research/bert/issues/352)
5. [**Bert on logs**](https://medium.com/rapids-ai/cybert-28b35a4c81c4)**, feature names as labels, finetune bert, predict.**
6. [**Bert scikit wrapper for pipelines**](https://towardsdatascience.com/build-a-bert-sci-kit-transformer-59d60ddd54a5)
7. [**What is bert not good at, also refer to the cited paper**](https://towardsdatascience.com/bert-is-not-good-at-7b1ca64818c5) **(is/is not)**
8. [**Jay Alamar on Bert**](http://jalammar.github.io/illustrated-bert/)
9. [**Jay Alamar on using distilliBert**](http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/)
10. [**sparse bert**](https://github.com/huggingface/transformers/tree/master/examples/movement-pruning)**,** [**paper**](https://arxiv.org/abs/2005.07683) **- When combined with distillation, the approach achieves minimal accuracy loss with down to only 3% of the model parameters.**
11. **Bert with keras,** [**blog post**](https://www.ctolib.com/Separius-BERT-keras.html)**,** [**colaboratory**](https://colab.research.google.com/gist/HighCWu/3a02dc497593f8bbe4785e63be99c0c3/bert-keras-tutorial.ipynb)
12. [**Bert with t-hub**](https://github.com/google-research/bert/blob/master/run_classifier_with_tfhub.py)
13. [**Bert on medium with code**](https://medium.com/huggingface/multi-label-text-classification-using-bert-the-mighty-transformer-69714fa3fb3d)
14. [**Bert on git**](https://github.com/SkullFang/BERT_NLP_Classification)
15. **Finetuning -** [**Better sentiment analysis with bert**](https://medium.com/southpigalle/how-to-perform-better-sentiment-analysis-with-bert-ba127081eda)**, claims 94% on IMDB. official code** [**here**](https://github.com/google-research/bert/blob/master/predicting_movie_reviews_with_bert_on_tf_hub.ipynb) **“ it creates a single new layer that will be trained to adapt BERT to our sentiment task (i.e. classifying whether a movie review is positive or negative). This strategy of using a mostly trained model is called** [**fine-tuning**](http://wiki.fast.ai/index.php/Fine_tuning)**.”**
16. [**Explain bert**](http://exbert.net/) **- bert visualization tool.**
17. **sentenceBERT** [**paper**](https://arxiv.org/pdf/1908.10084.pdf)
18. [**Bert question answering**](https://towardsdatascience.com/testing-bert-based-question-answering-on-coronavirus-articles-13623637a4ff?source=email-4dde5994e6c1-1586483206529-newsletter.v2-7f60cf5620c9-----0-------------------b506d4ba_2902_4718_9c95_a36e33d638e6---48577de843eb----20200410) **on covid19**
19. [**Codebert**](https://arxiv.org/pdf/2002.08155.pdf?fbclid=IwAR3XXrpuILgnqTHCI1-0LHPT39IJVVaBl9uGXTVAjUwb1xM8NGrKUHrEyac)
20. [**Bert multilabel classification**](http://towardsdatascience)
21. [**Tabert**](https://ai.facebook.com/blog/tabert-a-new-model-for-understanding-queries-over-tabular-data/) **-** [**TaBERT**](https://ai.facebook.com/research/publications/tabert-pretraining-for-joint-understanding-of-textual-and-tabular-data/) **is the first model that has been pretrained to learn representations for both natural language sentences and tabular data.**
22. [**All the ways that you can compress BERT**](http://mitchgordon.me/machine/learning/2019/11/18/all-the-ways-to-compress-BERT.html?fbclid=IwAR0X2g4VQDpN4otb7YPzn88r5XMg8gRd3NWfm3dd6P0aFZEEtOGKY9QU5ec)
**Pruning - Removes unnecessary parts of the network after training. This includes weight magnitude pruning, attention head pruning, layers, and others. Some methods also impose regularization during training to increase prunability (layer dropout).**
**Weight Factorization - Approximates parameter matrices by factorizing them into a multiplication of two smaller matrices. This imposes a low-rank constraint on the matrix. Weight factorization can be applied to both token embeddings (which saves a lot of memory on disk) or parameters in feed-forward / self-attention layers (for some speed improvements).**
**Knowledge Distillation - Aka “Student Teacher.” Trains a much smaller Transformer from scratch on the pre-training / downstream-data. Normally this would fail, but utilizing soft labels from a fully-sized model improves optimization for unknown reasons. Some methods also distill BERT into different architectures (LSTMS, etc.) which have faster inference times. Others dig deeper into the teacher, looking not just at the output but at weight matrices and hidden activations.**
**Weight Sharing - Some weights in the model share the same value as other parameters in the model. For example, ALBERT uses the same weight matrices for every single layer of self-attention in BERT.**
**Quantization - Truncates floating point numbers to only use a few bits (which causes round-off error). The quantization values can also be learned either during or after training.**
**Pre-train vs. Downstream - Some methods only compress BERT w\.r.t. certain downstream tasks. Others compress BERT in a way that is task-agnostic.**
1. [**Bert and nlp in 2019**](https://towardsdatascience.com/2019-year-of-bert-and-transformer-f200b53d05b9)
2. [**HeBert - bert for hebrwe sentiment and emotions**](https://github.com/avichaychriqui/HeBERT)
3. [**Kdbuggets on visualizing bert**](https://www.kdnuggets.com/2019/03/deconstructing-bert-part-2-visualizing-inner-workings-attention.html)
4. [**What does bert look at, analysis of attention**](https://www-nlp.stanford.edu/pubs/clark2019what.pdf) **- We further show that certain attention heads correspond well to linguistic notions of syntax and coreference. For example, we find heads that attend to the direct objects of verbs, determiners of nouns, objects of prepositions, and coreferent mentions with remarkably high accuracy. Lastly, we propose an attention-based probing classifier and use it to further demonstrate that substantial syntactic information is captured in BERT’s attention**
5. [**Bertviz**](https://github.com/jessevig/bertviz) **BertViz is a tool for visualizing attention in the Transformer model, supporting all models from the** [**transformers**](https://github.com/huggingface/transformers) **library (BERT, GPT-2, XLNet, RoBERTa, XLM, CTRL, etc.). It extends the** [**Tensor2Tensor visualization tool**](https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/visualization) **by** [**Llion Jones**](https://medium.com/@llionj) **and the** [**transformers**](https://github.com/huggingface/transformers) **library from** [**HuggingFace**](https://github.com/huggingface)**.**
6. **PMI-masking** [**paper**](https://openreview.net/forum?id=3Aoft6NWFej)**,** [**post**](https://www.ai21.com/pmi-masking) **- Joint masking of correlated tokens significantly speeds up and improves BERT's pretraining**
7. **(really good/)** [**Examining bert raw embeddings**](https://towardsdatascience.com/examining-berts-raw-embeddings-fd905cb22df7) **- TL;DR BERT’s raw word embeddings capture useful and separable information (distinct histogram tails) about a word in terms of other words in BERT’s vocabulary. This information can be harvested from both raw embeddings and their transformed versions after they pass through BERT with a Masked language model (MLM) head**
### **GPT2**
1. [**the GPT-2**](https://medium.com/dair-ai/experimenting-with-openais-improved-language-model-abf73bc123b9) **small algorithm was trained on the task of language modeling — which tests a program’s ability to predict the next word in a given sentence — by ingesting huge numbers of articles, blogs, and websites. By using just this data it achieved state-of-the-art scores on a number of unseen language tests, an achievement known as zero-shot learning. It can also perform other writing-related tasks, such as translating text from one language to another, summarizing long articles, and answering trivia questions.**
2. [**Medium code**](https://medium.com/dair-ai/explore-pretrained-language-models-with-pytorch-1b1e06b7510c) **for GPT=2 - big algo**
### **GPT3**
1. [**GPT3**](https://medium.com/swlh/all-hail-gpt-3-389c7f1fcb3b) **on medium - language models can be used to produce good results on zero-shot, one-shot, or few-shot learning.**
2. [**Fit More and Train Faster With ZeRO via DeepSpeed and FairScale**](https://huggingface.co/blog/zero-deepspeed-fairscale)
### **XLNET**
1. [**Xlnet is transformer and bert combined**](https://medium.com/logits/xlnet-sota-pre-training-method-that-outperforms-bert-26d4e9978983) **- Actually its quite good explaining it**
2. [**git**](https://github.com/zihangdai/xlnet)
3. **CLIP**
4. **(keras)** [**Implementation of a dual encoder**](https://keras.io/examples/nlp/nl_image_search/) **model for retrieving images that match natural language queries. - The example demonstrates how to build a dual encoder (also known as two-tower) neural network model to search for images using natural language. The model is inspired by the** [**CLIP**](https://openai.com/blog/clip/) **approach, introduced by Alec Radford et al. The idea is to train a vision encoder and a text encoder jointly to project the representation of images and their captions into the same embedding space, such that the caption embeddings are located near the embeddings of the images they describe.**
5. 1. **Adversarial methodologies**
6. **What is label** [**flipping and smoothing**](https://datascience.stackexchange.com/questions/55359/how-label-smoothing-and-label-flipping-increases-the-performance-of-a-machine-le/56662) **and usage for making a model more robust against adversarial methodologies - 0**
**Label flipping is a training technique where one selectively manipulates the labels in order to make the model more robust against label noise and associated attacks - the specifics depend a lot on the nature of the noise. Label flipping bears no benefit only under the assumption that all labels are (and will always be) correct and that no adversaries exist. In cases where noise tolerance is desirable, training with label flipping is beneficial.**
**Label smoothing is a regularization technique (and then some) aimed at improving model performance. Its effect takes place irrespective of label correctness.**
1. [**Paper: when does label smoothing helps?**](https://arxiv.org/abs/1906.02629) **Smoothing the labels in this way prevents the network from becoming overconfident and label smoothing has been used in many state-of-the-art models, including image classification, language translation and speech recognition...Here we show empirically that in addition to improving generalization, label smoothing improves model calibration which can significantly improve beam-search. However, we also observe that if a teacher network is trained with label smoothing, knowledge distillation into a student network is much less effective.**
2. [**Label smoothing, python code, multi class examples**](https://rickwierenga.com/blog/fast.ai/FastAI2019-12.html)
1. [**Label sanitazation against label flipping poisoning attacks**](https://arxiv.org/abs/1803.00992) **- In this paper we propose an efficient algorithm to perform optimal label flipping poisoning attacks and a mechanism to detect and relabel suspicious data points, mitigating the effect of such poisoning attacks.**
2. [**Adversarial label flips attacks on svm**](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.398.7446\&rep=rep1\&type=pdf) **- To develop a robust classification algorithm in the adversarial setting, it is important to understand the adversary’s strategy. We address the problem of label flips attack where an adversary contaminates the training set through flipping labels. By analyzing the objective of the adversary, we formulate an optimization framework for finding the label flips that maximize the classification error. An algorithm for attacking support vector machines is derived. Experiments demonstrate that the accuracy of classifiers is significantly degraded under the attack.**
3. **GAN**
4. [**Great advice for training gans**](https://medium.com/@utk.is.here/keep-calm-and-train-a-gan-pitfalls-and-tips-on-training-generative-adversarial-networks-edd529764aa9)**, such as label flipping batch norm, etc read!**
5. [**Intro to Gans**](https://medium.com/sigmoid/a-brief-introduction-to-gans-and-how-to-code-them-2620ee465c30)
6. [**A fantastic series about gans, the following two what are gans and applications are there**](https://medium.com/@jonathan_hui/gan-gan-series-2d279f906e7b)
1. [**What are a GANs?**](https://medium.com/@jonathan_hui/gan-whats-generative-adversarial-networks-and-its-application-f39ed278ef09)**, and cool** [**applications**](https://medium.com/@jonathan_hui/gan-some-cool-applications-of-gans-4c9ecca35900)
2. [**Comprehensive overview**](https://medium.com/@jonathan_hui/gan-a-comprehensive-review-into-the-gangsters-of-gans-part-1-95ff52455672)
3. [**Cycle gan**](https://medium.com/@jonathan_hui/gan-cyclegan-6a50e7600d7) **- transferring styles**
4. [**Super gan resolution**](https://medium.com/@jonathan_hui/gan-super-resolution-gan-srgan-b471da7270ec) **- super res images**
5. [**Why gan so hard to train**](https://medium.com/@jonathan_hui/gan-why-it-is-so-hard-to-train-generative-advisory-networks-819a86b3750b) **- good for critique**
6. [**And how to improve gans performance**](https://towardsdatascience.com/gan-ways-to-improve-gan-performance-acf37f9f59b)
7. [**Dcgan good as a starting point in new projects**](https://medium.com/@jonathan_hui/gan-dcgan-deep-convolutional-generative-adversarial-networks-df855c438f)
8. [**Labels to improve gans, cgan, infogan**](https://medium.com/@jonathan_hui/gan-cgan-infogan-using-labels-to-improve-gan-8ba4de5f9c3d)
9. [**Stacked - labels, gan adversarial loss, entropy loss, conditional loss**](https://medium.com/@jonathan_hui/gan-stacked-generative-adversarial-networks-sgan-d9449ac63db8) **- divide and conquer**
10. [**Progressive gans**](https://medium.com/@jonathan_hui/gan-progressive-growing-of-gans-f9e4f91edf33) **- mini batch discrimination**
11. [**Using attention to improve gan**](https://medium.com/@jonathan_hui/gan-self-attention-generative-adversarial-networks-sagan-923fccde790c)
12. [**Least square gan - lsgan**](https://medium.com/@jonathan_hui/gan-lsgan-how-to-be-a-good-helper-62ff52dd3578)
13. **Unread:**
1. [**Wasserstein gan, wgan gp**](https://medium.com/@jonathan_hui/gan-wasserstein-gan-wgan-gp-6a1a2aa1b490)
2. [**Faster training for gans, lower training count rsgan ragan**](https://medium.com/@jonathan_hui/gan-rsgan-ragan-a-new-generation-of-cost-function-84c5374d3c6e)
3. [**Addressing gan stability, ebgan began**](https://medium.com/@jonathan_hui/gan-energy-based-gan-ebgan-boundary-equilibrium-gan-began-4662cceb7824)
4. [**What is wrong with gan cost functions**](https://medium.com/@jonathan_hui/gan-what-is-wrong-with-the-gan-cost-function-6f594162ce01)
5. [**Using cost functions for gans inspite of the google brain paper**](https://medium.com/@jonathan_hui/gan-does-lsgan-wgan-wgan-gp-or-began-matter-e19337773233)
6. [**Proving gan is js-convergence**](https://medium.com/@jonathan_hui/proof-gan-optimal-point-658116a236fb)
7. [**Dragan on minimizing local equilibria, how to stabilize gans**](https://medium.com/@jonathan_hui/gan-dragan-5ba50eafcdf2)**, reducing mode collapse**
8. [**Unrolled gan for reducing mode collapse**](https://medium.com/@jonathan_hui/gan-unrolled-gan-how-to-reduce-mode-collapse-af5f2f7b51cd)
9. [**Measuring gans**](https://medium.com/@jonathan_hui/gan-how-to-measure-gan-performance-64b988c47732)
10. [**Ways to improve gans performance**](https://towardsdatascience.com/gan-ways-to-improve-gan-performance-acf37f9f59b)
11. [**Introduction to gans**](https://medium.freecodecamp.org/an-intuitive-introduction-to-generative-adversarial-networks-gans-7a2264a81394) **with tf code**
12. [**Intro to gans**](https://medium.com/datadriveninvestor/deep-learning-generative-adversarial-network-gan-34abb43c0644)
13. [**Intro to gan in KERAS**](https://towardsdatascience.com/demystifying-generative-adversarial-networks-c076d8db8f44)
7. **“GAN”** [**using xgboost and gmm for density sampling**](https://edge.skyline.ai/data-synthesizers-on-aws-sagemaker)
8. [**Reverse engineering**](https://ai.facebook.com/blog/reverse-engineering-generative-model-from-a-single-deepfake-image/)
# Deep Neural Machine Vision
## **TOOLS**
1. [Image deduplication](https://github.com/idealo/imagededup)
2. [Segment anything by Meta](https://segment-anything.com/demo#)
## **SUPER RESOLUTION**
1. [State of the art comparison](http://www.wisdom.weizmann.ac.il/~vision/zssr/)
## **DETECTION**
1. [**Review on DL technique applied to semantic segmentation**](https://arxiv.org/pdf/1704.06857.pdf)
2. [**Mastery on obj detection**](https://machinelearningmastery.com/object-recognition-with-deep-learning/) **- rcnn family and yolo family**
3. **Fair** [**detectron**](https://github.com/facebookresearch/Detectron)
4. [**Maskrcnn benchmark**](https://github.com/facebookresearch/maskrcnn-benchmark)**,** [**paper**](https://arxiv.org/abs/1703.06870)
5. [**Simpledet - obj detection and instance recognition**](https://github.com/TuSimple/simpledet)
6. [**Mmdetection**](https://github.com/open-mmlab/mmdetection?fbclid=IwAR1W0G-mhiNcCJk1YdnnFFozWY_j9QUNQo9Qevfdj6_PnnODfk-5iSWbMd0)
7. [**Blind image separation**](https://www.researchgate.net/publication/3938186_Blind_image_separation_through_kurtosis_maximization)
8. [**UNET**](https://heartbeat.fritz.ai/deep-learning-for-image-segmentation-u-net-architecture-ff17f6e4c1cf)
9. [**U^2 Net - using a detection network for pencil drawing generation and segmentation**](https://github.com/NathanUA/U-2-Net)
10. [**FastAI image segmentation**](https://gilberttanner.com/blog/fastai-image-segmentation)
11. 
12. 
13. [**You Only Look Once: Unified, Real-Time Object Detection**](https://arxiv.org/abs/1506.02640)**, 2015.**
14. [**YOLO9000: Better, Faster, Stronger**](https://arxiv.org/abs/1612.08242)**, 2016.**
15. [**YOLOv3: An Incremental Improvement**](https://arxiv.org/abs/1804.02767)**, 2018**
16. [**R-CNN: Regions with Convolutional Neural Network Features, GitHub**](https://github.com/rbgirshick/rcnn)**.**
17. [**Fast R-CNN, GitHub**](https://github.com/rbgirshick/fast-rcnn)**.**
18. [**Faster R-CNN Python Code, GitHub**](https://github.com/rbgirshick/py-faster-rcnn)**.**
19. [**YOLO, GitHub**](https://github.com/pjreddie/darknet/wiki/YOLO:-Real-Time-Object-Detection)**.**
20. [**Rich feature hierarchies for accurate object detection and semantic segmentation**](https://arxiv.org/abs/1311.2524)**, 2013.**
21. [**Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition**](https://arxiv.org/abs/1406.4729)**, 2014.**
22. [**Fast R-CNN**](https://arxiv.org/abs/1504.08083)**, 2015.**
23. [**Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks**](https://arxiv.org/abs/1506.01497)**, 2016.**
24. [**Mask R-CNN**](https://arxiv.org/abs/1703.06870)**, 2017.**
25. [**A Brief History of CNNs in Image Segmentation: From R-CNN to Mask R-CNN**](https://blog.athelas.com/a-brief-history-of-cnns-in-image-segmentation-from-r-cnn-to-mask-r-cnn-34ea83205de4)**, 2017.**
26. [**Object Detection for Dummies Part 3: R-CNN Family**](https://lilianweng.github.io/lil-log/2017/12/31/object-recognition-for-dummies-part-3.html)**, 2017.**
27. [**Object Detection Part 4: Fast Detection Models**](https://lilianweng.github.io/lil-log/2018/12/27/object-detection-part-4.html)**, 2018.**
28. [**Ikea ASM**](https://ikeaasm.github.io/)
29.
## **RECOGNITION**
1. [**Using image hashtags**](https://engineering.fb.com/ml-applications/advancing-state-of-the-art-image-recognition-with-deep-learning-on-hashtags/)
## Segmentation
1. [Vit](https://dino-vit-features.github.io/)
# Deep Neural Tabular
1. Tabnet
1. [papers with code](https://paperswithcode.com/paper/tabnet-attentive-interpretable-tabular/review/)
2. [paper](https://arxiv.org/abs/1908.07442)
3. [pytorch](https://github.com/dreamquark-ai/tabnet)
4. [mediun](https://towardsdatascience.com/tabnet-deep-neural-network-for-structured-tabular-data-39eb4b27a9e4)
2. [Survey on DNN and tabular data](https://arxiv.org/abs/2110.01889)
3. [Tabular data: deep learning is NOT all you need](https://arxiv.org/pdf/2106.03253.pdf)
# Deep Neural Time Series
1. Neural Propher
1. [NP Docs](https://neuralprophet.com/code/forecaster.html)
2. [Optimizing Time Series Forecasting with NeuralProphet: A Quick Guide](https://medium.com/@miritrope/optimizing-time-series-forecasting-with-neuralprophet-a-quick-guide-a08afe8216ac)
3. [Hyper Param Tuning](https://medium.com/eni-digitalks/unlock-the-neuralprophet-potential-hyperparameter-tuning-1905ccb87537)
# Basics
Series by ketan Doshi
1. [State-of-the-Art Techniques](https://towardsdatascience.com/audio-deep-learning-made-simple-part-1-state-of-the-art-techniques-da1d3dff2504) *(What is sound and how it is digitized. What problems is audio deep learning solving in our daily lives. What are Spectrograms and why they are all-important.)*
2. [Why Mel Spectrograms perform better](https://towardsdatascience.com/audio-deep-learning-made-simple-part-2-why-mel-spectrograms-perform-better-aad889a93505) *(Processing audio data in Python. What are Mel Spectrograms and how to generate them)*
3. [Data Preparation and Augmentation ](https://towardsdatascience.com/audio-deep-learning-made-simple-part-3-data-preparation-and-augmentation-24c6e1f6b52) *(Enhance Spectrograms features for optimal performance by hyper-parameter tuning and data augmentation)*
4. [Sound Classification](https://towardsdatascience.com/audio-deep-learning-made-simple-sound-classification-step-by-step-cebc936bbe5) *(End-to-end example and architecture to classify ordinary sounds. Foundational application for a range of scenarios.)*
5. [Automatic Speech Recognition](https://towardsdatascience.com/audio-deep-learning-made-simple-automatic-speech-recognition-asr-how-it-works-716cfce4c706) *(Speech-to-Text algorithm and architecture, using CTC Loss and Decoding for aligning sequences.)*
6. [Beam Search](https://towardsdatascience.com/foundations-of-nlp-explained-visually-beam-search-how-it-works-1586b9849a24) *(Algorithm commonly used by Speech-to-Text and NLP applications to enhance predictions)*
# Terminology
1. Audio Source Separation - Source separation techniques in audio processing can be classified into various methods, such as blind source separation, supervised source separation, and semi-supervised source separation. These techniques use algorithms and signal processing methods to extract specific sound sources from an audio mixture. It can be classified into various methods, such as blind source separation, supervised source separation, and semi-supervised source separation. (note: Usually these methods are for multi-channel. If we have a single channel audio recordings, this is more challenging.)
2. Sound event detection - Sound Event Detection (SED) is the task of recognizing the sound events and their respective temporal start and end time in a recording. Sound events in real life do not always occur in isolation, but tend to considerably overlap with each other. Recognizing such overlapping sound events is referred as polyphonic SED.
3. Query-based separation - Query-based separation in audio could involve using specific sounds or queries to identify and extract particular sounds from an audio recording. For example, in a crowded audio environment, you might use query-based separation to extract the voice of a specific speaker from a mixture of voices. By providing a query related to the speaker's voice characteristics or specific phrases they are saying, algorithms can be designed to identify and separate that particular speaker's voice from the overall audio recording.
# Feature Engineering
1. Fourier transform
1. [youtube](https://www.youtube.com/watch?v=7Tk6BAJ3mm8)
2. [Read this first](https://jonathan-hui.medium.com/speech-recognition-feature-extraction-mfcc-plp-5455f5a69dd9), by Jonathan Hui - a/d, boost, Mel-frequency cepstral coefficients (MFCC), Perceptual Linear Prediction (PLP), Hanning vs Hamming, window, discrete fourier transform (DFT), IDFT.
3. Learning from Audio Series by mlearnere
1. [Learning from Audio: Wave Forms](https://towardsdatascience.com/learning-from-audio-wave-forms-46fc6f87e016)
2. [Learning from Audio: Time Domain Features](https://towardsdatascience.com/learning-from-audio-time-domain-features-4543f3bda34c)
3. [Learning from Audio: Fourier Transformation](https://towardsdatascience.com/learning-from-audio-fourier-transformations-f000124675ee)
4. [Learning from Audio: The Mel Scale, Mel Spectrograms, and Mel Frequency Cepstral Coefficients](https://towardsdatascience.com/learning-from-audio-the-mel-scale-mel-spectrograms-and-mel-frequency-cepstral-coefficients-f5752b6324a8)
5. [Learning from Audio: Pitch and Chromagrams](https://towardsdatascience.com/learning-from-audio-pitch-and-chromagrams-5158028a505)
4. Spectrogram
1. [Understanding the Mel Spectrogram](https://medium.com/analytics-vidhya/understanding-the-mel-spectrogram-fca2afa2ce53), [2](https://towardsdatascience.com/getting-to-know-the-mel-spectrogram-31bca3e2d9d0), [3](https://medium.com/hacking-media/beginner-guide-to-visualizing-audio-as-a-spectogram-in-python-65dca2ab1e61), [4](https://importchris.medium.com/how-to-create-understand-mel-spectrograms-ff7634991056), [delta deltas](https://www.youtube.com/watch?v=zxEnuPolylY)
5. Mel-frequency cepstral coefficients (MFCC)
1. [youtube](https://www.youtube.com/watch?app=desktop\&v=SJo7vPgRlBQ)
# Deep Neural Audio
1. [Audio Deep Learning Made Simple: Automatic Speech Recognition (ASR), How it Works](https://towardsdatascience.com/audio-deep-learning-made-simple-automatic-speech-recognition-asr-how-it-works-716cfce4c706)
2. [Learning from Audio Pitch and Chromagrams](https://towardsdatascience.com/learning-from-audio-pitch-and-chromagrams-5158028a505)
3. Audio deep learning made simple by Ketan Doshi
1. [State-of-the-Art Techniques](https://towardsdatascience.com/audio-deep-learning-made-simple-part-1-state-of-the-art-techniques-da1d3dff2504) *(What is sound and how it is digitized. What problems is audio deep learning solving in our daily lives. What are Spectrograms and why they are all-important.)*
2. [Why Mel Spectrograms perform better](https://towardsdatascience.com/audio-deep-learning-made-simple-part-2-why-mel-spectrograms-perform-better-aad889a93505) *(Processing audio data in Python. What are Mel Spectrograms and how to generate them)*
3. [Data Preparation and Augmentation](https://towardsdatascience.com/audio-deep-learning-made-simple-part-3-data-preparation-and-augmentation-24c6e1f6b52) *(Enhance Spectrograms features for optimal performance by hyper-parameter tuning and data augmentation)*
4. [Sound Classification](https://towardsdatascience.com/audio-deep-learning-made-simple-sound-classification-step-by-step-cebc936bbe5) *(End-to-end example and architecture to classify ordinary sounds. Foundational application for a range of scenarios.)*
5. [Beam Search](https://towardsdatascience.com/foundations-of-nlp-explained-visually-beam-search-how-it-works-1586b9849a24) *(Algorithm commonly used by Speech-to-Text and NLP applications to enhance predictions)*
4. *Wav2Vec - paper* [*Youtube*](https://www.youtube.com/watch?v=fMqYul2TvBE)
5. [Whisper](https://github.com/openai/whisper)
1. [ZAC](https://github.com/jumon/zac) (Zero-shot Audio Classification using Whisper) allows you to assign audio files to ANY class you want without training.
6. [Neural Amp Modeler](https://www.neuralampmodeler.com/) - is an open-source project that uses deep learning to create models of guitar amplifiers and pedals with state-of-the-art accuracy. [Training notebook](https://colab.research.google.com/github/sdatkinson/neural-amp-modeler/blob/main/bin/train/easy_colab.ipynb#scrollTo=zrXbQY7vjZjk)
7. [NAM uses WaveNet](https://arxiv.org/abs/1609.03499)
8. Echo State Network
1. [Youtube](https://www.youtube.com/watch?v=uF4i9_7IQlI)
2. [Gentle introduction to Echo State Networks](https://towardsdatascience.com/gentle-introduction-to-echo-state-networks-af99e5373c68)
3. [Github code](https://github.com/ciortanmadalina/EchoStateNetwork/blob/master/EchoStateNetwork.ipynb)
[ ](https://www.neuralampmodeler.com/users)
# Algorithms
Sound Event Detection
1. [YamNet](https://github.com/tensorflow/models/tree/master/research/audioset/yamnet), and Real-time sound event detection [github](https://github.com/robertanto/Real-Time-Sound-Event-Detection), [Event types labels list](https://github.com/robertanto/Real-Time-Sound-Event-Detection/blob/main/keras_yamnet/yamnet_class_map.csv) - Relevant labels: 420:430
Query-based separation
1. [Zero Shot Audio Source Separation](https://github.com/RetroCirce/Zero_Shot_Audio_Source_Separation), [paper](https://arxiv.org/abs/2112.07891), [interface](https://replicate.com/retrocirce/zero_shot_audio_source_separation) - is a three-component pipeline that allows you to train an audio source separator to separate any source from the track. All you need is a mixture audio to separate, and a given source sample as a query. Then the model will separate your specified source from the track.
Audio Source Separation
1. [Audio Sep](https://github.com/Audio-AGI/AudioSep) - AudioSep is a foundation model for open-domain sound separation with natural language queries. AudioSep demonstrates strong separation performance and impressive zero-shot generalization ability on numerous tasks such as audio event separation, musical instrument separation, and speech enhancement"
2. Wave-U-net
1. [Original](https://github.com/f90/Wave-U-Net) Version 4y old
2. [Pytorch](https://github.com/f90/Wave-U-Net-Pytorch) Version 3y old
3. [TF2 / Keras](https://github.com/satvik-venkatesh/Wave-U-net-TF2) Version 2y old
4. [For speech enhancements](https://github.com/craigmacartney/Wave-U-Net-For-Speech-Enhancement)
Blind Source Separation
1. [Deep Audio Prior](https://github.com/adobe/Deep-Audio-Prior) - Our deep audio prior can enable several audio applications: blind sound source separation, interactive mask-based editing, audio textual synthesis, and audio watermarker removal.
2. BSS ([EM source separation](https://github.com/fgnt/pb_bss)) - This repository covers EM algorithms to separate speech sources in multi-channel recordings. In particular, the repository contains methods to integrate Deep Clustering (a neural network-based source separation algorithm) with a probabilistic spatial mixture model as proposed in the Interspeech paper "Tight integration of spatial and spectral features for BSS with Deep Clustering embeddings" presented at Interspeech 2017 in Stockholm.
Image embeddings and others
1. [Openl3](https://github.com/marl/openl3) - OpenL3: Open-source deep audio and image embeddings
2. [Pitch estimation](https://github.com/marl/crepe)
3. [Speaker recognition](https://github.com/Anwarvic/Speaker-Recognition) - Speaker recognition is the identification of a person given an audio file. It is used to answer the question "Who is speaking?" Speaker verification (also called speaker authentication) is similar to speaker recognition, but instead of returning the speaker who is speaking, it returns whether the speaker (who is claiming to be a certain one) is truthful or not. Speaker Verification is considered to be a little easier than speaker recognition.
4. [Voice activity detector](https://github.com/snakers4/silero-vad)
5. Taken from [here](https://www.mathworks.com/help/audio/referencelist.html?type=function\&category=pretrained-models\&s_tid=CRUX_topnav)
Other Tools
1. [KALDI](https://kaldi-asr.org/models.html) speech recognition toolkit with many SOTA models.
2. [isolating instruments from stereo music using Convolutional Neural Networks](https://towardsdatascience.com/audio-ai-isolating-vocals-from-stereo-music-using-convolutional-neural-networks-210532383785), [part 2](https://towardsdatascience.com/audio-ai-isolating-instruments-from-stereo-music-using-convolutional-neural-networks-584ababf69de)
3. [Sound classification using cnn, loading and normalizing sounds using librosa, converting to a 2d spectrogram image, using cnn on top.](https://medium.com/@mikesmales/sound-classification-using-deep-learning-8bc2aa1990b7)
4. [speech recognition with DL -](https://medium.com/@ageitgey/machine-learning-is-fun-part-6-how-to-do-speech-recognition-with-deep-learning-28293c162f7a) how to convert sounds to vectors, feeding into an RNN.
5. (Great) [Jonathan Hui on speech recognition](https://medium.com/@jonathan_hui/speech-recognition-series-71fd6784551a) - series.
6. [Gecko](https://medium.com/gong-tech-blog/introducing-gecko-an-open-source-solution-for-effective-annotation-of-conversations-2ecec0909941) - ([github.com/gong-io/gecko](https://github.com/gong-io/gecko)) [youtube](https://www.youtube.com/watch?v=CBYA0YC1NBI), is an open-source tool for the annotation of the linguistic content of conversations. It can be used for segmentation, diarization, and transcription. With Gecko, you can create and perfect audio-based datasets, compare the results of multiple models simultaneously, and highlight differences between transcriptions.
7.
# A Reality Check
[**A powerful benchmark**](https://github.com/KevinMusgrave/powerful-benchmarke)**,** [**paper**](https://arxiv.org/pdf/2003.08505.pdf)**,** [**medium**](https://medium.com/@tkm45/updates-to-a-metric-learning-reality-check-730b6914dfe7) **- normalizing data sets allows us to see that there wasn't any advancement in terms of metrics in many NLP algorithms.**
# NLP Tools
#### **SPACY**
1. [**Vidhaya on spacy vs ner**](https://www.analyticsvidhya.com/blog/2017/04/natural-language-processing-made-easy-using-spacy-%E2%80%8Bin-python/) **- tutorial + code on how to use spacy for pos, dep, ner, compared to nltk/corenlp (sner etc). The results reflect a global score not specific to LOC for example.**
2. **The** [**spaCy course**](https://course.spacy.io/)
3. **SPACY OPTIMIZATION -** [**LP using CYTHON and SPACY.**](https://medium.com/huggingface/100-times-faster-natural-language-processing-in-python-ee32033bdced)
### **NLP embedding repositories**
1. [**Nlpl**](http://vectors.nlpl.eu/repository/)
### **NLP DATASETS**
1. [**The bid bad**](https://datasets.quantumstat.com/) **600,** [**medium**](https://medium.com/towards-artificial-intelligence/600-nlp-datasets-and-glory-4b0080bf5ab)
2. [Amazon 51 Language datasets for NLU](https://www.amazon.science/blog/amazon-releases-51-language-dataset-for-language-understanding)
### **NLP Libraries**
1. [**Has all the known libraries**](https://nlpforhackers.io/libraries/)
2. [**Comparison between spacy, pytorch, allenlp**](https://luckytoilet.wordpress.com/2018/12/29/deep-learning-for-nlp-spacy-vs-pytorch-vs-allennlp/?fbclid=IwAR236Mrg4J4pBGSLlvQ8xNbEw21lvMeLi6CfqRB2x6BL1U9vJm7_mB7Q10E) **- very basic info**
3. [**Comparison spacy,nltk**](https://spacy.io/usage/facts-figures) **core nlp**
4. [**Comparing Production grade nlp libs**](https://www.oreilly.com/ideas/comparing-production-grade-nlp-libraries-accuracy-performance-and-scalability)
5. [**nltk vs spac**](https://blog.thedataincubator.com/2016/04/nltk-vs-spacy-natural-language-processing-in-python/)**y**
# Foundation NLP
## **Basic nlp**
1. [**Benchmarking tokenizers for optimalprocessing speed**](https://towardsdatascience.com/benchmarking-python-nlp-tokenizers-3ac4735100c5)
2. [**Using nltk with gensim** ](https://www.scss.tcd.ie/~munnellg/projects/visualizing-text.html)
3. [**Multiclass text classification with svm/nb/mean w2v/**](https://towardsdatascience.com/multi-class-text-classification-model-comparison-and-selection-5eb066197568)**d2v - tutorial with code and notebook.**
4. [**Basic pipeline for keyword extraction**](https://medium.com/analytics-vidhya/automated-keyword-extraction-from-articles-using-nlp-bfd864f41b34)
5. [**DL for text classification**](https://ahmedbesbes.com/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification.html)
1. **Logistic regression with word ngrams**
2. **Logistic regression with character ngrams**
3. **Logistic regression with word and character ngrams**
4. **Recurrent neural network (bidirectional GRU) without pre-trained embeddings**
5. **Recurrent neural network (bidirectional GRU) with GloVe pre-trained embeddings**
6. **Multi channel Convolutional Neural Network**
7. **RNN (Bidirectional GRU) + CNN model**
6. **LexNLP -** [**glorified regex extractor**](https://towardsdatascience.com/lexnlp-library-for-automated-text-extraction-ner-with-bafd0014a3f8)
## **Chunking**
1. [**Coding Chunkers as Taggers: IO, BIO, BMEWO, and BMEWO+**](https://lingpipe-blog.com/2009/10/14/coding-chunkers-as-taggers-io-bio-bmewo-and-bmewo/)
## **NLP for hackers tutorials**
1. [**How to convert between verb/noun/adjective/adverb forms using Wordnet**](https://nlpforhackers.io/convert-words-between-forms/)
2. [**Complete guide for training your own Part-Of-Speech Tagger -**](https://nlpforhackers.io/training-pos-tagger/) **using** [**Penn Treebank tagset**](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html)**. Using nltk or stanford pos taggers, creating features from actual words (manual stemming, etc0 using the tags as labels, on a random forest, thus creating a classifier for POS on our own. Not entirely sure why we need to create a classifier from a “classifier”.**
3. [**Word net introduction**](https://nlpforhackers.io/starting-wordnet/) **- POS, lemmatize, synon, antonym, hypernym, hyponym**
4. [**Sentence similarity using wordnet**](https://nlpforhackers.io/wordnet-sentence-similarity/) **- using synonyms cumsum for comparison. Today replaced with w2v mean sentence similarity.**
5. [**Stemmers vs lemmatizers**](https://nlpforhackers.io/stemmers-vs-lemmatizers/) **- stemmers are faster, lemmatizers are POS / dictionary based, slower, converting to base form.**
6. [**Chunking**](https://nlpforhackers.io/text-chunking/) **- shallow parsing, compared to deep, similar to NER**
7. [**NER -**](https://nlpforhackers.io/named-entity-extraction/) **using nltk chunking as a labeller for a classifier, training one of our own. Using IOB features as well as others to create a new ner classifier which should be better than the original by using additional features. Aso uses a new english dataset GMB.**
8. [**Building nlp pipelines, functions coroutines etc..**](https://nlpforhackers.io/building-a-nlp-pipeline-in-nltk/)
9. [**Training ner using generators**](https://nlpforhackers.io/training-ner-large-dataset/)
10. [**Metrics, tp/fp/recall/precision/micro/weighted/macro f1**](https://nlpforhackers.io/classification-performance-metrics/)
11. [**Tf-idf**](https://nlpforhackers.io/tf-idf/)
12. [**Nltk for beginners**](https://nlpforhackers.io/introduction-nltk/)
13. [**Nlp corpora**](https://nlpforhackers.io/corpora/) **corpuses**
14. [**bow/bigrams**](https://nlpforhackers.io/language-models/)
15. [**Textrank**](https://nlpforhackers.io/textrank-text-summarization/)
16. [**Word cloud**](https://nlpforhackers.io/word-clouds/)
17. [**Topic modelling using gensim, lsa, lsi, lda,hdp**](https://nlpforhackers.io/topic-modeling/)
18. [**Spacy full tutorial**](https://nlpforhackers.io/complete-guide-to-spacy/)
19. [**POS using CRF**](https://nlpforhackers.io/crf-pos-tagger/)
## **Synonyms**
1. **Python Module to get Meanings, Synonyms and what not for a given word using vocabulary (also a comparison against word net)** [**https://vocabulary.readthedocs.io/en/…**](https://vocabulary.readthedocs.io/en/latest/)
**For a given word, using Vocabulary, you can get its**
* **Meaning**
* **Synonyms**
* **Antonyms**
* **Part of speech : whether the word is a noun, interjection or an adverb et el**
* **Translate : Translate a phrase from a source language to the desired language.**
* **Usage example : a quick example on how to use the word in a sentence**
* **Pronunciation**
* **Hyphenation : shows the particular stress points(if any)**
### **Swiss army knife libraries**
1. [**textacy**](https://chartbeat-labs.github.io/textacy/) **is a Python library for performing a variety of natural language processing (NLP) tasks, built on the high-performance spacy library. With the fundamentals — tokenization, part-of-speech tagging, dependency parsing, etc. — delegated to another library, textacy focuses on the tasks that come before and follow after.**
## **Collocation**
1. **What is collocation? - “the habitual juxtaposition of a particular word with another word or words with a frequency greater than chance.”Medium** [**tutorial**](https://medium.com/@nicharuch/collocations-identifying-phrases-that-act-like-individual-words-in-nlp-f58a93a2f84a)**, quite good, comparing freq/t-test/pmi/chi2 with github code**
2. **A website dedicated to** [**collocations**](http://www.collocations.de/)**, methods, references, metrics.**
3. [**Text analysis for sentiment, doing feature selection**](https://streamhacker.com/tag/chi-square/) **a tutorial with chi2(IG?),** [**part 2 with bi-gram collocation in ntlk**](https://streamhacker.com/2010/05/24/text-classification-sentiment-analysis-stopwords-collocations/)
4. [**Text2vec**](http://text2vec.org/collocations.html) **in R - has ideas on how to use collocations, for downstream tasks, LDA, W2V, etc. also explains about PMI and other metrics, note that gensim metric is unsupervised and probablistic.**
5. **NLTK on** [**collocations**](http://www.nltk.org/howto/collocations.html)
6. **A** [**blog post**](https://graus.nu/tag/gensim/) **about keeping or removing stopwords for collocation, usefull but no firm conclusion. Imo we should remove it before**
7. **A** [**blog post**](http://n-chandra.blogspot.com/2014/06/collocation-extraction-using-nltk.html) **with code of using nltk-based collocation**
8. **Small code for using nltk** [**collocation**](http://compling.hss.ntu.edu.sg/courses/hg2051/week09.html)
9. **Another code / score example for nltk** [**collocation**](https://stackoverflow.com/questions/8683588/understanding-nltk-collocation-scoring-for-bigrams-and-trigrams)
10. **Jupyter notebook on** [**manually finding collocation**](https://github.com/sgsinclair/alta/blob/a482d343142cba12030fea4be8f96fb77579b3ab/ipynb/utilities/Collocates.ipynb) **- not useful**
11. **Paper:** [**Ngram2Vec**](http://www.aclweb.org/anthology/D17-1023) **-** [**Github**](https://github.com/zhezhaoa/ngram2vec) **We introduce ngrams into four representation methods. The experimental results demonstrate ngrams’ effectiveness for learning improved word representations. In addition, we find that the trained ngram embeddings are able to reflect their semantic meanings and syntactic patterns. To alleviate the costs brought by ngrams, we propose a novel way of building co-occurrence matrix, enabling the ngram-based models to run on cheap hardware**
12. **Youtube on** [**bigrams**](https://www.youtube.com/watch?v=3i5QEmaOtkU\&list=PLjTSKEJpqIeANubEWBo-z5TO89m7VtfG_)**,** [**collocation**](https://www.youtube.com/watch?v=QvrbsjwErMA)**, mutual info and** [**collocation**](http://www.let.rug.nl/nerbonne/teach/rema-stats-meth-seminar/presentations/Suster-2011-MI-Coll.pdf)
## **Language detection**
1. [**Using google lang detect**](https://github.com/Mimino666/langdetect) **- 55 languages af, ar, bg, bn, ca, cs, cy, da, de, el, en, es, et, fa, fi, fr, gu, he,**\
**hi, hr, hu, id, it, ja, kn, ko, lt, lv, mk, ml, mr, ne, nl, no, pa, pl,**\
**pt, ro, ru, sk, sl, so, sq, sv, sw, ta, te, th, tl, tr, uk, ur, vi, zh-cn, zh-tw**
## **Stemming**
**How to measure a stemmer?**
1. **References \[**[**1**](https://files.eric.ed.gov/fulltext/EJ1020841.pdf) [**2**](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.68.2870\&rep=rep1\&type=pdf)**(apr11)** [**3**](http://www.informationr.net/ir/19-1/paper605.html)**(Index compression factor ICF)** [**4**](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.16.8310\&rep=rep1\&type=pdf) [**5**](https://pdfs.semanticscholar.org/1c0c/0fa35d4ff8a2f925eb955e48d655494bd167.pdf)**]**
## **Phrase modelling**
1. [**Phrase Modeling**](https://github.com/explosion/spacy-notebooks/blob/master/notebooks/conference_notebooks/modern_nlp_in_python.ipynb) **- using gensim and spacy**
**Phrase modeling is another approach to learning combinations of tokens that together represent meaningful multi-word concepts. We can develop phrase models by looping over the the words in our reviews and looking for words that co-occur (i.e., appear one after another) together much more frequently than you would expect them to by random chance. The formula our phrase models will use to determine whether two tokens AA and BB constitute a phrase is:**
**count(A B)−countmincount(A)∗count(B)∗N>threshold**
1. [ **SO on PE.**](https://www.quora.com/Whats-the-best-way-to-extract-phrases-from-a-corpus-of-text-using-Python)
2.
## **Document classification**
1. [**Using hierarchical attention network**](https://www.cs.cmu.edu/~hovy/papers/16HLT-hierarchical-attention-networks.pdf)
## **Hebrew NLP tools**
1. [**HebMorph**](https://github.com/synhershko/HebMorph.CorpusSearcher) **last update 7y ago**
2. [**Hebmorph elastic search**](https://github.com/synhershko/elasticsearch-analysis-hebrew/wiki/Getting-Started) [**Hebmorph blog post**](https://code972.com/blog/2013/12/673-hebrew-search-done-right)**, and other** [**blog posts**](https://code972.com/hebmorph)**,** [**youtube**](https://www.youtube.com/watch?v=v8w32wC6ppI)
3. [**Awesome hebrew nlp git**](https://github.com/iddoberger/awesome-hebrew-nlp)**,** [**git**](https://github.com/synhershko/HebMorph/blob/master/dotNet/HebMorph/HSpell/Constants.cs)
4. [**Hebrew-nlp service**](https://hebrew-nlp.co.il/) [**docs**](https://docs.hebrew-nlp.co.il/#/README) [**the features**](https://hebrew-nlp.co.il/features) **(morphological analysis, normalization etc),** [**git**](https://github.com/HebrewNLP)
5. [**Apache solr stop words (dead)**](https://wiki.apache.org/solr/LanguageAnalysis#Hebrew)
6. [**SO on hebrew analyzer/stemming**](https://stackoverflow.com/questions/1063856/lucene-hebrew-analyzer)**,** [**here too**](https://stackoverflow.com/questions/20953495/is-there-a-good-stemmer-for-hebrew)
7. [**Neural sentiment benchmark using two algorithms, for character and word level lstm/gru**](https://github.com/omilab/Neural-Sentiment-Analyzer-for-Modern-Hebrew) **-** [**the paper**](http://aclweb.org/anthology/C18-1190)
8. [**Hebrew word embeddings**](https://github.com/liorshk/wordembedding-hebrew)
9. [**Paper for rich morphological datasets for comparison - rivlin**](https://aclweb.org/anthology/C18-1190)
## **Semantic roles:**
1. [**http://language.worldofcomputing.net/semantics/semantic-roles.html**](http://language.worldofcomputing.net/semantics/semantic-roles.html)
# Name Matching
## **Articles**
1. [Analytics Vidhya](https://medium.com/analytics-vidhya/fuzzy-name-matching-datasets-1ae28884f226) on fuzzy name matching datasets, by Zaki Jefferson
2. [Fuzzy matching people names](https://towardsdatascience.com/fuzzy-matching-people-names-6e738d6b8fe) by vadim markovtsev
3. [Name Matching Across datasets](https://ai.nic.in/AI/NameMatchingCaseML) - POC by Centere of Excellence in AI National Informatics Centre
4. [fuzzy name matching algorithms](https://towardsdatascience.com/python-tutorial-fuzzy-name-matching-algorithms-7a6f43322cc5) by felix kuestahler
## **Datasets**
1. [first and last name dataset](https://github.com/philipperemy/name-dataset), facebook 533M records, philippe remy
2. [data.world name datasets](https://data.world/datasets/names)
3. [Kaggle](https://www.kaggle.com/fivethirtyeight/fivethirtyeight-most-common-name-dataset/version/108), Name datasets, by fivethirtyeight\
4. [gender by name dataset](https://archive.ics.uci.edu/ml/datasets/Gender+by+Name)
5. [paper](http://www.lrec-conf.org/proceedings/lrec2008/pdf/291_paper.pdf) - a ground truth dataset for matching coltural diverse romanized person names
## Tools
1. [Dedupe](https://www.reddit.com/r/datasets/comments/4zrozk/request_name_matching_dataset/) - a python library for accurate and scalable fuzzy matching record deduplication and entity resolution
2. [name](https://github.com/bradhackinen/nama) - fast flexible name matching for large datasets
3. [name matcher](https://github.com/athenianco/names-matcher) by athenianco\
# String Matching
## Tools
1. [**Fuzzy string matching library - fuzzywuzzy - using edit-distance**](https://towardsdatascience.com/natural-language-processing-for-fuzzy-string-matching-with-python-6632b7824c49)
2. [**Difflib**](https://docs.python.org/3/library/difflib.html)**,** [**difflib**](https://pymotw.com/2/difflib/)**-2**
## **REGEX**
1. [**Why re is slow** ](https://swtch.com/~rsc/regexp/regexp1.html)
2. [**Benchmark**](https://github.com/mariomka/regex-benchmark)**,** [**comparisons**](https://rust-leipzig.github.io/regex/2017/03/28/comparison-of-regex-engines/)**,** [**more**](https://stackoverflow.com/questions/3544225/regular-expression-library-benchmarks)**,** [**many more**](https://stackoverflow.com/questions/11033190/regex-library-benchmark)**,**
3. [**Split on separator but keep the separator**](http://programmaticallyspeaking.com/split-on-separator-but-keep-the-separator-in-python.html)**, in Python**
4. [**Semantic versioning**](https://regexr.com/39s32)
5. [**Hyperscan**](https://github.com/intel/hyperscan) **-** [**Hyperscan**](https://www.hyperscan.io/) [**(paper)**](https://www.usenix.org/system/files/nsdi19-wang-xiang.pdf) **is a high-performance multiple regex matching library. It follows the regular expression syntax of the commonly-used libpcre library, but is a standalone library with its own C API.**
6. [**Re2**](https://github.com/google/re2/tree/abseil/python) **-** [**python**](https://pypi.org/project/re2/) **This is the source code repository for RE2, a regular expression library.**
1. [**Spacy’s Matcher & “regex”**](https://spacy.io/usage/rule-based-matching)
2. [**Flashtext**](https://github.com/vi3k6i5/flashtext)**- This module can be used to replace keywords in sentences or extract keywords from sentences. It is based on the** [**FlashText algorithm**](https://arxiv.org/abs/1711.00046)**.**
# TF-IDF
[**TF-IDF**](http://www.tfidf.com/) **- how important is a word to a document in a corpus**
**TF(t) = (Number of times term t appears in a document) / (Total number of terms in the document).**
**Frequency of word in doc / all words in document (normalized bcz docs have diff sizes)**
**IDF(t) = log\_e(Total number of documents / Number of documents with term t in it).**
**measures how important a term is**
**TF-IDF is TF\*IDF**
1. [**A much clearer explanation plus python code**](https://stevenloria.com/tf-idf/)**,** [**part 2**](http://blog.christianperone.com/2011/10/machine-learning-text-feature-extraction-tf-idf-part-ii/)
2. [**Get top tfidf keywords**](https://stackoverflow.com/questions/34232190/scikit-learn-tfidfvectorizer-how-to-get-top-n-terms-with-highest-tf-idf-score)
3. [**Print top features**](https://gist.github.com/StevenMaude/ea46edc315b0f94d03b9)
**Data sets:**
1. [**Fast text multilingual**](https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md)
2. [**NLP embeddings**](http://vectors.nlpl.eu/repository/#)
### **Sparse textual content**
1. **mean(IDF(i) \* w2v word vectors (i)) with or without reducing PC1 from the whole w2 average (amir pupko)**\
**def mean\_weighted\_embedding(model, words, idf=1.0):**
**if words:**
**return np.mean(idf \* model\[words], axis=0)a**
**else:**
**print('we have an empty list')**
**return \[]**
**idf\_mapping = dict(zip(vectorizer.get\_feature\_names(), vectorizer.idf\_))**
**logs\_sequences\_df\['idf\_vectors'] = logs\_sequences\_df.message.apply(lambda x: \[idf\_mapping\[token] for token in splitter(x)])**
**logs\_sequences\_df\['mean\_weighted\_idf\_w2v'] = \[mean\_weighted\_embedding(ft, splitter(logs\_sequences\_df\['message'].iloc\[i]), 1 / np.array(logs\_sequences\_df\['idf\_vectors'].iloc\[i]).reshape(-1,1)) for i in range(logs\_sequences\_df.shape\[0])]**\
1. [**Multiply by TFIDF**](https://towardsdatascience.com/supercharging-word-vectors-be80ee5513d)
2. **Enriching using lstm-next word (char or word-wise)**
3. **Using external wiktionary/pedia data for certain words, phrases**
4. **Finding clusters of relevant data and figuring out if you can enrich based on the content of the clusters**
5. [**Applying deep nlp methods without big data, i.e., sparseness**](https://towardsdatascience.com/lessons-learned-from-applying-deep-learning-for-nlp-without-big-data-d470db4f27bf?_branch_match_id=584170448791192656)
# Language Detection Identification Generation (NLD, NLI, NLG)
## Neural **Language Models**
1. [**Mastery on Word-based** ](https://machinelearningmastery.com/develop-word-based-neural-language-models-python-keras/)
## **NEURAL LANGUAGE GENERATION**
1. [**Using RNN**](https://www.aclweb.org/anthology/C16-1103)
2. [**Using language modeling**](https://medium.com/@shivambansal36/language-modelling-text-generation-using-lstms-deep-learning-for-nlp-ed36b224b275)
3. [**Word based vs char based**](https://datascience.stackexchange.com/questions/13138/what-is-the-difference-between-word-based-and-char-based-text-generation-rnns) **- Word-based LMs display higher accuracy and lower computational cost than char-based LMs. However, char-based RNN LMs better model languages with a rich morphology such as Finish, Turkish, Russian etc. Using word-based RNN LMs to model such languages is difficult if possible at all and is not advised. Char-based RNN LMs can mimic grammatically correct sequences for a wide range of languages, require bigger hidden layer and computationally more expensive while word-based RNN LMs train faster and generate more coherent texts and yet even these generated texts are far from making actual sense.**
4. [**mediu m on Char based with code, leads to better grammer**](https://towardsdatascience.com/besides-word-embedding-why-you-need-to-know-character-embedding-6096a34a3b10)
5. [**Git, keras language models, char level word level and sentence using VAE**](https://github.com/pbloem/language-models)
## **LANGUAGE DETECTION / IDENTIFICATION**
1. [**A qualitative comparison of google, azure, amazon, ibm LD LI**](https://medium.com/activewizards-machine-learning-company/comparison-of-the-most-useful-text-processing-apis-e4b4c1e6626a)
2. [**CLD2**](https://github.com/CLD2Owners/cld2/tree/master/docs)**,** [**CLD3**](https://github.com/google/cld3)**,** [**PYCLD**](https://github.com/aboSamoor/pycld2)**2,** [**POLYGLOT wraps CLD**](https://polyglot.readthedocs.io/en/latest/Detection.html)**,** [**alex ott cld stats**](https://gist.github.com/alexott/dd43fa8d1db4b8202d55c6325b2c69c2)**,** [**cld comparison vs tika langid**](http://blog.mikemccandless.com/2011/10/accuracy-and-performance-of-googles.html)
3. [**Fast text LI**](https://fasttext.cc/blog/2017/10/02/blog-post.html?fbclid=IwAR3dtJFRmpoZYq24U9ePlGeC65PT1Gy2Rsz9fH834CZ74Vs70utk2suuFsc)**,** [**facebook post**](https://www.facebook.com/groups/1174547215919768/permalink/1702123316495486/?comment_id=1704414996266318\&reply_comment_id=1705159672858517\¬if_id=1507280476710677\¬if_t=group_comment)
4. **OPENNLP**
5. [**Google detect language**](https://cloud.google.com/translate/docs/detecting-language)**,** [**github code**](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/translate/cloud-client/snippets.py)**,** [**v3beta**](https://cloud.google.com/translate/docs/detecting-language-v3)
6. [**Microsoft azure LD,**](https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/how-tos/text-analytics-how-to-language-detection) [**2**](https://westcentralus.dev.cognitive.microsoft.com/docs/services/TextAnalytics-v2-1/operations/56f30ceeeda5650db055a3c7)
7. [**Ibm watson**](https://cloud.ibm.com/apidocs/language-translator)**,** [**2**](https://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.1/com.ibm.swg.im.infosphere.dataexpl.engine.doc/c_vse_language_detection.html)
8. [**Amazon,**](https://docs.aws.amazon.com/comprehend/latest/dg/how-languages.html) [ **2**](https://aws.amazon.com/comprehend/)
9. [**Lingua - most accurate for java… doesn't seem like its accurate enough**](https://github.com/pemistahl/lingua)
10. [**LD with infinity gram 99.1 on a lot of data a benchmark for this 2012 method**](https://shuyo.wordpress.com/2012/02/21/language-detection-for-twitter-with-99-1-accuracy/)**,** [**LD with infinity gram**](https://github.com/shuyo/ldig)
11. [**WiLI dataset for LD, comparison of CLD vs others** ](https://arxiv.org/pdf/1801.07779.pdf)
12. [**Comparison of CLD vs FT vs OPEN NLP**](http://alexott.blogspot.com/2017/10/evaluating-fasttexts-models-for.html) **- beware based on 200 samples per language!!**
**Full results for every language that I tested are in table at the end of blog post & on** [**Github**](https://gist.github.com/alexott/dd43fa8d1db4b8202d55c6325b2c69c2)**. From them I can make following conclusions:**
* **all detectors are equally good on some languages, such as, Japanese, Chinese, Vietnamese, Greek, Arabic, Farsi, Georgian, etc. - for them the accuracy of detection is between 98 & 100%;**
* **CLD is much better in detection of "rare" languages, especially for languages, that are similar to more frequently used - Afrikaans vs Dutch, Azerbaijani vs. Turkish, Malay vs. Indonesian, Nepali vs. Hindi, Russian vs Bulgarian, etc. (it could be result of imbalance of training data - I need to check the source dataset);**
* **for "major" languages not mentioned above (English, French, German, Spanish, Portuguese, Dutch) the fastText results are much better than CLD's, and in many cases lingid.py's & OpenNLP's;**
* **for many languages results for "compressed" fastText model are slightly worse than results from "full" model (mostly only by 1-2%, but could be higher, like for Kazakh when difference is 33%), but there are languages where the situation is different - results for compressed are slight better than for full (for example, for German or Dutch);**
**OpenNLP has many misclassifications for Cyrillic languages - Russian/Ukrainian, ...**
**Rafael Oliveira** [**posted on FB**](https://www.facebook.com/groups/1174547215919768/permalink/1702123316495486/?comment_id=1704414996266318\&reply_comment_id=1705159672858517\¬if_id=1507280476710677\¬if_t=group_comment) **a simple diagram that shows what languages are detected better by CLD & what is better handled by fastText**
**Here are some additional notes about differences in behavior of detectors that I observe during analyzing results:**
* **fastText is more reliable than CLD on the short texts;**
* **fastText models & langid.py detect language as Hebrew instead of Jewish as in CLD. Similarly, CLD uses 'in' for Indonesian language instead of standard 'id' used by fastText & langid.py;**
* **fastText distinguish between Cyrillic- & Latin-based versions of Serbian;**
* **CLD tends to incorporate geographical & person's names into detection results - for example, blog post in German about travel to Iceland is detected as Icelandic, while fastText detects it as German;**
* **In extended detection mode CLD tends to select more rare language, like, Galician or Catalan over Spanish, Serbian instead of Russian, etc.**
* **OpenNLP isn't very good in detection for short texts.**
**The models released by fastText development team provides very good alternative to existing language detection tools, like, Google's CLD & langid.py - for most of "popular" languages, these models provides higher detection accuracy comparing to other tools, combined with high speed of detection (drawback of langid.py). Even using "compressed" model it's possible to reach good detection accuracy. Although for some less frequently used languages, CLD & langid.py may show better results.**
**Performance-wise, the langid.py is much slower than both CLD & fastText. On average, CLD requires 0.5-1 ms to perform language detection. For fastText & langid.py I don't have precise numbers yet, only approximates based on speed of execution of corresponding programs.**
**GIT:**
1. [**LD with infinity gram**](https://github.com/shuyo/ldig)
**Articles:**
1. [**Medium on training LI models**](https://towardsdatascience.com/how-i-trained-a-language-detection-ai-in-20-minutes-with-a-97-accuracy-fdeca0fb7724)
2.
**Papers:**
1. [**A comparison of lang ident approaches**](https://link.springer.com/chapter/10.1007/978-3-642-12275-0_59)
2. [**lI on code switch social media**](https://www.aclweb.org/anthology/W18-3206)
3. [**Comparing LI methods, has 6 big languages**](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.149.630\&rep=rep1\&type=pdf)
4. [**Comparing LI techniques**](https://dbs.cs.uni-duesseldorf.de/lehre/bmarbeit/barbeiten/ba_panich.pdf)
5. [**Radim rehurek LI on the web extending the dictionary**](https://radimrehurek.com/cicling09.pdf)
6. [**Comparative study of LI methods**](https://pdfs.semanticscholar.org/c422/3cc3765a1ac2e085b420e771d8022e6c244f.pdf)**,** [**2**](https://www.semanticscholar.org/paper/A-Comparative-Study-on-Language-Identification-Grothe-Luca/3f47b38b434f614d0cbf9af94cb4d74aa2bfe759)
## **LANGUAGE TRANSLATION**
1. [**State of the art methods for neural machine translation**](https://www.topbots.com/ai-nlp-research-neural-machine-translation/) **- a review of papers**
2. **LASER:** [**Zero shot multi lang-translation by facebook**](https://code.fb.com/ai-research/laser-multilingual-sentence-embeddings/)**,** [**github**](https://github.com/facebookresearch/LASER)
3. [**How to use laser on medium**](https://medium.com/the-artificial-impostor/multilingual-similarity-search-using-pretrained-bidirectional-lstm-encoder-e34fac5958b0)
4. **Stanford coreNLP language POS/NER/DEP PARSE etc for** [**53 languages**](https://www.analyticsvidhya.com/blog/2019/02/stanfordnlp-nlp-library-python)
5. [**Using embedding spaces**](https://rare-technologies.com/translation-matrix-in-gensim-python/) **w2v by gensim**
6. [**The risk of using bleu**](https://towardsdatascience.com/evaluating-text-output-in-nlp-bleu-at-your-own-risk-e8609665a213)
7. **Really good:** [**BLUE - what is it, how it is calculated?**](https://slator.com/technology/how-bleu-measures-translation-and-why-it-matters/)
**“\[BLEU] looks at the presence or absence of particular words, as well as the ordering and the degree of distortion—how much they actually are separated in the output.”**
**BLEU’s evaluation system requires two inputs: (i) a numerical translation closeness metric, which is then assigned and measured against (ii) a corpus of human reference translations.**
**BLEU averages out various metrics using an** [**n-gram method**](https://en.wikipedia.org/wiki/N-gram)**, a probabilistic language model often used in computational linguistics.**
**The result is typically measured on a 0 to 1 scale, with 1 as the hypothetical “perfect” translation. Since the human reference, against which MT is measured, is always made up of multiple translations, even a human translation would not score a 1, however. Sometimes the score is expressed as multiplied by 100 or, as in the case of Google mentioned above, by 10.**
**a BLEU score offers more of an intuitive rather than an absolute meaning and is best used for relative judgments: “If we get a BLEU score of 35 (out of 100), it seems okay, but it actually has no correlation to the quality of the output in any meaningful sense. If it’s less than 15, we can probably safely say it’s very bad. If it’s greater than 60, we probably have some mistake in our testing! So it will generally fall in there.”**
**“Typically, if you have multiple \[human translation] references, the BLEU score tends to be higher. So if you hear a very large BLEU score—someone gives you a value that seems very high—you can ask them if there are multiple references being used; because, then, that is the reason that the score is actually higher.”**
1. [**General talk**](https://slator.com/technology/google-facebook-amazon-neural-machine-translation-just-had-its-busiest-month-ever/) **about FAMG (fb, ama, micro, goog) and research direction atm, including some info about BLUE scores and the comparison issues with reports of BLUE (boils down to diff unmentioned parameters)**
2. **One proposed solution is** [**sacreBLUE**](https://arxiv.org/pdf/1804.08771.pdf)**, pip install sacreblue**
**Named entity language transliteration**
1. [**Paper**](https://arxiv.org/pdf/1808.02563.pdf)**,** [**blog post**](https://developer.amazon.com/blogs/alexa/post/ec66406c-094c-4dbc-8e9f-01050b27d43d/automatic-transliteration-can-help-alexa-find-data-across-language-barriers)**: English russian, hebrew, arabic, japanese, with data set and** [**github**](https://github.com/steveash/NETransliteration-COLING2018)
# Topics Modeling
## Misc
1. [**Word cloud**](http://keyonvafa.com/inauguration-wordclouds/) **for topic modellng**
2. [**Topic modeling with sentiment per topic according to the data in the topic**](https://www.slideshare.net/jainayush91/topic-modelling-tutorial-on-usage-and-applications)
3. **(TopSBM) topic block modeling,** [**Topsbm** ](https://topsbm.github.io/)
## NMF (Non Negative Matrix Factorization )
1. **Non-negative Matrix factorization (NMF)**
2. [**Medium Article about LDA and**](https://medium.com/ml2vec/topic-modeling-is-an-unsupervised-learning-approach-to-clustering-documents-to-discover-topics-fdfbf30e27df) **NMF (Non-negative Matrix factorization)+ code**
3. [**Sklearn LDA and NMF for topic modelling**](http://scikit-learn.org/stable/auto_examples/applications/plot_topics_extraction_with_nmf_lda.html#sphx-glr-auto-examples-applications-plot-topics-extraction-with-nmf-lda-py)
## **LSA (TFIDF + SVD)**
1. [**A very good article about LSA (TFIDV X SVD), pLSA, LDA, and LDA2VEC.**](https://medium.com/nanonets/topic-modeling-with-lsa-psla-lda-and-lda2vec-555ff65b0b05) **Including code and explanation about Dirichlet probability.** [**Lda2vec code**](http://nbviewer.jupyter.org/github/cemoody/lda2vec/blob/master/examples/twenty_newsgroups/lda2vec/lda2vec.ipynb#)
2. [**A descriptive comparison for LSA pLSA and LDA**](https://www.reddit.com/r/MachineLearning/comments/10mdtf/lsa_vs_plsa_vs_lda/)
## **LDA (Latent Dirichlet Allocation)**
* **A** [**great summation**](https://cs.stanford.edu/~ppasupat/a9online/1140.html) **about topic modeling, Pros and Cons! (LSA, pLSA, LDA)**
1. **(LDA) Latent Dirichlet Allocation**
2. **LDA is already taken by the above algorithm!**
3. [**Latent Dirichlet allocation (LDA) -**](https://algorithmia.com/algorithms/nlp/LDA) **This algorithm takes a group of documents (anything that is made of up text), and returns a number of topics (which are made up of a number of words) most relevant to these documents.**
4. [**Medium Article about LDA and**](https://medium.com/ml2vec/topic-modeling-is-an-unsupervised-learning-approach-to-clustering-documents-to-discover-topics-fdfbf30e27df) **NMF (Non-negative Matrix factorization)+ code**
5. [**Medium article on LDA - a good one with pseudo algorithm and proof**](https://medium.com/@jonathan_hui/machine-learning-latent-dirichlet-allocation-lda-1d9d148f13a4)
6. **In case LDA groups together two topics, we can influence the algorithm in a way that makes those two topics separable -** [**this is called Semi Supervised Guided LDA**](https://medium.freecodecamp.org/how-we-changed-unsupervised-lda-to-semi-supervised-guidedlda-e36a95f3a164)
7. [**LDA tutorials plus code**](https://www.machinelearningplus.com/nlp/topic-modeling-gensim-python/)**, used this to build my own classes - using gensim mallet wrapper, doesn't work on pyLDAviz, so use** [**this**](http://jeriwieringa.com/2018/07/17/pyLDAviz-and-Mallet/#comment-4018495276) **to fix it**
8. [**Introduction to LDA topic modelling, really good,**](http://www.vladsandulescu.com/topic-prediction-lda-user-reviews/) [**plus git code**](https://github.com/vladsandulescu/topics)
9. [**Sklearn examples using LDA and NMF**](http://scikit-learn.org/stable/auto_examples/applications/plot_topics_extraction_with_nmf_lda.html#sphx-glr-auto-examples-applications-plot-topics-extraction-with-nmf-lda-py)
10. [**Tutorial on lda/nmf on medium**](https://medium.com/mlreview/topic-modeling-with-scikit-learn-e80d33668730) **- using tfidf matrix as input!**
11. [**Gensim and sklearn LDA variants, comparison**](https://gist.github.com/aronwc/8248457)**,** [**python 3**](https://github.com/EricSchles/sklearn_gensim_example/blob/master/example.py)
12. [**Medium article on lda/nmf with code**](https://medium.com/mlreview/topic-modeling-with-scikit-learn-e80d33668730)
13. **One of the best explanation about** [**Tf-idf vs bow for LDA/NMF**](https://stackoverflow.com/questions/44781047/necessary-to-apply-tf-idf-to-new-documents-in-gensim-lda-model) **- tf for lda, tfidf for nmf, but tfidf can be used for top k selection in lda + visualization,** [**important paper**](http://www.cs.columbia.edu/~blei/papers/BleiLafferty2009.pdf)
14. [**LDA is a probabilistic**](https://stackoverflow.com/questions/40171208/scikit-learn-should-i-fit-model-with-tf-or-tf-idf) **generative model that generates documents by sampling a topic for each word and then a word from the sampled topic. The generated document is represented as a bag of words.**
**NMF is in its general definition the search for 2 matrices W and H such that W\*H=V where V is an observed matrix. The only requirement for those matrices is that all their elements must be non negative.**
**From the above definitions it is clear that in LDA only bag of words frequency counts can be used since a vector of reals makes no sense. Did we create a word 1.2 times? On the other hand we can use any non negative representation for NMF and in the example tf-idf is used.**
\
**As far as choosing the number of iterations, for the NMF in scikit learn I don't know the stopping criterion although I believe it is the relative improvement of the loss function being smaller than a threshold so you 'll have to experiment. For LDA I suggest checking manually the improvement of the log likelihood in a held out validation set and stopping when it falls under a threshold.**\
\
**The rest of the parameters depend heavily on the data so I suggest, as suggested by @rpd, that you do a parameter search.**\
\
**So to sum up, LDA can only generate frequencies and NMF can generate any non negative matrix.**
15. [**How to measure the variance for LDA and NMF, against PCA.**](https://stackoverflow.com/questions/48148689/how-to-compare-predictive-power-of-pca-and-nmf) **1. Variance score the transformation and inverse transformation of data, test for 1,2,3,4 PCs/LDs/NMs.**
16. [**Matching lda mallet performance with gensim and sklearn lda via hyper parameters**](https://groups.google.com/forum/#!topic/gensim/bBHkGogNrfg)
17. [**What is LDA?**](https://www.quora.com/Is-LDA-latent-dirichlet-allocation-unsupervised-or-supervised-learning)
1. **It is unsupervised natively; it uses joint probability method to find topics(user has to pass # of topics to LDA api). If “Doc X word” is size of input data to LDA, it transforms it to 2 matrices:**
2. **Doc X topic**
3. **Word X topic**
4. **further if you want, you can feed “Doc X topic” matrix to supervised algorithm if labels were given.**
18. **Medium on** [**LDA**](https://medium.com/ml2vec/topic-modeling-is-an-unsupervised-learning-approach-to-clustering-documents-to-discover-topics-fdfbf30e27df)**, explains the random probabilistic nature of LDA**
19. **Machinelearningplus on** [**LDA in sklearn**](https://www.machinelearningplus.com/nlp/topic-modeling-python-sklearn-examples/) **- a great read, dont forget to read the** [**mallet**](https://www.machinelearningplus.com/nlp/topic-modeling-gensim-python/) **article.**
20. **Medium on** [**LSA pLSA, LDA LDA2vec**](https://medium.com/nanonets/topic-modeling-with-lsa-psla-lda-and-lda2vec-555ff65b0b05)**, high level theoretical - not clear**
21. [**Medium on LSI vs LDA vs HDP, HDP wins..**](https://medium.com/square-corner-blog/topic-modeling-optimizing-for-human-interpretability-48a81f6ce0ed)
22. **Medium on** [**LDA**](https://medium.com/@samsachedina/effective-data-science-latent-dirichlet-allocation-a109742f7d1c)**, some historical reference and general high level how to use exapmles.**
23. [**Incredibly useful response**](https://www.quora.com/What-are-good-ways-of-evaluating-the-topics-generated-by-running-LDA-on-a-corpus) **on LDA grid search params and about LDA expectations. Must read.**
24. [**Lda vs pLSA**](https://stats.stackexchange.com/questions/155860/latent-dirichlet-allocation-vs-plsa)**, talks about the sampling from a distribution of distributions in LDA**
25. [**BLog post on topic modelling**](http://mcburton.net/blog/joy-of-tm/) **- has some text about overfitting - undiscussed in many places.**
26. [**Perplexity vs coherence on held out unseen dat**](https://stats.stackexchange.com/questions/182010/when-is-it-ok-to-not-use-a-held-out-set-for-topic-model-evaluation)**a, not okay and okay, respectively. Due to how we measure the metrics, ie., read the formulas.** [**Also this**](https://transacl.org/ojs/index.php/tacl/article/view/582/158) **and** [**this**](https://stackoverflow.com/questions/11162402/lda-topic-modeling-training-and-testing)
27. **LDA as** [**dimentionality reduction** ](https://stackoverflow.com/questions/46504688/lda-as-the-dimension-reduction-before-or-after-partitioning)
28. [**LDA on alpha and beta to control density of topics**](https://stats.stackexchange.com/questions/364494/lda-and-test-data-perplexity)
29. **Jupyter notebook on** [**hacknews LDA topic modelling**](http://nbviewer.jupyter.org/github/bmabey/hacker_news_topic_modelling/blob/master/HN%20Topic%20Model%20Talk.ipynb#topic=55\&lambda=1\&term=) **- missing code?**
30. [**Jupyter notebook**](http://nbviewer.jupyter.org/github/dolaameng/tutorials/blob/master/topic-finding-for-short-texts/topics_for_short_texts.ipynb) **for kmeans, lda, svd,nmf comparison - advice is to keep nmf or other as a baseline to measure against LDA.**
31. [**Gensim on LDA**](https://rare-technologies.com/what-is-topic-coherence/) **with** [**code** ](https://nbviewer.jupyter.org/github/dsquareindia/gensim/blob/280375fe14adea67ce6384ba7eabf362b05e6029/docs/notebooks/topic_coherence_tutorial.ipynb)
32. [**Medium on lda with sklearn**](https://medium.com/mlreview/topic-modeling-with-scikit-learn-e80d33668730)
33. **Selecting the number of topics in LDA,** [**blog 1**](https://cran.r-project.org/web/packages/ldatuning/vignettes/topics.html)**,** [**blog2**](http://www.rpubs.com/MNidhi/NumberoftopicsLDA)**,** [**using preplexity**](https://stackoverflow.com/questions/21355156/topic-models-cross-validation-with-loglikelihood-or-perplexity)**,** [**prep and aic bic**](https://stats.stackexchange.com/questions/322809/inferring-the-number-of-topics-for-gensims-lda-perplexity-cm-aic-and-bic)**,** [**coherence**](https://stackoverflow.com/questions/17421887/how-to-determine-the-number-of-topics-for-lda)**,** [**coherence2**](https://www.machinelearningplus.com/nlp/topic-modeling-gensim-python/#17howtofindtheoptimalnumberoftopicsforlda)**,** [**coherence 3 with tutorial**](https://datascienceplus.com/evaluation-of-topic-modeling-topic-coherence/)**, un**[**clear**](https://community.rapidminer.com/discussion/51283/what-is-the-best-number-of-topics-on-lda)**,** [**unclear with analysis of stopword % inclusion**](https://markhneedham.com/blog/2015/03/24/topic-modelling-working-out-the-optimal-number-of-topics/)**,** [**unread**](https://www.quora.com/What-are-the-best-ways-of-selecting-number-of-topics-in-LDA)**,** [**paper: heuristic approach**](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4597325/)**,** [**elbow method**](https://www.knime.com/blog/topic-extraction-optimizing-the-number-of-topics-with-the-elbow-method)**,** [**using cv**](http://freerangestats.info/blog/2017/01/05/topic-model-cv)**,** [**Paper: new stability metric**](https://github.com/derekgreene/topic-stability) **+ gh code,**
34. [**Selecting the top K words in LDA**](https://stats.stackexchange.com/questions/199263/choosing-words-in-a-topic-which-cut-off-for-lda-topics)
35. [**Presentation: best practices for LDA**](http://www.phusewiki.org/wiki/images/c/c9/Weizhong_Presentation_CDER_Nov_9th.pdf)
36. [**Medium on guidedLDA**](https://medium.freecodecamp.org/how-we-changed-unsupervised-lda-to-semi-supervised-guidedlda-e36a95f3a164) **- switching from LDA to a variation of it that is guided by the researcher / data**
37. **Medium on lda -** [**another introductory**](https://towardsdatascience.com/thats-mental-using-lda-topic-modeling-to-investigate-the-discourse-on-mental-health-over-time-11da252259c3)**,** [**la times**](https://medium.com/swiftworld/topic-modeling-of-new-york-times-articles-11688837d32f)
38. [**Topic modelling through time**](https://tedunderwood.com/category/methodology/topic-modeling/)
39. [**Mallet vs nltk**](https://stackoverflow.com/questions/7476180/topic-modelling-in-mallet-vs-nltk)**,** [**params**](https://github.com/RaRe-Technologies/gensim/issues/193)**,** [**params**](https://groups.google.com/forum/#!topic/gensim/tOoc1Q0Ump0)
40. [**Paper: improving feature models**](http://aclweb.org/anthology/Q15-1022)
41. [**Lda vs w2v (doesn't make sense to compare**](https://stats.stackexchange.com/questions/145485/lda-vs-word2vec/145488)**,** [**again here**](https://stats.stackexchange.com/questions/145485/lda-vs-word2vec)
42. [**Adding lda features to w2v for classification**](https://stackoverflow.com/questions/48140319/add-lda-topic-modelling-features-to-word2vec-sentiment-classification)
43. [**Spacy and gensim on 20 news groups**](https://www.shanelynn.ie/word-embeddings-in-python-with-spacy-and-gensim/)
44. **The best topic modelling explanation including** [**Usages**](https://nlpforhackers.io/topic-modeling/)**, insights, a great read, with code - shows how to find similar docs by topic in gensim, and shows how to transform unseen documents and do similarity using sklearn:**
1. **Text classification – Topic modeling can improve classification by grouping similar words together in topics rather than using each word as a feature**
2. **Recommender Systems – Using a similarity measure we can build recommender systems. If our system would recommend articles for readers, it will recommend articles with a topic structure similar to the articles the user has already read.**
3. **Uncovering Themes in Texts – Useful for detecting trends in online publications for example**
4. **A Form of Tagging - If document classification is assigning a single category to a text, topic modeling is assigning multiple tags to a text. A human expert can label the resulting topics with human-readable labels and use different heuristics to convert the weighted topics to a set of tags.**
5. [**Topic Modelling for Feature Selection**](https://www.analyticsvidhya.com/blog/2016/08/beginners-guide-to-topic-modeling-in-python/) **- Sometimes LDA can also be used as feature selection technique. Take an example of text classification problem where the training data contain category wise documents. If LDA is running on sets of category wise documents. Followed by removing common topic terms across the results of different categories will give the best features for a category.**
45. [**Another great article about LDA**](https://www.analyticsvidhya.com/blog/2016/08/beginners-guide-to-topic-modeling-in-python/)**, including algorithm, parameters!! And Parameters of LDA**
1. **Alpha and Beta Hyperparameters – alpha represents document-topic density and Beta represents topic-word density. Higher the value of alpha, documents are composed of more topics and lower the value of alpha, documents contain fewer topics. On the other hand, higher the beta, topics are composed of a large number of words in the corpus, and with the lower value of beta, they are composed of few words.**
2. **Number of Topics – Number of topics to be extracted from the corpus. Researchers have developed approaches to obtain an optimal number of topics by using Kullback Leibler Divergence Score. I will not discuss this in detail, as it is too mathematical. For understanding, one can refer to this\[1] original paper on the use of KL divergence.**
3. **Number of Topic Terms – Number of terms composed in a single topic. It is generally decided according to the requirement. If the problem statement talks about extracting themes or concepts, it is recommended to choose a higher number, if problem statement talks about extracting features or terms, a low number is recommended.**
4. **Number of Iterations / passes – Maximum number of iterations allowed to LDA algorithm for convergence.**
46. **Ways to improve LDA:**
1. **Reduce dimentionality of document-term matrix**
2. **Frequency filter**
3. **POS filter**
4. **Batch wise LDA**
47. [**History of LDA**](http://qpleple.com/bib/#Newman10a) **- by the frech guy**
48. [**Multilingual - alpha is divided by topic count, reaffirms 7**](http://mallet.cs.umass.edu/topics-polylingual.php)
49. [**Topic modelling with lda and nmf on medium**](https://medium.com/ml2vec/topic-modeling-is-an-unsupervised-learning-approach-to-clustering-documents-to-discover-topics-fdfbf30e27df) **- has a very good simple example with probabilities**
50. **Code:** [**great for top docs, terms, topics etc.**](http://nbviewer.jupyter.org/github/bmabey/hacker_news_topic_modelling/blob/master/HN%20Topic%20Model%20Talk.ipynb#topic=55\&lambda=1\&term=)
51. **Great article:** [**Many ways of evaluating topics by running LDA**](https://www.quora.com/What-are-good-ways-of-evaluating-the-topics-generated-by-running-LDA-on-a-corpus)
52. [**Difference between lda in gensim and sklearn a post on rare**](https://github.com/RaRe-Technologies/gensim/issues/457)
53. [**The best code article on LDA/MALLET**](https://www.machinelearningplus.com/nlp/topic-modeling-gensim-python/)**, and using** [**sklearn**](https://www.machinelearningplus.com/nlp/topic-modeling-python-sklearn-examples/) **(using clustering for getting group of sentences in each topic)**
54. [**LDA in gensim, a tutorial by gensim**](https://nbviewer.jupyter.org/github/rare-technologies/gensim/blob/develop/docs/notebooks/atmodel_tutorial.ipynb)
55. [**Lda on medium**](https://towardsdatascience.com/topic-modelling-in-python-with-nltk-and-gensim-4ef03213cd21)
56. [**What are the pros and cons of LDA and NMF in topic modeling? Under what situations should we choose LDA or NMF? Is there comparison of two techniques in topic modeling?**](https://www.quora.com/What-are-the-pros-and-cons-of-LDA-and-NMF-in-topic-modeling-Under-what-situations-should-we-choose-LDA-or-NMF-Is-there-comparison-of-two-techniques-in-topic-modeling)
57. [**What is the difference between NMF and LDA? Why are the priors of LDA sparse-induced?**](https://www.quora.com/What-is-the-difference-between-NMF-and-LDA-Why-are-the-priors-of-LDA-sparse-induced)
58. [**Exploring Topic Coherence over many models and many topics**](http://aclweb.org/anthology/D/D12/D12-1087.pdf) **lda nmf svd, using umass and uci coherence measures**
59. **\*\*\*** [**Practical topic findings for short sentence text**](http://nbviewer.jupyter.org/github/dolaameng/tutorials/blob/master/topic-finding-for-short-texts/topics_for_short_texts.ipynb) **code**
60. [**What's the difference between SVD/NMF and LDA as topic model algorithms essentially? Deterministic vs prob based**](https://www.quora.com/Whats-the-difference-between-SVD-NMF-and-LDA-as-topic-model-algorithms-essentially)
61. [**What is the difference between NMF and LDA? Why are the priors of LDA sparse-induced?**](https://www.quora.com/What-is-the-difference-between-NMF-and-LDA-Why-are-the-priors-of-LDA-sparse-induced)
62. [**What are the relationships among NMF, tensor factorization, deep learning, topic modeling, etc.?**](https://www.quora.com/What-are-the-relationships-among-NMF-tensor-factorization-deep-learning-topic-modeling-etc)
63. [**Code: lda nmf**](https://www.kaggle.com/rchawla8/topic-modeling-with-lda-and-nmf-algorithms)
64. [**Unread a comparison of lda and nmf**](https://wiki.ubc.ca/Course:CPSC522/A_Comparison_of_LDA_and_NMF_for_Topic_Modeling_on_Literary_Themes)
65. [**Presentation: lda sparse coding matrix factorization**](https://www.cs.cmu.edu/~epxing/Class/10708-15/slides/LDA_SC.pdf)
66. [**An experimental comparison between NMF and LDA for active cross-situational object-word learning**](https://ieeexplore.ieee.org/abstract/document/7846822)
67. [**Topic coherence in gensom with jupyter code**](https://markroxor.github.io/gensim/static/notebooks/topic_coherence_tutorial.html)
68. [**Topic modelling dynamic presentation**](http://chdoig.github.io/pygotham-topic-modeling/#/)
69. **Paper:** [**Topic modelling and event identification from twitter data**](https://arxiv.org/ftp/arxiv/papers/1608/1608.02519.pdf)**, says LDA vs NMI (NMF?) and using coherence to analyze**
70. [**Just another medium article about ™**](https://medium.com/square-corner-blog/topic-modeling-optimizing-for-human-interpretability-48a81f6ce0ed)
71. [**What is Wrong with Topic Modeling? (and How to Fix it Using Search-based SE)**](https://www.researchgate.net/publication/307303102_What_is_Wrong_with_Topic_Modeling_and_How_to_Fix_it_Using_Search-based_SE) **LDADE's tunings dramatically reduces topic instability.**
72. [**Talk about topic modelling**](https://tedunderwood.com/category/methodology/topic-modeling/)
73. [**Intro to topic modelling**](http://blog.echen.me/2011/08/22/introduction-to-latent-dirichlet-allocation/)
74. [**Detecting topics in twitter**](https://github.com/heerme/twitter-topics) **github code**
75. [**Another topic model tutorial**](https://github.com/derekgreene/topic-model-tutorial/blob/master/2%20-%20NMF%20Topic%20Models.ipynb)
76. **(didnt read) NTM -** [**neural topic modeling using embedded spaces**](https://github.com/elbamos/NeuralTopicModels) **with github code**
77. [**Another lda tutorial**](https://blog.intenthq.com/blog/automatic-topic-modelling-with-latent-dirichlet-allocation)
78. [**Comparing tweets using lda**](https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=2374\&context=sis_research)
79. [**Lda and w2v as features for some classification task**](https://www.kaggle.com/vukglisovic/classification-combining-lda-and-word2vec)
80. [**Improving ™ with embeddings**](https://github.com/datquocnguyen/LFTM)
81. [**w2v/doc2v for topic clustering - need to see the code to understand how they got clean topics, i assume a human rewrote it**](https://towardsdatascience.com/automatic-topic-clustering-using-doc2vec-e1cea88449c)
## **Mallet LDA**
1. [**Diff between lda and mallet**](https://groups.google.com/forum/#!topic/gensim/_VO4otCV6cU) **- The inference algorithms in Mallet and Gensim are indeed different. Mallet uses Gibbs Sampling which is more precise than Gensim's faster and online Variational Bayes. There is a way to get relatively performance by increasing number of passes.**
2. [**Mallet in gensim blog post**](https://rare-technologies.com/tutorial-on-mallet-in-python/)
3. **Alpha beta in mallet:** [**contribution**](https://datascience.stackexchange.com/questions/199/what-does-the-alpha-and-beta-hyperparameters-contribute-to-in-latent-dirichlet-a)
1. [**The default for alpha is 5.**](https://stackoverflow.com/questions/44561609/how-does-mallet-set-its-default-hyperparameters-for-lda-i-e-alpha-and-beta)**0 divided by the number of topics. You can think of this as five "pseudo-words" of weight on the uniform distribution over topics. If the document is short, we expect to stay closer to the uniform prior. If the document is long, we would feel more confident moving away from the prior.**
2. **With hyperparameter optimization, the alpha value for each topic can be different. They usually become smaller than the default setting.**
3. **The default value for beta is 0.01. This means that each topic has a weight on the uniform prior equal to the size of the vocabulary divided by 100. This seems to be a good value. With optimization turned on, the value rarely changes by more than a factor of two.**
## **Visualization**
1. **How to interpret topics using pyldaviz: Let’s interpret the topic visualization. Notice how topics are shown on the left while words are on the right. Here are the main things you should consider:**
1. **Larger topics are more frequent in the corpus.**
2. **Topics closer together are more similar, topics further apart are less similar.**
3. **When you select a topic, you can see the most representative words for the selected topic. This measure can be a combination of how frequent or how discriminant the word is. You can adjust the weight of each property using the slider.**
4. **Hovering over a word will adjust the topic sizes according to how representative the word is for the topic.**
5. **\*\*\*\***[**pyLDAviz paper\*\*\*!**](https://cran.r-project.org/web/packages/LDAvis/vignettes/details.pdf)
6. [**pyLDAviz - what am i looking at ?**](https://github.com/explosion/spacy-notebooks/blob/master/notebooks/conference_notebooks/modern_nlp_in_python.ipynb) **by spacy.** \
**There are a lot of moving parts in the visualization. Here's a brief summary:**
1. **On the left, there is a plot of the "distance" between all of the topics (labeled as the Intertopic Distance Map)**
2. **The plot is rendered in two dimensions according a** [**multidimensional scaling (MDS)**](https://en.wikipedia.org/wiki/Multidimensional_scaling) **algorithm. Topics that are generally similar should be appear close together on the plot, while dissimilar topics should appear far apart.**
3. **The relative size of a topic's circle in the plot corresponds to the relative frequency of the topic in the corpus.**
4. **An individual topic may be selected for closer scrutiny by clicking on its circle, or entering its number in the "selected topic" box in the upper-left.**
5. **On the right, there is a bar chart showing top terms.**
6. **When no topic is selected in the plot on the left, the bar chart shows the top-30 most "salient" terms in the corpus. A term's saliency is a measure of both how frequent the term is in the corpus and how "distinctive" it is in distinguishing between different topics.**
7. **When a particular topic is selected, the bar chart changes to show the top-30 most "relevant" terms for the selected topic. The relevance metric is controlled by the parameter λλ, which can be adjusted with a slider above the bar chart.**
1. **Setting the λλ parameter close to 1.0 (the default) will rank the terms solely according to their probability within the topic.**
2. **Setting λλ close to 0.0 will rank the terms solely according to their "distinctiveness" or "exclusivity" within the topic — i.e., terms that occur only in this topic, and do not occur in other topics.**
3. **Setting λλ to values between 0.0 and 1.0 will result in an intermediate ranking, weighting term probability and exclusivity accordingly.**
4. **Rolling the mouse over a term in the bar chart on the right will cause the topic circles to resize in the plot on the left, to show the strength of the relationship between the topics and the selected term.**
7. **A more detailed explanation of the pyLDAvis visualization can be found** [**here**](https://cran.r-project.org/web/packages/LDAvis/vignettes/details.pdf)**. Unfortunately, though the data used by gensim and pyLDAvis are the same, they don't use the same ID numbers for topics. If you need to match up topics in gensim's LdaMulticore object and pyLDAvis' visualization, you have to dig through the terms manually.**
8. [**Youtube on LDAvis explained**](http://stat-graphics.org/movies/ldavis.html)
9. **Presentation:** [**More visualization options including ldavis**](https://speakerdeck.com/bmabey/visualizing-topic-models?slide=17)
10. [**A pointer to the ldaviz fix**](https://github.com/RaRe-Technologies/gensim/issues/2069) **->** [**fix**](http://jeriwieringa.com/2018/07/17/pyLDAviz-and-Mallet/#comment-4018495276)**,** [**git code**](https://github.com/jerielizabeth/Gospel-of-Health-Notebooks/blob/master/blogPosts/pyLDAvis_and_Mallet.ipynb)
## **COHERENCE (Topic)**
1. [**What is?**](https://www.quora.com/What-is-topic-coherence)**,** [**Wiki on pmi**](https://en.wikipedia.org/wiki/Pointwise_mutual_information#cite_note-Church1990-1)
2. [**Datacamp on coherence metrics, a comparison, read me.**](https://datascienceplus.com/evaluation-of-topic-modeling-topic-coherence/)
3. **Paper:** [**explains what is coherence**](http://aclweb.org/anthology/J90-1003)
1. [**Umass vs C\_v, what are the diff?** ](https://groups.google.com/forum/#!topic/gensim/CsscFah0Ax8)
2. **Paper: umass, uci, nmpi, cv, cp etv** [**Exploring the Space of Topic Coherence Measures**](http://svn.aksw.org/papers/2015/WSDM_Topic_Evaluation/public.pdf)
3. **Paper:** [**Automatic evaluation of topic coherence**](https://mimno.infosci.cornell.edu/info6150/readings/N10-1012.pdf)
4. **Paper:** [**exploring the space of topic coherence methods**](https://dl.acm.org/citation.cfm?id=2685324)
5. **Paper:** [**Relation between mutial information / entropy and pmi**](https://svn.spraakdata.gu.se/repos/gerlof/pub/www/Docs/npmi-pfd.pdf)
6. **Stackexchange:** [**coherence / pmi how to calc**](https://stats.stackexchange.com/questions/158790/topic-similarity-semantic-pmi-between-two-words-wikipedia)
7. **Paper:** [**Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality**](http://www.aclweb.org/anthology/E14-1056) **- perplexity needs unseen data, coherence doesnt**
8. [**Evaluation of topic modelling techniques for twitter**](https://www.cs.toronto.edu/~jstolee/projects/topic.pdf) **lda lda-u btm w2vgmm**
9. **Paper:** [**Topic coherence measures**](https://svn.aksw.org/papers/2015/WSDM_Topic_Evaluation/public.pdf)
10. [**topic modelling from different domains**](http://proceedings.mlr.press/v32/chenf14.pdf)
11. **Paper:** [**Optimizing Semantic Coherence in Topic Models**](https://mimno.infosci.cornell.edu/papers/mimno-semantic-emnlp.pdf)
12. **Paper:** [**L-EnsNMF: Boosted Local Topic Discovery via Ensemble of Nonnegative Matrix Factorization** ](http://www.joonseok.net/papers/lensnmf.pdf)
13. **Paper:** [**Content matching between TV shows and advertisements through Latent Dirichlet Allocation** ](http://arno.uvt.nl/show.cgi?fid=145381)
14. **Paper:** [**Full-Text or Abstract? Examining Topic Coherence Scores Using Latent Dirichlet Allocation**](http://www.saf21.eu/wp-content/uploads/2017/09/5004a165.pdf)
15. **Paper:** [**Evaluating topic coherence**](https://pdfs.semanticscholar.org/03a0/62fdcd13c9287a2d4e1d6d057fd2e083281c.pdf) **- Abstract: Topic models extract representative word sets—called topics—from word counts in documents without requiring any semantic annotations. Topics are not guaranteed to be well interpretable, therefore, coherence measures have been proposed to distinguish between good and bad topics. Studies of topic coherence so far are limited to measures that score pairs of individual words. For the first time, we include coherence measures from scientific philosophy that score pairs of more complex word subsets and apply them to topic scoring.**
**Conclusion: The results of the first experiment show that if we are using the one-any, any-any and one-all coherences directly for optimization they are leading to meaningful word sets. The second experiment shows that these coherence measures are able to outperform the UCI coherence as well as the UMass coherence on these generated word sets. For evaluating LDA topics any-any and one-any coherences perform slightly better than the UCI coherence. The correlation of the UMass coherence and the human ratings is not as high as for the other coherences.**
1. **Code:** [**Evaluating topic coherence, using gensim umass or cv parameter**](https://datascienceplus.com/evaluation-of-topic-modeling-topic-coherence/) **- To conclude, there are many other approaches to evaluate Topic models such as Perplexity, but its poor indicator of the quality of the topics.Topic Visualization is also a good way to assess topic models. Topic Coherence measure is a good way to compare difference topic models based on their human-interpretability.The u\_mass and c\_v topic coherences capture the optimal number of topics by giving the interpretability of these topics a number called coherence score.**
2. **Formulas:** [**UCI vs UMASS** ](http://qpleple.com/topic-coherence-to-evaluate-topic-models/)
3. [**Inferring the number of topics for gensim's LDA - perplexity, CM, AIC, and BIC**](https://stats.stackexchange.com/questions/322809/inferring-the-number-of-topics-for-gensims-lda-perplexity-cm-aic-and-bic)
4. [**Perplexity as a measure for LDA**](https://groups.google.com/forum/#!topic/gensim/tgJLVulf5xQ)
5. [**Finding number of topics using perplexity**](https://groups.google.com/forum/#!topic/gensim/TpuYRxhyIOc)
6. [**Coherence for tweets**](http://terrierteam.dcs.gla.ac.uk/publications/fang_sigir_2016_examine.pdf)
7. **Presentation** [**Twitter DLA**](https://www.slideshare.net/akshayubhat/twitter-lda)**,** [**tweet pooling improvements**](http://users.cecs.anu.edu.au/~ssanner/Papers/sigir13.pdf)**,** [**hierarchical summarization of tweets**](https://www.researchgate.net/publication/322359369_Hierarchical_Summarization_of_News_Tweets_with_Twitter-LDA)**,** [**twitter LDA in java**](https://sites.google.com/site/lyangwww/code-data) [**on github**](https://github.com/minghui/Twitter-LDA)\
**Papers:** [**TM of twitter timeline**](https://medium.com/@alexisperrier/topic-modeling-of-twitter-timelines-in-python-bb91fa90d98d)**,** [**in twitter aggregation by conversatoin**](https://www.aaai.org/ocs/index.php/ICWSM/ICWSM16/paper/download/13162/12778)**,** [**twitter topics using LDA**](http://uu.diva-portal.org/smash/get/diva2:904196/FULLTEXT01.pdf)**,** [**empirical study**](https://snap.stanford.edu/soma2010/papers/soma2010_12.pdf) **,**
8. [**Using regularization to improve PMI score and in turn coherence for LDA topics**](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.230.7738\&rep=rep1\&type=pdf)
9. [**Improving model precision - coherence using turkers for LDA**](https://pdfs.semanticscholar.org/1d29/f7a9e3135bba0339b9d70ecbda9d106b01d2.pdf)
10. [**Gensim**](https://radimrehurek.com/gensim/models/coherencemodel.html) **-** [ **paper about their algorithm and PMI/UCI etc.**](http://svn.aksw.org/papers/2015/WSDM_Topic_Evaluation/public.pdf)
11. [**Advice for coherence,**](https://gist.github.com/dsquareindia/ac9d3bf57579d02302f9655db8dfdd55) **then** [**Good vs bad model (50 vs 1 iterations) measuring u\_mass coherence**](https://markroxor.github.io/gensim/static/notebooks/topic_coherence_tutorial.html) **-** [**2nd code**](https://gist.github.com/dsquareindia/ac9d3bf57579d02302f9655db8dfdd55) **- “In your data we can see that there is a peak between 0-100 and a peak between 400-500. What I would think in this case is that "does \~480 topics make sense for the kind of data I have?" If not, you can just do an np.argmax for 0-100 topics and trade-off coherence score for simpler understanding. Otherwise just do an np.argmax on the full set.”**
12. [**Diff term weighting schemas for topic modeling, code plus paper**](https://github.com/cipriantruica/TM_TESTS)
13. [**Workaround for pyLDAvis using LDA-Mallet**](http://jeriwieringa.com/2018/07/17/pyLDAviz-and-Mallet/#comment-4018495276)
14. [**pyLDAvis paper**](http://www.aclweb.org/anthology/W14-3110)
15. [**Visualizing LDA topics results** ](https://de.dariah.eu/tatom/topic_model_visualization.html)
16. [**Visualizing trends, topics, sentiment, heat maps, entities**](https://github.com/Lissy93/twitter-sentiment-visualisation) **- really good**
17. **Topic stability Metric, a novel method, compared against jaccard, spearman, silhouette.:** [**Measuring LDA Topic Stability from Clusters of Replicated Runs**](https://arxiv.org/pdf/1808.08098.pdf)
## **LDA2VEC**
1. **“if you want to rework your own topic models that, say, jointly correlate an article’s topics with votes or predict topics over users then you might be interested in** [**lda2vec**](https://github.com/cemoody/lda2vec)**.”**
2. [**Datacamp intro**](https://www.datacamp.com/community/tutorials/lda2vec-topic-model)
3. [**Original blog**](https://multithreaded.stitchfix.com/blog/2016/05/27/lda2vec/#topic=38\&lambda=1\&term=) **- I just learned about these papers which are quite similar:** [**Gaussian LDA for Topic Word Embeddings**](http://www.aclweb.org/anthology/P15-1077) **and** [**Nonparametric Spherical Topic Modeling with Word Embeddings**](http://arxiv.org/abs/1604.00126)**.**
4. [**Moody’s Slide Share**](https://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec-57135994) **(excellent read)**
5. [**Docs**](http://lda2vec.readthedocs.io/en/latest/?badge=latest)
6. [**Original Git**](https://github.com/cemoody/lda2vec) **+** [**Excellent notebook example**](http://nbviewer.jupyter.org/github/cemoody/lda2vec/blob/master/examples/twenty_newsgroups/lda2vec/lda2vec.ipynb#topic=0\&lambda=1\&term=)
7. [**Tf implementation**](https://github.com/meereeum/lda2vec-tf)**,** [**another more recent one tf 1.5**](https://github.com/nateraw/Lda2vec-Tensorflow)
8. [**Another blog explaining about lda etc**](https://datawarrior.wordpress.com/tag/lda2vec/)**,** [**post**](https://datawarrior.wordpress.com/2016/02/15/lda2vec-a-hybrid-of-lda-and-word2vec/)**,** [**post**](https://datawarrior.wordpress.com/2016/04/20/local-and-global-words-and-topics/)
9. [**Lda2vec in tf**](https://github.com/meereeum/lda2vec-tf)**,** [**tf 1.5**](https://github.com/nateraw/Lda2vec-Tensorflow)**,**
10. [**Comparing lda2vec to lda**](https://medium.com/scaleabout/a-gentle-introduction-to-doc2vec-db3e8c0cce5e)
11. **Youtube:** [**lda/doc2vec with pca examples**](https://www.youtube.com/watch?v=i3Opb3-QNX4)
12. [**Example on gh**](https://github.com/BoPengGit/LDA-Doc2Vec-example-with-PCA-LDAvis-visualization/blob/master/Doc2Vec/Doc2Vec2.py) **on jupyter**
## **TOP2VEC**
1. [Git](https://github.com/ddangelov/Top2Vec), [paper](https://arxiv.org/pdf/2008.09470.pdf)
2. Topic modeling with distillibert [on medium](https://towardsdatascience.com/topic-modeling-with-bert-779f7db187e6), [bertTopic](https://towardsdatascience.com/interactive-topic-modeling-with-bertopic-1ea55e7d73d8)!, c-tfidf, umap, hdbscan, merging similar topics, visualization, [berTopic (same method as the above)](https://github.com/MaartenGr/BERTopic)
3. [Medium with the same general method](https://towardsdatascience.com/topic-modeling-with-bert-779f7db187e6)
4. [new way of modeling topics](https://towardsdatascience.com/top2vec-new-way-of-topic-modelling-bea165eeac4a)
# Named Entity Recognition (NER)
1. [**State of the art LSTM architectures using NN**](https://blog.paralleldots.com/data-science/named-entity-recognition-milestone-models-papers-and-technologies/)
2. **Medium:** [**Ner free datasets**](https://towardsdatascience.com/deep-learning-for-ner-1-public-datasets-and-annotation-methods-8b1ad5e98caf) **and** [**bilstm implementation**](https://towardsdatascience.com/deep-learning-for-named-entity-recognition-2-implementing-the-state-of-the-art-bidirectional-lstm-4603491087f1) **using glove embeddings**
3. **Easy to implement in keras! They are based on the following** [**paper**](https://arxiv.org/abs/1511.08308)
4. [**Medium**](https://medium.com/district-data-labs/named-entity-recognition-and-classification-for-entity-extraction-6f23342aa7c5)**: NLTK entities, polyglot entities, sner entities, finally an ensemble method wins all!**
* [**Comparison between spacy and SNER**](https://medium.com/@dudsdu/named-entity-recognition-for-unstructured-documents-c325d47c7e3a) **- for terms.**
* **\*\*\*** [**Unsupervised NER using Bert**](https://towardsdatascience.com/unsupervised-ner-using-bert-2d7af5f90b8a)
* [**Custom NER using spacy**](https://towardsdatascience.com/custom-named-entity-recognition-using-spacy-7140ebbb3718)
* [**Spacy Ner with custom data**](https://medium.com/@manivannan_data/how-to-train-ner-with-custom-training-data-using-spacy-188e0e508c6)
* [**How to create a NER from scratch using kaggle data, using crf, and analysing crf weights using external package**](https://towardsdatascience.com/named-entity-recognition-and-classification-with-scikit-learn-f05372f07ba2)
* [**Another comparison between spacy and SNER - both are the same, for many classes.**](https://towardsdatascience.com/a-review-of-named-entity-recognition-ner-using-automatic-summarization-of-resumes-5248a75de175)
* [**Vidhaya on spacy vs ner**](https://www.analyticsvidhya.com/blog/2017/04/natural-language-processing-made-easy-using-spacy-%E2%80%8Bin-python/) **- tutorial + code on how to use spacy for pos, dep, ner, compared to nltk/corenlp (sner etc). The results reflect a global score not specific to LOC for example.**
**Stanford NER (SNER)**
* [**SNER presentation - combines HMM and MaxEnt features, distributional features, NER has** ](https://nlp.stanford.edu/software/jenny-ner-2007.pdf)
* [**many applications.**](https://nlp.stanford.edu/software/jenny-ner-2007.pdf)
* [**How to train SNER, a FAQ with many other answers (read first before doing anything with SNER)**](https://nlp.stanford.edu/software/crf-faq.shtml#a)
* [**SNER demo - capital letters matter, a minimum of one.**](http://nlp.stanford.edu:8080/ner/process)
* [**State of the art NER benchmark**](https://github.com/magizbox/underthesea/wiki/TASK-CONLL-2003)
* [**Review paper, SNER, spacy, stanford wins**](http://www.aclweb.org/anthology/W16-2703)
* [**Review paper SNER, others on biographical text, stanford wins**](https://arxiv.org/ftp/arxiv/papers/1308/1308.0661.pdf)
* [**Another NER DL paper, 90%+**](https://openreview.net/forum?id=ry018WZAZ)
**Spacy & Others**
* [**Spacy - using prodigy and spacy to train a NER classifier using active learning**](https://www.youtube.com/watch?v=l4scwf8KeIA)
* [**Ner using DL BLSTM, using glove embeddings, using CRF layer against another CRF**](http://nlp.town/blog/ner-and-the-road-to-deep-learning/)**.**
* [**Another medium paper on the BLSTM CRF with guillarue’s code**](https://medium.com/intro-to-artificial-intelligence/entity-extraction-using-deep-learning-8014acac6bb8)
* [**Guillaume blog post, detailed explanation**](https://guillaumegenthial.github.io/sequence-tagging-with-tensorflow.html)
* [**For Italian**](https://www.qcri.org/app/media/4916)
* [**Another 90+ proposed solution**](https://arxiv.org/pdf/1603.01360.pdf)
* [**A promising python implementation based on one or two of the previous papers**](https://github.com/deepmipt/ner)
* [**Quora advise, the first is cool, the second is questionable**](https://www.quora.com/How-can-I-perform-named-entity-recognition-using-deep-learning-RNN-LSTM-Word2vec-etc)
* [**Off the shelf solutions benchmark**](https://www.programmableweb.com/news/performance-comparison-10-linguistic-apis-entity-recognition/elsewhere-web/2016/11/03)
* [**Parallel api talk about bilstm and their 2mil tagged ner model (washington passes)**](https://blog.paralleldots.com/data-science/named-entity-recognition-milestone-models-papers-and-technologies/)
# SEARCH
1. **Bert** [**search engine**](https://towardsdatascience.com/covid-19-bert-literature-search-engine-4d06cdac08bd)**, cosine between paragraphs and question.**
2. **Semantic search, autom completion, filtering, augmentation, scoring. Problems: Token matching, contextualization, query misunderstanding, image search, metric. Solutions: synonym generation, query autocompletion, alternate query generation, word and doc embedding, contextualization, ranking, ensemble, multilingual search**
# Neural NLP
## **CONVOLUTION NEURAL NETS (CNN)**
1. [**Cnn for text**](https://medium.com/@TalPerry/convolutional-methods-for-text-d5260fd5675f) **- tal perry**
2. [**1D CNN using KERAS**](https://blog.goodaudience.com/introduction-to-1d-convolutional-neural-networks-in-keras-for-time-sequences-3a7ff801a2cf)
## **SEQ2SEQ SEQUENCE TO SEQUENCE**
1. [**Keras blog**](https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html) **- char-level, token-using embedding layer, teacher forcing**
2. [**Teacher forcing explained**](https://towardsdatascience.com/what-is-teacher-forcing-3da6217fed1c)
3. [**Same as keras but with token-level**](https://towardsdatascience.com/machine-translation-with-the-seq2seq-model-different-approaches-f078081aaa37)
4. [**Medium on char, word, byte-level**](https://medium.com/@petepeeradejtanruangporn/experimenting-with-neural-machine-translation-for-thai-1681fd2b375a)
5. [**Mastery on enc-dec using the keras method**](https://machinelearningmastery.com/develop-encoder-decoder-model-sequence-sequence-prediction-keras/)**, and on** [**neural translation**](https://machinelearningmastery.com/define-encoder-decoder-sequence-sequence-model-neural-machine-translation-keras/)
6. [**Machine translation git from eng to jap**](https://github.com/samurainote/seq2seq_translate_slackbot/blob/master/seq2seq_translate.py)**,** [**another**](https://github.com/samurainote/seq2seq_translate_slackbot)**, and its** [**medium**](https://towardsdatascience.com/how-to-implement-seq2seq-lstm-model-in-keras-shortcutnlp-6f355f3e5639)
1. [**Incorporating Copying Mechanism in Sequence-to-Sequence Learning**](https://arxiv.org/abs/1603.06393) **- In this paper, we incorporate copying into neural network-based Seq2Seq learning and propose a new model called CopyNet with encoder-decoder structure. CopyNet can nicely integrate the regular way of word generation in the decoder with the new copying mechanism which can choose sub-sequences in the input sequence and put them at proper places in the output sequence.**
# Tokenization
1. [Tokenization In OpenAI API : Let’s Explore Tiktoken Library](https://medium.com/@basics.machinelearning/tokenization-in-openai-api-lets-explore-tiktoken-library-d02d3ce94b0a), [Github](https://github.com/openai/tiktoken) - Pair encoding (BPE) for tokenization, BPE is a data compression algorithm.
2. (really good) [Hugging Face on tokenization](https://huggingface.co/docs/transformers/tokenizer_summary)
3. [An Explanatory Guide to BERT Tokenizer](https://www.analyticsvidhya.com/blog/2021/09/an-explanatory-guide-to-bert-tokenizer/)
# Decoding Algorithms For NLP
1. (great) [greedy, beam search, pure sampling, top k sampling](https://medium.com/voice-tech-podcast/visualising-beam-search-and-other-decoding-algorithms-for-natural-language-generation-fbba7cba2c5b) by Katnoria
2. (great) [How to generate text by HuggingFace](https://huggingface.co/blog/how-to-generate)
3. what is [beam](https://angelina-yang.medium.com/what-is-beam-search-decoding-in-a-neural-machine-translation-model-adaab30c6579) search by angelina yang
4. [understanding greedy and beam](https://medium.com/@jessica_lopez/understanding-greedy-search-and-beam-search-98c1e3cd821d) by Jessica lopez
5. [beam in seq2seq](https://angelina-yang.medium.com/what-is-beam-search-decoding-in-a-neural-machine-translation-model-adaab30c6579) by Dharti Dhami
# Multi Language
1. [**Fb’s laser**](https://engineering.fb.com/ai-research/laser-multilingual-sentence-embeddings/)
2. [**Xlm**](https://github.com/facebookresearch/XLM)**,** [**xlm-r**](https://ai.facebook.com/blog/-xlm-r-state-of-the-art-cross-lingual-understanding-through-self-supervision/)
3. **Google universal embedding space.**
# Augmentation
1. [**Synonyms**](https://towardsdatascience.com/data-augmentation-in-nlp-2801a34dfc28)**, similar embedded words (w2v), back translation, contextualized word embeddings, text generation**
2. **Yonatan hadar also has a medium post about this**
# Knowledge Graphs
1. [**Automatic creation of KG using spacy**](https://towardsdatascience.com/auto-generated-knowledge-graphs-92ca99a81121) **and networx**\
**Knowledge graphs can be constructed automatically from text using part-of-speech and dependency parsing. The extraction of entity pairs from grammatical patterns is fast and scalable to large amounts of text using NLP library SpaCy.**
2. [**Medium on Reconciling your data and the world of knowledge graphs**](https://towardsdatascience.com/reconciling-your-data-and-the-world-with-knowledge-graphs-bce66b377b14)
3. **Medium Series:**
1. [**Creating kg**](https://towardsdatascience.com/knowledge-graphs-at-a-glance-c9119130a9f0)
2. [**Building from structured sources**](https://towardsdatascience.com/building-knowledge-graphs-from-structured-sources-346c56c9d40e)
3. [**Semantic models**](https://towardsdatascience.com/semantic-models-for-constructing-knowledge-graphs-38c0a1df316a)
# Annotation & Disagreement
## Tools
1. [**Snorkel**](https://www.snorkel.org/use-cases/) **- using weak supervision to create less noisy labelled datasets**
1. [**Git**](https://github.com/snorkel-team/snorkel)
2. [**Medium**](https://towardsdatascience.com/introducing-snorkel-27e4b0e6ecff)
2. [**Snorkel metal**](https://jdunnmon.github.io/metal_deem.pdf) **weak supervision for multi-task learning.** [**Conversation**](https://spectrum.chat/snorkel/help/hierarchical-labelling-example~aa4d8617-d287-43a6-865e-7c9034888363)**,** [**git**](https://github.com/HazyResearch/metal/blob/master/tutorials/Multitask.ipynb)
1. **Yes, the Snorkel project has included work before on hierarchical labeling scenarios. The main papers detailing our results include the DEEM workshop paper you referenced (**[**https://dl.acm.org/doi/abs/10.1145/3209889.3209898**](https://dl.acm.org/doi/abs/10.1145/3209889.3209898)**) and the more complete paper presented at AAAI (**[**https://arxiv.org/abs/1810.02840**](https://arxiv.org/abs/1810.02840)**). Before the Snorkel and Snorkel MeTaL projects were merged in Snorkel v0.9, the Snorkel MeTaL project included an interface for explicitly specifying hierarchies between tasks which was utilized by the label model and could be used to automatically compile a multi-task end model as well (demo here:** [**https://github.com/HazyResearch/metal/blob/master/tutorials/Multitask.ipynb**](https://github.com/HazyResearch/metal/blob/master/tutorials/Multitask.ipynb)**). That interface is not currently available in Snorkel v0.9 (no fundamental blockers; just hasn't been ported over yet).**
2. **There are, however, still a number of ways to model such situations. One way is to treat each node in the hierarchy as a separate task and combine their probabilities post-hoc (e.g., P(credit-request) = P(billing) \* P(credit-request | billing)). Another is to treat them as separate tasks and use a multi-task end model to implicitly learn how the predictions of some tasks should affect the predictions of others (e.g., the end model we use in the AAAI paper). A third option is to create a single task with all the leaf categories and modify the output space of the LFs you were considering for the higher nodes (the deeper your hierarchy is or the larger the number of classes, the less apppealing this is w/r/t to approaches 1 and 2).**
3. [**mechanical turk calculator**](https://morninj.github.io/mechanical-turk-cost-calculator/)
4. [**Mturk alternatives**](https://moneypantry.com/amazon-mechanical-turk-crowdsourcing-alternatives/)
1. [**Workforce / onespace**](https://www.crowdsource.com/workforce/)
2. [**Jobby**](https://www.jobboy.com/)
3. [**Shorttask**](http://www.shorttask.com/)
4. [**Samasource**](https://www.samasource.org/team)
5. **Figure 8 -** [**pricing**](https://siftery.com/crowdflower/pricing) **-** [**definite guide**](https://www.earnonlineguys.com/figure-eight-tasks-guide/)
5. [**Brat nlp annotation tool**](http://brat.nlplab.org/?fbclid=IwAR1bDCM3j3nEQb3Hrf9dGCwyRvDVMBXoob4WtVLCWAMBgPraZmkSi123IrI)
6. [**Prodigy by spacy**](https://prodi.gy/)**,**
1. [**seed-small sample, many sample tutorial on youtube by ines**](https://www.youtube.com/watch?v=5di0KlKl0fE)
2. [**How to use prodigy, tutorial on medium plus notebook code inside**](https://medium.com/@david.campion/text-classification-be-lazy-use-prodigy-b0f9d00e9495)
7. [**Doccano**](https://github.com/chakki-works/doccano) **- prodigy open source alternative butwith users management & statistics out of the box**
8. **Medium** [**Lighttag - has some cool annotation metrics\tests**](https://medium.com/@TalPerry/announcing-lighttag-the-easy-way-to-annotate-text-afb7493a49b8)
9. [Loopr](https://loopr.ai/products/labeling-platform).ai - An AI powered semi-automated and automated annotation process for high quality data.object detection, analytics, nlp, active learning.
10. **Medium** [**Assessing annotator disagreement**](https://towardsdatascience.com/assessing-annotator-disagreements-in-python-to-build-a-robust-dataset-for-machine-learning-16c74b49f043)
11. [**A great python package for measuring disagreement on GH**](https://github.com/o-P-o/disagree)
12. [**Reliability is key, and not just mechanical turk**](https://www.youtube.com/watch?v=ktZLuXPXPEI)
13. [**7 myths about annotation**](https://www.aaai.org/ojs/index.php/aimagazine/article/viewFile/2564/2468)
14. [**Annotating twitter sentiment using humans, 3 classes, 55% accuracy using SVMs.**](http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0155036) **they talk about inter agreement etc. and their DS is** [**partially publicly available**](https://www.clarin.si/repository/xmlui/handle/11356/1054)**.**
15. [**Exploiting disagreement** ](https://s3.amazonaws.com/academia.edu.documents/8026932/10.1.1.2.8084.pdf?AWSAccessKeyId=AKIAIWOWYYGZ2Y53UL3A\&Expires=1534444363\&Signature=3dHHw3EmAjPXFxwutVbtsZWEIzw%3D\&response-content-disposition=inline%3B%20filename%3DExploiting_agreement_and_disagreement_of.pdf)
16. [**Vader annotation**](http://comp.social.gatech.edu/papers/icwsm14.vader.hutto.pdf)
1. **They must pass an english exam**
2. **They get control questions to establish their reliability**
3. **They get a few sentences over and over again to establish inter disagreement**
4. **Two or more people get a overlapping sentences to establish disagreement**
5. **5 judges for each sentence (makes 4 useless)**
6. **They dont know each other**
7. **Simple rules to follow**
8. **Random selection of sentences**
9. **Even classes**
10. **No experts**
11. **Measuring reliability kappa/the other kappa.**
17. [**Label studio** ](https://labelstud.io/)
**Ideas:**
1. **Active learning for a group (or single) of annotators, we have to wait for all annotations to finish each big batch in order to retrain the model.**
2. **Annotate a small group, automatic labelling using knn**
3. **Find a nearest neighbor for out optimal set of keywords per “category,**
4. **For a group of keywords, find their knn neighbors in w2v-space, alternatively find k clusters in w2v space that has those keywords. For a new word/mean sentence vector in the ‘category’ find the minimal distance to the new cluster (either one of approaches) and this is new annotation.**
## Myths
1. [**7 myths of annotation**](https://www.aaai.org/ojs/index.php/aimagazine/article/viewFile/2564/2468)
1. **Myth One: One Truth Most data collection efforts assume that there is one correct interpretation for every input example.**
2. **Myth Two: Disagreement Is Bad To increase the quality of annotation data, disagreement among the annotators should be avoided or reduced.**
3. **Myth Three: Detailed Guidelines Help When specific cases continuously cause disagreement, more instructions are added to limit interpretations.**
4. **Myth Four: One Is Enough Most annotated examples are evaluated by one person.**
5. **Myth Five: Experts Are Better Human annotators with domain knowledge provide better annotated data.**
6. **Myth Six: All Examples Are Created Equal The mathematics of using ground truth treats every example the same; either you match the correct result or not.**
7. **Myth Seven: Once Done, Forever Valid Once human annotated data is collected for a task, it is used over and over with no update. New annotated data is not aligned with previous data.**
## Crowd Sourcing
#### [**Crowd Sourcing** ](https://www.youtube.com/watch?v=ktZLuXPXPEI)
* **Conclusions:**
* **Experts are the same as a crowd**
* **Costs a lot less $$$.**
## Disagreement
## **Inter agreement**
**\*\*\*** [**The best tutorial on agreements, cohen, david, kappa, krip etc.**](https://dkpro.github.io/dkpro-statistics/inter-rater-agreement-tutorial.pdf)
1. **Cohens kappa (two people)**
**but you can use it to map a group by calculating agreement for each pair**
1. [**Why cohens kappa should be avoided as a performance measure in classification**](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0222916)
2. [**Why it should be used as a measure of classification**](https://thedatascientist.com/performance-measures-cohens-kappa-statistic/)
3. [**Kappa in plain english**](https://stats.stackexchange.com/questions/82162/cohens-kappa-in-plain-english)
4. [**Multilabel using kappa**](https://stackoverflow.com/questions/52272901/multi-label-annotator-agreement-with-cohen-kappa)
5. [**Kappa and the relation with accuracy**](https://gis.stackexchange.com/questions/110188/how-are-kappa-and-overall-accuracy-related-with-respect-to-thematic-raster-data) **(redundant, % above chance, should not be used due to other reasons researched here)**
**The Kappa statistic varies from 0 to 1, where.**
* **0 = agreement equivalent to chance.**
* **0.1 – 0.20 = slight agreement.**
* **0.21 – 0.40 = fair agreement.**
* **0.41 – 0.60 = moderate agreement.**
* **0.61 – 0.80 = substantial agreement.**
* **0.81 – 0.99 = near perfect agreement**
* **1 = perfect agreement.**
1. **Fleiss’ kappa, from 3 people and above.**
**Kappa ranges from 0 to 1, where:**
* **0 is no agreement (or agreement that you would expect to find by chance),**
* **1 is perfect agreement.**
* **Fleiss’s Kappa is an extension of Cohen’s kappa for three raters or more. In addition, the assumption with Cohen’s kappa is that your raters are deliberately chosen and fixed. With Fleiss’ kappa, the assumption is that your raters were chosen at random from a larger population.**
* [**Kendall’s Tau**](https://www.statisticshowto.datasciencecentral.com/kendalls-tau/) **is used when you have ranked data, like two people ordering 10 candidates from most preferred to least preferred.**
* **Krippendorff’s alpha is useful when you have multiple raters and multiple possible ratings.**
1. **Krippendorfs alpha**
* [**Ignores missing data entirely**](https://deepsense.ai/multilevel-classification-cohen-kappa-and-krippendorff-alpha/)**.**
* **Can handle various sample sizes, categories, and numbers of raters.**
* **Applies to any** [**measurement level**](https://www.statisticshowto.datasciencecentral.com/scales-of-measurement/) **(i.e. (**[**nominal, ordinal, interval, ratio**](https://www.statisticshowto.datasciencecentral.com/nominal-ordinal-interval-ratio/)**).**
* **Values range from 0 to 1, where 0 is perfect disagreement and 1 is perfect agreement. Krippendorff suggests: “\[I]t is customary to require α ≥ .800. Where tentative conclusions are still acceptable, α ≥ .667 is the lowest conceivable limit (2004, p. 241).”**
* [**Supposedly multi label**](https://stackoverflow.com/questions/57256287/calculate-kappa-score-for-multi-label-image-classifcation)
1. **MACE - the new kid on the block. -**
**learns in an unsupervised fashion to**
1. **a) identify which annotators are trustworthy and**
2. **b) predict the correct underlying labels. We match performance of more complex state-of-the-art systems and perform well even under adversarial conditions**
3. [**MACE**](https://www.isi.edu/publications/licensed-sw/mace/) **does exactly that. It tries to find out which annotators are more trustworthy and upweighs their answers.**
4. [**Git**](https://github.com/dirkhovy/MACE) **-**
**When evaluating redundant annotatio**
**ns (like those from Amazon's MechanicalTurk), we usually want to**
1. **aggregate annotations to recover the most likely answer**
2. **find out which annotators are trustworthy**
3. **evaluate item and task difficulty**
**MACE solves all of these problems, by learning competence estimates for each annotators and computing the most likely answer based on those competences.**
1.
**Calculating agreement**
1. **Compare against researcher-ground-truth**
2. **Self-agreement**
3. **Inter-agreement**
1. [**Medium**](https://towardsdatascience.com/inter-rater-agreement-kappas-69cd8b91ff75)
2. [**Kappa**](https://stats.stackexchange.com/questions/82162/cohens-kappa-in-plain-english) **cohen**
3. [**Multi annotator with kappa (which isnt), is this okay?**](https://stackoverflow.com/questions/52272901/multi-label-annotator-agreement-with-cohen-kappa)
4. **Github computer Fleiss Kappa** [**1**](https://gist.github.com/skylander86/65c442356377367e27e79ef1fed4adee)
5. [**Fleiss Kappa Example**](https://www.wikiwand.com/en/Fleiss%27_kappa#/Worked_example)
6. [**GWET AC1**](https://stats.stackexchange.com/questions/235929/fleiss-kappa-alternative-for-ranking)**,** [**paper**](https://s3.amazonaws.com/sitesusa/wp-content/uploads/sites/242/2014/05/J4_Xie_2013FCSM.pdf)**: as an alternative to kappa, and why**
7. [**Website, krippensorf vs fleiss calculator**](https://nlp-ml.io/jg/software/ira/)
## **Troubling shooting agreement metrics**
1. **Imbalance data sets, i.e., why my** [**Why is reliability so low when percentage of agreement is high?**](https://www.researchgate.net/post/Why_is_reliability_so_low_when_percentage_of_agreement_is_high)
2. [**Interpretation of kappa values**](https://towardsdatascience.com/interpretation-of-kappa-values-2acd1ca7b18f)
3. [**Interpreting agreement**](http://web2.cs.columbia.edu/~julia/courses/CS6998/Interrater_agreement.Kappa_statistic.pdf)**, Accuracy precision kappa**
## **Machine Vision annotation**
1. [**CVAT**](https://venturebeat.com/2019/03/05/intel-open-sources-cvat-a-toolkit-for-data-labeling/)
# Sentiment Analysis
## **Databases**
1. [**Sentiment databases**](https://medium.com/@datamonsters/sentiment-analysis-tools-overview-part-1-positive-and-negative-words-databases-ae35431a470c)
2. **Movie reviews:** [**IMDB reviews dataset on Kaggle**](https://www.kaggle.com/c/word2vec-nlp-tutorial/data)
3. **Sentiwordnet – mapping wordnet senses to a polarity model:** [**SentiWordnet Site**](http://sentiwordnet.isti.cnr.it/)
4. [**Twitter airline sentiment on Kaggle**](https://www.kaggle.com/crowdflower/twitter-airline-sentiment)
5. [**First GOP Debate Twitter Sentiment**](https://www.kaggle.com/crowdflower/first-gop-debate-twitter-sentiment)
6. [**Amazon fine foods reviews**](https://www.kaggle.com/snap/amazon-fine-food-reviews)
## **Tools**
1. **\*\* Many** [**Sentiment tools,** ](https://medium.com/@datamonsters/sentiment-analysis-tools-overview-part-2-7f3a75c262a3)
2. [**NTLK sentiment analyzer**](http://www.nltk.org/api/nltk.sentiment.html)
3. **Vader (NTLK, standalone):**
1. [**Vader/Sentiwordnet/etc python code examples - possibly good for ensembles**](https://nlpforhackers.io/sentiment-analysis-intro/)
2. **\*\***[**Intro into Vader**](http://t-redactyl.io/blog/2017/04/using-vader-to-handle-sentiment-analysis-with-social-media-text.html)
3. [**Why vader?**](https://www.quora.com/Which-is-the-superior-Sentiment-Analyzer-Vader-or-TextBlob)
4. **\*\***[**Vader - a clear explanation about the paper’s methodology** ](https://www.ijariit.com/manuscripts/v4i1/V4I1-1307.pdf)
5. **Simple Intro to** [**Vader**](https://medium.com/@aneesha/quick-social-media-sentiment-analysis-with-vader-da44951e4116)
6. [**A very lengthy and overly complex explanation about using NTLK vader**](https://programminghistorian.org/en/lessons/sentiment-analysis)
7. [**Vader tutorial, +-0.2 for neutrals.**](https://www.learndatasci.com/tutorials/sentiment-analysis-reddit-headlines-pythons-nltk/)
4. **Text BLob:**
1. [**Text blob classification**](http://rwet.decontextualize.com/book/textblob/)
2. [**Python code**](https://planspace.org/20150607-textblob_sentiment/)
3. [**More code**](https://textminingonline.com/getting-started-with-textblob)
4. [**A lengthy tutorial**](https://www.analyticsvidhya.com/blog/2018/02/natural-language-processing-for-beginners-using-textblob/)
5. **\*\***[**Text blob sentiment analysis tutorial on medium**](https://medium.com/@rahulvaish/textblob-and-sentiment-analysis-python-a687e9fabe96)
6. [**A lengthy intro plus code about text blob**](https://aparrish.neocities.org/textblob.html)
5. [**Comparative opinion mining a review paper - has some info about unsupervised as well**](https://arxiv.org/pdf/1712.08941.pdf)
6. [**Another reference list, has some unsupervised.**](http://scholar.google.co.il/scholar_url?url=http://www.nowpublishers.com/article/DownloadSummary/INR-011\&hl=en\&sa=X\&scisig=AAGBfm0NN0Pge4htltclF-D6H4BpxocqwA\&nossl=1\&oi=scholarr)
7. **Sentiwordnet3.0** [**paper**](https://www.researchgate.net/profile/Fabrizio_Sebastiani/publication/220746537_SentiWordNet_30_An_Enhanced_Lexical_Resource_for_Sentiment_Analysis_and_Opinion_Mining/links/545fbcc40cf27487b450aa21.pdf)
8. [**presentation**](https://web.stanford.edu/class/cs124/lec/sentiment.pdf)
9. [Hebrew Psychological Lexicons](https://github.com/natalieShapira/HebrewPsychologicalLexicons) by Natalie Shapira
This is the official code accompanying a paper on the [Hebrew Psychological Lexicons](https://www.aclweb.org/anthology/2021.clpsych-1.6.pdf) was presented at CLPsych 2021.
**Reference papers:**
1. [**Twitter as a corpus for SA and opinion mining**](http://crowdsourcing-class.org/assignments/downloads/pak-paroubek.pdf)
## **Ground Truth**
1. **For sentiment In Vader -**
1. **“Screening for English language reading comprehension – each rater had to individually score an 80% or higher on a standardized college-level reading comprehension test.**
2. **Complete an online sentiment rating training and orientation session, and score 90% or higher for matching the known (prevalidated) mean sentiment rating of lexical items which included individual words, emoticons, acronyms, sentences, tweets, and text snippets (e.g., sentence segments, or phrases).**
3. **Every batch of 25 features contained five “golden items” with a known (pre-validated) sentiment rating distribution. If a worker was more than one standard deviation away from the mean of this known distribution on three or more of the five golden items, we discarded all 25 ratings in the batch from this worker.**
4. **Bonus to incentivize and reward the highest quality work. Asked workers to select the valence score that they thought “most other people” would choose for the given lexical feature (early/iterative pilot testing revealed that wording the instructions in this manner garnered a much tighter standard deviation without significantly affecting the mean sentiment rating, allowing us to achieve higher quality (generalized) results while being more economical).**
5. **Compensated AMT workers $0.25 for each batch of 25 items they rated, with an additional $0.25 incentive bonus for all workers who successfully matched the group mean (within 1.5 standard deviations) on at least 20 of 25 responses in each batch. Using these four quality control methods, we achieved remarkable value in the data obtained from our AMT workers – we paid incentive bonuses for high quality to at least 90% of raters for most batches.**
[**Multilingual Twitter Sentiment Classification: The Role of Human Annotators**](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0155036)
* **1.6 million tweets labelled**
* **13 languages**
* **Evaluated 6 pretrained classification models**
* **10 CFV**
* **SVM / NB**
* **Annotator agreements.**
* **about 15% were intentionally duplicated to be annotated twice,**
* **by the same annotator**
* **by two different annotators**
* **Self-agreement from multiple annotations of the same annotator**
* **Inter-agreement from multiple annotations by different annotators**
* **The confidence intervals for the agreements are estimated by bootstrapping \[**[**12**](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0155036#pone.0155036.ref012)**].**
* **It turns out that the self-agreement is a good measure to identify low quality annotators,**
* **the inter-annotator agreement provides a good estimate of the objective difficulty of the task, unless it is too low.**
**Alpha was developed to measure the agreement between human annotators, but can also be used to measure the agreement between classification models and a gold standard. It generalizes several specialized agreement measures, takes ordering of classes into account, and accounts for the agreement by chance. Alpha is defined as follows:**
[**Method cont here**](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0194317) **in a second paper**
# Question Answering
1. [**Pythia, qna for images**](https://github.com/facebookresearch/pythia) **on colab**
2. [**Building a Q\&A system part 1**](https://towardsdatascience.com/building-a-question-answering-system-part-1-9388aadff507)
3. [**Building a Q\&A model**](https://towardsdatascience.com/nlp-building-a-question-answering-model-ed0529a68c54)
4. [**Vidhya on Q\&A**](https://medium.com/analytics-vidhya/how-i-build-a-question-answering-model-3548878d5db2)
5. [**Q\&A system using**](https://medium.com/voice-tech-podcast/building-an-intelligent-qa-system-with-nlp-and-milvus-75b496702490) [**milvus - An open source embedding vector similarity search engine powered by Faiss, NMSLIB and Annoy**](https://github.com/milvus-io/milvus)
6. [**Q\&A system**](https://medium.com/@akshaynavalakha/nlp-question-answering-system-f05825ef35c8)
---
[Next Page](/llms-full.txt/1)
