Interpretable & Explainable AI (XAI)
Last updated
Was this helpful?
Last updated
Was this helpful?
Interpretability and Explainability in Machine Learning / slides. Understanding, evaluating, rule based, prototype based, risk scores, generalized additive models, explaining black box, visualizing, feature importance, actionable explanations, casual models, human in the loop, connection with debugging.
a tutorial by Hima Lakkaraju (tutorial , , )
by Cinthia rudin
A great on the topic by Shir Meir Lador
From the above image: - a really good review for everything XAI - “a survey to help industry practitioners (but also data scientists more broadly) understand the field of explainable machine learning better and apply the right tools. Our latter sections build a narrative around a putative data scientist, and discuss how she might go about explaining her models by asking the right questions. From an organization viewpoint, after motivating the area broadly, we discuss the main developments, including the principles that allow us to study transparent models vs opaque models, as well as model-specific or model-agnostic post-hoc explainability approaches. We also briefly reflect on deep learning models, and conclude with a discussion about future research directions.”
,
(great) transparent (simultability, decomposability, algorithmic transparency) post-hoc interpretability (text explanation, visual local, explanation by example,), evaluation, utility.
“A growing number of techniques provide model interpretations, but can lead to wrong conclusions if applied incorrectly. We illustrate pitfalls of ML model interpretation such as bad model generalization, dependent features, feature interactions or unjustified causal interpretations. Our paper addresses ML practitioners by raising awareness of pitfalls and pointing out solutions for correct model interpretation, as well as ML researchers by discussing open issues for further research.” - mulner et al.
*** This is very good, includes eli5, lime, shap, many others.
Book:
- White-box and black-box ML model explanation library. is an open source Python library aimed at machine learning model inspection and interpretation. The focus of the library is to provide high-quality implementations of black-box, white-box, local and global explanation methods for classification and regression models.
youtube,
are not always consistent and do not agree with each other, this article has a make-sense explanation and flow for using shap and its many plots.
- Path attribution methods are a gradient-based way of explaining deep models. These methods require choosing a hyperparameter known as the baseline input. What does this hyperparameter mean, and how important is it? In this article, we investigate these questions using image classification networks as a case study. We discuss several different ways to choose a baseline input and the assumptions that are implicit in each baseline. Although we focus here on path attribution methods, our discussion of baselines is closely connected with the concept of missingness in the feature space - a concept that is critical to interpretability research.
WHAT IF TOOL - GOOGLE, ,
The Language Interpretability Tool (LIT) is an open-source platform for visualization and understanding of NLP models.
- “trying to \textit{explain} black box models, rather than creating models that are \textit{interpretable} in the first place, is likely to perpetuate bad practices and can potentially cause catastrophic harm to society. There is a way forward -- it is to design models that are inherently interpretable. This manuscript clarifies the chasm between explaining black boxes and using inherently interpretable models, outlines several key reasons why explainable black boxes should be avoided in high-stakes decisions, identifies challenges to interpretable machine learning, and provides several example applications where interpretable models could potentially replace black box models in criminal justice, healthcare, and computer vision.”
from
() was designed to automatically perform feature selection on a dataset using randomized features, i.e., measuring valid features against their shadow/noisy counterparts.
by Microsoft, .
,
- explain transformers with 2 lines of code.
Theory:
A series on Shap, Lime.
Shapash
Consistency - do different explainability methods give, on average, similar explanations?
Stability - for similar instances, are the explanations similar?
Compacity - do fewer features drive the model?
Partial Shap
Shap residuals
SHAP advanced
How to calculate Shap values per class based on this graph
NLP and IMAGE, - In the experiments in , we demonstrate that both machine learning experts and lay users greatly benefit from explanations similar to Figures 5 and 6 and are able to choose which models generalize better, improve models by changing them, and get crucial insights into the models' behavior.
- An anchor explanation is a rule that sufficiently “anchors” the prediction locally – such that changes to the rest of the feature values of the instance do not matter. In other words, for instances on which the anchor holds, the prediction is (almost) always the same.
How Shap values are calculated - .
Cooporative game theory & Shapely values, ,
Intro to shap and lime, ,
Part I:
Part II:
Part III:
Part VI:
Part V:
Part VI:
Part VII:
Part VIII:
Medium
**** In depth
(lime and shap)
- Yann Golhen
- francesco marini using 3 new metrics
? by Samuele Mazzanti - "Discover “ParShap”: an advanced method to detect which columns make your model underperform on new data" implemented in -stats.
- whatever you do start with this
- continue with this
