Fairness, Accountability, and Transparency
An explained model can still discriminate, leak private data, or break the law, and those are the failures this page is about. It starts with the regulation that sets the rules, moves through the FAT research community and the bias it studies, then the methods and tools that remove bias, and ends with privacy, differential privacy, anonymization, and how anonymized data gets recovered.
The same notes are in Fairness, Accountability, and Transparency In Prompts.
REGULATION FOR AI
The rules come first, because they decide what a company has to prove about its models. Preparing for EU regulations is the post on how AI leaders should prepare for the looming EU regulations, and the EU regulation DOC is the AI Regulation draft itself. Sector rules exist too, such as EIOPA - regulation for insurance companies, whose report is kept at the end of the page.
Ethics and regulations in Israel came in a sequence of reports:
First Report by the intelligence committee headed by prof. Itzik ben israel and prof. evyatar matanya
Third by meizam leumi for AI systems in, the national subcommittee report on ethics and regulation in israel, published on Machine Learning Israel
The same site also hosts a lecture on ethics and AI at Google: אתיקה ו-AI בגוגל: ג'ף דין מסביר על התהליך הפנימי של פיתוח מודלים מתקדמים.
FAIRNESS, ACCOUNTABILITY & TRANSPARENCY
Regulation says what must hold; the FAT research community works out how to measure and enforce it. FATML website - The past few years have seen growing recognition that machine learning raises novel challenges for ensuring non-discrimination, due process, and understandability in decision-making. In particular, policymakers, regulators, and advocates have expressed fears about the potentially discriminatory impact of machine learning, with many calling for further technical research into the dangers of inadvertently encoding bias into automated decisions.
At the same time, there is increasing alarm that the complexity of machine learning may reduce the justification for consequential decisions to “the algorithm made me do it.”
FAT ML publishes its Principles and best practices and a list of projects. The related conference is FAccT - A computer science conference with a cross-disciplinary focus that brings together researchers and practitioners interested in fairness, accountability, and transparency in socio-technical systems.
Fairness is not always a cost. The OpenReview Paper - there is no fairness, enforcing fairness can improve accuracy makes that case. Companies state their own frameworks: Google on responsible ai practices is Google's AI Principles, a guiding framework for responsible development and use of AI alongside transparency and accountability in its development process. And the explanation problem from the previous page returns in Bengio on ai, Will Knight's Wired piece on how deep learning is good at finding patterns in reams of data but can't explain how they're connected, and on Yoshua Bengio wanting his algorithms to understand the "why".
Fairness can also be attacked. Poisoning attacks on fairness - Research in adversarial machine learning has shown how the performance of machine learning models can be seriously compromised by injecting even a small fraction of poisoning points into the training data. We empirically show that our attack is effective not only in the white-box setting, in which the attacker has full access to the target model, but also in a more challenging black-box scenario in which the attacks are optimized against a substitute model and then transferred to the target model
For a guided tour, the AI Bias & Fairness series of articles about Bias Fairness Johnathan Hui runs in five parts:
In Clinical research - Selection , Sample , Time , Attrition , Survivorship, reporting, funding, citation, Volunteer , self-selection , non-response, pre-screening , healthy person, membership, ascertainment, performance, berkson admission, neyman, measurement, observer, expectation, response, self reporting, social desirability, recall, acquiescence agreement, leading, courtesy, attention verification, lead time, immortal time, misclassification, chronological, detection, spectrum, cofounder, susceptibility, collider, simpson, ommited, allocation, channeling.
AI - known cases in Vision, NLP - sentiment, embedding, language models, historical, compass, recommender, datasets.
address AI Bias with Fairness criteria and tools - per population, predictive parity, calibration by group
Caveats and limitations of AI Fairness Approaches - sample bias, label bias, miscalibration outcome test, redlining, etc.
AI Fairness Approaches - statistical fairness, equalizing acceptance rate, error rate, etc.
BIAS
Once the kinds of bias have names, they have to be watched in a running model. Gabe Barcelos's "Understanding Bias in Machine Learning Models", arize.ai on model bias, covers the types of bias in developing ML models, methods to detect them, their impact, and best practices for avoiding them at each stage of the pipeline. The figure below is the monitoring view.
DEBIASING MODELS
Detecting bias is half the job; the other half is taking it out of the representation. The first attempt most people try is adversarial training, and the first paper is a warning about it. Adversarial removal of demographic features - “We show that demographic information of authors is encoded in -- and can be recovered from -- the intermediate representations learned by text-based neural classifiers. The implication is that decisions of classifiers trained on textual data are not agnostic to -- and likely condition on -- demographic attributes. “ It continues: “we explore several techniques to improve the effectiveness of the adversarial component. Our main conclusion is a cautionary one: do not rely on the adversarial training to achieve invariant representation to sensitive features.”
A linear alternative is Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection (paper) , github, presentation by Shauli et al. - removing biased information such as gender from an embedding space using nullspace projection. The presentation is Shauli Ravfogel's ACL 2020 talk from Bar-Ilan university.
The objective is this: give a representation of text, for example BERT embeddings of many resumes/CVs, we want to achieve a state where a certain quality, for example a gender representation of the person who wrote this resume is not encoded in X. they used the light version definition for “not encoded”, i.e., you cant predict the quality from the representation with a higher than random score, using a linear model. I.e., every linear model you will train, will not be able to predict the person’s gender out of the embedding space and will reach a 50% accuracy.
This is done by an iterative process that includes:
Linear model training to predict the quality of the concept from the representation.
Performing ‘projection to null space’ for the linear classifier, this is an acceptable linear algebra calculation that has a meaning of zeroing the representation from the projection on the separation place that the linear model is representing, making the model useless. I.e., it will always predict the zero vector.
This is done iteratively on the neutralized output, i.e., in the second iteration we look for an alternative way to predict the gender out of X, until we reach 50% accuracy (or some other metric you want to measure) at this point we have neutralized all the linear directions in the embedding space, that were predictive to the gender of the author.
For a matrix W, the null space is a sub-space of all X such that WX=0, i.e., W maps X to the zero vector, this is a linear projection of the zero vector into a subspace. For example you can take a 3d vectors and calculate its projection on XY.
Removing a direction from the embedding leaves the training data untouched, which raises the next question. Can we extinct predictive samples? Its an open question, Maybe we can use influence functions?
Understanding Black-box Predictions via Influence Functions - How can we explain the predictions of a blackbox model? In this paper, we use influence functions — a classic technique from robust statistics — to trace a model’s prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction.
We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually indistinguishable training-set attacks.
Simpler tricks exist for word embeddings. Removing ‘gender bias using pair mean pca is the question about identifying the gender subspace by taking the principal components of gender-pair difference vectors in word2vec. And to measure bias in a product's outputs, Bias detector by intuit - Based on first and last name/zip code the package analyzes the probability of the user belonging to different genders/races. Then, the model predictions per gender/race are compared using various bias metrics.
FAIRNESS TOOLS
The methods above are research code; the tools below package fairness checks for everyday use. The first step is knowing where sensitive data sits: PII tools, by gensim analyzes personal data and sensitive information at scale, sensitive data discovery tools for internal PII compliance and MSPs. Then Fair-learn A Python package to assess and improve fairness of machine learning models.
Sk-lego adds fairness utilities to scikit-learn, shown first as an overview.
Its first use is regression.
The second is classification, shown over the next four figures.
The third is the information filter.
The papers behind those constraints are:
M. Zafar et al. (2017), Fairness Constraints: Mechanisms for Fair Classification
M. Hardt, E. Price and N. Srebro (2016), Equality of Opportunity in Supervised Learning
The Sk-lego fairness documentation, Fairness — scikit-lego latest documentation, survives as an archived copy: https://web.archive.org/web/2020/https://scikit-lego.readthedocs.io/en/latest/fairness.html.
PRIVACY
Fairness tools protect groups; privacy protects the individual whose data trained the model. Two podcast episodes on privacy in data science and on fairness in AI used to sit here; they are kept at the end of the page.
DIFFERENTIAL PRIVACY
The most rigorous answer to that individual risk is differential privacy. Differential privacy has emerged as a major area of research in the effort to prevent the identification of individuals and private data. It is a mathematical definition for the privacy loss that results to individuals when their private information is used to create AI products. It works by injecting noise into a dataset, during a machine learning training process, or into the output of a machine learning model, without introducing significant adverse effects on data analysis or model performance. It achieves this by calibrating the noise level to the sensitivity of the algorithm. The result is a differentially private dataset or model that cannot be reverse engineered by an attacker, while still providing useful information. Uses BOTLON & EPSILON
For a quick visual version, "Differential Privacy - Simply Explained" is on youtube. The business framing, how do you aggregate data across your customer base without violating data rights contracts or making your customers angry, is Georgian Partners' "What is Differential privacy?", archived at https://web.archive.org/web/2020/https://georgianpartners.com/what-is-differential-privacy/.
ANONYMIZATION
When noise is too blunt, the alternative is to find and remove the identifying text itself. The same notes are in Named Entity Recognition (NER).
Using NER (omri mendels) is the NLP approach to data anonymization. The same post, practical ways for de-identifying real-world private data, by Omri Mendels, is also at https://towardsdatascience.com/nlp-approaches-to-data-anonymization-1fb5bde6b929.
DE-ANONYMIZATION
Anonymization only matters if the data cannot be pulled back out, and large language models show that it can. GPT2 - Of language datasets is the paper on extracting training data from large language models; the figure below is from it.
That brings the page back to where it started, regulation. Preparing for EU regulations by MonaLabs, Itai Bar Sinai on the new obligations, including post-market monitoring, on all companies using "high-risk" AI in the EU, is also at https://towardsdatascience.com/how-ai-leaders-should-prepare-for-the-looming-eu-regulations-99e9d4f4c039.
Deprecated links
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
EIOPA - regulation for insurance companies. This address no longer opens: https://www.eiopa.europa.eu/content/eiopa-publishes-report-artificial-intelligence-governance-principles_en
Sk-lego. This address no longer opens: https://scikit-lego.readthedocs.io/en/latest/fairness.html
Differential privacy. This address no longer opens: https://georgianpartners.com/what-is-differential-privacy/
Privacy in DataScience. This address now points to an unrelated site: http://www.unsupervised-podcast.xyz/ab55d406
Fairness in AI. This address now points to an unrelated site: http://www.unsupervised-podcast.xyz/5d7fc118
Last updated
Was this helpful?