> For the complete documentation index, see [llms.txt](https://www.mlcompendium.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.mlcompendium.com/responsible-ai/fairness-accountability-and-transparency.md). # Fairness, Accountability, and Transparency An explained model can still discriminate, leak private data, or break the law, and those are the failures this page is about. It starts with the regulation that sets the rules, moves through the FAT research community and the bias it studies, then the methods and tools that remove bias, and ends with privacy, differential privacy, anonymization, and how anonymized data gets recovered. The same notes are in [Fairness, Accountability, and Transparency In Prompts](/responsible-ai/fairness-accountability-and-transparency-in-prompts.md). ### REGULATION FOR AI The rules come first, because they decide what a company has to prove about its models. [Preparing for EU regulations](https://medium.com/@itai_88363/how-ai-leaders-should-prepare-for-the-looming-eu-regulations-99e9d4f4c039) is the post on how AI leaders should prepare for the looming EU regulations, and the [EU regulation DOC](https://drive.google.com/file/d/1ZaBPsfor_aHKNeeyXxk9uJfTru747EOn/view) is the AI Regulation draft itself. Sector rules exist too, such as EIOPA - regulation for insurance companies, whose report is kept at the end of the page. Ethics and regulations in Israel came in a sequence of reports: 1. [First Report by the intelligence committee](https://www.globes.co.il/news/article.aspx?did=1001307714) headed by prof. Itzik ben israel and prof. evyatar matanya 2. [Second report by AI and data science committee](https://innovationisrael.org.il/sites/default/files/%D7%93%D7%95%D7%97%20%D7%A1%D7%95%D7%A4%D7%99%20%D7%A1%D7%99%D7%9B%D7%95%D7%9D%20%D7%95%D7%95%D7%A2%D7%93%D7%AA%20%D7%AA%D7%9C%D7%9D%20%D7%9C%D7%AA%D7%9B%D7%A0%D7%99%D7%AA%20%D7%9E%D7%95%D7%A4%20%D7%9C%D7%90%D7%95%D7%9E%D7%99%D7%AA%20%D7%91%D7%91%D7%99%D7%A0%D7%94%20%D7%9E%D7%9C%D7%90%D7%9B%D7%95%D7%AA%D7%99%D7%AA%20-.pdf) 3. Third by meizam leumi for AI systems in, the national subcommittee report on [ethics and regulation in israel](https://machinelearning.co.il/4330/israelaiethicsreport/#more-4330), published on Machine Learning Israel The same site also hosts a [lecture](https://machinelearning.co.il/3349/googleai/) on ethics and AI at Google: אתיקה ו-AI בגוגל: ג'ף דין מסביר על התהליך הפנימי של פיתוח מודלים מתקדמים. ### FAIRNESS, ACCOUNTABILITY & TRANSPARENCY Regulation says what must hold; the FAT research community works out how to measure and enforce it. FATML [website](https://www.fatml.org/) - The past few years have seen growing recognition that machine learning raises novel challenges for ensuring non-discrimination, due process, and understandability in decision-making. In particular, policymakers, regulators, and advocates have expressed fears about the potentially discriminatory impact of machine learning, with many calling for further technical research into the dangers of inadvertently encoding bias into automated decisions. At the same time, there is increasing alarm that the complexity of machine learning may reduce the justification for consequential decisions to “the algorithm made me do it.” FAT ML publishes its [Principles and best practices](https://www.fatml.org/resources/principles-and-best-practices) and a list of [projects](https://www.fatml.org/resources/relevant-projects). The related conference is [FAccT](https://facctconference.org/) - A computer science conference with a cross-disciplinary focus that brings together researchers and practitioners interested in fairness, accountability, and transparency in socio-technical systems. Fairness is not always a cost. The OpenReview [Paper - there is no fairness, enforcing fairness can improve accuracy](https://openreview.net/forum?id=wXoHN-Zoel) makes that case. Companies state their own frameworks: [Google on responsible ai practices](https://ai.google/responsibilities/responsible-ai-practices/) is Google's AI Principles, a guiding framework for responsible development and use of AI alongside transparency and accountability in its development process. And the explanation problem from the previous page returns in [Bengio on ai](https://www.wired.com/story/ai-pioneer-algorithms-understand-why/), Will Knight's Wired piece on how deep learning is good at finding patterns in reams of data but can't explain how they're connected, and on Yoshua Bengio wanting his algorithms to understand the "why". Fairness can also be attacked. [Poisoning attacks on fairness](https://arxiv.org/pdf/2004.07401.pdf) - Research in adversarial machine learning has shown how the performance of machine learning models can be seriously compromised by injecting even a small fraction of poisoning points into the training data. We empirically show that our attack is effective not only in the white-box setting, in which the attacker has full access to the target model, but also in a more challenging black-box scenario in which the attacks are optimized against a substitute model and then transferred to the target model For a guided tour, the AI Bias & Fairness [series of articles](https://jonathan-hui.medium.com/ai-bias-fairness-series-ce21ebf7b2e9) about Bias Fairness Johnathan Hui runs in five parts: 1. [In Clinical research](https://jonathan-hui.medium.com/bias-in-clinical-research-data-science-machine-learning-deep-learning-40a8786a5046) - Selection , Sample , Time , Attrition , Survivorship, reporting, funding, citation, Volunteer , self-selection , non-response, pre-screening , healthy person, membership, ascertainment, performance, berkson admission, neyman, measurement, observer, expectation, response, self reporting, social desirability, recall, acquiescence agreement, leading, courtesy, attention verification, lead time, immortal time, misclassification, chronological, detection, spectrum, cofounder, susceptibility, collider, simpson, ommited, allocation, channeling. 2. [AI](https://jonathan-hui.medium.com/ai-bias-b85c86bbca90) - known cases in Vision, NLP - sentiment, embedding, language models, historical, compass, recommender, datasets. 3. [address AI Bias with Fairness criteria and tools](https://jonathan-hui.medium.com/address-ai-bias-with-fairness-criteria-tools-9af1ab8e4289) - per population, predictive parity, calibration by group 4. [Caveats and limitations of AI Fairness Approaches](https://jonathan-hui.medium.com/caveats-limitations-on-ai-fairness-approaches-8628e6a992fd) - sample bias, label bias, miscalibration outcome test, redlining, etc. 5. [AI Fairness Approaches](https://jonathan-hui.medium.com/ai-fairness-approaches-mathematical-definitions-49cc418feebd) - statistical fairness, equalizing acceptance rate, error rate, etc. ### BIAS Once the kinds of bias have names, they have to be watched in a running model. Gabe Barcelos's "Understanding Bias in Machine Learning Models", arize.ai on [model bias](https://arize.com/understanding-bias-in-ml-models/#MLMonitoring), covers the types of bias in developing ML models, methods to detect them, their impact, and best practices for avoiding them at each stage of the pipeline. The figure below is the monitoring view.

Model bias monitoring.

### DEBIASING MODELS Detecting bias is half the job; the other half is taking it out of the representation. The first attempt most people try is adversarial training, and the first paper is a warning about it. [Adversarial removal of demographic features](https://arxiv.org/abs/1808.06640) - “We show that demographic information of authors is encoded in -- and can be recovered from -- the intermediate representations learned by text-based neural classifiers. The implication is that decisions of classifiers trained on textual data are not agnostic to -- and likely condition on -- demographic attributes. “ It continues: “we explore several techniques to improve the effectiveness of the adversarial component. Our main conclusion is a cautionary one: do not rely on the adversarial training to achieve invariant representation to sensitive features.” A linear alternative is [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://arxiv.org/abs/2004.07667) (paper) , [github](https://github.com/shauli-ravfogel/nullspace_projection), [presentation](https://docs.google.com/presentation/d/1Xi5HLpvvRE8BqcNBZMyPS4gBa0i0lqZvRebz-AZxAPA/edit) by Shauli et al. - removing biased information such as gender from an embedding space using nullspace projection. The presentation is Shauli Ravfogel's ACL 2020 talk from Bar-Ilan university. The objective is this: give a representation of text, for example BERT embeddings of many resumes/CVs, we want to achieve a state where a certain quality, for example a gender representation of the person who wrote this resume is not encoded in X. they used the light version definition for “not encoded”, i.e., you cant predict the quality from the representation with a higher than random score, using a linear model. I.e., every linear model you will train, will not be able to predict the person’s gender out of the embedding space and will reach a 50% accuracy. This is done by an iterative process that includes: 1. Linear model training to predict the quality of the concept from the representation. 2. Performing ‘projection to null space’ for the linear classifier, this is an acceptable linear algebra calculation that has a meaning of zeroing the representation from the projection on the separation place that the linear model is representing, making the model useless. I.e., it will always predict the zero vector. This is done iteratively on the neutralized output, i.e., in the second iteration we look for an alternative way to predict the gender out of X, until we reach 50% accuracy (or some other metric you want to measure) at this point we have neutralized all the linear directions in the embedding space, that were predictive to the gender of the author. For a matrix W, the null space is a sub-space of all X such that WX=0, i.e., W maps X to the zero vector, this is a linear projection of the zero vector into a subspace. For example you can take a 3d vectors and calculate its projection on XY. Removing a direction from the embedding leaves the training data untouched, which raises the next question. Can we extinct predictive samples? Its an open question, Maybe we can use influence functions? [Understanding Black-box Predictions via Influence Functions](https://arxiv.org/pdf/1703.04730.pdf) - How can we explain the predictions of a blackbox model? In this paper, we use influence functions — a classic technique from robust statistics — to trace a model’s prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually indistinguishable training-set attacks. Simpler tricks exist for word embeddings. [Removing ‘gender bias using pair mean pca](https://stackoverflow.com/questions/48019843/pca-on-word2vec-embeddings) is the question about identifying the gender subspace by taking the principal components of gender-pair difference vectors in word2vec. And to measure bias in a product's outputs, [Bias detector by intuit](https://github.com/intuit/bias-detector) - Based on first and last name/zip code the package analyzes the probability of the user belonging to different genders/races. Then, the model predictions per gender/race are compared using various bias metrics. ### FAIRNESS TOOLS The methods above are research code; the tools below package fairness checks for everyday use. The first step is knowing where sensitive data sits: [PII tools, by gensim](https://pii-tools.com/) analyzes personal data and sensitive information at scale, sensitive data discovery tools for internal PII compliance and MSPs. Then [Fair-learn](https://github.com/fairlearn/fairlearn) A Python package to assess and improve fairness of machine learning models.

Fairlearn.

Sk-lego adds fairness utilities to scikit-learn, shown first as an overview.

Sk-lego fairness.

Its first use is regression.

Regression fairness view.

The second is classification, shown over the next four figures.

Classification fairness view.

Fairness tools.

Fairness tools.

Fairness tools.

The third is the information filter.

Information filter.

The papers behind those constraints are: M. Zafar et al. (2017), Fairness Constraints: Mechanisms for Fair Classification M. Hardt, E. Price and N. Srebro (2016), Equality of Opportunity in Supervised Learning The Sk-lego fairness documentation, Fairness — scikit-lego latest documentation, survives as an archived copy: . ### PRIVACY Fairness tools protect groups; privacy protects the individual whose data trained the model. Two podcast episodes on privacy in data science and on fairness in AI used to sit here; they are kept at the end of the page. ### DIFFERENTIAL PRIVACY The most rigorous answer to that individual risk is differential privacy. Differential privacy has emerged as a major area of research in the effort to prevent the identification of individuals and private data. It is a mathematical definition for the privacy loss that results to individuals when their private information is used to create AI products. It works by injecting noise into a dataset, during a machine learning training process, or into the output of a machine learning model, without introducing significant adverse effects on data analysis or model performance. It achieves this by calibrating the noise level to the sensitivity of the algorithm. The result is a differentially private dataset or model that cannot be reverse engineered by an attacker, while still providing useful information. Uses BOTLON & EPSILON For a quick visual version, "Differential Privacy - Simply Explained" is on [youtube](https://www.youtube.com/watch?v=gI0wk1CXlsQ\&feature=emb_title). The business framing, how do you aggregate data across your customer base without violating data rights contracts or making your customers angry, is Georgian Partners' "What is Differential privacy?", archived at . ### ANONYMIZATION When noise is too blunt, the alternative is to find and remove the identifying text itself. The same notes are in [Named Entity Recognition (NER)](/natural-language-processing-nlp/named-entity-recognition-ner.md). [Using NER (omri mendels)](https://medium.com/data-science/nlp-approaches-to-data-anonymization-1fb5bde6b929) is the NLP approach to data anonymization. The same post, practical ways for de-identifying real-world private data, by Omri Mendels, is also at . ### DE-ANONYMIZATION Anonymization only matters if the data cannot be pulled back out, and large language models show that it can. GPT2 - [Of language datasets](https://arxiv.org/pdf/2012.07805.pdf) is the paper on extracting training data from large language models; the figure below is from it.

Of language datasets.

That brings the page back to where it started, regulation. Preparing for EU regulations by MonaLabs, Itai Bar Sinai on the new obligations, including post-market monitoring, on all companies using "high-risk" AI in the EU, is also at . ## Deprecated links {% hint style="warning" %} These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above. {% endhint %} * EIOPA - regulation for insurance companies. This address no longer opens: * Sk-lego. This address no longer opens: * Differential privacy. This address no longer opens: * Privacy in DataScience. This address now points to an unrelated site: * Fairness in AI. This address now points to an unrelated site: --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://www.mlcompendium.com/responsible-ai/fairness-accountability-and-transparency.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.