An open-source toolkit for auditing bias and experimenting with fairness methods in machine learning models.
Aequitas is an open-source bias auditing and Fair ML toolkit that helps data scientists and researchers detect and mitigate bias in machine learning models. It provides tools to audit model predictions across sensitive attributes and experiment with fairness-enhancing methods throughout the ML pipeline. The toolkit supports binary classification tasks and integrates various pre-, in-, and post-processing techniques to promote equitable outcomes.
Data scientists, machine learning researchers, and policymakers working on binary classification models who need to assess and improve fairness. It's also suitable for teams implementing responsible AI practices in production systems.
Developers choose Aequitas for its comprehensive, all-in-one approach to fairness—combining bias auditing with mitigation experimentation in a single, extensible toolkit. Its integration of multiple Fair ML methods and visualization tools simplifies the process of evaluating and enhancing model equity.
Bias Auditing & Fair ML Toolkit
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Provides a wide range of confusion matrix-based fairness metrics (e.g., TPR, FPR, Precision) and visualization tools like summary and disparity plots, enabling in-depth analysis across sensitive attributes.
Includes pre-, in-, and post-processing techniques such as Data Repairer and FairGBM, allowing for end-to-end experimentation with bias mitigation via a streamlined interface like DefaultExperiment.
Supports adding user-implemented methods with intuitive interfaces, and components can be used individually or in integrated workflows, as shown in the tutorial notebook for method addition.
Offers the ability to save experiment artifacts, from transformed data to fitted models, ensuring that audits and corrections can be replicated and validated.
The toolkit is explicitly limited to binary classification tasks, as stated in the description, making it unsuitable for multi-class or regression problems without significant adaptation.
Requires data in a specific pandas DataFrame format with categorical sensitive attributes, which can be restrictive and add preprocessing overhead for datasets with continuous or non-standard attributes.
The integration of hyperparameter optimization via Optuna and extensive experimentation (e.g., with 'large' experiment sizes) can lead to high computational costs and slower runtimes, especially on large datasets.
Only includes two dataset families (BankAccountFraud and FolkTables), which may not cover diverse use cases, forcing users to prepare and adapt their own data more frequently.