Robustness Metrics
Lightweight Python library for evaluating classification model robustness across out-of-distribution generalization, stability, and uncertainty metrics.
What is Robustness Metrics?
Robustness Metrics is a Python library developed by Google Research that provides standardized modules for evaluating classification model robustness. It measures how well models perform across three critical dimensions: out-of-distribution generalization, stability under input perturbations, and uncertainty calibration. The library solves the problem of inconsistent robustness evaluation by offering lightweight, framework-agnostic tools that work with any model producing logits.
Machine learning researchers and practitioners who need to systematically evaluate and compare the robustness of classification models across different architectures and training regimes. This includes teams benchmarking model performance for research papers or production deployment.
Developers choose Robustness Metrics because it provides standardized, reproducible evaluation across multiple robustness dimensions with minimal integration effort. Unlike custom evaluation scripts, it offers pre-built datasets, measurements, and reports while remaining framework-agnostic—working with TensorFlow, PyTorch, JAX, or any model that produces logits.
Overview
Robustness Metrics provides lightweight modules to evaluate the robustness of classification models across three key dimensions: out-of-distribution generalization, stability under natural perturbations, and uncertainty calibration. It includes popular benchmark datasets and works with any model that maps inputs to logits, making it applicable beyond vision models.
Key Features
- Out-of-distribution evaluation — Assesses model performance on shifted datasets like ImageNetV2 and ImageNet-C
- Stability measurement — Evaluates prediction consistency under natural input perturbations
- Uncertainty quantification — Measures how well predicted probabilities reflect true probabilities
- Framework-agnostic — Works with TensorFlow, PyTorch, JAX, or any model producing logits
- Pre-built datasets — Includes standardized out-of-distribution datasets for benchmarking
- Flexible reporting — Supports custom measurement combinations or predefined robustness reports
Philosophy
The library emphasizes lightweight, modular evaluation that works across different model frameworks while providing standardized benchmarks for meaningful robustness comparisons.
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
