A database of reproducible real-world Java bugs and a framework for controlled software engineering experiments.
Defects4J is a database and experimental infrastructure for software engineering research. It provides a collection of real, reproducible bugs from open-source Java projects, along with tools to checkout, compile, test, and analyze these bugs in a controlled manner. It solves the problem of inconsistent or synthetic benchmarks by offering a standardized platform for evaluating techniques like automated testing and program repair.
Software engineering researchers and practitioners focusing on empirical studies, automated testing, program repair, fault localization, and test generation who need a reliable benchmark of real-world Java faults.
Developers and researchers choose Defects4J because it offers a large, curated set of real faults with guaranteed reproducibility, a comprehensive CLI for controlled experimentation, and rich metadata—making it the de facto standard benchmark for many software engineering research areas.
A Database of Real Faults and an Experimental Infrastructure to Enable Controlled Experiments in Software Engineering Research
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Contains over 854 real, minimized bugs from 17 popular Java projects like JFreeChart and Gson, each validated with a single fixing commit and triggering test, ensuring authenticity for research.
Provides a command-line interface to checkout, compile, test, and analyze bugs, with built-in scripts for coverage, mutation analysis, and test generation, enabling consistent experimental setups.
Allows exporting detailed properties via commands like export and query, including modified classes, classpaths, and trigger tests, facilitating deep analysis without manual extraction.
Specifies Java 11 and timezone (America/Los_Angeles) requirements, and excludes broken/flaky tests, guaranteeing bug reproducibility across controlled environments as noted in the README.
Requires Java 11, Git, Subversion, Perl with cpanm dependencies, and manual initialization via init.sh, which can be time-consuming and error-prone for quick adoption.
Must run in America/Los_Angeles timezone and with Java 11; deviations can cause non-reproducible bugs, and deprecated bugs exist due to Java version changes, limiting flexibility.
Primarily tailored for academic experiments like automated program repair; lacks features for practical development tools such as IDE integration or real-time debugging aids.