Defects4J
A database of reproducible real-world Java bugs and a framework for controlled software engineering experiments.
What is Defects4J?
Defects4J is a database and experimental infrastructure for software engineering research. It provides a collection of real, reproducible bugs from open-source Java projects, along with tools to checkout, compile, test, and analyze these bugs in a controlled manner. It solves the problem of inconsistent or synthetic benchmarks by offering a standardized platform for evaluating techniques like automated testing and program repair.
Software engineering researchers and practitioners focusing on empirical studies, automated testing, program repair, fault localization, and test generation who need a reliable benchmark of real-world Java faults.
Developers and researchers choose Defects4J because it offers a large, curated set of real faults with guaranteed reproducibility, a comprehensive CLI for controlled experimentation, and rich metadata—making it the de facto standard benchmark for many software engineering research areas.
Overview
A Database of Real Faults and an Experimental Infrastructure to Enable Controlled Experiments in Software Engineering Research
Use Cases
Best For
- Evaluating automated program repair (APR) techniques on real Java bugs
- Benchmarking fault localization algorithms against known defects
- Assessing test generation tools like EvoSuite or Randoop
- Conducting controlled experiments in software testing research
- Studying bug patterns and characteristics in open-source Java projects
- Teaching software testing and debugging with reproducible examples
Not Ideal For
- Teams needing a live bug-tracking system for ongoing development workflows
- Projects using languages other than Java or older Java versions (pre-11)
- Environments where strict timezone or dependency management is infeasible
Pros & Cons
Pros
Contains over 854 real, minimized bugs from 17 popular Java projects like JFreeChart and Gson, each validated with a single fixing commit and triggering test, ensuring authenticity for research.
Provides a command-line interface to checkout, compile, test, and analyze bugs, with built-in scripts for coverage, mutation analysis, and test generation, enabling consistent experimental setups.
Allows exporting detailed properties via commands like export and query, including modified classes, classpaths, and trigger tests, facilitating deep analysis without manual extraction.
Specifies Java 11 and timezone (America/Los_Angeles) requirements, and excludes broken/flaky tests, guaranteeing bug reproducibility across controlled environments as noted in the README.
Cons
Requires Java 11, Git, Subversion, Perl with cpanm dependencies, and manual initialization via init.sh, which can be time-consuming and error-prone for quick adoption.
Must run in America/Los_Angeles timezone and with Java 11; deviations can cause non-reproducible bugs, and deprecated bugs exist due to Java version changes, limiting flexibility.
Primarily tailored for academic experiments like automated program repair; lacks features for practical development tools such as IDE integration or real-time debugging aids.
Frequently Asked Questions
Related Projects
Unix historyContinuous Unix commit history from 1970 until today
mzdataMulti-extract and Multi-level Dataset of Mozilla Issue Tracking History
OCL Expressions on GitHubA Data Set of OCL Expressions on GitHub
Findbugs-mavenThe Bug Catalog of the Maven Ecosystem
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.