Question 1

How do I set up Bencher with GitHub Actions?

Accepted Answer

Use the bencherdev/bencher GitHub Action from the marketplace, add the BENCHER_API_TOKEN secret, and run the bencher CLI command in your workflow YAML. The README provides a complete example with steps for error handling and PR comments.

Question 2

Bencher vs CodeCov for performance metrics?

Accepted Answer

Bencher is often compared to CodeCov for coverage, but it focuses exclusively on performance benchmarking. It tracks metrics over time, detects regressions in CI, and offers a web console, whereas CodeCov is for code coverage analysis.

Question 3

What languages and tools does Bencher support?

Accepted Answer

Bencher supports many languages and tools, including Rust with Criterion, C++ with Google Benchmark, Python with pytest-benchmark, and JavaScript with Benchmark.js, as listed in the adapters section. It also has JSON support for custom cases.

Question 4

Can Bencher track custom benchmarks like build times?

Accepted Answer

Yes, Bencher can track custom metrics such as build time and file size using its JSON adapter, detailed in the 'How To' documentation for tracking these specific aspects without a standard harness.

Question 5

How does Bencher detect performance regressions?

Accepted Answer

Bencher uses customizable analytics and thresholds to compare current benchmark results with historical data. You can set upper and lower boundaries, and it generates alerts when these are exceeded, as explained in the thresholds documentation.

Question 6

Is Bencher free or is there a paid version?

Accepted Answer

Bencher has open-source components under Apache/MIT licenses, but some features are under the Bencher Plus License. The cloud version may have pricing, while self-hosting is free but requires you to manage infrastructure and updates.

bencher

What is bencher?

Overview

Use Cases

Best For

Related Projects

Found a gem we're missing?

Not Ideal For

Pros & Cons

Pros

Cons

Frequently Asked Questions