Question 1

How do I gradually ramp up an experiment with Scientist?

Accepted Answer

Control rollout via the enabled? method in your custom experiment class, using percent-based logic or run_if blocks to conditionally execute candidates. Start with a low percentage and increase as confidence grows, monitoring mismatches closely.

Question 2

What's the difference between Scientist and feature flags for refactoring?

Accepted Answer

Scientist focuses on real-time comparison of code behavior and performance, providing detailed metrics and error handling. Feature flags simply toggle between code paths without built-in validation, making Scientist better for ensuring correctness but more complex to set up.

Question 3

How do I handle timeouts in candidate code with Scientist?

Accepted Answer

Scientist does not protect against timeouts; you must ensure your candidate code has its own timeout handling. The README suggests running experiments on a low percentage to minimize user impact if timeouts occur.

Question 4

Can I use Scientist for A/B testing new user features?

Accepted Answer

Scientist is designed for refactoring internal code, not user-facing A/B tests. It compares backend behaviors, so for feature rollouts, dedicated A/B testing tools that handle user segmentation and analytics are more appropriate.

Question 5

How do I clean up data for analysis in Scientist experiments?

Accepted Answer

Use the clean method in your experiment to transform values before publishing, such as extracting usernames from User objects. This reduces storage needs and makes mismatch debugging easier, as shown in the 'Keeping it clean' example.

Question 6

What monitoring systems integrate well with Scientist?

Accepted Answer

Scientist is agnostic; you can publish to any system. The README examples use statsd for timing metrics and redis for storing mismatches, but you can adapt it to tools like Graphite, databases, or log aggregators based on your infrastructure.

Question 7

How to test Scientist experiments in a Rails app?

Accepted Answer

Enable raise_on_mismatches in your test environment to fail tests on mismatches. For easier Rails integration, consider using the lab_tech wrapper, which provides ActiveRecord support for storing and analyzing results.

Scientist

What is Scientist?

Overview

Use Cases

Best For

Related Projects

Found a gem we're missing?

Not Ideal For

Pros & Cons

Pros

Cons

Frequently Asked Questions