compare-mt

Provides word-level accuracy by frequency, sentence bucket analysis, and n-gram differences, moving beyond aggregate scores to pinpoint specific weaknesses, as demonstrated in the example comparing neural vs. phrase-based MT.

Supports BLEU, ROUGE, COMET, and word likelihoods, enabling cross-task evaluation like summarization with ROUGE scores, as shown in the summarization example.

Includes bootstrap resampling for metrics, helping assess the reliability of differences between systems based on data sampling, with configurable samples and probability thresholds.

Allows analysis by POS tags, source word features, and numerical labels, as evidenced by examples that reveal differences in content vs. function words or sentence position effects.