Pax

Achieves state-of-the-art Model FLOPs Utilization on TPU v4 Pods for large language models, as shown in benchmark graphs scaling to trillions of parameters with weak scaling patterns.

Uses Python dataclass-based HParams for fully configurable experiments, enabling clean separation between setup and implementation with Fiddle integration for shared parameters and eager error checking.

Supports multi-slice distributed training across TPU pods with pjit and pmap parallelism, detailed in multislice setup guides and example runs for scaling to massive models.

Provides BaseInput abstractions compatible with SeqIO and Lingvo, automatically handling sharding and padding for eval data, as explained in the inputs documentation.