kvax

Implements block-wise attention masks built once per forward-backward pass using a Triton kernel, preventing cross-sequence contamination in packed sequences and avoiding O(seq_len²) GPU memory usage, as highlighted in the Key Concepts.

Leverages an all-gather based method from the Llama 3 training paper to distribute sequence processing across GPUs, reducing activation memory and balancing computational load, which is critical for distributed training setups.

Automatically skips blocks consisting entirely of padding tokens to save computation, with clear guidelines in the How to Use section for marking padding tokens with PADDING_SEGMENT_ID.

Uses FlashAttention 2 algorithms implemented in Triton for optimized GPU performance, as demonstrated in the benchmarks showing speedups in forward and backward passes with long sequences.