Inference Framework

JAX, Flax & TransformersPython

A model-definition framework for state-of-the-art machine learning models across text, vision, audio, and multimodal tasks.

sglangPython

A high-performance serving framework for large language models and multimodal models, delivering low-latency and high-throughput inference.

candle-wasm-examplesRust

A minimalist, high-performance machine learning framework for Rust with a focus on serverless inference and GPU support.

nndeployC++

A visual workflow-based AI deployment framework for multi-platform and multi-backend inference, supporting large models and edge devices.