Ambrosia
A cross-platform CLI tool for cleaning and improving text datasets for machine learning, with fast operations and LLM-based filtering.
What is Ambrosia?
Ambrosia is a cross-platform command-line utility designed to enhance the quality of text datasets used for training machine learning models. It provides fast, standard data cleaning operations like deduplication and filtering, alongside a unique capability to leverage large language models (LLMs) for intelligent sorting and filtering of dataset entries.
Machine learning engineers and data scientists who need to preprocess and clean text datasets efficiently, particularly those working with instruction-tuning datasets or any structured text data requiring quality filtering.
Developers choose Ambrosia for its combination of high-speed traditional data hygiene tools and programmable LLM-based filtering in a single, dependency-free binary, enabling both foundational cleaning and intelligent, prompt-driven dataset curation.
Overview
clean up your LLM datasets
Use Cases
Best For
- Performing fast deduplication on text datasets using exact matching or semantic similarity via ROUGE-L.
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
