trafilatura
A Python library and CLI tool for web crawling, scraping, and extracting main text, metadata, and comments from web pages.
What is trafilatura?
Trafilatura is a Python package and command-line tool for gathering text and metadata from the web. It performs web crawling, scraping, and extraction of main texts, comments, and structured data, outputting results in formats like CSV, JSON, HTML, and Markdown. It solves the problem of turning raw HTML into clean, meaningful content by focusing on actual text and avoiding page noise.
Developers, researchers, and data scientists who need to extract and process web content at scale for projects like text analysis, data acquisition, or building datasets.
Developers choose Trafilatura for its robust and configurable extraction that balances precision and recall, its modular design without database dependencies, and its consistent outperformance of other open-source libraries in text extraction benchmarks.
Overview
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Use Cases
Best For
- Extracting main article text and metadata from news websites
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
