Robots Txt
5 projects
Showing 5 of 5 projects
collyGo
A fast and elegant scraping and crawling framework for Go, designed for extracting structured data from websites.
#spider#crawler#scraper
Stars25.3k
Forks1.8k
Last commit4 days ago
Crawler4jJava
An open-source Java web crawler that provides a simple interface for multi-threaded web crawling.
#java-library#open-source#crawling-framework
Stars4.6k
Forks1.9k
Last commit4 years ago
Next-SitemapTypeScript
A sitemap generator for Next.js that creates sitemaps and robots.txt for static, pre-rendered, dynamic, and server-side pages.
#server-side-rendering#nextjs#search-engine-optimization
Stars3.7k
Forks137
Last commit13 days ago
Heritrix Q&AJava
An open-source, extensible, web-scale, archival-quality web crawler from the Internet Archive.
#webcrawling#digital-preservation#warc
Stars3.2k
Forks783
Last commit6 days ago
HeritrixJava
An open-source, extensible, web-scale, archival-quality web crawler from the Internet Archive.
#webcrawling#digital-preservation#warc
Stars3.2k
Forks783
Last commit6 days ago
Related Tags
Community-curated · Updated weekly · 100% open source
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.