Web Crawling
5 projects
Showing 5 of 5 projects
trafilaturaPython
A Python library and CLI tool for web crawling, scraping, and extracting main text, metadata, and comments from web pages.
#text-extraction#readability#article-extractor
Stars5.8k
Forks359
Last commit7 months ago
DotnetSpiderC#
A lightweight, efficient, and fast high-level web crawling and scraping framework for .NET.
#web-crawling#distributed#redis
Stars4.1k
Forks1.1k
Last commit21 days ago
SitemapGeneratorRuby
A framework-agnostic Ruby gem for generating XML sitemaps with Rails integration and support for multiple sitemap extensions.
#web-crawling#search-engine-optimization#ruby-gem
Stars2.5k
Forks280
Last commit1 month ago
FerrumRuby
A high-level Ruby API for controlling Chrome/Chromium browsers directly via the Chrome DevTools Protocol.
#web-crawling#developer-tools#chrome
Stars2.0k
Forks159
Last commit2 days ago
domain_analyzerPython
A Python security analysis tool that automatically discovers and reports comprehensive information about a given domain.
#python-tool#dns-analysis#web-crawling
Stars1.9k
Forks237
Last commit3 years ago
Related Tags
Community-curated · Updated weekly · 100% open source
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.