xidel
A command-line tool to extract data from HTML/XML pages and JSON APIs using CSS, XPath, XQuery, JSONiq, and pattern matching.
What is xidel?
Xidel is a command-line tool for downloading and extracting data from HTML, XML, and JSON sources, including web pages and REST APIs. It supports multiple query languages like CSS selectors, XPath, XQuery, and JSONiq, allowing users to scrape, transform, and generate structured documents. It solves the problem of programmatically accessing and processing web data without relying on browser automation or complex scripting.
Developers, data engineers, and researchers who need to automate web scraping, extract data from APIs, or transform web content into structured formats from the command line.
Xidel stands out by supporting a wide range of query languages in a single tool, offering dependency-free binaries for major platforms, and enabling complex data extraction and transformation workflows with simple commands.
Overview
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
Use Cases
Best For
- Scraping data from websites using CSS selectors or XPath
- Extracting JSON data from REST APIs with JSONiq queries
- Transforming HTML/XML documents into JSON or other formats
- Automating multi-page web scraping with link following
- Command-line data extraction without browser automation
- Processing web data in shell scripts or pipelines
Not Ideal For
- Projects scraping JavaScript-heavy single-page applications (SPAs) that require browser automation
- Large-scale distributed web scraping tasks needing built-in concurrency and job management
- Teams integrated into Python or JavaScript ecosystems preferring language-specific libraries like Scrapy or Puppeteer
Pros & Cons
Pros
Supports CSS selectors, XPath 2.0/3.0, XQuery 1.0/3.0, and JSONiq, enabling flexible data extraction from HTML, XML, and JSON, as demonstrated in the README's language examples.
Provides pre-compiled executables for Windows, Linux, and macOS, simplifying installation without external dependencies, as highlighted in the README.
Automatically follows links with commands like `--follow //a` for multi-page scraping, streamlining workflows as shown in the simple example.
Can create and transform XML, HTML, and JSON documents from extracted data, supporting complex processing, per the key features description.
Cons
Lacks ability to render or execute JavaScript, making it ineffective for dynamic websites that load content client-side, a limitation not addressed in the README.
Written in Pascal and wraps Pascal Internet Tools, resulting in a smaller community and fewer resources compared to mainstream tools like Python libraries.
Compiling from source requires Lazarus IDE or build scripts, which can be cumbersome for users not familiar with Pascal, as noted in the installation section.
Frequently Asked Questions
Related Projects
jqCommand-line JSON processor
fxTerminal JSON viewer & processor
yqyq is a portable command-line YAML, JSON, XML, CSV, TOML, HCL and properties processor
visidataA terminal spreadsheet multitool for discovering and arranging data
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.