QuackOSM
An open-source Python and CLI tool for reading OpenStreetMap PBF files using DuckDB and exporting to GeoParquet.
What is QuackOSM?
QuackOSM is an open-source Python library and command-line tool designed to read and process OpenStreetMap Protocol Buffer (PBF) files efficiently. It uses DuckDB with its Spatial extension to parse OSM data, enabling fast filtering, conversion to GeoParquet format, and direct export to DuckDB databases. It solves the problem of slow, single-threaded OSM data processing by leveraging DuckDB's multithreading and spatial capabilities.
Geospatial developers, data engineers, and researchers who work with OpenStreetMap data and need to perform large-scale extractions, filtering, or conversions into analytical formats like GeoParquet.
Developers choose QuackOSM for its performance advantages over traditional tools like GDAL, its seamless integration with DuckDB for spatial queries, and its ability to automatically fetch and filter OSM data without manual preprocessing.
Overview
QuackOSM: an open-source Python and CLI tool for reading OpenStreetMap PBF files using DuckDB
Use Cases
Best For
- Converting large OSM PBF files to GeoParquet for cloud analytics
- Filtering OpenStreetMap data by tags and geometry in a single step
- Automating OSM data downloads for specific regions or cities
- Integrating OSM data into DuckDB workflows for spatial analysis
- Building ETL pipelines for geospatial applications using Python
- Replacing GDAL for faster, multithreaded OSM data processing
Not Ideal For
- Environments with strict disk space limitations, as QuackOSM requires ~10x free disk space relative to PBF file size for intermediate processing.
- Real-time applications needing immediate OSM data access without caching or file I/O latency.
- Projects solely requiring simple OSM data extraction without DuckDB or GeoParquet integration, where lighter tools like Osmium suffice.
- Teams operating in Python <3.10 environments, since QuackOSM mandates Python 3.10 or higher.
Pros & Cons
Pros
Utilizes DuckDB's Spatial extension for parallel PBF reading, explicitly benchmarking faster than single-threaded GDAL, as stated in the README.
Can download OSM PBF files automatically for geometries or text queries from sources like Geofabrik, eliminating manual data acquisition steps.
Exports directly to GeoParquet format, facilitating seamless use in modern data lakes and analytics stacks like DuckDB or PySpark.
Supports combined geometry and tag-based filtering in one step, including negations and wildcards, without pre-clipping with ogr2ogr.
Cons
Demands substantial disk space (10x PBF file size) and memory for intermediate files, as documented in the 'Memory usage' section, limiting use on constrained systems.
Relies on numerous Python packages (e.g., DuckDB with Spatial, geoarrow-pyarrow, geopandas) and specific versions, increasing installation friction and potential conflicts.
The intelligent caching system, while reducing repeat computations, adds complexity and storage overhead for frequently changing OSM data or ad-hoc queries.
Frequently Asked Questions
Related Projects
osmnxDownload, model, analyze, and visualize street networks and other geospatial features from OpenStreetMap.
prettymapp🖼️ Create beautiful maps from OpenStreetMap data in a streamlit webapp
overpass-wrapperPython bindings for the OpenStreetMap Overpass API
osmapiPython wrapper for the OpenStreetMap API
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.