octillery
A Go package for database sharding that works with any ORM or raw SQL.
What is octillery?
Octillery is a Go library for implementing database sharding, designed to be compatible with any ORM that uses the `database/sql` interface or raw SQL queries. It provides a flexible, declarative approach to distributing database load across multiple shards, making it suitable for scaling applications with high data volumes.
Go developers building applications that require horizontal database scaling through sharding, particularly those already using ORMs like xorm, gorp, gorm, or dbr, or raw SQL.
Developers choose Octillery for its ORM-agnostic design and pluggable architecture, allowing integration into existing Go applications without being locked into a specific ORM or database technology, along with declarative YAML configuration for sharding setup.
Overview
Go package for sharding databases ( Supports every ORM or raw SQL )
Use Cases
Best For
- Scaling Go applications with high data volumes by distributing load across multiple database shards.
- Integrating database sharding into existing Go projects that use various ORMs like xorm, gorp, gorm, or dbr.
- Implementing custom sharding algorithms or database adapters beyond the preinstalled modulo and hashmap options.
- Managing database migrations and seed data import for sharded databases via CLI tools.
- Ensuring unique ID generation across all shards using a sequencer for consistent data distribution.
- Testing sharding configurations with SQLite3 in development while using MySQL in production.
Not Ideal For
- Projects needing out-of-the-box support for databases like PostgreSQL or MongoDB without custom adapter development
- Teams seeking fully automated sharding with dynamic rebalancing and built-in monitoring tools
- Applications using ORMs that bypass the standard `database/sql` interface, such as some NoSQL or graph database drivers
Pros & Cons
Pros
Works with any ORM implementing `database/sql`, including xorm, gorp, gorm, and dbr, as highlighted in the README's compatibility features, allowing integration into diverse codebases.
Supports custom sharding algorithms and database adapters, with preinstalled modulo and hashmap algorithms, enabling flexibility for specific use cases beyond defaults.
Sharding setup is defined in YAML, including shard key, algorithm, and sequencer usage, making it easy to manage and version control, as demonstrated in the usage examples.
Includes a CLI tool for schema migration powered by schemalex and seed import from CSV, streamlining database operations for sharded environments.
Cons
Only supports MySQL and SQLite3 out of the box; adding new databases requires implementing the DBAdapter interface and placing files in a plugin directory, as admitted in the 'How It Works' section.
Requires using the CLI to transpose import statements and writing detailed YAML configurations, which can be error-prone and time-consuming for large or legacy projects.
Relies on a custom vitess-sqlparser fork, which may introduce compatibility issues or lack updates compared to mainstream SQL parsers, potentially affecting long-term maintenance.
Frequently Asked Questions
Related Projects
vitessVitess is a database clustering system for horizontal scaling of MySQL.
PgwebCross-platform client for PostgreSQL databases
go-mysqla powerful mysql toolset with Go
prestPostgreSQL ➕ REST, low-code, simplify and accelerate development, ⚡ instant, realtime, high-performance on any Postgres application, existing or new, MCP server
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.