Google HCLS Data Harmonization
A mapping language and engine for converting complex, nested data between schemas, with extensibility via plugins.
What is Google HCLS Data Harmonization?
Whistle is a data transformation engine and mapping language that converts complex, nested data from one structure to another based on configuration files. It solves the problem of data schema conversion, particularly in healthcare for harmonizing data like FHIR, but is applicable to any domain requiring structured data mapping.
Developers and data engineers working on data integration, ETL pipelines, or healthcare data interoperability projects, especially those needing to map between different data schemas like HL7v2 to FHIR.
Developers choose Whistle for its concise, efficient mapping syntax, extensible plugin architecture, and robust tooling like LSP support and unit testing capabilities, making complex data transformations more maintainable and scalable.
Overview
This is an engine that converts data of one structure to another, based on a configuration file which describes how. There is an accompanying syntax to make writing mappings easier and more robust.
Use Cases
Best For
- Converting healthcare data between different standards like HL7v2 to FHIR
- Building ETL pipelines for nested or complex data structures
- Creating maintainable mapping configurations for data schema conversions
- Extending data transformation logic with custom plugins
- Writing unit tests for data mapping rules directly in the mapping language
- Integrating data harmonization into Java-based applications
Not Ideal For
- Simple, flat data transformations that could be handled with basic scripting or SQL
- Real-time streaming data pipelines requiring sub-second latency
- Teams lacking Java expertise or unwilling to manage JVM toolchain dependencies
Pros & Cons
Pros
Whistle's terse language is specifically designed for complex nested data, making transformations concise and easier to maintain compared to verbose alternatives like XSLT.
The architecture supports custom plugins for native transformations and integrations, as evidenced by first-party plugins for FHIR harmonization and unit testing.
Includes an LSP implementation for IDE support, a linter for code formatting, and documentation generation, which enhance productivity in mapping development.
Built-in plugins for FHIR code translation and reconciliation reduce the need for external tools in healthcare data interoperability projects.
Cons
Requires JDK 11 and Gradle 7 for building and running, imposing significant setup and maintenance overhead for teams not already invested in the JVM stack.
As a domain-specific language focused on data transformation, it lacks the broad community support and third-party libraries of general-purpose ETL tools like Apache NiFi.
The from-scratch rewrite to Whistle 2 introduces breaking changes, forcing existing Whistle 1 users to invest effort in migration, as noted in the README.
Frequently Asked Questions
Related Projects
OpenWearablesSelf-hosted platform to unify wearable health data through one AI-ready API.
NextGen Connect Integration EngineThe swiss army knife of healthcare integration.
FHIR ConverterConversion utility to translate legacy data formats into FHIR
Open eHealth Integration PlatformOpen eHealth Integration Platform
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.