How to fine-tune HyenaDNA on my own genomic dataset?

You need to create a custom dataset class and dataloader, similar to the GenomicBenchmarks example, and adjust configs in `configs/experiment/`. The README provides steps but requires familiarity with Pytorch Lightning and Hydra.

HyenaDNA vs Nucleotide Transformer: which is better for long sequences?

HyenaDNA excels at ultra-long contexts (up to 1M tokens) with single nucleotide resolution, while Nucleotide Transformer is optimized for shorter ranges. HyenaDNA includes pretrained weights for long sequences, but integration requires using provided Docker images or custom setup.

What GPU do I need for the HyenaDNA large model?

For the large-1m model, you'll need high-end GPUs like an A100, as noted in the HuggingFace GPU requirements. The Colab example recommends paid tier for 1M sequences, and local setups require similar hardware.

Can HyenaDNA handle RNA or protein sequences?

No, HyenaDNA is specifically designed for DNA sequences at single nucleotide resolution, using a tokenization for DNA bases. The README focuses on genomic data, and adapting it for other biomolecules would require significant modification.

Is there pretrained support for non-human genomes?

Currently, pretrained weights are only available for the human reference genome (hg38). For other species, you'd need to pretrain from scratch using custom data, as mentioned in the 'Pretraining on your own data' section.

How to extract embeddings from a HyenaDNA model?

Use the provided `huggingface.py` script or the standalone code for embeddings, which loads pretrained weights without the model head. The README includes examples for inference and getting logits or embeddings.

What are the main downstream tasks HyenaDNA is tested on?

It's tested on tasks like GenomicBenchmarks classification, Nucleotide Transformer datasets, chromatin profiling, and species classification, with configs and commands provided in the README for replication.

Open-Awesome

HyenaDNA

Apache-2.0Assembly

A long-range genomic foundation model that processes DNA sequences up to 1 million nucleotides at single nucleotide resolution.

Visit Website GitHub

791 stars107 forks0 contributors

What is HyenaDNA?

HyenaDNA is a long-range genomic foundation model that processes DNA sequences up to 1 million nucleotides at single nucleotide resolution. It is designed to handle ultra-long genomic contexts, enabling tasks like classification, prediction, and in-context learning on DNA data. The model is pretrained on the human reference genome and can be fine-tuned for various downstream genomic applications.

Target Audience

Genomics researchers, bioinformaticians, and machine learning practitioners working with DNA sequence data who need to model long-range dependencies and fine-grained nucleotide interactions.

Value Proposition

HyenaDNA uniquely combines extreme sequence length capability (up to 1M tokens) with single nucleotide resolution, outperforming existing models on long-range genomic tasks. Its open-source implementation and pretrained weights lower the barrier to applying deep learning to genomics.

Overview

Official implementation for HyenaDNA, a long-range genomic foundation model built with Hyena

Use Cases

Best For

Analyzing ultra-long DNA sequences (e.g., full chromosomes)
Fine-tuning on genomic classification tasks like enhancer prediction
Species classification from DNA sequences
Chromatin profile prediction
In-context learning experiments on genomic benchmarks
Pretraining custom genomic foundation models

Not Ideal For

Projects needing out-of-the-box genomic analysis without deep learning expertise
Applications focused exclusively on short DNA sequences (under 1k bases) where long-range modeling adds unnecessary complexity
Teams without access to high-performance GPUs (e.g., A100/V100) for training or inference on large models
Researchers requiring extensive, beginner-friendly documentation and pre-built pipelines for common genomic workflows

Pros & Cons

Pros

Ultra-Long Context

Handles sequences up to 1 million tokens, enabling analysis of full chromosomes or large genomic regions, as highlighted in the intro and key features.

Single Nucleotide Resolution

Processes DNA at individual base level for fine-grained analysis, allowing precise genomic feature detection as stated in the description.

Pretrained Weights Available

Offers multiple model sizes on HuggingFace pretrained on hg38, reducing initialization time for downstream tasks, with GPU requirements specified.

Flexible Fine-Tuning

Supports various downstream tasks like species classification and chromatin profiling, with example configs and dataloaders provided in the README.

Cons

Complex Setup Process

Requires Docker or manual installation of dependencies like Flash Attention, and familiarity with Pytorch Lightning and Hydra, making onboarding challenging.

High Resource Demands

Large models need powerful GPUs (e.g., A100 for 1M sequences in Colab paid tier), and pretraining or fine-tuning can be computationally intensive.

Limited Documentation and Maturity

The repo is described as a 'work in progress,' with users needing to dig into code for custom dataloaders, and experimental features like bidirectional implementation are not fully supported.

Steep Learning Curve

Assumes advanced ML knowledge, with custom configs and dataloaders required for new datasets, as noted in sections on setting up downstream experiments.

Frequently Asked Questions

Related Projects

Evo

Biological foundation modeling from molecular to genome scale

Stars1,519

Forks177

Last commit3 months ago

Nucleotide Transformer

Foundation Models for Genomics & Transcriptomics

Stars887

Forks95