Whisper-AT
A joint audio tagging and speech recognition model that adds audio event detection to OpenAI Whisper with minimal computational overhead.
What is Whisper-AT?
Whisper-AT is an extension of OpenAI's Whisper speech recognition model that adds audio event tagging capabilities. It can transcribe speech while simultaneously identifying non-speech audio events like music, animal sounds, or environmental noises from 527 AudioSet categories. The model solves the problem of needing separate systems for speech recognition and audio content analysis by providing both functions in a single efficient model.
Researchers and developers working on audio understanding applications, particularly those needing both speech transcription and environmental sound recognition in noisy real-world audio.
Developers choose Whisper-AT because it provides audio event detection with negligible computational overhead compared to standalone Whisper, uses the same familiar API, and offers insights into how robust ASR models represent non-speech sounds.
Overview
Code and Pretrained Models for Interspeech 2023 Paper "Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong Audio Event Taggers"
Use Cases
Best For
- Adding audio event detection to existing Whisper-based transcription pipelines
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
