Speech-Based Idiom Classifier

This project takes raw speech audio through transcription and language modeling to produce a literal-or-idiomatic classification in one system.

The full pipeline achieved 80.0% end-to-end accuracy on 459 custom audio recordings.

Written idiom classification assumes clean text, but a speech-based system must absorb transcription errors, acoustic variation, and ambiguity before it can reason about phrase meaning.

I built the complete audio-to-label pipeline, fine-tuned the language model, established a classical baseline, recorded the evaluation set, and performed per-class error analysis on the combined system.

OpenAI Whisper transcribes the audio before a fine-tuned BERT classifier predicts whether the target phrase is literal or idiomatic. Audio processing uses librosa, and Gradio provides an interactive interface.

I fine-tuned BERT on 39,924 examples from the MAGPIE dataset, reaching a weighted F1 of 0.92 and outperforming a TF-IDF plus logistic-regression baseline by 14 percentage points.

The largest gap appeared between clean-text classifier performance and end-to-end audio performance. Error analysis identified Whisper transcription noise as the dominant failure mode rather than the downstream classifier.

The project demonstrated why component metrics are not enough for multi-stage ML systems. End-to-end evaluation and error attribution are necessary to identify which stage actually limits user-facing performance.