Differentiable Allophone Graphs for Language Universal Speech Recognition
(3 minutes introduction)
| Brian Yan (Carnegie Mellon University, USA), Siddharth Dalmia (Carnegie Mellon University, USA), David R. Mortensen (Carnegie Mellon University, USA), Florian Metze (Carnegie Mellon University, USA), Shinji Watanabe (Carnegie Mellon University, USA) |
|---|
Building language-universal speech recognition systems entails producing phonological units of spoken sound that can be shared across languages. While speech annotations at the language-specific phoneme or surface levels are readily available, annotations at a universal phone level are relatively rare and difficult to produce. In this work, we present a general framework to derive phone-level supervision from only phonemic transcriptions and phone-to-phoneme mappings with learnable weights represented using weighted finite-state transducers, which we call differentiable allophone graphs. By training multilingually, we build a universal phone-based speech recognition model with interpretable probabilistic phone-to-phoneme mappings for each language. These phone-based systems with learned allophone graphs can be used by linguists to document new languages, build phone-based lexicons that capture rich pronunciation variations, and re-evaluate the allophone mappings of seen language. We demonstrate the aforementioned benefits of our proposed framework with a system trained on 7 diverse languages.
