Towards a General-Purpose Linguistic Annotation Backend

  • Graham Neubig Carnegie Mellon University
  • Patrick Littell National Research Council Canada
  • Chian-Yu Chen Carnegie Mellon University
  • Jean Lee Carnegie Mellon University
  • Zirui Li Carnegie Mellon University
  • Yu-Hsiang Lin Carnegie Mellon University
  • Yuyan Zhang Carnegie Mellon University


Language documentation is inherently a time-intensive process; transcription, glossing, and corpus management consume a significant portion of documentary linguists’ work. Advances in natural language processing can help to accelerate this work, using the linguists’ past decisions as training material, but questions remain about how to prioritize human involvement.

In this extended abstract, we describe the beginnings of a new project that will attempt to ease this language documentation process through the use of natural language processing (NLP) technology. It is based on (1) methods to adapt NLP tools to new languages, based on recent advances in massively multilingual neural networks, and (2) backend APIs and interfaces that allow linguists to upload their data (§2). We then describe our current progress on two fronts: automatic phoneme transcription, and glossing (§3). Finally, we briefly describe our future directions (§4).