Publications

LINSPECTOR WEB: A Multilingual Probing Suite for Word Representations

Max Eichler, Gözde Gül Şahin, Iryna Gurevych

We present LINSPECTOR WEB, an open source multilingual inspector to analyze word representations. Our system provides researchers working in low-resource settings with an easily accessible web based probing tool to gain quick insights into their word embeddings especially outside of the English language. To do this we employ 16 simple linguistic probing tasks such as gender, case marking, and tense for a diverse set of 28 languages. We support probing of static word embeddings along with pretrained AllenNLP models that are commonly used for NLP downstream tasks such as named entity recognition, natural language inference and dependency parsing. The results are visualized in a polar chart and also provided as a table. LINSPECTOR WEB is available as an offline tool or at https://linspector.ukp.informatik.tu-darmstadt.de.

LINSPECTOR: Multilingual Probing Tasks for Word Representations

Gözde Gül Şahin, Clara Vania, Ilia Kuznetsov, Iryna Gurevych

Despite an ever growing number of word representation models introduced for a large number of languages, there is a lack of a standardized technique to provide insights into what is captured by these models. Such insights would help the community to get an estimate of the downstream task performance, as well as to design more informed neural architectures, while avoiding extensive experimentation which requires substantial computational resources not all researchers have access to. A recent development in NLP is to use simple classification tasks, also called probing tasks, that test for a single linguistic feature such as part-of-speech. Existing studies mostly focus on exploring the information encoded by the sentence-level representations for English. However, from a typological perspective the morphologically poor English is rather an outlier: the information encoded by the word order and function words in English is often stored on a subword, morphological level in other languages. To address this, we introduce 15 word-level probing tasks such as case marking, possession, word length, morphological tag count and pseudoword identification for 24 languages. We present experiments on several state of the art word embedding models, in which we relate the probing task performance for a diverse set of languages to a range of classic NLP tasks such as semantic role labeling and natural language inference. We find that a number of probing tests have significantly high positive correlation to the downstream tasks, especially for morphologically rich languages. We show that our tests can be used to explore word embeddings or black-box neural models for linguistic cues in a multilingual setting. We release the probing datasets and the evaluation suite with https://github.com/UKPLab/linspector.

Citations

If you use LINSPECTOR, please cite us:

@inproceedings{eichler-etal-2019-linspector,
    title = "{LINSPECTOR} {WEB}: A Multilingual Probing Suite for Word Representations",
    author = {Eichler, Max  and
      {\c{S}}ahin, G{\"o}zde G{\"u}l  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations",
    month = nov,
    year = "2019",
    address = "Hong Kong, China",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/D19-3022",
    doi = "10.18653/v1/D19-3022",
    pages = "127--132",
}
@article{DBLP:journals/corr/abs-1903-09442,
  author    = {G{\"{o}}zde G{\"{u}}l Sahin and
               Clara Vania and
               Ilia Kuznetsov and
               Iryna Gurevych},
  title     = {{LINSPECTOR:} Multilingual Probing Tasks for Word Representations},
  journal   = {CoRR},
  volume    = {abs/1903.09442},
  year      = {2019},
  url       = {http://arxiv.org/abs/1903.09442},
  archivePrefix = {arXiv},
  eprint    = {1903.09442},
  timestamp = {Mon, 01 Apr 2019 14:07:37 +0200},
  biburl    = {https://dblp.org/rec/bib/journals/corr/abs-1903-09442},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}