Abstract:

This article focuses on the study of word embedding, a feature-learning technique in natural language processing that maps words or phrases to low-dimensional vectors. Beginning with the linguistic theories concerning contextual similarities — “distributional hypothesis” and “context of situation”, this article introduces two ways of numerical representation of text: one-hot and distributed representation. In addition, this article presents statistical-based language models (such as co-occurrence matrix and singular value decomposition) as well as neural network language models (NNLM, such as continuous bag-of-words and skip-gram). This article also analyzes how word embedding can be applied to the study of word-sense disambiguation and diachronic linguistics.

Key words: natural language processing, text representation, word embedding