Abstract
Using pretrained word embeddings has been shown to be a very effective way in improving the performance of natural language processing tasks. In fact almost any natural language tasks that can be thought of has been improved by these pretrained embeddings. These tasks range from sentiment analysis, translation, sequence prediction amongst many others. One of the most successful word embeddings is the Word2vec model proposed by Mikolov. We will be using the negative sampling objective of Mikolov to train our novel word embeddings, while replacing the way context words are formed. We model our context words not as a sum, but as a series of transitions from an initial uniform distribution using rate matrices. We call this novel word embedding modelWord2rate. Doing so allows us to gain greater insight by grounding our embeddings in a statistical foundation. We find that using a Taylor series of rate matrices allows our embeddings to have a variety of modes, where each mode has unique properties. We compare the different modes with exisiting embeddings like CBOW andCMOWand find them to be competitive across many tasks. We also gain novel properties from some of the modes which are useful in syntactic tasks. Hyperdimensional Computing is a model of computation proposed by Kanerva that involves representing concepts as database entries stored as key-value pairs. These key-value pairs are represented by hyperdimensional binary vectors and combined using addition and Hadarmard product. We show that this representation can be extended to our Word2rate embeddings. We introduce a novel model based in Hyperdimensional Computing called HyperRate. In HyperRate, we replace the usual binary vector key-value pair of Kanerva with a rate matrix-normalised vector pair. This allows us to incorporate statistical theory into Hyperdimensional Computing. We show that HyperRate is comparable to the usual key-value pair representation of Kanerva in both database retrieval and language recognition tasks. We conclude by linking Word2rate and HyperRate to natural language. We argue that natural language is not just words, but is a candidate to form the basis of human cognition. We then link natural language, Markov chains and with Hyperdimensional Computing to form a model of human cognition.