Abstract
Graphs and sentences play important roles in the field of natural language processing (NLP). In this thesis, we present some of our recent works on graph representation learning and sentence representation learning for NLP applications. Graph convolution networks (GCNs) have been widely used for NLP tasks. However, vanilla GCNs exclude efficient non-local information propagation as they follow a local information aggregation scheme. To address this problem, we introduce Lightweight, Dynamic Graph Convolutional Networks (LDGCNs), which capture richer non-local interactions by synthesizing higher order information. Our following work addresses another issue on the effective extraction of relevant information from graphs for NLP tasks. Unlike previous works which relied on a rule-based pruning method, we propose Attention Guided Graph Convolutional Networks (AGGCNs), which can automatically learn how to selectively focus on the relevant sub-structures useful for specific tasks. Empirically, our models achieve better performance than baselines. We also researched on representation learning of sentence. BERT has achieved new state-of-the-art performance on many NLP tasks. However, it is inefficient for sentencepair tasks. We propose a lightweight extension of BERT and a novel self-supervised learning objective to derive meaningful sentence embeddings in an unsupervised manner. Empirically, the proposed method outperforms other unsupervised baselines and achieves competitive performance with supervised methods on various tasks.