Abstract
The quest for biologically inspired learning is driven, not just by the desire to explain experimentally-observed properties of biological neural networks, but also by the hope of discovering more efficient methods to train neural networks or providing insights for high-level general intelligence. Though deep learning has achieved significant breakthroughs in various areas, it demands heavy computational power (MWs in AlphaGo v.s. 20W in human brain) and the extent to which it explains learning in biological networks remains undetermined. From a practical standpoint, biological plausibility need not be enforced when building an AI system. However, integrating it into learning is preferred to formalize a comprehensive agent with human-level intelligence, as human is the only instance of general AI now. For example, the realization and implementation of binary spikes between biological neurons and the local update rules, such as the spike-timing dependent plasticity, could lead to more efficient training of massive neural networks with faster inference speed but consume less energy and memory storage. In this thesis, we will give a brief overview on the intertwined history between AI and neuroscience, explore the basics in deep learning, and then present our work on biologically inspired algorithms for neural networks, which provides theoretical support for the systematic design of simple local learning rules for massive networks and the implementation of energy-efficient neuromorphic hardware. We first study the missing value imputation problem in heterogeneous sensor networks, providing an overview on deep neural networks. Second, we propose a learning algorithm, variational probability flow learning, to train deep Boltzmann machines for generative tasks. It demonstrates a local update rule whose asymmetric version gives rise to spike-timing-dependent plasticity, an experimentally observed update rule in biological neurons which reveals that changes in synaptic weights of some neurons depend on the timing difference between presynaptic and postsynaptic. This local update rule only depends on the states and firing rates of the adjacent neurons. Unlike contrastive divergence, there is no need for Gibbs confabulations; and unlike backpropagation, alternating feedforward and feedback phases are not required. Finally, we introduce a novel local learning rule in spiking neural networks, the McCulloch-Pitts network, for continuous-time sequence learning. It further explores the asymmetry connections in binary neurons to directly give rise to the spike-timing-dependent plasticity rule. As a consequence, the McCulloch- Pitts network could generalize from static Hopfield networks to robustly memorize multiple spatiotemporal patterns of binary vectors. We regard this thesis as an initial attempt towards biologically plausible learning. Bringing together advanced variational inference techniques and homotopy type theory, our long-term goal is to bridge the gap between neuroscience and machine learning to design novel computational architectures that are different from current deep neural networks but more suitable for sequence learning tasks. The local learning rules and similarity to biological neurons can provide insights for developing novel brain-like chips that are as powerful as present processing units in specific real-world problems but consume much less energy.