Abstract
We decode conceptual representations from brain activities, including Functional magnetic resonance imaging (fMRI) measured Blood-oxygen-level-dependent (BOLD) signal and Functional near-infrared spectroscopy (fNIRS) measured oxygenated hemoglobin (HBO) signal. As the primary approach, we use linear regression to build a decoder to decode BOLD and HBO into various conceptual representations. For the fMRI, we reuse the available dataset. For the fNIRS, we design and conduct two experiments. We decode fMRI and fNIRS data into various text-derived word vectors and image and audio-derived multi-modal conceptual representations. We show that fMRI data has better decodability than fNIRS due to its high spatial resolution. Nevertheless, fNIRS can still provide comparable results at a low cost. Furthermore, fNIRS has a higher temporal resolution than fMRI, thus can be extended to a broader range of language research. We demonstrate that both fMRI and fNIRS data can be decoded into image-derived feature spaces or multi-modal feature spaces that include image features. In addition, we present words, pictures, and sounds stimuli to the subjects separately and analyzed the induced brain activity patterns. We observe that picture stimuli elicit the most significant brain response, followed by text and sound.