Executive Summary : | Perceptual Brain Decoding (PBD) refers to an approach that uses the responses from the brain (evoked by different perceptual stimuli) to identify the original stimulus or some of its characteristics. These types of applications require an external perceptual stimulus (visual, in this case). As compared to various other EEG based applications, e.g. identification of mental states corresponding to different emotions, intentions to perform a task (e.g. in case of motor imagery), attention, eye movements, decision making in certain tasks, mental load, stress etc., the problem of perceptual brain decoding with EEG is relatively less explored. However, with effective use of deep learning methods, there have been a handful of recent works reported for PBD which use deep learning methods on EEG signals, to perform identification / reconstruction of the input stimuli or imagination related to the input stimuli. While this is quite encouraging, considering that this is a new area, naturally, there is much scope of improvement in terms of performance of such methods, and variation in the inputs that is considered, the generalizability of such methods on larger datasets, and on analysis of such methods. In this work, we target the PBD using deep learning methods, across three types of visual stimuli: a) images (static stimuli), b) video (dynamic stimuli) (without audio), and c) Imagined handwritten characters and numbers, by capturing EEG data under well-defined protocols involving each of the stimuli modality. Note that the decoding for of the three cases is treated independently. Since this work involves decoding with visual stimuli, we term this as visual brain decoding (VBD). For each type visual modality, we would including different categories of stimuli (e.g. images of different objects, videos of different types, and characters and words). After suitable pre-processing of the EEG signals, the machine learning and deep learning methods will take the input as the EEG, for performing two tasks i) classification of the input stimuli (e.g. into different types of images, or videos, or characters, respectively), and ii) reconstruction of the input stimuli (e.g. reconstruction of images, videos, and characters and words, respectively) In addition to the development of the deep learning methods for achieving good quality classification and reconstruction, the work would also involve some analysis for channel localization, connectivity, feature importance etc. which can provide important insights from a neuroscience perspectives and can also serve to enable low-cost devices. |