Analysis of EEG signals & cognitive activity in 3D modeling for a multi modal interface system

The human brain uses a complex network of billions of neurons functioning together. Through learning and experience, the human brain establishes millions of connections between neurons. Although the individual functions of neurons are known, how these neurons work in a network to perform cognitive processes still requires research and investigation. Every human being has their own learning rate to understand things and develop a skill-set. There is a need for adaptive systems to change the pace of learning according to the user's competency level to have an impact on performance. This thesis explores the application of EEG signals to estimate the cognitive activity of competent and novice users in a design task. The main goal of this thesis is to identify the user's competency using cognitive activities acquired through EEG signals in an MMIS. We developed a multimodal interface system (MMIS) (xDe-SIGN v2) that allows the users to model a 3D object using speech and gesture modalities. We used Microsoft speech recognition API to detect and decode speech input and a Leap Motion sensor and API for gesture recognition. Research questions are classified into 6 groups: input modality, psychophysiological analysis, cognitive activity, information processing, and competency classification. The research questions are investigated in four major parts: a) the design and development of an MMIS (Chapter 4) b) qualitative evaluation of MMIS using speech and gestures for 3D modelling (Chapter4) c) quantitative evaluation of MMIS using EEG signals (Chapter 6-9) d)classification of user's competency level for adaptive systems design (Chapter10). We tested the usability of the system in 2 sets of experiments with 12 participants. We used EEG signals to record users' mental states and cognitive activity. First, we analyzed users' cognitive activity in a unimodal system(using keyboard and mouse inputs), and then, in a multimodal system (using speech and gesture inputs). We used a combination of qualitative methods such as questionnaires and quantitative methods such as EEG bands, Power Spectral Density (PSD) and Functional Brain Networks (FBN) to investigate the cognitive activity of novice and competent users. Our qualitative evaluation results supported by questionnaires indicate that speech and gestures were well-coordinated in human to human communication but not in human-computer interaction (HCI). However, speech and gestures could be used in HCI with proper pre-processing and optimization techniques, as 90% of the participants completed the given task with reasonable precision in xDe-SIGN v2. Our quantitative evaluation results supported by EEG power analysis showed that there are significant differences in the alpha, beta, and theta band activity of novice and competent users. The results also suggest that physical actions such as drawing, manipulation and moving 3D models have a direct impact on users' performance defined by task completion time, as competent users performed 1.5 times more physical actions than novices who had twice as many conceptual actions as competent users. These findings suggest that the structure of cognitive actions is the key to high performance. Directional FBN analysis also indicates significant differences in cognitive activity in both novice and competent users in various states. The cognitive activity is more intense while the participants use speech and gestures for 3D modelling. The frontal region of the brain is mostly active, which indicates the use of short-term memory. The thesis provides experimental evidence that EEG based measures can be used as a quantitative metric to analyze cognitive activity in HCI. Finally, we have proposed a method to classify user's competency levels using convolutional neural networks and EEG signals. We obtained a classification accuracy of more than 88%, which shows the effectiveness of the proposed method. Thus, we conclude that the proposed method has a clear potential for developing state-of-the-art adaptive systems that can adapt to users' competency levels.