Perceptual and psychological mechanisms of gaze-based communication across virtual, artificial, and neurodiverse social interactions

<p>Eye gaze is central to social interactions as it fulfils a dual role: it can both convey and interpret social cues. However, these cues rarely occur in isolation, and they need to be integrated with information from other modalities, such as speech or gestures. Recent technological advancements have made it possible to investigate the dynamics of gaze-based communication through interactions with artificial agents. Building on these developments, the current thesis explores how neurodiversity as well as psychological and perceptual factors influence the interpretation and use of gaze as a communicative cue in non-verbal virtual interactions. Specifically, through three interconnected experiments, this work investigates how neurodiverse individuals (i.e., autistic and non-autistic people) distinguish between communicative and private gaze, and how both beliefs <em>about </em>and the appearance <em>of </em>artificial agents shape the interpretation of their gaze cues.</p> <p>Delays or difficulties in gaze-based joint attention have been considered key behavioural markers of autism. However, studies using objective measures of behaviour or attention have not consistently identified or explained ‘impairments’ in autism. To address this challenge, the first experimental study of this thesis (<strong>Chapter 2</strong>) explores the perceptual dynamics of neurodiverse gaze-based communication. It investigates how eye contact, repeated averted gaze shifts, and gaze duration influence the perception of communicative intent. In a semi-interactive web-based task, autistic and non-autistic participants evaluated whether an on-screen agent was engaging in private inspection or requesting an object. The appearance of the agent varied between human-like and robotic. The findings suggest that both groups rely on similar gaze cues to infer communicative intent, challenging the traditional assumption that autistic people have a diminished sensitivity to gaze-based social cues. However, the agent's appearance influenced the perception of communicative intent and subjective experiences differently for autistic and non-autistic people, in that autistic people perceived eye contact with the robotic agent as more communicative than the humanlike agent and perceived the robotic agent also more positively than the human-like agent.</p> <p>Building on the findings presented in <strong>Chapter 2</strong>, the second study (<strong>Chapter 3</strong>) extends the experimental paradigm to investigate how user beliefs shape the interpretation of gaze dynamics. By manipulating, within-subjects, whether participants believed the agent was human- or AI-controlled, the study examined how perceived human agency influences the interpretation of ambiguous gaze cues. We found that believing the agent to be human-controlled increased perceived communicativeness when gaze cues were ambiguous, but this effect was attenuated in the second block of the experiment. These results suggest that perceived human agency influences the interpretation of ambiguous gaze cues. However, learned expectations about an agent's behaviour can override initial assumptions.</p> <p>The third experimental study (<strong>Chapter 4</strong>) expands this investigation into a fully interactive virtual reality setting, incorporating eye- and motion-tracking to explore (1) how gaze cues are perceived in the context of hand gestures and (2) whether user beliefs (top-down) interact with agent appearance (bottom-up) in shaping perception. In a joint attention game with a virtual partner, participants both initiated and responded to joint attention bids using hand gestures, while coordination could be implicitly facilitated by attending to the agent's eye gaze. Each participant interacted with both a human-like and a robotic virtual agent and was informed that their partner—regardless of appearance—was either human- or AI-controlled. While gaze cues were consistently used for coordination, subjective experiences varied based on both agent appearance and participants' beliefs about whether the agent was controlled by a human or AI. The findings suggest that AI-driven agents are approached more like humans than mere computational systems, yet, their physical form influences social expectations and interaction dynamics.</p> <p>Throughout the PhD project, I interacted with neurodivergent participants gaining valuable insights into neurodivergent experiences of the research process. These insights motivated the final study (<strong>Chapter 5</strong>) in which we partnered with autistic people to revise and validate the self-report measure of autistic traits used throughout this thesis (‘The Comprehensive Autistic Trait Inventory’ or CATI). Our revisions aimed to enhance accessibility, reduce stigma, and better capture the diversity of autistic experiences. The revised inventory (CATI-R) was validated in a large sample, demonstrating strong psychometric properties. We also developed guidelines for creating/designing future self-report inventories of subclinical neurodivergent traits.</p> <p>Together, these four empirical studies offer new insights into the mechanisms of neurodiverse gaze-based communication, the role of agency expectations in social interactions with artificial agents, and the importance of inclusive approaches in science. By integrating perspectives from psychology, cognitive science, and human– computer interaction, this thesis contributes to a more nuanced understanding of joint attention and social cognition in neurodiverse human and artificial contexts.</p>