Priming In Human Brain and in Language Models

Recent research has shown that large language models (LLMs) simulate human-like grammatical performance without explicit grammatical instruction (1). Yet the nature of these grammatical representations and their similarity to representations learnt by humans is unclear. Assessing the degree of similarity between human and LLM grammatical performance can lead to two key outcomes, benefiting fundamental research in both neuro-cognitive and AI disciplines.

From the neuro-cognitive perspective, we aim to provide a mechanistic account of how grammatical regularities can be learnt by humans without previously assumed in-built abstract hierarchical syntax, thus expanding and challenging decades of cognitive theory. In turn, AI systems need insights from human data to guide their design to ultimately match human grammatical performance for translational applications in human-AI interaction.

Importantly, this project will generate unique and valuable open-source data and computational methods that will align LLM and human performance metrics and facilitate transformative bioscience research.

To achieve this, we will, for the first time, compare indicators of syntactic representation encoded in LLMs to those encoded in the human cognitive system and detectable through cortical activity (EEG) and behaviour (eye-tracking). In doing so, we will advance fundamental research in multiple disciplines (LLMs, psycholinguistics, cognitive neuroscience) and create tools for translational application in human-AI interaction technologies.

Paradigm: We will compare human reading metrics and LLM performance (conditional predictions) in a structural priming task, given the same sentences. The prime and target sentences used will have direct-object syntax ("The man gave a bottle to the boy") and prepositional-object syntax ("The man gave the boy a bottle"). We expect facilitated processing of target sentences after structurally similar primes, replicating previous human and LLM findings (Experiment 1).

To examine the nature of the syntactic representations in humans and LLMs, we will manipulate relative frequency and plausibility of the structures and gauge the degree of similarity in human and LLM responses. (Experiments 2,3).

Human priming effects will be assessed by per-word measures of processing difficulty across conditions: gaze durations and scan paths in eye-tracking, EEG evoked-response amplitudes. In LLMs, processing difficulty will be derived from theoretically comparable measures of surprisal. Human and LLM data will be directly compared using multivariate and machine learning approaches in a novel cross-disciplinary analysis pipeline.