Research & Projects

Paper Link. Doctoral dissertation, defended July 2026.

Within the last decade, large language models (LLMs) have made remarkable strides in size and capabilities. Despite the performance improvements in their linguistic, stylistic, and social proficiency, our understanding of the inner workings of LLMs still lags behind, leaving the “black box” problem unsolved. One possibility is that models’ performance derives from their ability to identify surface-level patterns from the statistical distributions of the data, and that massive amounts of training data and parameters are necessary to identify and model these surface-level patterns. An alternative possibility, however, is that LLMs are able to extract the sub-surface structures responsible for the consequent patterns exhibited in the data, rather than solely modeling the statistical probabilities themselves. One area to examine such hypotheses lies within syntax: language is produced and perceived linearly, but its surface form is derived from the rule-governed structures that underpin language, namely syntax. This duality of deep, subsurface structure and its linearized output situates syntax as an ideal ground for LLM interpretability research to discover whether LLMs are “stochastic parrots” that acquire the superficial statistical distributions and probabilities of language, or if they are capable of acquiring the deep structures causally responsible for generating that linearized output.

This dissertation supports the latter hypothesis as our research finds not only that LLMs trained on linearized textual data are capable of extracting the deep, sub-surface hierarchical syntactic structures proposed by generative syntactic theories, but that these encoded representations of deep syntax are furthermore utilized by the models.

Paper Link. Presented at the Second BabyLM Workshop (2026), hosted at EMNLP.

Inspired by linguistic theory and recent work exploring linguistic biasing of neural networks, we propose a "Universal Grammar" (UG) pretraining paradigm for transformer-based neural networks, preparing the network to learn language more effectively, especially in low data volume scenarios. We accomplish this "universal" pretraining with a combination of synthetic language-like sequences created with a context-free grammar, biological sequence sources like protein amino acids, and material sources like crystal structures. Experiments with the BabyLM Challenge datasets show improved language acquisition in English, Dutch, and Chinese with our MissUniverse models, compared to not using "UG" pretraining. This pretraining paradigm has interesting implications for both the future of language models, and potentially how we think about human Universal Grammar.

Paper Link. Presented at the Conference on Natural Langauge Learning (2025), co-hosted with ACL.

The syntactic probing literature has been largely limited to shallow structures like dependency trees, which are unable to capture the subtle differences in sub-surface syntactic structures that yield semantic nuances. These structures are captured by theories of syntax like generative syntax, but have not been researched in the LLM literature due to the difficulties in probing these complex structures that have many silent, covert nodes. Our work presents a method for overcoming this limitation by deploying Hewitt and Manning (2019)’s dependency-trained probe on sentence constructions whose structural representation is identical in a dependency parse, but differs in theoretical syntax. If a pretrained language model has captured the theoretical syntax structure, then the probe’s predicted distances should vary in syntactically-predicted ways. Using this methodology and a novel dataset, we find evidence that LLMs have captured syntactic structures far richer than previously realized, indicating LLMs are able to capture the nuanced meanings that result from sub-surface differences in structural form.

Paper Link. Presented at the Society for Computation in Linguistics (2025).

Our research provides empirical support that LLM’s contextualized word embeddings have captured deep and hierarchical syntactic structure. In 2019, Hewitt and Manning found evidence that LLMs have captured features of structural dependency parses within their word representations; we extend this work by deploying their methodology on sentence structures that are differentiated only in a constituency-based account like Minimalism rather than a dependency-based account. Our novel work creates a dataset containing several carefully selected sentence structures whose dependency parses are identical, but whose constituency trees differ due to to the size of the complement (vP versus TP versus CP). We find differences in the probe’s predicted distances that can only be explained if the embeddings have indeed captured some Minimalist structural difference between these sentence types. The impact of our work helps to realize Linzen (2019)’s argument that linguists can further the study and understanding of LLMs and that the field of NLP provides novel tools for further linguistic research.

Awarded funding by the Center for Computational Language Sciences, this research is currently on-going.

This experiment identifies and test a novel garden path construction. Alternating unaccusative verbs are verbs that can appear in a transitive sentence in which the subject is the "do-er" (e.g. The acrobat dropped the rings where the rings are the things that dropped and the acrobat is the cause of this dropping), or in an intransitive sentence in which the subject is the "do-ee" (e.g. The acrobat dropped in which the acrobat is the thing that drops and the cause is not given). In this self-paced reading experiment, we test what providing an overt causative like make (e.g. The trapeze artist made the acrobat drop....) does for people's anticipated sentence structures. Our findings suggest that the overt causative make biases readers into anticipating a simpler single-causation intransitive completion (The trapeze artist made the acrobat drop) rather than a more complex double-causation transitive realization (e.g. The tapeze artist made the acrobat drop the rings). While more research is needed to clarify the full spectrum of results, the work presents a novel garden path construction that provides insights into how humans process causation events and their associated syntactic structure.

In this reproduction study, we retrain and test Amini, Liu, and Cotterell's (2023) Hexatagger dependency parse method using garden path structures as a means to test performance. Amini et al's left-corner shift-reduce dependency parse SOTA performance, but predicted tags independently and in parallel in order to improve efficiency. We test this parsing method on garden path constructions as these sentences require a full and global understanding of the sentence in order to properly parse. For example, the garden path sentence the old man the boat requires that the phrase the old man be reinterpreted two pieces in which the old constitutes the nominal subject and man constitutes the main verb, thus yielding the correct parse in which a group of old people man/operate the boat. Compared to the hexatagger's performance on the Penn Treebank's (PTB) dependency test set, our results found that errors stemming solely from critical misparses of garden path structures account for a 13.62% performance drop in unlabeled attachment scores (UAS) and a 14.78% performance drop in labeled attachment scores (LAS). Garden path sentences are grammatical structures that subvert typical sentence expectations and require a dep understanding of syntax in order to parse. Our work highlights the need to challenge models to recover deeper syntactic structure rather than rely on probable word orders and typical phrase groupings.