LIN389C is a research course for students who work in computational linguistics. It is aimed at students with advanced knowledge in natural language processing and machine learning techniques who are doing research in the area. In the course, we discuss current research by course participants, review foundational knowledge that is relevant to participants' research, and talk about big-picture issues and current research in field.
Course: LIN389C Research in Computational Linguistics. Unique number: 39050
Time and Location: Mondays 9:15-10:10, 10:45-12, RLP 4.422
Instructor: Katrin Erk, email katrin.erk@utexas.edu, office RLP 4.734
Office hours: Mondays 2-3, Wednesdays 10-11, 4--5
Course Canvas page: https://utexas.instructure.com/courses/1388268
Jan 22
We talk about topics to cover in this semester's class. We'll review the topic list that you compiled at the end of the last semester (Google doc: Link on Canvas). But also please bring suggestions for topics that are relevant to your research.
We also do a round-table.
Jan 29: Philosophy of LLMs
9:15 We read Harvey and Kyle's paper: Are Language Models More Like Libraries or Like Librarians? Bibliotechnism, the Novel Reference Problem, and the Attitudes of LLMs
10:30 Venkat is giving a practice job talk.
Feb 5: Fact-checking and entailment
We read: Jean-Philippe Bernardy, Stergios Chatzikyriakidis, A Type-Theoretical system for the FraCaS test suite: Grammatical Framework meets Coq.
Note: You do not need to look up all the methods they use in the paper, we'll discuss them in class. Just get an overall impression of the approach. If you do want to dive in further, here is information about GF, and information about Coq.
Feb 12: Entailment, and neural-symbolic methods
We are reading: Abulhair Saparov, Tom M. Mitchell, Towards General Natural Language Understanding with Probabilistic Worldbuilding
Also: Kyle gives a practice talk
We also talk some more about topics for the semester
Feb 19: Entailment in LLMs
We are reading: Abulhair Saparov & He He, Language models are greedy reasoners: A systematic formal analysis of chain-of-thought.
Feb 26: LLMs and language acquisition
We are reading: Wentao Wang, Wai Keen Vong, Najoung Kim, Brenden M. Lake: Finding Structure in One Child's Linguistic Experience
Long update Juan Diego
Mar 4 : LLMs and language acquisition
We are reading: Chengxu Zhuang, Evelina Fedorenko, Jacob Andreas: Visual Grounding Helps Learn Word Meanings in Low-Data Regimes
Mar 11 Spring break
Mar 18: LLMs and language acquisition
We are reading: Hassan Shahmohammadi, Maria Heitmeier, Elnaz Shafaei-Bajestan, Hendrik P. A. Lensch, and Harald Baayen, Language with Vision: a Study on Grounded Word and Sentence Embeddings
Long update Yating
Mar 25 LLMs and theories of meaning
Talk by Ryan Nefdt
Long update Hongli
April 1: Situation knowledge in LLMs: Theory of Mind
We are reading:
An overview paper: Ma, Sansom, Peng, and Chai 2023, Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models
and a technical paper on the same topic: Shapira, Levy, Alavi, Zhou, Choi, Goldberg, Sap, and Shwartz 2024, Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models
Long update: Venkat
April 8 Situation knowledge in LLMs: Theory of Mind
We are reading: I Cast Detect Thoughts: Learning to Converse and Guide with Intents and Theory-of-Mind in Dungeons and Dragons (Zhou et al. 2023 ACL) https://aclanthology.org/2023.acl-long.624.pdf
Long update Will
April 15
Talk by Hans Uszkoreit: Large language models from an applied perspective
Too much is being spoken and written on LLMs these days. And yet, some of the most burning questions are still waiting for satisfactory answers. I will talk about LLMs from an applied perspective. We have investigated several models with respect to the role they could play in modern business and production processes and made some useful observations for our own work on training data.
I will also touch on these questions:
· Are LMMs over-hyped or underrated?
· How many will be needed?
· Can they be made reliably truthful, creative, explainable, consistent?
· How far away are they from AGI?
· For which jobs will they be successful?
· Can they become problem solvers?
LLMs behave a little closer to people than traditional NLP systems with respect to various psycholinguistic phenomena such as garden paths and center-self-embedding. I will show examples. But could the transformer architecture indeed exhibit a higher degree of cognitive plausibility?
I do not expect to really cover all these topics in a single talk. I am fully satisfied if I manage to add some thoughts and insights that have shaped our perspective on this first generation of broad AI models.
Hans Uszkoreit is Scientific Director at the German Research Center for Artificial Intelligence (DFKI) in Berlin. Uszkoreit has worked in AI for more than 40 years as professor, research leader and entrepreneur. He has (co-)authored more than 250 publications in AI and computational linguistics and he is Member of the European Academy of Sciences. 1984, he received his PhD in Linguistics from UT Austin and worked several years in AI research at SRI and Stanford U. He was then professor for computational linguistics in Saarbrücken and honorary professor at TU Berlin. For three years, he worked in Beijing as Chief AI Advisor for Lenovo and as entrepreneur. He co-founded several AI start-ups, the last one dedicated to LLMs.
Long update Sooji
April 22 Metalinguistic Abilities of LLMs
We are reading: Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, Luke Melas-Kyriazi: A Benchmark for Learning to Translate a New Language from One Grammar Book
Long update Yejin
April 29: Wrap-up and preparation for next semester
We do:
A big round-table discussion: What are everybody's research plans for the summer?
Discussion of possible topics for next semester
Final paper due date: Friday May 4, end of day.
To teach about, encourage, and give students time for research. Also to encourage discussion and collaboration among students interested in the same subfield.
There are six constituencies who will not be treated exactly equally in the course because their needs are different:
first-year students: need general context and help on first research projects and first-year papers
second-year students: need feedback on the research they have done, prepare for further research, and write second-year papers
third-year students: need to write their Qualifying Papers, write grant proposals
post-candidacy pre-proposal students: need to write and present their dissertation proposals
post-candidacy dissertation students: need to write dissertations, get feedback
students from other departments: needs will vary
The course consists of two main parts: a research seminar, and discussion of ongoing student research.
Research seminar: Topics and readings will be given under Schedule.
Format: Each student prepares 3 bullets about what you like about the paper, 3 bullets about what you don’t, plus questions/other comments
We'll randomize the order in which people get to discuss their bullet poinst
Discussion of ongoing student research:
Round-table: short presentations by all participants about their current research. This will happen almost every week.
On-going research: longer presentations (30 minutes or an hour including discussion), students, faculty, auditors if they wish.
Dissertation proposal presentations.
Dissertation progress talks.
Practice talks for conference presentations.
First-year students: Attend all classes/activities. Talk about research.
First semester, submit literature discussion sketch halfway through the semester (see schedule), submit literature discussion at end of semester.
Second semester, submit first-year paper draft halfway through the semester (see schedule), submit first-year paper at end of semester.
Second-year students: Attend all classes/activities. Talk about research.
First semester, submit research discussion draft halfway through the semester (see schedule), submit research discussion paper at end of semester.
Second semester, submit second-year paper draft halfway through the semester (see schedule), submit second-year paper at end of semester.
Third-year students: Attend all classes/activities. Talk about research.
First semester, submit QP proposal halfway through the semester (see schedule), submit QP progress report at end of semester.
Second semester, submit QP draft halfway through the semester (see schedule), submit QP at end of semester.
Post-candidacy, pre-proposal students: Attend all presentations. Talk about research.
Dissertation-writing students: Attend all presentations, give at least one presentation during semester on doctoral research).
Students from other departments: a course project, with 2 documents: intermediate report (2-3 pages), final report (8 pages), deadlines as given in the schedule.
Grading will be based on the course requirement listed above.
This course does not have a final exam or midterm exam.
The course will use plus-minus grading, using the following scale:
A >= 93%
A- >= 90%
B+ >= 87%
B >= 83%
B- >= 80%
C+ >= 77%
C >= 73%
C- >= 70%
D+ >= 67%
D >= 63%
D- >= 60%
The University of Texas at Austin provides upon request appropriate academic accommodations for qualified students with disabilities. Please contact the Division of Diversity and Community Engagement, Services for Students with Disabilities, 5121-471-6259.
A student who misses an examination, work assignment, or other project due to the observance of a religious holy day will be given an opportunity to complete the work missed within a reasonable time after the absence, provided that he or she has properly notified the instructor. It is the policy of the University of Texas at Austin that the student must notify the instructor at least fourteen days prior to the classes scheduled on dates he or she will be absent to observe a religious holy day. For religious holy days that fall within the first two weeks of the semester, the notice should be given on the first day of the semester. The student will not be penalized for these excused absences, but the instructor may appropriately respond if the student fails to complete satisfactorily the missed assignment or examination within a reasonable time after the excused absence.
Occupants of buildings on The University of Texas at Austin campus are required to evacuate buildings when a fire alarm is activated. Alarm activation or announcement requires exiting and assembling outside. Familiarize yourself with all exit doors of each classroom and building you may occupy. Remember that the nearest exit door may not be the one you used when entering the building. Students requiring assistance in evacuation shall inform their instructor in writing during the first week of class. In the event of an evacuation, follow the instruction of faculty or class instructors. Do not re-enter a building unless given instructions by the following: Austin Fire Department, The University of Texas at Austin Police Department, or Fire Prevention Services office. Information regarding emergency evacuation routes and emergency procedures can be found at http://www.utexas.edu/emergency.
If you are worried about someone who is acting differently, you may use the Behavior Concerns Advice Line to discuss by phone your concerns about another individual's behavior. This service is provided through a partnership among the Office of the Dean of Students, the Counseling and Mental Health Center (CMHC), the Employee Assistance Program (EAP), and The University of Texas Police Department (UTPD). Call 512-232-5050 or visit http://www.utexas.edu/safety/bcal