Chat reference can be evaluated both quantitatively and qualitatively to help determine best practices for staffing and quality of service. At one public university, both librarians and undergraduate student workers provide chat reference service to the campus community and the public. Coding a sample of 160 chat transcripts from the beginning, middle, and end of the term allowed us to see how well our librarians and student workers were providing this service. Two librarians coded our sample of chat transcripts by category and difficulty level of each question as well as the accuracy and quality of the answer. Referral and customer service behaviors were also examined. Analyzing these chat transcripts gave us a better idea of what kinds of questions are being asked by our patrons and how we could improve our librarian and student worker training to address those needs. In particular, we found that our student workers could improve their chat service quality by better understanding our interlibrary loan process and knowing when to refer a question to a subject librarian. After adjusting student training based on our findings, we repeated the process a year later and found substantial improvement in the quality of our student workers' chat reference interactions. This suggests that doing qualitative analysis of chat transcripts can be a worthwhile way to periodically assess and improve the quality of chat reference service.