Large Language Models, Artificial Intelligence and the Future of Law
Session 9: How will AI change the legal profession?
Session 9: How will AI change the legal profession?
An Experiment
Objective
How good are current LLMs compared to humans on legal tasks?
Which LLMs are good for which kinds of legal tasks?
What prompts can we use in LLMs to do well on human tasks?
Design
Case Development:
A synthetic legal case, simulating a real-life consumer complaint, was constructed to serve as the basis for the tasks.
A law student with practical experience in consumer law was responsible for creating initial drafts for each legal task, which served as the human-generated content baseline.
LLM Intervention:
Different LLMs, including ChatGPT 3.5, ChatGPT 4, Gemini, Claude 3, and LLaMA 2, were tasked with generating outputs for the same set of legal tasks using tailored prompt engineering.
Each LLM received a general but specific prompt designed to elicit comprehensive responses for each task.
Blind Evaluation:
The outputs from the human expert and the LLMs were anonymized and randomized.
A total of six drafts (one from each LLM and the human) were presented for each of the five tasks, without revealing the source of the draft to the evaluators.
Assessment Criteria:
Respondents, who were other law students or practitioners, rated each draft based on accuracy, helpfulness, and comprehensiveness on a 5-point Likert scale.
Additionally, respondents were asked to guess whether each draft was generated by a human or an LLM and to provide reasons for their guesses.
Prompt Engineering Challenge:
At the end of the evaluation, respondents were asked to suggest how they would engineer prompts to optimize LLM performance for each task, based on their insights from the evaluation.
Sample:
17 Respondents. All students of the course.
On 04/02/2022 (4th February, 2022), at around 2:30 pm, I visited the Jeevan Nagar branch of ABC hotels, located in Bangalore, along with three of my friends. I placed an order for two sandwiches and one mosambi juice for which we paid 525 Rs. After about 10 minutes, the food arrived. When I had drunk almost half of the juice, much to my shock and dismay, I found insects in it. My friends also saw this. I complained to the branch manager who said that the insect would have entered the glass after serving. He also denied the refund. I left the juice and the restaurant. When I came home I vomited twice. I want to complain against ABC Hotels and receive compensation for my agony. Please tell me how.
On 04/02/2022 (4th February, 2022), at around 2:30 pm, I visited the Jeevan Nagar branch of ABC hotels, located in Bangalore, along with three of my friends. I placed an order for two sandwiches and one mosambi juice for which we paid 525 Rs. After about 10 minutes, the food arrived. When I had drunk almost half of the juice, much to my shock and dismay, I found insects in it. My friends also saw this. I complained to the branch manager who said that the insect would have entered the glass after serving. He also denied the refund. I left the juice and the restaurant. When I came home I vomited twice. I want to complain against ABC Hotels and receive compensation for my agony. Please tell me how. My details are: K. Gayathri 46 years of age, female 20/918, III street, Vimala Nagar, Bengaluru – 100 Gayathri1888@gmail.com 9234567891 The Opposing Party details are: ABC Hotels Private Limited Represented by its Manager 2nd Cross Street, Simala Nagar, Bengaluru-60 abchotels@gmail.com 9456716832
You are clerk with a judge in the district consumer redressal commission in India. The above is a case before you where the parties have filed a complaint and written statement. Based on these documents, the judge asks you to prepare a one-page material summary of the case. The summary should contain the following:
Summary: A brief summary of the case.
Issues addressed: Key questions are in dispute between the parties
List of Evidence: List of evidence between the parties
Relief claimed: Relief Claimed by the parties
Jurisdiction: Where pecuniary and territorial jurisdiction is satisfied
Limitation: Whether the complaint was filed within the limitation period
Give me the one-page material summary.
You are a legal intern in India assisting a lawyer specialising in consumer law. You must give the lawyer relevant cases that are related to the query described by the consumer. Find cases pertinent to the issue or the sector of the grievance, and that can be used by the lawyer to persuade the court. Only refer to cases that have been actually decided by the Supreme Court, the High Court or the National Consumer Dispute Resolution Commission in India. Giving cases not decided by these courts or made-up cases, will lead the lawyer to lose the client. Try and list atleast five of the most appropriate cases for the issue though you may list more, summarising the relevant ratio of the case.
Query: """On 04/02/2022 (4th February, 2022), at around 2:30 pm, I visited the Jeevan Nagar branch of ABC hotels, located in Bangalore, along with three of my friends. I placed an order for two sandwiches and one mosambi juice for which we paid 525 Rs. After about 10 minutes, the food arrived. When I had drunk almost half of the juice, much to my shock and dismay, I found insects in it. My friends also saw this. I complained to the branch manager who said that the insect would have entered the glass after serving. He also denied the refund. I left the juice and the restaurant. When I came home I vomited twice. I want to complain against ABC Hotels and receive compensation for my agony."""
You are a judge in the district consumer redressal commission in India. The above is a case before you where the parties have filed a complaint and written statement. Based on these documents, you are asked to prepare a judgment. Assume that the documentary evidence provided by both parties is true. Assume also that no oral evidence was given by either party. Write an elaborate judgment on the matter taking into account all facts and the law. If necessary, for a legal proposition, you may use relevant case-laws in your judgment.
Next Session: To what extent will the legal community accept AI?