Task 1: Legal Document Retrieval

Task 1’s goal is to return the article(s) that are related to a given question. The article(s) are considered “relevant” to a question iff the question can be answered using the article(s).

• The training data is in JSON format as follows:

[

    {

        "question_id": "DS-101",

        "question_type": "Đúng/Sai",

        "text": "Cơ sở điện ảnh phát hành phim phải chịu trách nhiệm trước pháp luật về nội dung phim phát hành, đúng hay sai?",

        "relevant_articles": [

            {

                "law_id": "05/2022/QH15",

                "article_id": "15"

            }

        ]

    }

]

• The test data is in JSON format as follows:

[

    {

        "question_id": "DS-1",

        "question_type": "Đúng/Sai",

        "text": "Phim đã được Bộ Văn hóa, Thể thao và Du lịch, Ủy ban nhân dân cấp tỉnh cấp giấy phép phân loại phim sẽ có giá trị trên toàn quốc, đúng hay sai?"

    }

]

• The system should retrieve all the relevant articles. Please see the Submission Details section below for the format of the submissions.

Note that “relevant_articles”  is the list of all relevant articles to the questions.

The evaluation methods are precision, recall, and F2-measure as follows:

Precisioni = the number of correctly retrieved articles of question ith
            the number of retrieved articles of question ith

Recalli = the number of correctly retrieved articles of question ith
          the number of relevant articles of question ith

F2i= (5 x Precisioni x Recalli)
      (4Precisioni + Recalli)

F2 = average of (F2i)

In addition to the above evaluation measures, ordinal information retrieval measures such as Mean Average Precision and R-precision can be used for discussing the characteristics of the submission results. The macro-average F2-measure is the principal measure for Task 1.

Task 2: Legal Question Answering

Given a legal question, the goal is to answer the question. In ALQAC 2024, there are three types of questions:

• True/False questions (Câu hỏi Đúng/Sai). Here is a training example:

[

    {

        "question_id": "DS-101",

        "question_type": "Đúng/Sai",

        "text": "Cơ sở điện ảnh phát hành phim phải chịu trách nhiệm trước pháp luật về nội dung phim phát hành, đúng hay sai?",

        "relevant_articles": [

            {

                "law_id": "05/2022/QH15",

                "article_id": "15"

            }

        ],

"answer": "Đúng"

    }

]

For the True/False questions, the answer must be "Đúng" or "Sai".

• Multiple-choice questions (Câu hỏi trắc nghiệm). Here is a training example:

[

    {

        "question_id": "TN-102",

        "question_type": "Trắc nghiệm",

        "text": "Nam, nữ kết hôn với nhau phải từ đủ bao nhiêu tuổi trở lên?",

"choices": {

"A": "Nam từ đủ 20 tuổi trở lên, nữ từ đủ 18 tuổi trở lên.",

"B": "Nam từ đủ 18 tuổi trở lên, nữ từ đủ 20 tuổi trở lên.",

"C": "Nam từ đủ 21 tuổi trở lên, nữ từ đủ 19 tuổi trở lên.",

"D": "Nam từ đủ 19 tuổi trở lên, nữ từ đủ 21 tuổi trở lên."

},

        "relevant_articles": [

            {

                "law_id": "52/2014/QH13",

                "article_id": "8"

            }

        ],

"answer": "A"

    }

]

For the multiple-choice questions, the answer must be "A", "B", "C" or "D".

• Free-text questions (Câu hỏi tự luận). Here is a training example:

[

    {

        "question_id": "TL-103",

        "question_type": "Tự luận",

        "text": "Cơ quan nào có trách nhiệm thống nhất quản lý nhà nước về điện ảnh?",

        "relevant_articles": [

            {

                "law_id": "05/2022/QH15",

                "article_id": "45"

            }

        ],

"answer": "Chính phủ"

    }

]

For the free-text questions, the answer is free-text and will be evaluated by human experts.

The principal evaluation measure is accuracy:

Accuracy = (the number of questions that were correctly answered)
(the number of questions)

Note: 

• Once the submission deadline for Task 1 has passed, the gold labels for Task 1 will be made available and can be utilized for Task 2.

• The output submitted by participants will be published to a public GitHub repository so that legal and AI experts can refer to this information for analysis purposes. In situations of uncertainty, expert evaluation serves as the authorized measure for determining the participants' system performance.

• You are free to crawl Internet data as additional data. There are no limitations on the use of externally sourced data. 

• Participants may use any large language models (LLMs) and pre-trained models. However, all participating teams are required to submit their source code for verification.