Patent Claims Translation / Evaluation Tasks 2026

Sponsors

Introduction

The performance of machine translation using NMT and LLMs has improved dramatically, and in some cases, it can even surpass human translation depending on the language and domain. However, there is currently no universal method for accurately evaluating the performance of machine translation. Even widely used metrics such as COMET have been reported to yield unstable or inaccurate evaluation results when applied to translations of texts from domains other than those used in COMET's training.

The same applies to patent document translation. While the average translation quality has significantly improved, it remains difficult to accurately evaluate aspects such as appropriate terminology usage and term consistency. In particular, patent claims present additional challenges due to their length and distinctive writing style, making accurate evaluation even more difficult.

Therefore, we will conduct a Shared Task focusing on Japanese-English patent claim translation. The goal is not only to compete on translation quality, but also to ultimately develop an automatic evaluation method that can accurately assess translation results.

Last year, we conducted Japanese-to-English and English-to-Japanese translation tasks for patent claims, collected translation outputs from various systems, and performed error annotation on those outputs. This year, we will conduct an evaluation task using this annotation data. Participants are asked to annotate the machine translation outputs using the same schema as last year's annotation.

Important Dates

• Test Period September 4 - September 17, 2026

• System Description Paper for Shared Tasks Submission Deadline September 21, 2026

• Review Feedback of System Description Papers October 1, 2026

• Camera-ready Deadline October 12, 2026

• Workshop Dates November 9, 2026

Evaluation Tasks

This shared task evaluates patent claim translation for JA-EN and EN-JA.

Task Requirements:

1. Error Annotation: Following the Annotation Criteria below, annotate translation errors in each translation file (output/) against its corresponding source text (input/) provided in the test set.

2. Quality Scoring: Assign a quality score from 0 to 100 for each translation output.

Submission Format

• Please use the same JSON format to the training data

• The output file must use the same basename as the input file with a .json extension (e.g., input: 01.txt -> output: 01.json).

• Please organize the files into two directories: ja-en and en-ja.

How to Submit

• Please submit the zip-compressed evaluation results to pat-claim-mt .at. googlegroups.com

Evaluation

Automatic Evaluation

Human Evaluation

Past Editions

Patent Claims Translation / Evaluation Tasks 2025