BioPIE: A Biomedical Protocol Information Extraction Dataset for Experiment Understanding

🎉 Congratulations! Our paper has been accepted to EMNLP 2026 Findings!

Understanding biomedical experiments provides a foundation for downstream tasks, e.g., laboratory automation, and facilitates effective cross-disciplinary communication. Two challenges, High Information Density (HID) and Multi-Step Reasoning (MSR), pose unique difficulties for precise automatic experimental understanding. Extracting structured knowledge, e.g., Knowledge Graphs (KGs), is an effective approach to address the HID and MSR. However, existing biomedical datasets for structured knowledge Information Extraction (IE) are limited to a general or coarse-grained level, hindering fine-grained experimental understanding. To address this gap, we introduce Biomedical Protocol Information Extraction Dataset (BioPIE), a dataset providing procedure-centric KGs that captures entities, actions, and relations at a scale sufficient for reasoning across biomedical protocols. We evaluate both supervised and LLM-based IE methods on BioPIE to verify its effectiveness, and implement a biomedical question answering system to provide a quantitative illustration of BioPIE's effectiveness for downstream understanding tasks. The experimental results demonstrate improved understanding performance on both the HID and MSR question sets.

Figure 1. An illustration of BioPIE. (A) An annotated example of a biomedical experimental protocol for plasmid DNA preparation, illustrating how diverse laboratory operations are decomposed into structured procedural entities and relations under our annotation scheme, independent of the domain-specific biological semantics. This design choice ensures that the scheme remains generalizable across diverse biomedical workflows, such as cell culture, sequencing, and fabrication, without being constrained by the specialized terminology of any single sub-domain. (B) Statistics of entity types and relation types in the BioPIE dataset. (C) Representative entity and relation labels in our annotation scheme, with definitions and examples. Different protocol examples are shown across panels to illustrate the framework's applicability to diverse experimental procedures.

Table 1. Test F1 scores of different baselines on our proposed dataset. "Joint" denotes joint IE, while "Pipeline" refers to performing NER and RE separately. "Rel" and "Rel+" indicate relation extraction from original text under boundary and strict evaluation, respectively, and "RE" denotes relation extraction with gold entities, applicable only to the pipeline methods. The number of in-context examples was tuned on the validation set. ¹ Denotes the use of scibert (Beltagy et al., 2019) as the encoder; ² denotes biobert (Lee et al., 2020).

Figure 2. BioPIE enables knowledge integration in the lab automation. BioPIE can be used to extract large volume of biomedical protocols into structured knowledge, which can then be used by knowledge systems.