Award Purpose

The 2026 RoboSoft Award for Best Paper in Manufacturing or Testing Reproducibility is being offered to encourage development of more substantial and informative procedure sections which make appropriate use of supplementary information offered via online file repositories, e.g., GitHub, IEEE, university-hosted. 

While the working group recognizes and appreciates the importance of reproducibility in software and algorithms, this award is targeted at hardware-focused reproducibility in manufacturing and testing. If you would like to see a version of the award aimed at software, algorithms or other digital products (particularly if you would be willing to help create it!), please let us know!

Criteria for 2026 Award in Manufacturing and Testing Reproducibility

All full papers accepted for publication at RoboSoft will be reviewed for the award using their final version. Eligibility requires presence of a manufacturing or testing procedure (or both) that is specific to the submitted paper, e.g., papers that reference a manufacturing section published in a prior work but contain no manufacturing section or supplementary information in the RoboSoft publication would not be eligible. However, a paper that references an older work but contains an independent manufacturing procedure would be eligible. 

We strongly encourage use of repositories (GitHub, IEEE, university-hosted, etc) and supplementary information that extends beyond what is typically included in a standard IEEE conference publication. Information contained within these repositories will be included in the review, presuming the files are directly referenced in the publication.

Criteria

All categories are equally weighted and receive a score of 1 to 5. A score of 5 is intentionally very difficult to achieve, to help separate submissions. 

1. Completeness of presented information. This score is a average of three subcategories, each scored 1 to 5: Purchased Items; Manufactured Items; Procedures.  At their discretion, a reviewer may determine that one subcategory (e.g., manufactured items), is not relevant to a paper and exempt it from the average. 

2. Quality of procedures, in the context of reproducibility. 

3. Clarity of presented information. 

Criteria #1: Completeness

Completeness will be reviewed for three subcategories, with the understanding that not all papers will necessarily make use of all three. The score from each subcategory will be averaged to produce the final completeness score, with subcategories that are not relevant removed from the average at the reviewers' discretion. 

Completeness, Purchasable Items: Are all components sufficiently well identified to allow a reader to purchase the same items to replicate the work (notwithstanding availability)?

5 - All pertinent items are listed, including all details (e.g., screw type, material batch information, etc.) A rating of 5 should be difficult to achieve, and requires significant SI.

3 - Major components are well identified, but minor components (types of screws, etc) may be neglected. Some models may be incompletely identified (missing brand, batch number, etc), but sufficient information is likely present to acquire equivalent components. 

1- Many purchased items are not identified. 

N/A - no items were purchased, and this section should be removed from the average score. Very few papers with manufacturing and testing procedures will fall in this category. 

Completeness, Manufactured Items: Is sufficient detail provided for manufactured items to determine if a similar item, once produced, matches what was used in the paper? Note that this is distinct from procedure in that this criteria focuses on features and geometries, e.g., drawings, CAD files, assembly diagrams, connections, etc. 

5 - Full detail is provided. A rating of 5 should be difficult to achieve. E.g., for manufacturing a printed part, a rating of 5 should go beyond CAD files and include print parameters, and a molded component should include mold files. 

3 - Significant detail is provided, e.g., CAD files for printed parts, mold files for molded parts, CAD or diagrams for testing jigs. Some details may be missing, but sufficient information is likely present to identify if any reproduced components are a reasonable facsimile of those used in the work. 

1 - Notably insufficient detail is provided. 

N/A - no manufactured items were produced, and this section should be removed from the average score. This score should not be used if a test procedure made use of custom jigs in testing. 

Completeness, Procedures: Is sufficient procedure detail provided to reproduce the full manufacturing procedure and / or testing process, presuming identical materials, components and geometries are already known? This subcategory is distinguished from manufactured items in that it focuses on the process instead of geometry, i.e., how to produce an item or run a test. 

5 - The process is completely described. Note that clarity and completeness are somewhat interrelated, even though they are judged separately. A video helps with clarity, but it is also generally more information dense. To score a 5, all steps of the process must be described, with no evidence of ‘hidden’ steps. Common failure points are identified, and reasoning for key steps is provided. As with other sections, a 5 should be difficult to earn. 

3 - The process is well described. There may be some evidence of assumed, omitted or 'hidden' steps (e.g., was the container of platinum-catalyzed silicone Part A stirred prior to pouring? Was sand paper placed in the grip to hold the sample better?), but a researcher experienced in the field would likely assume them. Reasoning, failure points or identification of critical points is generally omitted.   

1 - Notably insufficient detail is provided, to the point where the test or manufacturing procedure could not be reproduced even by an experienced researcher in the field. 

N/A - no processes or procedures were present. Very few papers will fall into this category.

Criteria #2: Quality of procedures, in the context of reproducibility. 

The goal of this criteria is to address aspects of the procedures or processes that inherently affect reproducibility, without regard to research novelty, device performance, etc. While reviewers are granted some discretion in determining what those aspects may be, we encourage consideration of the following:

• Number of samples tested or manufactured, and, specific to testing, number of “runs” on a given sample. 

• Procedural vs skill-based manufacturing steps, e.g., “stir until all bubbles are gone”; “stir for 30 seconds”. NOTE: reviewers may determine which is most appropriate for a given step. 

• Consideration of whether system behavior is inherently deterministic or stochastic and, in that context, appropriateness of statistical approaches. 

• Extent to which environmental data (e.g., temperature, humidity) was recorded. 

Reviewers should NOT attempt to judge impact of procedures or processes on device or component effectiveness or novelty, only on reproducibility.