LLM-3D Print: Large Language Models To Monitor and Control 3D Printing

Yayati Jadhav Peter Pak Amir Barati Farimani

Carnegie Mellon University Mechanical Engineering

Overview

Industry 4.0 has revolutionized manufacturing by driving digitalization and shifting the paradigm toward additive manufacturing (AM). Fused Deposition Modeling (FDM), a key AM technology, enables the creation of highly customized, cost-effective products with minimal material waste through layer-by-layer extrusion, posing a significant challenge to traditional subtractive methods. However, the susceptibility of material extrusion techniques to errors often requires expert intervention to detect and mitigate defects that can severely compromise product quality. While automated error detection and machine learning models exist, their generalizability across diverse 3D printer setups, firmware, and sensors is limited, and deep learning methods require extensive labeled datasets, hindering scalability and adaptability. To address these challenges, we present a process monitoring and control framework that leverages pre-trained Large Language Models (LLMs) alongside 3D printers to detect and address printing defects. The LLM evaluates print quality by analyzing images captured after each layer or print segment, identifying failure modes and querying the printer for relevant parameters. It then generates and executes a corrective action plan. We validated our framework's effectiveness by comparing it against a control group of engineers with diverse AM expertise, demonstrating that LLM-based agents can accurately identify and correct common 3D printing errors—such as inconsistent extrusion, stringing, warping, and layer adhesion—without human intervention.

LLMs in continuous improvement cycle: LLM-based supervisor agents can be employed at each step of the continuous improvement cycle. The cycle involves evaluating print quality, identifying failure modes, gathering relevant information, and planning and solving the issues by adjusting the print parameters, ensuring high-quality defect-free parts.

Schematic of the Proposed Framework

In this study, we introduce a framework that leverages the multimodal and emergent reasoning capabilities of LLMs to detect and resolve issues during 3D printing. The system uses specialized LLM agents for specific tasks, coordinated by a supervisory LLM to ensure efficient workflow and communication. By utilizing the strengths of LLMs in reasoning and optimization, the framework identifies errors, assesses print quality, gathers necessary information, and addresses issues, enhancing efficiency and reducing material waste by correcting errors in subsequent layers without discarding the entire part.

The hierarchical machine-to-machine framework operates by capturing two images of the ongoing 3D print—one from the top and one from the front—after a layer is completed and the print is paused. These images, along with the part description, are analyzed by an LLM to evaluate print quality, identify defects, and make relevant observations.

If an issue is detected, the supervisory LLM invokes a planner to generate a detailed plan, outlining the necessary information and queries for the printer. Another LLM agent retrieves this information via the printer’s API, and based on the data, the LLM generates a solution plan, which is then implemented through direct communication with the printer's API. After executing the solution, the supervisory LLM verifies the parameters and resumes the print, ensuring coordinated and efficient system operation.

This framework is highly adaptable, working across various 3D printers and optimizing print parameters specific to each part without needing a pre-existing dataset. It can fine-tune process parameters on-the-fly, accommodating different materials, geometries, and printer settings. Additionally, the LLM provides detailed process commentary, enhancing traceability and supporting quality assurance by comprehensively documenting the manufacturing process. This reduces the need for destructive testing and increases trust in the final product, ensuring compliance with industry standards and regulations.

Schematic of the Proposed Framework: The process begins with a G-code file (a) being uploaded to the 3D printer, which is equipped with two frame-mounted cameras (b). After each layer is printed, the extruder moves to the home position, and two images of the current print state are captured (c). These images are analyzed by the LLM, which evaluates the print, makes observations, and identifies any failures. If failures are detected, the LLM supervisor (d) invokes the information planner (e). The executor (f) then carries out the information gathering plan, after which the solution planner (g) is activated by the supervisor. The solution plan is executed by another executor (h), and finally, the supervisor invokes the handoff module (j) to resume the print.

Prompt Pipelines

In this work, we utilize GPT-4o's capabilities to detect anomalies in 3D prints, improving accuracy and reliability by identifying and addressing issues in near real-time. After each layer is printed, the printer pauses, and two cameras capture images from the top (extruder view) and the front after the extruder returns to its home position. These images, along with a part description and images from the previous layer, are fed into the LLM to ensure it only identifies new issues, avoiding redundant error detection.

The planning agent is structured with two internal modules. The first module analyzes known information, observations, and detected failures to select, adapt, and refine reasoning prompts, ensuring the planner uses the most optimized and relevant prompts. The second module then uses these tailored prompts to develop a specific, actionable plan. This plan is executed by the executor agents, which interact with the 3D printer via API to gather data and implement the solution. This dual-module design enables the planning agent to efficiently resolve any issues that arise during the printing process.

The Agent Executor uses a predefined Python function to communicate with the printer via its API. Upon receiving a detailed plan from the planning agent, it translates the plan into operational steps by calling specific API endpoints to run G-code scripts and macros. The ReAct method ensures the executor not only issues commands but also actively monitors printer responses, reasoning through each step. If a response is insufficient or indicates an incomplete action, the executor reassesses and adjusts by selecting alternative endpoints or modifying the G-code script. This iterative process continues until the desired outcome is achieved, ensuring accurate and efficient execution.

Prompt Pipeline for individual agents: (a) The Error Detection Agent begins by processing images from the paused printer (1), which are formatted into a predefined prompt (2) and passed to the LLM (3). The LLM analyzes these images to assess print quality and detect any failures, generating a response (4) in the specified format.

(b) In the Planning Agent, the adapter module uses an LLM to select and adapt reasoning prompts based on observations from the Error Detection Agent. These adapted prompts, combined with existing information, guide the planner agent in creating an actionable plan.

(c) The Executor Agent: The LLM selects the appropriate API endpoint to execute the plan on the printer, utilizing the given plan and available printer objects. It then validates the command's success based on the printer's response and adjusts the endpoints as necessary. The internal reasoning and tool use by the LLM are also employed during this stage.

Results

Comparison of LLM-Optimized Print with Baseline: The LLM-optimized prints of the (a) wrench and (e) raised text show cleaner, well-defined edges, a smooth surface finish, and consistent extrusions, whereas the baseline prints (b) and (f) have rough, uneven edges and inconsistent material deposition. The LLM-optimized prints (c) and (g) demonstrate continuous improvements across layers, with better adhesion and more precise infill patterns, while the baseline prints show a steady decline in quality, perpetuating errors in (d) and (h).

Failures detected: (a) Observations and failures detected by the LLM in single-layer print during print optimization and for (b) Multi-Layer print of wrench. (b)(1) Comparison of collected human response and LLM response to multilayer print for each layer and (b)(2) the agreement score.

LLM print parameter optimization: Main parameters selected by the LLM for optimization: (a) Single-layer PLA print, (b) Single-layer TPU print, (c) Multilayer raised text print, and (d) Multilayer wrench print.

Video:

Bibtex

@misc{jadhav2024llm3dprintlargelanguage,

title={LLM-3D Print: Large Language Models To Monitor and Control 3D Printing},

author={Yayati Jadhav and Peter Pak and Amir Barati Farimani},

year={2024},

eprint={2408.14307},

archivePrefix={arXiv},

primaryClass={cs.CL},

url={https://arxiv.org/abs/2408.14307},

}

Page updated

Report abuse