APPLYING LLMs to design

Project done in collaboration with Autodesk and as part of research with the Design Research Collective, advised by Prof. Chris McComb.

Major problems addressed:

• To address the gap of manufacturing first approach

• To aid designers with low/no material knowledge to take data driven decisions

• Application of LLMs in workflow(s) – NLP driven decisions

• Aiding materials engineering experts to take decisions early in the cycle

The field of material selection plays a pivotal role in various industries, from manufacturing to construction. Usually, conceptual design takes the form of iterative design cycles where iterations are made in designs and manufacturing method and after this cycle, a material selection cycle kicks in. Leveraging Language Models (LLMs) in material selection introduces a novel approach to streamline the decision-making process. In this project, our objective was to integrate LLMs into the material selection workflow, focusing on enhancing the efficiency and effectiveness of decision-making and solving the issue of manufacturing first approach. Responsible for Technical Stack (Input prompts -> LLMs -> Survey Creation).

The overall timeline of this project consists is divided into four phases, which are as follows:

1. Assembly Details extraction

1. • Standard Models are selected and used as benchmark for the study

   • Standard assembly is combined with user intent and input

   • Assembly details are also extracted as ground truth

2. Large Language Model

• This is further divided into multiple categories as shown below

  1. Baseline Language model implementation

  2. Prompt Engineering and Fine-tuning

  3. Standard formatting

  4. Evaluation of Language Model outputs

3. Query from material database (TBD)

• The language model outputs a standard format of filters for query

• Ranked material suggestions is provided

• Evaluation on set criterion done

4. Expert Survey and Confidence score (TBD)

• Survey design on evaluation of output (Ranked suggestion)

• Survey rollout and data analysis

• Results reported

To assess the efficacy of the integrated LLM, a dual-phase evaluation was conducted, encompassing both qualitative and quantitative analyses.

Quantitative metrics were employed to measure the performance of the LLM in comparison to traditional material selection methods. Metrics included domain specific estimates such as Weight, Cost and Safety Factor. Specific evaluation of the Language Model for this use case is to be done.

Qualitative analysis involved identifying and addressing shortcomings in the LLM. It was noticed that hallucination errors were pretty common and it was mitigated by 30% of the baseline using qualitative methods.

In conclusion, the integration of LLMs into the material selection process looks promising in improving decision-making accuracy and addressing inherent limitations in the conventional approach. This innovative application has the potential to reshape how industries approach material selection in the iterative conceptual design phase, providing more informed and efficient solutions. Future steps involve further refinement of the LLM, expanding the dataset for increased diversity, and exploring avenues for real-time decision support in material selection processes (RLHF).