Generative AI with Scientific Diagrams

General Discussion

When we recall the way we were educated in Science and Math Education, our first resource for our learning was our textbook.  One of the biggest aides when learning an abstract science or math topic was the scientific diagram or figure attached to the concept or problem.  In this area, we explore (1) how well generative AI makes scientific diagrams and (2) does a scientific diagram attached to a picture help the LLM to solve a science or math problem?

Research on scientific diagram generation


Research on MLLM reasoning with diagrams

• GPT-4 CoT: Let’s think step by step.

• GPT-4 w/o Viz: Don’t use visualization. Let’s think step by step.

• GPT-4V CoT: Let’s think step by step.

• GPT-4 VoT: Visualize the state after each reasoning step.

One can try this with science and math problems in multi-modal LLMs - and see if this improves task performance on problem solving.