Abstract

Teaching to improve student models (e.g., knowledge distillation) is an extensively studied methodology in LLMs. However, in human education, teaching enhances not only the students but also the teachers by fostering more rigorous and clearer reasoning, as well as deeper knowledge building. We ask: Can LLMs also learn by teaching (LbT) for better reasoning? If the answer is yes, we can potentially unlock the possibility of continuously advancing the models without solely relying on human-produced data or stronger models. 

In this paper, we provide a preliminary exploration of this question. We show that LbT ideas can be incorporated into existing LLM training/prompting pipelines and bring improvements. Specifically, we design three methods, each mimicking one of the three levels of LbT: observing students' feedback, learning from the feedback, and learning iteratively, with the goal of improving answer accuracy without training or improving models' inherent capability with fine-tuning. We reveal some findings: 

• Teaching materials that make it easier for students to learn (via in-context learning) have clearer and more accurate logic.

• Weak-to-strong generalization: LbT might help improve strong models by teaching weak models.

• Diversity in students might help: teaching multiple students could be better than teaching a single student or the teacher alone. 

We hope that our exploration can inspire future research on LbT and, more broadly, the adoption of advanced education techniques to improve LLMs.

Method 1

Approach. One common teaching strategy in education is that the teacher first teaches students how to solve a class of problems by giving them the example rationale (named Teaching Rationale, or TR in short) and the answer (named Teaching Answer, or TA in short) to a particular question (named Teaching Problem, or TP in short). Then, the teacher asks students to solve other similar problems (named Exam Problem, or EP in short) to test if the students understand the concepts. The teacher can also learn from this process by observing the feedback (i.e., LbT level 1): if the students can answer EPs well, then it likely means that the TR-TA pair is of high quality. Method 1 simulates this process to select the best answer TA for a given TP, as depicted in the following figure.

Results. The results on MATH are shown in the table above. Key takeaways are:

• M1 is effective with various model settings and surpasses baselines. M1 exceeds self-consistency (SC) with various model settings: strong-teach-weak (e.g., GPT-4o teaches GPT-4o mini), weak-teach-strong (e.g., Mistral-7B teaches LLaMA3-8B), and self-teaching (e.g., LLaMA3-8B teaches itself).

• M1 can further benefit from multiple students. Using GPT-3.5 to teach both LLaMA3-8B and Mistral-7B achieves a significant improvement than teaching LLaMA3-8B or Mistral-7B separately.

These suggest the promise of M1 as a tool to continuously advance the models.

Please refer to the paper for results on the competition-level code synthesis task. We make a small interactive demo as follows:

Method 2

Approach. In education, after identifying which teaching materials (e.g., TR-TA pairs) can enhance student performance, teachers can use this information to improve their knowledge or teaching strategies. For example, if students perform poorly due to unclear or inaccurate teaching materials, teachers can correct their knowledge and avoid generating similar TR-TA pairs in the future. M2 simulates this approach to finetune and improve the LLM, as depicted in the following figure.

Results. The results on MATH are shown in the table above.  We can see that M2 achieves better results compared to solely using the correctness scores in DPO. This improvement is because LbT provides more informative scores than those purely based on correctness. This experiment demonstrates that LbT can also be used to improve the inherent capability of LLMs via finetuning.

Method 3

Approach. In the previous two methods, we utilize students' exam scores as the learning signal. In M3, we explore whether reflecting on students' detailed exam responses can help the teacher iteratively refine its teaching materials. Notably, we aim to verify whether these refinements can enhance the teacher's own performance by providing more effective knowledge. If so, we can assert that the iterative process of teaching, reflection, and material refinement facilitates some form of "knowledge building'' for the teacher as in humans. Additionally, we are interested in whether having multiple and diverse LLMs as students offers further benefits. The method is depicted in the following figure. 

Results. The above table shows that LLMs are able to reflect on the failure cases of students and propose revised exemplars that improve the teacher’s performance. More importantly, we observe a performance gain brought by having dedicated students (as opposed to using a single LLM in prompt optimization as in previous work). Comparing to the scenario where the teacher and student are the same, having one or multiple LLMs different to the teacher as the student improves the quality of the teaching material faster. This demonstrates LbT as a case of weak-to-strong generalization. We speculate that the benefits are brought by more diverse error types made by a different (weaker) student model.

Discussions and Future Work

Insights into In-Context Learning. Currently, we conduct student “learning” with ICL, based on the assumption that students can effectively “learn” from ICL examples and apply similar strategies to solve EPs. Interestingly, prior work found that a correct input-output pairing in ICL examples does not matter much. At first glance, this finding seems to challenge our design, as it suggests that the TA accuracy may not affect the EA accuracy, which means the LbT score cannot reflect the quality of TR+TA. However, we find that, as opposed to only providing labels in the ICL examples in prior work, providing rationales is important. LLMs can follow the problem-solving logic in the detailed rationale in the ICL examples well. This may be because the rationale provides more information, making the ICL examples easier to follow. Consequently, we see that students can use similar logic as the TR when solving the EP. This means that better TR+TA can indeed lead to improved ER and thus higher EA accuracy (i.e., LbT score).

Weak-to-Strong Generalization. Improving models with human-generated/annotated data or synthetic data from stronger models is the dominant paradigm. However, how can we continuously improve the strongest model without relying on human-generated and annotated data? A recent work conducts an exploration on using weak model supervision to train a larger model. Our work is another attempt towards the “weak-to-strong generalization” prospect by drawing from how humans continuously acquire new knowledge without direct instruction. We demonstrate that stronger models can further improve their own results (M1), parameters (M2), and prompts (M3) by utilizing the feedback of weaker models.

M1 and M2 rely on generating/selecting similar EPs. We verify that LbT-based scoring can help select high-quality TR-TAs but require the TP and EPs to have similar problem-solving strategies. In our experiments, suitable EPs are selected according to human-provided information in the dataset. One extension is to let a model automatically identify EPs similar to a TP from a large pool. Another direction is to synthesize similar problems based on a group of problems and exploit the LbT principle to score many rationales for the new problems. Specifically, as a “self-instruct” extension to M2, we can generate a new problem P based on a group of problems S = {P1, · · · , Pk} that are already known to be similar. The generating-scoring pipeline can then be applied to P to obtain rationale-score pairs, where the LbT score can be easily obtained using S as the EPs.

Additional inference cost. LbT-based scoring in M1 and M2 requires additional inference costs, which aligns with recent studies that show that increasing inference costs might be a promising way to improve models’ reasoning capabilities (e.g., OpenAI O1). Nevertheless, designing efficient inference algorithms and systems is needed to make these approaches more usable.

Borrowing Education Strategies to Improve LLMs. We believe that this work only scratches the surface of the potential of LbT, and other more general educational principles, in LLMs. As LLMs are becoming increasingly powerful, more advanced approaches in pedagogy can potentially help with the inference and training of LLMs. In the paper, we provide a roadmap towards this broader research agenda.

Acknowledgements

We thank Sergey Yekhanin from Microsoft Research for their support and suggestions for the work. We thank Zixuan Zhou, Chao Yu, Boxun Li for their discussions. We thank the anonymous reviewers for their insightful suggestions.

BibTeX

@inproceedings{ning2024lbt,

title={Can {LLM}s Learn by Teaching for Better Reasoning? A Preliminary Study},

author={Xuefei Ning and Zifu Wang and Shiyao Li and Zinan Lin and Peiran Yao and Tianyu Fu and Matthew B. Blaschko and Guohao Dai and Huazhong Yang and Yu Wang},

booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},

year={2024},

url={https://openreview.net/forum?id=0ZZMUjZJYF}

}