The project aims to enhance sentiment analysis by applying knowledge distillation to BERT models using the SST2 dataset, with a focus on creating efficient yet comprehensive models. It explores various BERT configurations, such as the 6-6-510-BERT with 28.6 million parameters, and the more compact 4-4-256-BERT with 11.17 million parameters, each showing a trade-off between training accuracy and validation performance. Other models like the 8-6-384-1536-BERT and 6-6-384-BERT also demonstrate this balance between complexity and performance. While the project faces limitations such as dataset dependency, potential loss of complexity from the larger BERT model, sensitivity to hyperparameter tuning, high computational demands, and interpretability issues, future improvements could include integrating knowledge graph distillation. This approach promises enhanced semantic understanding and model explainability, though it introduces challenges like the need for high-quality knowledge graphs and increased computational complexity. Overall, these efforts indicate that further optimization of model architecture and hyperparameters could lead to better sentiment analysis performance.

Click the buttons below to view the full report and technical article