Embedded Files
Description of the tasks
This challenge will adopt part of the recently developed TeleQnA dataset [1], composed by multiple-choice questions related to different classes of telecom knowledge domains [2], and will require participants to work on (at least one of) the following independent tasks:
Specialize Falcon on telecom knowledge: In this task, the participants will download Falcon model [3], and improve such model on their local computing facilities. The participants will be required to enhance the accuracy of the baseline model when answering to the multiple-choice questions included in the TeleQnA dataset by developing novel solutions or combining existing methods such as Retrieval Augmented Generation (RAG) and prompt engineering.
Specialize Phi-2 on telecom knowledge: In this task, the participants will download Phi-2 [4], and improve such model on their local computing facilities. The participants will be required to enhance the baseline model accuracy when answering to the multiple-choice questions included in the TeleQnA dataset by developing novel solutions or combining existing methods such as fine tuning, RAG and prompt engineering.
Further information are available on Zindi
Enrolling
Register on Zindi and access to the competition webpage. Participate to the webinar to know more about the competition.
Evaluation rules
The participants to the challenge will be evaluated based on three important criteria:
The accuracy that their solutions will reach on the final test set.
NOTE: The final test set is not part of the TeleQnA currently available online.
The readability and reproducibility of the delivered code.
The quality of the scientific paper, presenting the proposed solution, and its novelty, as well as the tackled problem(s) during the challenge, and the related experiments done.
In addition, the paper submission, acceptance, and presentation to the IEEE Globecom 2024 (https://globecom2024.ieee-globecom.org/) workshop (NOTE: at this stage, the workshop acceptance at IEEE Globecom is not yet been confirmed) associated with this challenge are strong requirements to win the competition, and they will demonstrate the participants capability to present their work, provide model explainability, and allow the community to reproduce their experiments. This challenge seeks to promote and encourage openness, rigor, reproducibility, and explainability of AI-based models; therefore, the participants will be strongly encouraged to share their solution on GitHub.
Timeline
References
[1] TeleQnA: https://github.com/netop-team/TeleQnA
[2] A. Maatouk, F. Ayed, N. Piovesan, A. De Domenico, M. Debbah, and Z.-Q. Luo, “Teleqna: A benchmark dataset to assess large language models telecommunications knowledge,” arXiv preprint arXiv:2310.15051, 2023 https://arxiv.org/pdf/2310.15051.pdf
[3] E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Hesslow, J. Launay, Q. Malartic et al., “The falcon series of open language models,” arXiv preprint arXiv:2311.16867, 2023 https://arxiv.org/pdf/2311.16867.pdf
[4] M. Javaheripi, and S. Bubeck, “Phi-2: The surprising power of small language models,” Dec. 2023 https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/
Page updated
Google Sites
Report abuse