Hands-on Whisper-Taiwanese Model Tv0.5

Whisper-Taiwanese model V0.5 (Tv0.5): This model is a fine-tuned version of OpenAI’s openai/whisper-large-v3-turbo. It was developed by the National University of Tainan (NUTN), Taiwan, as part of a National Science and Technology Council (NSTC)-funded industry-academia collaboration project. We carried out the Taiwanese-English Co-Learning Pilot Project from September 2024 to June 2025 in collaboration with JEN-PIN ENTERPRISE CO., LTD. The model is trained for Taiwanese language recognition tasks using JEN-PIN educational materials generated through Student–Machine Co-Learning during the Fall 2024 semester. Additionally, the NUTN is collaborating with the National Center for High-performance Computing (NCHC) of the National Applied Research Laboratories (NARLabs) in Taiwan to provide computational and storage resources and co-develop an AI learning model for elementary and high school students. 

Model Details

• Base Model: openai/whisper-large-v3-turbo

Fine-tuned for: Taiwanese Hokkien Automatic Speech Recognition (ASR)

Fine-tuning Framework: Hugging Face Transformers

Training Duration: Approximately 180 hours using two V100 GPUs

Dataset: Custom dataset, including the Dictionary of Frequently-Used Taiwanese Taigi released by the Ministry of Education, Taiwan, totaling approximately 90 hours of audio data.

Input Format: 16kHz mono WAV

License: CC BY-NC 4.0

Step 1: Download the Sample Code from here

Step 2: Log in to your Google account

Step 3: Connect to Colab

Step 4: Click on the File menu, then select Upload notebook to upload the sample code

Step 5: Click on the Runtime menu and select Change runtime type. Set it to Python 3 and T4GPU

• Step 5.1: Download Whisper-Taiwanese Tv0.5 Model

• Step 5.2: Learn how to speak Taiwanese by referring to this webpage 

• Step 5.3: Speak in Taiwanese and record your audio

• Step 5.4: Execute ASR (Automatic Speech Recognition) to convert your Taiwanese audio to Chinese text

• Step 5.5: Download the Chinese-to-English translation model

• Step 5.6: Translate the Chinese text into English

Step 6: End