Transformer

Transformer :

This project fine-tunes a GPT-2 Transformer-based Language Model using a custom dataset of poems. The aim is to train the model to generate coherent and contextually rich poetic text based on user-provided prompts.

 How It Works 

 Dataset:

poem.txt file containing lines of poetry.

Loaded and processed using Hugging Face Datasets library, converted into a custom dataset.

 Key Components:

 Model and Tokenizer:

• Pretrained GPT-2 model and tokenizer loaded via transformers library.

• Tokenizer adjusted to set the pad token to the end-of-sequence (eos) token to avoid padding issues.

  Data Preparation:

• Text data read line-by-line from the poem file.

• Each line tokenized to a fixed max_length of 128.

• Truncation and padding applied for consistent input size.

Data Collator:

• Uses DataCollatorForLanguageModeling with mlm=False since GPT-2 uses causal (autoregressive) language modeling.

 Training Arguments:

• Output directory for checkpoints and model

• 50 epochs, batch size of 2

• Custom logging and saving steps

• Training loss only (no evaluation metrics)

 Trainer:

Hugging Face’s Trainer API orchestrates model training with the tokenized dataset and collator.

 Poem Generation Function:

generate_poem() function takes a text prompt and generates a continuation based on specified parameters:

• temperature for randomness

• top_k and top_p for sampling strategies

• repetition_penalty to avoid repetitive phrases

 Results:

• Fine-tuned GPT-2 model saved to disk.

• Generates unique and creative poetic lines based on user prompts like "Beneath the stars".

 Libraries Used:

• transformers

• datasets

• torch

• pandas

• Google Colab (for cloud storage mount)

 Notable Features:

✅ Custom fine-tuning of a large pretrained model

✅ Poem generation with controlled sampling parameters

✅ Usage of Hugging Face Trainer API for efficient training

✅ Personalized poem generation based on any prompt

Google colab Notebook page :