European Smart Tourism Tools Observatory

Before detailing the classification, it is imperative to define a prompt.

Prompt - a set of instructions or a question given to a LLM (Large Language Model) to elicit a specific response or action from it. 

There are three possible approaches to classifying STTs using LLMs:

• Fine-tuning - is the process of adjusting a pre-trained LLM by continuing the training phase with a smaller, specialized dataset to improve its performance on specific tasks or domains.

• Prompt-guided - is the method where we provide the LLM with specific prompts that guide it to generate the desired output or perform a particular task without additional training.

• Fine-tuning combined with prompt-guided 

Due to the lack of computing resources required to carry out the fine-tuning, we decided for a prompt-guided approach.

Within this approach there are several ways to proceed.  We could employ the following techniques:

• Chain-of-Thought (CoT) - Involves prompting the LLM to provide intermediate steps or reasoning paths leading to an answer. 

• Tree-of-Thought (ToT) - Involves prompting the LLM to consider various reasoning pathways and potential outcomes, much like branches on a tree, to reach a comprehensive answer.

• Zero-shot learning - The LLM generates a response without any prior examples, based solely on the prompt. 

• Few-shot learning - The LLM is provided with a few examples to guide its response. 

• Role-Based Expert Assignment - Involves assigning experts to specific solution domains, ensuring that each domain is represented by knowledgeable individuals who specialize in their respective areas. 

After our research to decide on the most suitable technique(s) for classifying STTs, we used few-shot learning combined with role-based expert assignment.

Our methodology with the LLM to classify STTs is summarized as follows:

1. The first prompt is to explain the STT taxonomy.

2. The following prompts are the few classification examples to guide the LLM (one per prompt). The number of examples given is sufficient to cover all the categories of the taxonomy.

3. Finally, unclassified STTs are provided (one per prompt). In each prompt and before providing the STT to be classified, we ask the LLM to create an expert for each application domain of the STT taxonomy (in this case, 3 experts). The task of each expert is to assign a score from 1 to 5 to each of their categories. Those with a score higher than 3, if any, are the expert's answer to the classification problem.

Results

Our experiments with Copilot (Bing Chat), ChatGPT, and Gemini have proved that this classification is feasible. However, the performance metrics obtained so far are not good enough for large-scale classification.

Possible future directions to get better results are to apply fine-tuning to an LLM and, perhaps to get the best possible results, combine it with our methodology so far. In that case, the first prompt would no longer be necessary since the LLM would already be an expert in the subject.

Content from the following catalogs has been added to the Observatory with permission from the organizations listed below.