Definition of evaluation 

- ‘’Evaluation is a process that critically examines a program. It involves collecting and analyzing information about a program’s activities, characteristics, and outcomes. Its purpose is to make judgments about a program, to improve its effectiveness, and/or to inform programming decisions’’ 27. 

- ‘’Event evaluation is concerned with assessment, which usually involves measuring a set of key variables, as well as monitoring those variables to determine positive and negative outcomes. It is a subjective determination that can utilise objective quantitative measures’’ 28. 

Evaluation is often one of the last considerations of festivals in terms of time, budget and human resources. An evaluation can be conducted by the internal team (do-it-yourself process) or an external party (full-scale and professional study), it depends on: 

- Scale of the festival: a weekend festival and a month-long festival will require different level of resources 

- Funding source: certain funding scheme/programmes will require the type of evaluation they want to see 

- Your festival’s need: what do you want to know from the evaluation? 

- Resources: this refers to budget, time frame and human resources you have. Do you have enough budget to hire someone from an agency to conduct the evaluation or your staff has to work on it? How much time do you have to conduct and analyse the evaluation? 

Before you decide to have an evaluation for your festival, you need to set the objective clearly so that you could select the most appropriate tools to get the outcome you desire 29. 

Your objectives can be based on: 

a) Process: to measure the programmes in the festival, programme quality and who it is reaching; useful in identifying inefficiencies and streamlining processes; 

b) Impact & Outcome: the immediate and long term effect of the festival and how it is aligned with the festival objectives/goals, it reflects how well the festival objectives have been achieved through the curated programmes;

c) Summative: considers the entire festival and future perspective; useful to justify the project on what they have achieved and lobby for project continuation 30;

The outcomes can be: 

a) Quantitative 

- to measure tangible data, ie how many, how much, how big 

- the outcomes are normally data that can be presented in numbers or percentages 

b) Qualitative 

- to measure intangible values, such as awareness, appreciation and change of behaviour 

- more complex 

The outcome of the data will be able to: 

- provide a better understanding of your target audiences’ (demographics), their needs and satisfaction 

- monitor the creative process to deliver the objectives 

- increase the productivity and effectiveness of your festival 

- collect tangible data for impact report or grant report to partners and sponsors 

- analyse the impact of the festival 

- understand the improvement for the following festival 

- build a database of knowledge for developing new programmes in the future 31. 

27. Michael Quinn Patton. Qualitative Research Evaluation Methods. Thousand Oaks, 1987, CA: Sage Publishers. 

28. Donald Getz. Event management and event tourism (1st ed.), 1997, New York: Cognizant Communications Corp. 

29.  Vipul Nanda, “The 7 Types of Evaluation You Need to Know,” Atlan, August 7, 2019, https://humansofdata.atlan.com/2017/04/7-types-of-evaluation/

30. Nanda 2019. 

31. Nanda 2019. 

Questionnaire / Surveys 

‘’A questionnaire is a specific set of written questions which aims to gather specific information from the respondents. Questions can be open or closed-ended. This is commonly used to gather information from mass samples, for example, festival goers/visitors/participants/audiences in order to have enough data to understand the pattern’’ 32. 

Questionnaire can be conducted in different manners: 

• Online 

- Can be conducted on-site and off-site; 

- On-site: online questionnaire can be conducted by having a few gadgets in common areas/information counter/ticketing counter in the festival sites; 

- Off-site: email the questionnaire to your audiences or collecting the data via a form (ie Google Form, SurveyMonkey, jotform) but not everyone who receive the links will click on it to fill up the form; 

- A small reward, ie promo code on tickets, small gift might be able to encourage the audiences to take part in the questionnaire. 

• Paper and pencil 

- This can be an alternative if you don't have the budget to have a few gadgets on-site 

- Cons: you need to key in the data before you could proceed with the data analysis. 

• Person-to-person interview

-If you have sufficient number of surveyors, you could consider having person-to-person interview which will give you more in-depth information, but it’s more time consuming compared to the above two 33. 

Focus Group Discussion (FGD)

- Normally consists of 8-12 individuals from similar backgrounds or experiences to discuss a specific topic of interest in a loosely structured discussion with a moderator/interviewer. 

- Qualitative tool where questions are asked about the respondents’ perceptions, attitudes, beliefs or opinions. 

- Respondents are able to talk with other group members instead of to the interviewer only. 

- This can be used to understand the feedback from certain groups, ie artists, contractors/suppliers 

- Cons: Respondents may be reluctant to share some sensitive or personal ideas and concerns among the group members 34.

Stakeholder evaluation: audience, artists, partners, stallholders, etc. 

In order to have an overview of the festival and more inclusive, you may consider to involve people from different aspects:

• Internal

- working team (permanent, project/contract basis, part time) 

- volunteers 

• • • External 

- festival goers/visitors/participants/audiences 

- artists/performers 

- contractors/suppliers, ie venues, logistics 

- stallholders/vendors 

- Business owners around the sites/venues (Food & Beverage, Hotel, Retail) 

- partners/sponsors 

Links: 

- Thundering Hooves - a research into the 2005/6 Festival season comprising the 11 festivals that made up the then fledgling Festivals Edinburgh. 

- Evaluation and Audience Research Toolkit - Supporting Evaluation of Family Arts Festival Event 

32. Josiah Kaplan, “Questionnaires,” BetterEvaluation, August 31, 2015, www.betterevaluation.org/en/evaluation-options/questionnaire

33. Kaplan 2015. 

34. Sushil Baral, Sudeep Uprety , and Bipul Lamichhane, “Focus Group Discussion” (Health Research And Social Development Forum, March 2016), www.herd.org.np/uploads/frontend/Publications/PublicationsAttachments1/1485497050-Focus%20Group%20Discussion_0.pdf.