Embedded Files
Where to start with data collection?
There are many, many ways to collect data - but it is important to be able to assess your own situation and choose the right tools for your circumstances.
Often, data collection tools are described in terms of the kind of data they collect:
Qualitative data is descriptive, non-numerical information that helps you understand people’s experiences, perspectives, and meanings. It answers questions like how and why. Examples include interview responses, open-ended survey answers, observations, stories, and reflections.
Quantitative data is numerical information that can be counted, measured, and analysed statistically. It answers questions like how many, how often, or to what extent. Examples include attendance numbers, survey ratings (e.g. 1–5 scales), completion rates, and demographic data.
Often, using mixed or multi methods allows Hubs to be flexible, and demonstrate a well-rounded picture of the program or concept they are evaluating. This simply means that rather than using only quantitative or only qualitative approaches, you have a mix of both.
Selecting the right tools for your context should be driven by:
- Program Logic and Evaluation Plan
Go back to the program logic you created in Stage 4. Your program logic should be treated as a live document, so don't be afraid to make some changes through an evaluation lens.
You should also have developed an evaluation plan in the last section, ahead of planning and developing tools. An evaluation plan draws out your outcomes, and connects them to the tools, and ensures that you don't make evaluation plans for the sake of it.
Program Logic Evaluation Plan Template.pptx2. An understanding of what is meaningful to measure
2. An understanding of what is meaningful to measure
There is a saying in evaluation - not everything that matters can be measured, and not everything that can be measured matters.
Having a solid Program Logic and Evaluation Plan before designing tools helps with this, but even with good plans in place, use your community knowledge about what is valuable and feasible to collect at any particular time, with any particular tool.
Across Study Hubs in Place (SHIP) programs, we use four shared measures to support consistency across all programs. However, these are applied proportionately, meaning that depending on the outreach activity, not all the shared measures will be used.
The SHIP Shared Measures graphic below outlines what kind of things may be meaningful to measure, and therefore key to designing the right tools.
Even outside of the SHIP Partnership, this approach may be useful to create a consistent approach to tool design in your WP evaluation.
Selecting the right evaluation tool
One of the most common evaluation challenges is deciding how to collect meaningful evidence without placing unnecessary burden on participants or staff.
Surveys can be useful, but they are only one tool available to widening participation practitioners. Not everything needs a survey. People often show us more through activities, stories, conversations and observation than they do through formal questionnaires.
Many programs benefit from combining methods. For example, a short survey may provide evidence of increased confidence, while participant stories help explain how that change occurred.
Good evaluation design begins by asking the following questions before deciding on the data collection tool or method:
What do we want to know?
Who are we working with?
What is realistic in our context?
What information will be useful?
The graphic below can give you some suggestions on how to collect different types of data. However, there are hundreds of evaluation methods out there - this is not a limited list!
Developing the right tools
Once you have selected the most appropriate tools, you need to build them!
The options below provide a summary of some commonly used data collection methods, and some resources to help you develop your own. You may also want to take a look at some more specific examples on the following page in Stage 5, called "Examples and Inspiration".
This list is not exhaustive - you can try other data collection methods! It is also important that we acknowledge that traditional research methods have typically been defined and developed by universities and researchers. In Hub settings, our practice will often be necessarily different, as we work to bring rural knowledges, decolonial perspectives, and community need to the table.
Administrative Data
This is data collected for a broad, administrative purpose. It is often not collected solely for the purpose of widening participation evaluation, but it can be useful to give contextual information. It is also often required in Government or other grant reporting.
On a large scale, the Australian Bureau of Statistics is an example of administrative data, such as in the Education link to the left.
Examples in a smaller Hub context:
Number of students registered at a Hub or for a program
Number of activities run under a specific program
Repeat attendance
Social media engagement
Surveys
Surveys can collect qualitative and quantitative data, and are the most commonly used evaluation tool in Hub-led WP. They should be carefully developed against your program logic and KPIs. The link to the left provides 10 key steps to designing a good survey. QuestionPro software is availble to all RUSH participating in the SHIP Partnership.
Consider:
What platform will you use to deliver the survey?
How will you protect confidentiality?
How will you analyse the data at the other end?
Do you need to compare data from multiple surveys?
Examples:
Pre and post surveys for programs or activities
Pulse surveys (quick check-ins)
General feedback surveys
One-on-one qualitative collection
There are a variety of tools under this banner, and many Hubs use these in their normal operations without realising! You can read about many different methods in the article to the left.
It is important to always consider your purpose, and design interactions carefullyto use them as part of formal evaluation.
Examples:
Interviews
Collecting/creating impact stories from specific students or participants.
Case studies, similar to the above.
Post-event discussions or reflections with participants. These can occur in group settings too.
Read more about interview types via the link below:
Interviews and focus groups
Interviews and focus groups can be conducted in many different ways. Hubs have used interviews or focus groups to successfully add depth to quantitative WP evaluation data.
Examples:
Structured interviews - everyone gets asked the same questions.
Semi-structured interviews - the interview follows a guide of themnes and questions, but responds to the participant's answers.
Focus groups - a group of people respond to prompts and each others' responses. The people have often been chosen specifically by the Hub/evaluator/researcher for who or what they represent and the perspectives they bring.
Observation R.Krueger 10.17.pdfFurther reading
Observation in Evaluation, Richard Krueger
Observation
Structured observation involves intentionally recording participant behaviours, interactions and engagement against agreed criteria.
It is useful for understanding participation, engagement, peer interaction and help-seeking behaviours.
Tip
Record specific observations rather than general impressions. For example, instead of "Students seemed engaged", record "18 of 21 students voluntarily contributed to discussion activities; 12 students asked questions."
Page updated
Google Sites
Report abuse