Method: Thematic Analysis, Qualitative Data Coding

Team: UX Researcher, Product Manager, Data Analyst

Duration: Q3 - Q4 2021

Role: UX Researcher

CONTEXT

There were over 3,107 open-ended responses to analyze

In response to Mayor Eric Garcetti's Executive Order on racial equity in City Government (June 19, 2020), the Department conducted a survey of NC board members. The goal was to gauge their perspectives on social and economic disparities in their communities.

A dashboard was created to summarize the survey's quantitative data, but the most impactful insights were hidden in lengthy text responses. HackforLA was assigned the task of analyzing this text data and extracting meaningful findings. The aim was to provide EmpowerLA with actionable insights that could guide city initiatives.

APPROACH

Using Thematic Analysis to make sense of the overloaded amounts of data

Having large sets of long open ended responses was a big challenge. Mainly because there were large sets of regions within North California and the neighborhoods within those regions.

After carefully discussing and how to approach this massive data set, the team and I decided to first categorize and data-code all the open-ended responses. Afterwards we carefully connected similar patterns and themes with one another.

Going deeper into the "6-step" process

STEP 1

Organize the information in a workable format

We imported all the datapoint into excel sheet and created 3 important filters:

Regions
Neighborhood
Question

These three filters helped to easily look over questions and prioritize what regions to look over. Thus, allowing us to easily identify specific themes that emerged out.

Excel Plan

Summarize themes by regions + neighborhood + question
With small sample sizes, avoid creating themes and instead aggregate data at a higher level, such as the regional level.
After all neighborhoods have been summarized, move up a level and summarize themes at the region level

Theme was only created if it's been mentioned by 2+ survey respondents

STEP 2

Transfer from Excel to Miro

Once information was condensed, we transfered the contents from Excel onto Miro. When information is pasted from, Miro will automatically create one sticky per Excel cell

General guideline for our sticky notes

One datapoint per stick
Add a relevant statement or a question that comes from the codes/notes
Make sure each note is understandable on its own
Be concrete (short) and concise

What information we should include within our sticky notes

Factual statements
Participant quotations (in quotation marks - good quotes are either respresentative or unexpected)
Observations, as statements
Questions

🚨 Problem we encountered 🚨

As we were tranferring our data into the Miro, there were handful of open-ended responses where one person would write a lengthy response which can contain various themes. Even though it might one single person, we have decided to prioritize open-ended response with lengthy and in-depth details

The above response includes at least three distinct themes. To effectively use this response as a datapoint, we needed to condense it into a usable format.

Example of condensing one long passage (response) into three distinct sticky notes

STEP 3

Organizing the stickies

Because there were so many individual sticky notes, we decided to color code and tag stickies based on the region and neighorhood. This allowed to make the pattern detection much more efficient and keep in acocunt for each neighboord and region

Example of the stickies with Region and Neighborhood

STEP 4

Clustering the stickies

Once each sticky was prepared, Reseracher and I started dragging them around the board to group theme by theme.

However, we did not have a predetermined list of themes or number of groups. Themes naturally emerged from this coding process and were at times modified iteratively

In the end, we settled on 11 broad themes to classify our data

Civic Engagement
Policing
Housing & Homelessness
Mental health & Health services
Environment & Sustainability
Infrastructure & Transport
Education & Training
Jobs & Business
Minority support & Welfare
Safety & Emergency preparedness
Miscellaneous

Another Problem I faced...

Some stickies included content that could pertain to more than one theme

"Teach civics in schools" can be classified as both "Education & Training" or "Civic Engagement"

So it was important to rely on my judgement as to how to classify a sticky and which bucket it was more closely tied to

Conversations between researchers are important during this process to ensure inter-coder reliability

Because each theme contained multiple sub-themes (e.g., civic engagement included responses relevant to city hall, city council, and neighborhood councils), we could have come up with finer or cruder classifications depending on our needs

STEP 5

Looking for Finer Themes and Patterns

Each theme contained numerous stickies that told a deeper story and added details to each theme

So the team and I paid attention to those finer details and further grouped the stickies

For example, for the theme of "Policing", many people wanted LAPD to be defunded, but there were also a lot of people who did not want police defunding.

We separated those stickies and made a note of the common detailed themes we saw emerging on top of each group

STEP 6

Analyzing meanings in clusters and organic relationship between themes

Once finer clustering was done, we looked for relationships between those micro-clusters. Some clusters of information seemed to back other clusters or provide a deeper actionable suggestions.

Our team decided to use boxes, arrows, and a flow organization on Miro to create a clear information

Few examples are provided below:

FINDINGS

Themes (in order of "importance" based on frequency of mentions in survey)

Civic Engagement
Housing & Homelessness
Police Reform
Education & Training
Environment & Sustainability
General Mental Health
Infrastructure & Transport
Jobs & Business
Minority Support & Welfare
Safety & Emergecy Preparedness

Indirect findings:

People feel angry and frustrated - alot of grievances
People are providing their emails for follow-ups
Few respondent listed out detailed actionable plans for future improvement
Other provide a list of grievances without an actual plan or suggestion for action
Comments on how the survey was written (ambiguity in questions, redundancy, etc)

Actionable Insight

Then we have organized our findings into a visualized chart where our data team can triangulate our data set with their own. Within each themes, we have gave a bullet points of specific needs that people needed.

Next Steps

Visualization of ranked themes by region
Align the UX insight with the previous finding on quantitative data analysis

Based on our findings, the designer used our insight into the creation of SEIE Survey Summary Map

Illustration of our final product

My Learnings

Conversations between researchers are important during this process to ensure inter-coder reliability
Iterative design and participatory research approaches are critical when working on a project that involves active communication and collaboration
Using thematic analysis is useful when trying to find a meaning within a large set of data set

Page updated

Google Sites

Report abuse