As a UX researcher in Schibsted Media Group, I try to get everyone in the team to tag along, observe or facilitate research sessions. Journalists working on our app Peil frequently moderate usability tests, and we collaborate on the analysis of these.
However, there is still one part of the research process that too often falls to the researcher alone: the analysis of in-depth interviews. And why shouldn’t everyone be involved in this as well? In this article, I’ll share why and how we do this in our team, with examples of two ways of collaborating on the analysis.
Benefits of collaborative analysis
Qualitative research is about understanding an experience, and each participant will give you a different perspective on it. But problems can occur when we put more weight on one participants’ account over the others. It’s an easy thing to do — it could be that you personally only talked to one person for the study, or you got along so much better with one participant than the others, or someone’s opinions were closer to your own, and the list goes on.
Meetings become shorter and decisions are made faster, with less time wasted arguing about what we think the users would want.
Overcoming these biases is the hardest thing to teach. Bringing the team members together, comparing notes and finding commonalities — or the explanations for discrepancies — can be a great way of getting everyone on the same page. It allows everyone an opportunity to get to know all the research participants, as well as get an idea about how the experiences they had with their interview compares to others.
First-hand experiences are always more valuable than passively listening to a presentation. This goes not only for the interviews themselves but for analysis as well. By doing it together, everyone becomes more invested in the outcomes of a study.
The simplest way of doing a collaborative analysis
When I do interview analyses by myself, I prefer to colour code all the notes by interviewee, and then cut sections from the notes into a master document arranged by theme. Once I have everything in order, with a happy rainbow of notes and quotes for each research question, I look for patterns and meanings. With this in mind, I simply tried to create a workshop out of this process.
Our first collaborative analysis workshop was for a study on push-notifications, and I just printed the colour coded interview notes. Going one research question at the time, we cut out the corresponding sections and quotes from all six interviews and looked at them together. What does this all mean, do they have common opinions? If not, do we understand why? I felt we had succeeded in doing a proper analysis as a team, and everyone was on the same page on how to use the findings. I couldn’t wait to try this again! My one improvement would be to bring enough scissors, so we didn’t have to wait to use them.
Our research goals in the notifications study were very specific, and the interview notes were only about a page for each participant. These two points turned out to be crucial to the success of this collaborative analysis method. The next time we attempted this analysis workshop, the material was in-depth interviews exploring a new theme, with five pages of notes per interviewee. The shortcomings of the process were immediately evident: it took too long to read through the unsorted, raw notes, and the amount of information for each research question was too much to digest. We ended up with most of the workshop participants just defending the standpoint of their interviewee — one of the things we were trying to avoid by analysing the material together.
It was clear we needed to do more preparation to make the analysis more efficient and distribute the knowledge better. So for the next project, we iterated on the process and came up with something that works for more substantial material.
Collaborative analysis of in-depth interviews — step by step method
Step 1: Make sure your notes are all in the same format
For the collaborative analysis workshop to be efficient, you need to sort the notes before the session. You also need to consider how to break the material down into chunks that are digestible in a discussion.
For our team’s first go at this method, our project was understanding retention for our news app Peil. Six different team members interviewed 8 of our users, and the extensive notes followed the winding path of the interviews themselves.
To ensure that all the information was structured in the same way, I had all the interviewers fill in a Google form with their notes. Sure, it takes extra time for everyone in preparing for the workshop, but it’s an essential step for the session to be efficient. It also serves as the first round of digestion of the interview, where the flavourful but uninteresting things are left out. So never mind the grumbling about how research is stealing more of people’s time, this is an essential step!
In the Google Form (or another tool of your choice), write questions that are in between the research questions and the questions from the interview guide. I tend to write many small interview questions to break up difficult topics, so in those cases, I prefer to gather them into one or two overarching questions in the form. It’s a balance between having big blocks of text that are hard to digest in the workshop, and having to look for information across different columns with the potential risk of missing things. You probably have to do this a few times to find the level of granularity that works best for your team.
I recommend making the form after you’ve done at least one interview. This allows you to adjust the questions in the form according to things that came up, and the things the interviewees were the most interested in. You might need to add a question to break up the detail level for some of the hypotheses. Or maybe you can remove some detail questions about an idea that wasn’t interesting at all. Make sure the form is ready for when people have written up their notes. If someone fills in the form and then you change it afterwards, it will mess up the data output.
Step 2: Prepare the room and materials
During the workshop, you want all participants to have equal access to the data, and for all of them to interact with it. Our solution is printing everything and putting it up on the wall.
I usually download the .csv file from the Google Form, then adjust the text wrapping for all cells and make the columns wider to increase readability. For the first workshop, I printed this on A3 paper in font size 18 and hung them up on the wall of our meeting room. If you have a small team, 2–3 people, you can get away with using a smaller font and less paper.
You can still identify each interviewee as their responses are represented horizontally, and the research questions vertically.
Step 3: Assign key roles
You want all team members to be part of doing the analysis. However, consider assigning someone to be in charge of taking notes and leading the discussion. A small team may do well without a leader, but for groups larger than 3 it’s great to have someone deciding which points to discuss first, and when to move on to the next thing.
Everyone in the Peil team was invited to participate in our workshop, and we ended up with an analysis squad of five. Anyone who has done qualitative analysis a few times should be able to lead the discussion. We were super fortunate to have research lead Veronica Heltne as our discussion leader.
The note taker’s job is to filter the discussion into actionable learnings, so it’s helpful if this is also someone who has done qualitative analyses before. Alternatively, these two roles could be held by the same person — just write notes as you go and then make the report or presentation afterwards.
Step 4: Work through the material
For each question, discuss what you can learn. Agree on points for the presentation as you go.
Moving from one column to the next, our analysis squad took a few minutes to read, then looked for patterns, discussed similarities and differences, and what it all meant. We had pens, post-its and markers available, and it was sometimes helpful to be able to highlight quotes, underline or annotate and draw on the print-outs to aide the discussion. The annotations and highlights were especially helpful for the complex research goals where you needed to combine information from several columns to understand the full picture.
As the note taker, I was listening keenly to the discussions and made summaries of the main points. After every section, I read the points I had noted aloud so the analysis squad could adjust if they didn’t agree, or add a good quote to highlight the findings. I was writing directly into a powerpoint presentation with one research question on each slide, as we wanted to share with the rest of the team as soon as we finished. It’s a good idea to also have a separate slide (or place on the wall) for “other insights” that come up unexpectedly.
At the end of the session, we went through the full slide-deck together to see if there was anything else we wanted to add. We also prioritised the action points that had come up for further work. This collaborative analysis workshop took us 3,5 hours, but the team really did get a common understanding of the research findings. For less complicated studies we have since done it all in 2 hours, so the timing will vary both with experience and the amount of material you have.
The future of collaboration
Having tried this a few times, I think this process of collaborative analysis works pretty well for our team, as well as for other teams in Schibsted Media Group that we’ve shared it with. Depending on the material, the workshop is sometimes faster than our first one, but never longer. We haven’t gone back to using scissors to cut up the interview notes during the session, but that could be a quick option for lighter studies.
I am happy to say we are also over the initial resistance to the “extra work” of filling in the google forms. Having gone through it once, the benefits of structuring the material is clear to everyone, and the team is happy to be able to go through the analysis together to collect first-hand knowledge.
I am sure we will continue iterating on the process, as one always does in product development. For now though, this works really well for us.
Is your team ready to try analysing collaboratively?
A big thank you to Veronica Heltne for sparring with me, making great suggestions and in more ways than I can count helping to develop this process.
Ane Sharma
Ane Sharma is a UX researcher at Schibsted Media Group, the biggest media group in Scandinavia. In her role, she plans and performs qualitative research for new initiatives as well as established brands. She also facilitates training sessions for colleagues from other disciplines who want to learn about UX research. Previously she was the user researcher at appear.in, a video chat startup.
