Opening a new product in a new market raised lots of questions. We needed to find answers, fast. Cue the Insights Bonanza!
A bit of background as way of introduction
At Dreams, a Stockholm fintech company, we recently launched our first B2B2C app. Having had prior experience only in delivering our offering directly to customers, the new product offering had us wondering: who are our new potential users? How do they currently solve their financial problems and how do they perceive our new offering?
As our company moved into unfamiliar B2B2C territory, insecurities bloomed. With just a few months to launch, the teams who manage, develop, design, and sell this new offering had to make a million decisions a minute without answers to important questions. In daily’s and syncs we heard people ask, “Are we even building the right thing?”
We in the insights team saw this as an opportunity for impact. To help answer those questions and reassure our colleagues, we came up with an initiative to bring product teams closer to potential users. Thus the ‘Insights Bonanza’ was born. An event that would allow – through research democratization – different teams to get the answers they needed by speaking, observing, and listening to potential users themselves.
Our goal was to have our colleagues take part in user tests of the new app. Them being moderators, observers, or note-takers, empowering them with the opportunity to get the answers they need first-hand. The insights team recruited, scheduled and created research material for the Bonanza. Our colleagues from the product teams simply had to show up and participate.
We circulated the idea to different teams, gauging their interest in such an event. Colleagues with previous experience with user testing knew the value this would bring. And they helped us highlight the need for such an event.
First, we planned
Creating focus through consensus
The Bonanza, like any research project, needed clear objectives and research questions. Using a simple form, we started collecting input from those who work on the B2B2C product, asking them what they see as the main challenges and what information they would need to overcome these challenges.
After going through the responses, we identified four main themes. The need to:
- better understand users’ financial needs in different markets.
- know how potential end-users in different markets differ from those in our home market.
- understand how our offering compares to the way people solve their financial problems.
- understand how potential users would use our savings product.
In the form, we asked about potential roles our colleagues wanted to explore. We hoped that, by asking them about potential roles they wanted, enough of our colleagues would voluntarily sign up to be moderators. This approach wasn’t very successful. Only one colleague volunteered to moderate while all the others asked to be observers or note-takers. We found that our colleagues needed more information about the event and about the different roles to feel comfortable signing up.
We shared the results of the survey with our colleagues. And we found that some team members actually did want to moderate, but didn’t want to take the opportunity from others.
But once we expressed our need, those with previous user testing experience stepped up and filled the roles. We learned that sometimes being more direct is helpful when signing a team up for research democratization.
Equipping folks with the tools to do the work
Once we formulated our objectives and research questions, we created a shared Bonanza toolkit. This toolkit was available via our intranet to all who would participate. The toolkit included everything a participant would need to know before taking part in a research session; the group they belonged to, our moderation guide, tips on how to take good notes and the overall schedule for the week.
In the toolkit, we designated the roles by teams, so everyone knew what was expected of them. Once we started seeing the toolkit page views increase, we asked if participants were missing anything, and we booked sessions with those who needed more clarification. Overall, the toolkit was appreciated by our colleagues and did its job of informing and setting the right expectations.
Putting together the teams
Roughly 80% of our colleagues who work with this product participated in the Bonanza. We divided them into four cross-functional teams of 5-7 people. We wanted to ensure that each group had a mix of competencies and experience working on the product. It was also essential for us to have C-level participants in different groups, observing how users interact with our product and hearing about their financial needs.
Recruiting the right testers
With the help of the sales team, we narrowed down two countries that made the most strategic sense to investigate. Our goal was to recruit twelve participants so that each team could conduct three sessions each.
Besides recruiting from two different countries, we also screened test participants based on their confidence in saving money. This was allowing our colleagues to see how people outside of our home market and with different financial situations feel about our product.
Sharing the recruitment details in the toolkit made it so our colleagues could connect to the overall objective of the Bonanza and better understand the value of what we were trying to accomplish.
Method & session experience
As we wanted to get insights into potential user’s financial needs in different markets, the natural choice was to conduct remote testing. We decided on a hybrid between an interview and an informal usability test. Since we wanted to know more about the user’s needs and their first experience with our product.
Each session started with getting to know more about the participant’s financial situation, struggles, and solutions for saving and investing their money. Then we shared a prototype with them, along with a scenario and task. Once they completed their tasks, we ended with questions around their overall feedback on the product and final thoughts.
Though we provided each moderator with an interview script, we wanted them to get to the answers they felt they and their team needed. It was great to see them dig deeper into their assumptions, demystifying things they’ve heard from our home market and building new perspectives on challenges a new market brings.
Observers and note-takers
The majority of the bonanza teams were taking on the role of observers and note-takers. Therefore, we had to be sure to create clear and simple formats for them to conduct their roles. Our whole aim with this project is that it should be as easy as possible for people to join in and contribute.
To keep things simple, we used something teams were familiar with from previous research projects: the ‘Wall of Justice’. It is a template we’ve used to collect feedback during user testing that makes for simple note-taking as participants move through a prototype.
We adapted the wall to include all of the manuscript questions and the screens of the prototype. Which made it easy for everyone to know where their notes should go. It also included the market and confidence level of the group of testers the team would be speaking to that day. We included short profiles of the testers provided by the recruitment service we used. That allowed our colleagues to get a preview of whom they were speaking with before and after the session.
At the end of each participant’s row on the board, we added our research questions. Which would become useful during the debrief and analysis. We found that adding this at the end of each session made it so our colleagues left each session with something valuable. Especially since many of them wouldn’t be able to participate in the analysis.
Sessions were scheduled to have a 30-min debrief afterward. We started with a round table where we shared our thoughts on the overall session and who we had spoken to, followed by going over the notes and bringing over the evidence we gathered that helps us answer our research questions.
It was during this period where Bonanza participants got to speak about what they witnessed, their thoughts and feelings on the experience of our product and how relevant we are according to different user needs. Looking back, it was the most valuable part of the entire Bonanza and according to our colleagues, they really felt open to discuss previous beliefs they held about our product and how people experienced it.
The event should be as easy as possible for your colleagues to simply show up and participate. Sharing the event background info as early as possible helps the participants manage their expectations. Especially for those who haven’t taken part in UX research in the past. Sharing the overall purpose, schedule, and outline of the event sets expectations. Clear up some time in your calendar to address any questions or coaching that might be needed.
Avoid having one tester represent the “truth”. For some colleagues, this might be the first time they take part in UX research. Listening to user interviews or usability tests can really influence them. We took time during the debrief to take a step back and look at the bigger picture, making sure no one single response/feedback was influencing us too much. Having a researcher present to help lead the debrief helps keep bias in check.
Encourage colleagues to moderate a session. For some, moderating a session comes naturally, while for others it’s a big leap from their comfort zone. What we found is that running pilot interviews helped our colleagues build their confidence in asking questions and active listening, learning more about when to dig deeper. Running pilot tests with moderators also gives people a chance to become familiar with the interview guide, something they can rely on if an interview is going off course.
Include colleagues in the analysis. This can be challenging depending on schedules, but if possible the impact and confidence in the findings is much greater. If time doesn’t allow it, ending sessions with a rapid analysis in the ‘wall of justice’ allows for people to leave with meaningful information they can feel confident in.
Collect reflections and questions. During the discussions, it’s important to capture what questions are left, and how our colleagues would use the information they’ve just taken in. This allows for us as a research team to keep track of insight needs and potential future research.
In the end, the Insights Bonanza was a four day long empathy exercise. Four teams were assigned a day to test with potential users from two markets. We pinpointed the main questions and problems we were facing and created an opportunity for them to get answers themselves. The product teams were grateful for the insights gained from the research. And our job persuading them of research’s value was made that much easier because of their participation. Win, win.