Research Debt: Report your study or it didn’t happen
Have you ever felt like a detective, trying to track down the source of a peculiar myth about your users? Or learned only half-way into a research project that something similar has already been done recently? Maybe a stakeholder asked a simple question and only the researcher on leave knew the answer?
One might call it a side-effect of lean, agile research. I’d rather call it ‘Research Debt’. I’ve been there quite often. It’s frustrating. It’s inefficient. And it’s unnecessary.
Admittedly, running research for agile teams is tough: small budgets, few resources, constantly changing business priorities, and little time to finish anything before you need to hop on the next project. Often, UX Researchers (and researching PMs or Designers) are tempted to compromise on documentation to get more done in the limited time they have. Whilst skipping documentation and sharing can speed you up short-term, the price to pay mid and long-term is very high.
What is ‘Research Debt’?
Borrowed from the term Tech Debt, the fast pace of product development can also take its toll on User Research. Since the term is not fully established yet, it could mean three different things:
Research Debt as accumulated bias because all studies are conducted with the quickest and dirtiest Guerilla methods and convenience samples. This might be the most dangerous one, because it creates a false sense of security for business and product decisions.
Research Debt as lack of strategic user understanding, because there’s never time for generative and big picture studies. Most research done is late-stage usability testing.
And then there is the third way to read the term, which you might come across even in organisations with relatively advanced UX maturity and a dedicated research team:
Research Debt as Insights Amnesia, where perfectly good insights are not accessible to all parties concerned. Reporting and sharing are skipped to get more studies done. Stephanie Rosenbaum and Caroline Jarret called this ‘neglected after-research activities’. Considering, how many team resources go into any research, losing the final result – knowledge – is a hefty price to pay for a bit of time saved on the last mile.
Consequences of Research Debt
For today, let’s focus on the challenges of the Insights-Amnesia type because it is a headache even for quite seasoned researchers and a topic that gets increasing attention from the Research Ops community. A variety of consequences may become visible only months after poor documentation became a habit:
Individuals become bottlenecks: You think it’s fine, because you can always ask the researcher about missing details? Well, think twice. As soon this person is unavailable or left the company, this knowledge that cost so much to collect is lost forever. The bus factor, known in Tech teams, can be a risk for Research teams, too.
Limited weight as evidence: Well-written research reports can be an incredibly powerful tool when it comes to disagreements between decision-makers, or when critical issues out of scope need to be flagged. No investor or C-Level would change their position or priorities based on “It’s obvious that …” or “A user called and said…” - Product leaders nowadays are very data-savvy and pride themselves that their decisions are based on evidence, not just gut feeling. Hence, documentation your argument is sound and evidence-based is your best shot to get a foot in the door.
Limited reach: Excellent research is done, but only directly involved Stakeholders know about it. The rest of the company may continue making wrong decisions since they are not aware of the things you learned about your users.
Wasted resources: Or, similar topics are researched in parallel or over and over again, because existing insights are not known by other teams. Peers miss many opportunities for knowledge synergy and desk research in previous studies.
Room for abuse: Without a reference document providing a clear interpretation of the data, it can also be tempting for decision-makers to cherry-pick small bits in order to serve a specific agenda. And it also makes it easier to dismiss uncomfortable research insights as unreliable if the way they were collected is not documented properly.
Cloud amnesia: Insights are stored on a cloud platform that is abandoned after a while or is only accessible to a few team members (e.g., when companies move on from Confluence, ban Google Docs, or limit the number of Miro accounts). Hence, learnings cannot be shared with a wider audience nor easily migrated to the next platform. I’ve also seen key documents completely disappear from the cloud when the creator left the company and their accounts were disabled.
Context amnesia: Insights reports lack key context that is required to understand the takeaways, e.g., what the product looked like back then or how the data was collected. A person from a different team or someone joining a year later has no chance to understand what this research was about.
Lack of engaging storytelling: Sharing reliable insights is one challenge, sharing it in an engaging way is another one. Research storytelling is a key strategy to make your work visible and impactful. But just as detailed reporting, storytelling takes time. There are documentation methods that work well for documentation but are incredibly dry. And others that are more engaging but can be misleading in terms of key takeaways. Ideally, your documentation strategy serves both.
TACKLING RESEARCH DEBT - A Documentation checklist
So, what can we do about it?
Well, start documenting and take ownership of sharing. There’s no way around it. Slow down a little. Documentation needs to become part of your project planning and time management. Sharing needs to be owned by your team. Don’t rely on stakeholders to share with the rest of the team - it’s not their job nor their priority.
The good news is that there are many different formats of documentation (e.g., Toplines, Affinity Diagrams, a Slack Channel, sophisticated Research Repositories, etc.), some of them quite fast, some of them quite engaging. What they have in common is that they allow the critical reader to decide for themselves if the insights are sound or not. This can help tackling the Insights-Amnesia type of Research Debt.
A good rule of thumb could be:
Document in a way that a person new to company and product can access the information, understand the point of your study and to what degree your claims are reliable.
You can achieve this, by dedicating at least a bit of energy into answering the following questions.
Why did you do the research? What was the goal?
When, where and by whom was the data collected?
What research method was applied?
Who were the participants? How many? To what degree do they represent real users?
What did the product look like at the time?
What limitations and potential research biases should the reader be aware of?
What were the most important takeaways and action items of this study?
What detailed data justifies the key takeaways?
There is a lot of wiggle-room for the level of detail - or if you even cover the last point at all. However, if you have the slightest feeling your takeaways could be counter-intuitive, could cause resistance within the company, and/or are relevant for future product initiatives, more detailed insights are a must.
Then, there’s also the point of accessibility:
Is there easy access for all stakeholders?
Will the document stay if the creator leaves and the cloud platform is discontinued? Quick fix: PDF backup.
Do they have a chance to learn about it without actively looking?
It’s a tough list. But it is unlikely you are completely starting from scratch. Odds are you or your peers have already tried some methods that tick several of these boxes. And you can combine several sharing and documentation formats to compensate for a couple of these points. Additionally, you can use different documentation formats for different types of studies and audiences.
Tackling Research Debt is a continuous challenge. If the research itself was flawed or stakeholders are just ignorant to research in general, a detailed report or well-produced highlight video alone cannot fix that either. But it’s an important part of it. Below, I put together only a few of the many options out there.
Curious to read more? Follow our social media to learn when we release our next pieces. In the meantime, check our blog and online courses for more about best practises in UX and Market research.
