First Steps in Research #3: Hard Lessons
Welcome to the final edition of the series First Steps in Research.
User Research as a discipline is still in the rather early stages of standardising training for Junior Researchers, even though there have been huge leaps in recent years. For juniors without mentors and another research discipline as background, it can be a very tiresome journey of trial and error. Often, their role demands them to hit the ground running.
For our 3-piece mini-series, we spoke to very experienced researchers (many of them textbook authors, engaged mentors, and/or UX thought-leaders) about their personal stories of learning from mistakes, peers, and mentors.
Check out our blog to learn where they came from and catch up on the first two parts of the series.
What rather naïve misconception did you have about UX Research before you started doing it yourself?
I thought I would be doing research all day and can focus on doing so… Nope. A lot of my daily work is education and communication. Educating the people I work with about research, what it can do for them and its limitations. Communicating insights and help, or gently push, their implementation to create better experiences.
I sort of imagined that just doing the user research will magically tell us what to build while creating a software and that will also make sure that software is successful. Once we started to do usability tests and interviews, I instantly realised how acting on insights (sometimes even just agreeing on what the insight is) is not that straightforward. Working with research is so much more than just the act of executing research.
I came from academia with a lot of arrogance about qualitative research. To me, it was chaotic, results were random, and qualitative researchers were just people who were afraid of numbers. Turns out, just like with coffee, I simply had never experienced it done well until my mid-twenties. Nowadays, qualitative research is my bread and butter. There are so many real-life research questions in the blind spot of quantitative research - we just cannot afford to refuse an answer just because we are more comfortable with the (pseudo) precision of numbers.
I thought UX research is only about prototype testing and finding little bugs in the design. Little did I know.
What important lesson about UX Research did you learn from your first mentor?
One of my professors always talked about ‘real world empathy’. We learn by experience, we cannot anticipate all the different types of users we study, so we need to expand our horizon. “Slip into their shoes!” - I literally did this by participating in search and rescue training missions to understand how these people think, talk, and operate. This gave me the most valuable insights into understanding their needs and how to develop valuable, usable and accepted tools for them.
Always let the participant know early that they are on track, this is exactly the kind of detail and information you came to learn about. A lot of times participants aren’t familiar with the research interview situation and positive feedback makes them feel more at ease. You need to make them feel comfortable. This invites them to open up and share their stories.
It was back at the university where I first learned about usability. I worked with Gergely Krammer who was a professor there. He talked about the importance of getting better by keeping a log of how each session went and reflecting on the outcome of the session. This way further sessions and nuanced things like how we ask each question can be improved. By now, I know that introspection by itself is not enough for growth, but checking back on past issues and events is usually insightful.
You should always make sure your participant leaves the session with the warm and fuzzy feeling that their input made your day. It's a matter of research ethics. And actually - even hearing the same feedback for the 5th time is not boring, it cements your findings - which is a great thing.
It is all about storytelling! You might be a star in setting up research, you might be an amazing moderator or interviewer. But if you cannot present your results in an engaging story, nobody will want to listen or act upon it.
Tom Illmensee, Product Strategist and Founder of Orbital Music Park
The quality of research outcomes reflects the amount of preparation before the first session begins. Have a research plan and a protocol mapped out. Never wing it.
Any major fuck-ups during your first studies? Want to share?
In my early days as a UX Research Intern at SoundCloud, I was tasked with writing my first findings report to conclude an intensive discovery research project. I thought to myself '“Well, Basim, now you have to write the perfect report.” One month later, my perfect report was born. I organised a 60 minute meeting with the project stakeholders to share the full findings and just as I opened up the slide deck, one of my colleagues said: “Yeah, we don’t have time to go through this whole thing, can you just share the highlights with us?” Pretty eye opening realisation. Here’s a selection of some more report-related blunders: 30+ boring slides of 10 pt text. No overview of insights or takeaways. Sharing findings weeks after the team needed them. Anecdotes and details from users that had little relevance for the project. So yeah, I’d definitely advise against doing all of the above. And definitely check in with your stakeholders and team to set mutual expectations about research deliverables!
I was and very likely still am pretty bad at directly recruiting people for a study. I remember during my early days in the gaming industry we were looking for participants for a user study and we didn't really have a budget for professional recruitment. So we had to do it directly ‘on the street’ ourselves. A colleague and I went to a park, where we knew we could find people from our target group, who usually spent their break from work or university there. And we approached this young woman, who was sitting on a bench and started to make our introductions. And at that very moment this lady starts crying - we were completely shocked and puzzled because we basically had only said “Hello” and that couldn't have triggered this emotional reaction, so we asked her if something was wrong and whether we could help somehow. It turned out that this woman just received some bad news from her doctors and we had stumbled into her when she started processing these news - long story short, we did not recruit anyone that day, we sat with this woman for - I don't know how long - to just listen and comfort her and afterwards we ended the day because my colleague and I just didn't have the heart to continue recruiting. On the next day, we sent someone else out to recruit participants for our study.
Major fuck-up during my first studies - no. As a Market Researcher, I was trained in a system where I learnt on the job, meaning: I accompanied a senior and shadowed her. Later on, when running my own studies without a senior supervisor, I have endless stories of fuck-ups. Biggest one? We tested visuals for a shower gel in the USA - the agency was from the UK, the client headquarter in Germany, I am German as well. No American native speaker had a last look at the concepts we tested. We did test shower gel concepts and the participants in the USA went crazy about it, loved it, wanted to have it, it felt so new and innovative. Well, that's always great. However I felt something was wrong. And it was - the term ‘shower gel’ (which is used in Germany and UK) is hardly known in the US. The correct term is 'body wash'. The research design had created a huge bias and research outcomes needed to be taken with a lot of caution.
I may have fucked up almost as many times as I did studies, so I guess that's the name of the game. Maybe not a major fuckup, but when I first started to do usability tests I was not very good at asking questions, or indeed setting up or guiding the user through the session. We were sitting in a meeting room with huge glass windows and the user just aced the test, performing each task flawlessly. I noticed only after the session that a developer was playing with the same exact prototype right before the window on two huge monitors. We needed to repeat this session.
The first years as a User Researcher are just as much (if not more) about learning how to be a better researcher as they are about learning the answers to your research problem. Get all the support you can get. Put the well-being of your participants above all else. Ask for help. Steal processes. Watch what other researchers do and ask them why they do it that way.
Mistakes happen, they only turn into failures if they are handled badly. Meaning, if you don’t learn how to do the same thing better next time. If you are not open to feedback and tips. They turn into failures when you try to cover them up at the cost of reliable insights. We just heard a number of stories where great researchers acknowledged their mistakes and either redid the study or flagged the results as biased.
Preparation, sanity checks, and dry-runs are key. And make no mistake - even guerrilla research requires a great deal of preparation. Just as Katharina pointed out, we can never anticipate everything users will do - which is obviously why we need to do user research in the first place. Yet, trying to understand in advance what is expected of you, your research, and what could be the pitfalls of your study design, is key to flexibility and improvisation when the unexpected happens. Not just during the research, but also when sharing the results - as we learned from Basim.
Or as Tom put it in three words:
“Never wing it.”
You enjoyed the read? Let us know on our social channels, spread the word and reach out to us if you have other questions you always wanted to ask senior researchers.
