A few months ago, I watched a video on how to evaluate whether or not research was being conducted effectively. The video, which I can not find anywhere for the life of me, outlines a number of questions to consider before diving into user research. I have found these questions particularly useful when evaluating my user research process for software applications. Whenever I start doing a contextual inquiry, I first ask myself this set of questions:
- Are we measuring reasonably? Have we defined relevant and reasonable metrics to measure results with? I have found that this question is not always applicable if you are doing very qualitative research and looking to get an understanding of what shapes a users way of working.
- Are the people in the study representative? When doing user research, it is important that the users are representative of those who will be using the product, service, or application. If they are not, then the results will be not be as relevant as they could be. The risk you could run into is that the results will point your design in a different direction than it should go.
- Are the tasks representative? Asking users to perform the tasks we expect them to use the product/service for will give us relevant answers. If you ask them to perform outlier tasks or tasks that are not representative of what the system will do, our data will not help you understand how the majority of users will use the product/service.
- Is the context representative? Is the research happening within the context that the product/service would normally be used? People may not be able to remember the way they would use something as they would be replying on recall rather than recognition. For example, if you are conducting a contextual inquiry on how someone uses an application on their mobile device but they are sitting at their work desk, then you will not get as rich of information as you would by being with them when they do normally use it.
- How likely are these results to be flukes? It is important to know that the people you conducting your research on are representative and normally perform the same tasks that you are the designing the system for.
- If representatives are broken, was it replaced by a good theory? If you are not able to do user research in the same environment and with the same tools as would represent the real situation, then it is important to find a good “representative situation” that would answer the same questions as closely as possible. For example, if you need to test an application with clients, but it is impossible to reach out to clients due to sensitive information. How could you get around this obstacle? You could test client representatives, who are the first point of contact for the clients and have a good understanding of who they are and what they want.
As part of my standard UX process I apply these questions to projects in order to identify areas of improvements. I have found that in doing so, I can ensure that I am getting more accurate data to work with and and ultimately design a system that works for the users.