Analytics vs. traditional user research, why do you need both?

Funnily enough, soon after publishing my previous post about measuring the business value of design, I came across to a excellent blog post about the negative side effects of design metrics and data driven design.

In the recent years there has been quite a buzz inside and outside the design community about Big Data, A/B testing, analytics and data driven design. On one hand, Big Data and analytics are seen as sources of “hard facts” and thus a better basis for making objective business decisions than the “softer” data provided by most of the traditional user research methods. Especially some managers feel that in this era of analytics we can forget about the traditional user research methods and rely on analytics only. On the other hand, there are some designers & design researchers, who see basing design decisions on the analytics data as a killer for creativity & originality of the design solutions. So where lies the truth? As usual, it’s not black and white. It’s not the data itself that is good or evil, it’s more about how you use the data and different types of data have different uses.

What vs. Why

Both traditional user research methods and analytics provide tools to capture behavioral data i.e. data about what people actually do with your product, which can be quite different from what they say. This data is very important in order to identify potential usability issues and in some extent, analytics can also be used to investigate the potential causes for these issues.

However, using analytics it is nearly impossible to capture attitudinal data i.e. data used to understand and measure people’s motivations, expectations, perceptions, emotions and mental models regarding the product. Without this kind of data it is difficult to understand why people use the product the way they do or why they do not use it. This kind of data can also be utilized in design phase in order to define e.g. the content architecture, navigation structure, brand and visual style of the product, which are design artefacts heavily relying on the user’s mental models, experiences and emotions.

Quantitative vs. Qualitative

It has been often argued, that analytics data is better than traditional user research data, because with analytics it is easier to gather and analyze larger amounts of data. But is bigger really better? Or is it just different? Quantitative data is really good for answering questions like how many, how often and how much. This kind of data allows designers to prioritize identified issues and focus their resources on improving those, that have the most significant impact on the usability and user experience. Also, when measuring something subjective, like emotional response, it is easier to feel confident about the results, if the amount of data is big enough.

However, quantitative data does not give any insight on why people do what they do and how to actually fix the identified problem and this is where qualitative data gathered using traditional user research methods can help. Also, quantitative, analytics based data is only about the existing users, whereas qualitative data can bring valuable insight about people who currently are not using the product. Having different types of data from multiple sources creates a more diverse picture about the actual and potential users of the product and generates actionable insights.

Before vs. After

The context of use varies between different data gathering methods. Using the traditional user research it is typically only possible to gather data about de-contextualized and/or scripted use of the product. This makes the data more inaccurate compared to analytics data, which measures the actual, natural use of the product in real context. However, you first need to build the product in order to get analytics data.

Therefore, different types of data become useful in different phases of product development. In the beginning, when you are trying to gather new ideas and opportunities for the future development, user insights gathered using traditional methods like customer interviews can be a great inspiration to this work. Also, in the early phases it is possible to validate the initial business model, value proposition and intended customer journey with targeted end users which greatly reduces the risk of developing the wrong thing. During the product development phase it is useful to be able to validate the design decisions before actually investing into developing them and in this phase basic usability testing can greatly reduce the cost of development. Finally, in the end the product is released and in this phase analytics is a great tool for measuring the actual user behaviours and how the product is doing in relation to the business goals and KPIs set in the beginning.

UX is not just about numbers, it’s about feelings

As a summary, we can’t expect any type of data alone to address all the possible aspects of usability, user behaviour and user experience. As designers, we need to find a way to get all the different data types working together so that the data creates actionable insights to justify the design decisions. We need to understand what type of data is needed in different phases of the product development and use appropriate data gathering methods in each phase.

But even when we have all the possible data, user experience is not about numbers, it’s about feelings. So in addition to gathering data & facts to tweak the design, the designers need to use their creativity and imagination to provide experiences the users will fall in love with. Let the data be the inspiration to great design, but not the only guiding principle!