A UX COMMUNICATION METRIC: DATA CONFIDENCE SCORE
Background. Some consumers of UX research (our internal stakeholders) may not be familiar with our research methodologies or may be skeptical of our findings, especially of qualitative research that uses smaller sample sizes. Our team wanted to develop a multi-dimensional metric to communicate to stakeholders that the more confident we are in our data, the more assured stakeholders can be to make decisions based on the research findings.
From this we developed the Data Confidence Score, a subjective evaluation of a qualitative data set. We identified 6 dimensions of our research design process that influence how reliable we feel the data are:
Quantity of participants
Expertise level with product
Representativeness
Response quality
Data consistency
Making inferences from data
Method. Researchers conduct this evaluation at the end of the project which includes a subject score from 1-5 plus comments about lower scores (risks). Importantly, a researcher does not review their work alone. The score is an average of all the dimensions included in the research project. This score is reported along with the findings. A rubric (below) is also shared.
Results. We rolled this out with an introduction to the metric. Stakeholders, most of whom are product and engineering folks, were excited to be able to have a quick way to know how successful the research project was at meeting the goals, which is interpreted as higher ratings on each dimensions. We found there was less questioning of our methods and findings, and more alignment on next steps. Our goal is collect this metric over time and to begin to evaluate its validity and reliability over the long term.
Rubric for the Data Confidence Score communication metric.
Thumbnail photo: www.tcnorth.com