The Problem with Validity

How much of what I just discovered is my own beliefs reflecting back on me — lightly but insignificantly embellished by the research completed?

This is a concern with any research-driven discovery process — and one that I’ve constantly struggled with in building independent products. An insight might seem novel or part of a meaningful process of synthesis, but there is always considerable risk that you’re simply reflecting your own biases, beliefs, and existing knowledge back onto your problem. You engage in the motions of research, while actually just subconsciously rearranging your findings to fit what you already believe. Independent work suffers from this most because we don’t have the teams that form structural safeguards to prevent that sealing off of our growth.

This is often what we talk about when we talk about validity — the degree to which our research findings map on to reality. But validity isn’t so easily measured in the type of qualitative research in which many designers and applied researchers engage. Thankfully, we have a natural bulwark that can help surface shield us from our worst impulses on this — if we know where to look.

The Utility of Theory

Validity. Usually, we hear the words reliability (are the findings consistent and accurate?), generalizability (are the findings generalizable/relevant to the population being studied?) and validity (are the findings reflective of what we intended to measure?) tossed around together, and they can’t really be separated, but we’re going to focus on validity. The usual visual model is that validity is hitting the bull’s eye with a date, and reliability is hitting that same bull’s eye five times.

Research is fundamentally concerned with validity because it’s really at the foundation of what we do — and the value that our work offers. Research with high validity offers a deep and previously inaccessible understanding of the world, whereas low validity research presents as a false understanding of that same world — wrapped in the garb of legitimacy.

The problem is that you can do everything else right and still learn absolutely nothing. Worse yet, others might look to your research process and see that process as legitimating for the invalid research findings — pulling others into the same erroneous understanding. Thankfully, this tends to be where theory can come to the rescue.

There’s a number of philosophical traditions in qualitative research, but the starting point for validity’s dominance sits in the positivist tradition, which (oversimplified) suggests that there is a single truth that can be derived from research — basically that there is a truth waiting to be discovered and mapped by the researcher. Personally, I disagree with this premise in qualitative studies, but it’s worth understanding that that foundation in qualitative research did emerge from principles of scientific knowledge and discovery — and that this idea of “truths” (objective realities) has been superimposed overtop of human experience and culture.

Post-positivism works to address some of this thinking and is more what we tend to see these days — acknowledging the subjective nature of researcher observation and arguing that truth itself will always be uncertain. This is because the instrument of measurement (the researcher’s observation) can only have a subjective and biased viewpoint. This is useful because it shifts the baselines for validity into the realm of possibility (i.e. something CAN actually be valid since the benchmark for validity isn’t impossible), but still situate the researcher as observing some objective truth imperfectly.

Thankfully, post-modernism comes to the rescue at this point (from my point of view, anyway). Post-modernism (and many of the traditions that sit under that umbrella) doesn’t suggest that reality and truth are constants — but rather that the observers of those realities co-equally construct the world around them from the limited positions that they occupy as observers of the world. If two people are standing on opposite ends of a train station at a moment in time as a train arrives, to the person on the far end, the train might appear flattened and short. To the person nearest to the train’s entry point, the train might appear long and loud — stretching far into the distance. One might experience that train from a stance of fear as a huge machine thunders past them; another might experience it relative to hope and opportunity thinking about where it might take them when it pulls into the station. Both observations of the train hold equal value in this case, and so we can derive a certain amount of validity in our research by collecting and recording these observations so that enough of the phenomenon of the train can be recounted with validity — enough that others might partially understand what it is to experience Train.

Post-modernist qualitative research then situates the researcher as observing and co-constructing phenomena with those who observe and experience them. Because there isn’t a single truth, but rather many truths co-constructed by the observers and experiencers of the phenomenon under study, the validity of any given research endeavour is itself subjective in relation to the phenomenon being observed and the construction of validity itself.

Why Protection

Post-modernist qualitative research can — initially — seem like a bit of a cop-out to the more positivist-leaning disciplines. It can feel a bit like you’re simply ducking the validity question altogether — arguing that because there’s no objective truth, we needn’t even bother. How can someone be confident with the results then?

But what’s missed here is that while we don’t believe that there is some objective truth about people waiting to be discovered, we do believe that there are shared and common experiences to be understood and communicated. These experiences in turn can service the creation of the new and the maintenance of the old — building for and with those communities who participated in the research. Because post-modernist research is starting all conversations with the assumption that the experience of the participant is valid and worthy of consideration, it empowers the practitioner to more fully engage with the landscape of information and experience surrounding the domain of interest. This naturally — almost inevitably — leads to more exhaustive and (arguably) valid findings.

This gets back to that question of how this frame of research protects us from some of our own biases. Doing qualitative research in the postmodernist vein is necessarily participatory and reflective — situating the observer (you) both within the context of observation, and acknowledging one’s own biases and role in transforming the experience of the participants. What naturally tends to emerge is a documented and synthesized reflection on one’s own positioning relative to the research that — ideally — can serve as an effective foil to unconscious bias as the study progresses.

If we acknowledge that we can’t naturally observe experiences objectively, and instead include that assumption as part of our research, it forces us to design around the lack of objectivity. When we’re designing around that fallibility, then we’re building natural safeguards against it. But if we don’t take that step — and instead assume an objective stance in lieu of perceived expertise or role — then we’ve left our research unprotected against its greatest threat: the researcher.