Linda Neuhauser, Dr.P.H., Crohnology.MD Co-Principal Investigator, School of Public Health, University of California, Berkeley
One of the big challenges we have faced in the Crohnology.MD project is developing an evaluation plan. We are all familiar with the trade-offs between using qualitative and quantitative methods. If we choose a primarily quantitative approach, we may have statistical results, but may not really understand the phenomena being tested. If we select a qualitative approach, it’s hard to know whether the results would be similar if tested on a larger cohort. This is the typical tension in evaluation design. My view is that traditional evaluation methods are not well suited to novel e-health experiments, like the Project HealthDesign projects. I’ll start with a personal story:
Last March, I participated in a conference on Artificial Intelligence and Health Communication at Stanford University. There were many fascinating state-of-the-art projects. I was especially intrigued by a project using avatars as health coaches. The developers described how they gathered some preliminary (formative) information at the project outset, implemented the project, and then quantitatively evaluated the outcomes. The developers and the funder considered this an acceptable design, as would many evaluators. However, the results uncovered an important problem—the users did not find the avatars adequately believable and that interfered with their engagement with the virtual coaching. Similar surprises cropped up in other projects that were presented, even though the developers of the projects were highly talented, had given careful thought to the evaluation, and had used both qualitative and quantitative methods. As a result, a number of these projects will now need to be re-funded, redesigned and re-evaluated. I left the conference with the strong impression that some of these problems could have been avoided if the developers had used different evaluation methods. Certainly, e-health evaluation is an area that needs a lot more attention.
I believe the roots of the evaluation conundrum lie in the foundations of scientific inquiry. Before the mid-20th century, the dominant “positivist” view was that truth is knowable and generalizable and the focus was on discovering the laws that govern the physical world. This perspective advanced the idea that quantitative methods (like randomized controlled trials) should be the gold standard of research and that their findings could be broadly extrapolated. In the past 50 years, the scientific paradigm has shifted. In the “post-positivist” world, the “critical realists” have emerged and posit that it is impossible for humans to adequately perceive the real world, and that claims about reality must be subjected to the widest possible examination.
There is now a greater understanding that it can be very difficult to define what we want to study, much less quantify the results. This is particularly true when we move from the natural sciences to human sciences—people don’t follow the more predictable laws of the physical world. In the health arena, we now have a host of social sciences to help us understand the connections between people and health. Changes in evaluation design have paralleled changes in views about scientific inquiry. Qualitative methods are considered increasingly important to achieve a deeper understanding of the complicated nature of human health and behavior. Likewise, there is a growing recognition that formative work needs to take place before investigators even know enough to finalize a research design that focuses on outcomes.
Scientists have identified a newer research challenge: one posed by what is now called “design (or artificial) sciences.” This field is concerned “not with how things are, but with how they might be.” (Simon, 1996). Those who study design sciences say that they pose special challenges for research methods because understanding a problem and solving that problem often happen at the same time. Similarly, there is no “end” to the design and problem-solving process. Obviously, design science efforts are not well suited to traditional research approaches that test a priori hypotheses. For this reason, design sciences emphasize real-time, iterative investigative approaches, such as usability testing, over the more simplistic “before” and “after” methods of traditional evaluations. In other words, just as we are experimenting with observations of daily living (ODLs) to get finer, time-sensitive information about participant data, we need to do the same in our evaluations.
In my view, the Project HealthDesign projects, which create new technologies and measure complex daily interactions between people, their environments, and their health, lie at the intersection of human and design sciences. Effective evaluation designs for these projects will need to include a strong focus on usability and other iterative testing over time, artfully linked with formative and outcome measures. There is no definitive guidance for such evaluation designs. Fortunately, the Project HealthDesign teams have taken a pioneering experimental approach to finding a better path forward.
Have you successfully employed both quantitative and qualitative evaluation techniques on a project? What advice can you share with us?