Lucia Rojas-Smith, M.P.H., Dr.P.H., BreathEasy Evaluation Lead, RTI International
Summer arrived in Maryland over the holiday weekend with searing temperatures in the 90s, high humidity and a plague of black biting flies. I don’t ever recall seeing spring leave in such a huff without so much as an “until we meet again.” My only mental escape was to begin planning for my annual retreat to Deer Isle, Maine. The escape was short-lived as I also needed to finalize the BreathEasy evaluation protocol and write this post about it before the weekend was over. It occurred to me, as I was pulling weeds in the sweltering heat, that planning for evaluation and planning for a trip are really very similar. In planning an evaluation or a trip, what you take and what you leave behind set the stage for the journey. If you’ve planned well, your chances of achieving your goal without undue stress are exponentially enhanced. If planning goes by the wayside, you may find yourself buying an overpriced pair of dress shoes (assuming you can even find a department store two hours before the wedding) or wearing running shoes with your suit and becoming the butt of family jokes for years to come.
As I pack for a trip, every single item in my suitcase is put in and taken out at least twice. Each item has to pass three criteria: 1) It must be essential; 2) It must complement at least two other outfits; 3) The amount of space it takes up must be commensurate to its utility. The criteria for selecting the final measures for the evaluation are essentially the same. All items that make the final cut must: 1) Be critical to the measuring of the process or outcome; 2) Complement other items to form a more cohesive understanding of the process or outcome; 3) Be worth the burden of collection.
Good evaluation, much like packing well, involves trade-offs, so identifying what is critical involves careful attention to a logic model and revisiting assumptions periodically. We began developing the evaluation many months ago. Over that time, we became quite attached to all the measures we selected, because each represented something important to each member of the evaluation team. It was hard giving them up! However, we now have a much clearer sense than we did at the beginning of the project of what the BreathEasy application will do and how it will be used. As we weighed the pros and cons of dropping each item, we had to ask: Do we really think BreathEasy will lead to this outcome or engage this process and, if so, why? Is it even the most important outcome or process? Without a logic model and a set of explicit selection criteria it would have been very difficult for us to make the cuts in a reasoned fashion. Even so, we may have more measures than we can collect, so the decisions become even more focused on the critical pathways to key outcomes.
Deciding what to bring and what to leave behind is largely determined by what you want to do once you reach your destination. The BreathEasy application, we hypothesize, will lead clinicians to make better – or at least more timely — treatment decisions, which will result in improved asthma-related health outcomes for their patients. The health outcome measures easily make the critical list, but as we envision how BreathEasy will be used by clinicians it becomes obvious that we have to understand how the application will impact communication and clinical workflow. If we reach the end of the journey and cannot explain why clinicians changed (or didn’t change) their treatment practices, the whole trip would be a bust.
However, capturing communication and workflow in a clinical setting or, as Co-Investigator Stephen Rothemich says, “following the bouncing ball” is a daunting task. We developed a laundry list of measures for our clinician baseline and follow-up surveys, but concluded that these items were impractical and highly imperfect methods of capturing a complex process that changes daily and from patient to patient.
So, short of following clinicians around the office with clipboards for six months, what is the alternative? The alternative is leveraging the analytics from the BreathEasy dashboard and developing a process to document communications among the nurses, doctors and patients in response to BreathEasy. The timestamp and unique user identification number allows us to readily track the frequency and duration of BreathEasy use by clinicians. More granular information is captured in a message abstraction form that nurses complete after they review the dashboard. The form is designed to track each communication to a physician or patient to its final disposition. We will be able to assess, for example, how many dashboard reviews resulted in prescription refills or calls to the patients, as well as the reasons for the calls. We’ve tried to keep the form simple, but we recognize that it may be burdensome to complete – especially if there are multiple points of communication and interaction involving the physician, nurse and patient that need to be documented.
We also recognize that the message abstraction form may overlook some key step or process. Another truism of trip planning applies here as well: some of the best moments on a trip are unplanned, and being open to the moment is as essential as a guidebook or GPS. BreathEasy is moving into uncharted waters; we can’t possibly plan for or foresee every possible impact it will have on communication and workflow. Validating our hypotheses about these processes may be gratifying, but it’s the prospect of learning something new and unexpected that makes the trip memorable.
Learn more about BreathEasy.
Read more posts from the BreathEasy team.
