On November 19th, 2019, the Agency for Healthcare Research and Quality (AHRQ) presented Recent Insights into CAHPS Survey Modes and Response Rates. The AHRQ moderator set the stage by clearly articulating that the goal of this research is to evaluate strategies for improving survey responsiveness & survey representativeness, as these qualities are the foundation of reliable, accurate survey data collection.
The research, which many attendees (including the PRC Team) were anxious about hearing, focused on the use of web survey methodologies in the Emergency Department.
The Emergency Department Patient Experiences with Care (EDPEC) survey has been in development for several years. The survey has gone through a mode experiment and two feasibility tests. The most recent feasibility test, administered in 2018, tested web methodologies to improve response rates. Nine variations of survey administration were tested using combinations of mail, phone, and web. The results were summarized in the Effects of Push-to-Web Mixed Mode Approaches on Survey Response Rates: Evidence from a Randomized Experiment in Emergency Departments.
The 2018 feasibility test found that none of the web-survey variations resulted in significantly higher response rates from the control method of mail with phone follow-up. The overall response rate for all nine methods tested was 18.6%, which is much lower than the 2018 calendar year response rate for HCAHPS at 25.5%. The highest response rate on the ED study was 27.3%, from the approach using up to 4 emails, then a mailing, and then a telephone follow-up. In the test, the only methodologies that obtained a response rate greater than 20% were those that included a telephone follow-up component.
The most effective mode for completing surveys was phone, followed by mail, and then web. In both this study, and one presented on the use of incentives, researchers concluded that email is only effective working in conjunction with a traditional mode follow-up (mail or phone). Email has been found to have the lowest response rate and will likely be insufficient if used alone.
In terms of representativeness, telephone survey completions reached a more representative age distribution of the sampled population. Researchers noted specifically that the telephone mode was the most successful for gathering responses from younger patients. Telephone surveys also reached more patients in poor health, lower education, and minority groups. The 2018 feasibility test results article states that, “…results suggest that future protocols for this survey [EDPEC] should include a telephone component.”
In general, hospitals are eager to use web as a survey methodology. Leaders are hopeful that web will increase response rate and improve representativeness of young adults, which are currently a hard-to-reach population. Web would also allow for cost savings to hospitals compared to mail and telephone.
At the conclusion of the webinar, one of the speakers noted that web surveying has not shown the progress in effectiveness that they had hoped to see over the 10+ years that they have been testing. While using web alone is not the best option for official CAHPS measurement at this time, using a web component in a mixed methodology approach may be desirable for the cost savings it can bring. By starting with a web survey invitation, organizations can begin their data collection at a lower cost and reduce the number of patients who need to be contacted with traditional phone or mail activities. This approach would allow for better representativeness of the sample population, a higher response rate to improve data reliability, and a lower cost for administering hospitals.
We were very pleased to hear AHRQ articulate a strongly-held PRC belief, that representativeness of the data is the core of survey research, and the findings presented were consistent with what we have observed in our own multi-modal studies. There are situations, however, when cost may be an overriding factor for health care organizations, and leaders are comfortable utilizing a web-only approach, accepting the caveats that come with it, for the sake of meeting their budget. Understanding the objectives of the research are the first step in evaluating the relative importance of cost versus quality. What are these results for, how will they be used?
For official HCAHPS measurement, CMS knows that millions of dollars are on the line for hospitals across the country. As a result, they cannot take shortcuts on methodological rigor; their reported scores must be iron-clad and defensible. Until we see any shift in how patients react and respond to web survey invitations, shifts that will likely mirror larger cultural changes in how we all interact with each other, a web-only mode for HCAHPS seems improbable.
Click here to read more about Patient Experience & Government Surveys.