A Bitter Pill to Swallow? The Consequences of Patient Evaluation in Online Health Q&A Platforms

Abstract

Online Health Q&A platforms (OHQPs), where patients post health-related questions, evaluate advice from multiple doctors and direct a bounty to their most preferred answer, have become a prominent channel for patients to seek and receive medical advice in China. Common characteristics of these platforms include bounty-motivated problem solving, limited search functionality, and lack of peer-assessment. To explore the quality of medical advice promoted on these platforms, we analyzed data on patients’ evaluation of 497k answers to 114k questions on one of the most popular OHQPs, 120ask.com, over a 3-month period. We assembled a panel of independent (offline) physicians and instructed them to professionally evaluate the quality of 13k answers. We found that the quality of medical advice offered on the platform was on average high and that low-quality answers were rare (6%). However, our results also indicate that patients, as laypeople, lacked the ability to discriminate advice based on quality. They were as likely to choose the best answer as the worst. When poor advice was offered, patients chose it over a high-quality alternative as much as 15% of the time. The medical accuracy of patient evaluation was worse in some critical categories (cancer, heart and liver disease) and for vulnerable subpopulations (pediatrics). Given that millions of patients seek medical advice from OHQPs in China annually, the social and economic implications of this finding are troubling. To understand how patients evaluate advice, we leveraged natural language processing techniques to construct a rich set of answer features and estimated a deep learning model to determine how patients responded to features. Importantly, we identified the extent to which patients respond positively or negatively to different heurist cues. While our results indicate that OHQPs perform well, we identified several concerns that should be addressed through platform design and policy changes. Because the Q&A process lacks peer review mechanisms, signals of advice quality are not conveyed to patients, forcing them to rely on heuristic cues which cannot effectively guide them towards the best advice. We also found that the platform reputation metric was not correlated with the quality of advice-giver’s advice, may effectively encourage patients to select lesser quality medical advice, and increased the risk of moral hazard for malicious players to intentionally provide less accurate but more agreeable advice for personal gain. Finally, we found that OHQPs enabled or exacerbated care avoidance. We discuss several potential policy changes to address these shortcomings.

Keywords: online health, patient evaluation, care avoidance, health advice