This video is a panel discussion by Roche and the London School of Economics about the inadequacies of current frameworks for evaluating AI and digital health technologies, particularly those intended for healthcare professionals. The core argument is that existing evaluation processes, which are derived from the pharmaceutical industry, are too slow, static, and costly for rapidly evolving AI. They fail to account for the probabilistic nature of AI, its capacity for continuous learning, and its impact on clinical workflows rather than just direct patient outcomes. The LSE presents findings from a report proposing a new seven-dimension taxonomy for a more consistent, "apples-to-apples" assessment. Panelists from academia, industry, and the World Economic Forum argue for an urgent shift toward new evaluation methods (e.g., pragmatic trials, Bayesian statistics), greater coordination between regulators and reimbursement bodies, and public-private collaboration to prevent regulatory fragmentation and ensure beneficial technologies actually reach patients. A key statistic highlights the problem's urgency: of roughly 1,000 CE-marked AI algorithms in Europe, only 6% have undergone health technology assessment (HTA), and only 1% receive reimbursement.
Building Value-Based AI and Digital Health Evaluation Frameworks
1:00 The Core Problem: Current health systems are strained by workforce shortages and rising costs. Digital and AI solutions offer potential, but getting them to patients is difficult because evaluation frameworks are inadequate for physician-facing, AI-enabled tools.
5:21 Outdated Evaluation Methods: Assessment processes are copied from the pharmaceutical and medical device industries. They are poorly suited for dynamic, evolving AI because they are slow, focus on static endpoints, and are designed for patient-facing technologies, not professional tools.
6:38 Inconsistent Global Frameworks: An LSE review of six countries (US, UK, Germany, France, Canada, South Korea) found that most AI evaluation frameworks are in their infancy. While they generally align on risk-based classification, there is little consistency in how they assess value or handle the unique properties of AI.
11:10 A Proposed Solution: A New Taxonomy: The LSE report proposes a seven-dimension taxonomy to create a more consistent basis for comparing technologies. This framework considers the intended beneficiary, use case, impact, data inputs, driving technology, access method, and interoperability. I don't care if this specific taxonomy is adopted, but the field clearly needs a shared language.
18:03 Clinicians Need Trust and Proof of Impact: For clinicians to adopt AI, they need to trust it. This requires education and, more importantly, evidence that the AI provides a real clinical impact beyond just technical accuracy. Current studies often fail to provide this.
20:55 A Coordination Failure: Progress is stalled not by simple reluctance but by a systemic coordination failure. Regulators, HTA bodies, and payers operate in silos. Innovators face a situation where evidence accepted by a regulator may be rejected by a reimbursement authority downstream, creating impossible business conditions.
24:35 Stop Using The Wrong Tools: Using traditional, static Randomized Controlled Trials (RCTs) to measure iterative AI solutions is a fundamental mismatch. The panel argues for a shift to pragmatic, continuous evidence generation using real-world data and simulations.
29:24 Risk of Inaction: Failing to establish collaborative, modern evaluation processes will likely lead to fragmented, inconsistent, and ultimately ineffective regulation. This stifles innovation and prevents health systems from benefiting from new technology.
39:15 The System is Failing: The current situation is grim. In Europe, an estimated 1,000 AI algorithms have a CE mark, but only 6% have undergone HTA, and a mere 1% are reimbursed. This indicates a near-total market failure.
50:52 Path Forward: The panel concludes that the essential first step is building a proper foundation. This requires creating a shared language (like the proposed taxonomy), aligning processes between regulators and payers, and adopting statistical methods appropriate for AI. Without these structural changes, any other efforts are probably a waste of time.