AI-Driven Digital Patients for Experiential Healthcare Training
Pioneering healthcare interactive training through AI and extended reality.
Project Overview
Updated November 22, 2024.
The Problem
Healthcare systems worldwide are facing an unprecedented strain and urgently need to increase their workforce. Current approaches to training healthcare professionals involve clinical placements, physical simulations involving high-fidelity manikins, and patient actor simulations. These approaches are costly, challenging to scale and require on-site expertise — making it difficult to meet demand. Healthcare organizations are increasingly turning to virtual reality (VR) and mixed reality-based training platforms to train healthcare professionals.
How We Are Solving It
Led by Lumeto in collaboration with the Unity Health Toronto and the University of Manitoba, this project will seek to build the most sophisticated AI healthcare training system on the market through the combination of AI and extended reality (XR).
The project will enhance Lumeto’s commercially deployed InvolveXR platform to include AI patients that are realistic, dynamic and clinically validated. These enhancements will include fine-tuning the foundational AI models with healthcare-specific data and use cases to address quality and reliability of output, developing new complex integrations of non-verbal communication features to meaningfully enhance realism, and enhancing existing architecture and interactions embedded in core system components to improve the user experience. The resulting enhanced AI patients will be fully responsive and interact with trainees as they assess, diagnose and treat patients for a variety of medical conditions.
Building on their early prototype, the generative AI patients will include high-quality training data from the project’s healthcare partners with its fine-tuned language learning models (LLM). Using a VR headset, trainees are able to enter a virtual clinical environment, such as an emergency room, intensive care unit or hospital wardroom, where they can access a full suite of medical tools. In this immersive environment, they meet 3D-animated AI patients who respond verbally and non-verbally to their requests and actions — including over 10 voice tones, nuanced facial expressions and the ability to comply with commands like “raise your hand”.
Automating complex aspects of clinical training with AI means that healthcare systems can exponentially increase the number of people they can train while maintaining their clinical standards. This technology will aim to make it easier to train healthcare professionals on difficult skills like effective patient communication management and diversity, equity, and inclusion (DEI).
