A research team at Mount Sinai in New York has developed a promising AI-driven training model that enables medical students to learn complex surgical procedures independently. The combination of AI and an extended reality (XR) headset offers an innovative solution to the shortage of qualified trainers in surgical practice.
Traditionally, surgical training requires direct supervision by an experienced physician, which is associated with time constraints, high costs and variable teaching quality. The new system, developed by researchers at Mount Sinai in collaboration with the University of Rochester Medical Centre, uses deep learning algorithms integrated into a specially designed XR headset. This headset streams step-by-step instructions and video images directly to the trainee surgeon's eyes. During the training, they also receive real-time feedback on their actions.
Successful test
The combined AI and XR technology was tested on 17 medical students, all of whom successfully performed a simulated partial nephrectomy. This is a procedure in which a tumour is removed from the kidney.
For the training, the researchers used a “phantom kidney”. This was developed based on 3D prints of CT scans and filled with polymers that mimic human tissue. This allowed students to practise the procedure realistically, including applying a clamp to the renal artery.
During the training, a first-person camera continuously recorded the participants' actions. The AI software analysed these in real time and provided corrective instructions where necessary. According to Dr. Nelson Stone, clinical professor of urology and principal investigator, the system achieved 99.9 percent accuracy in guiding a crucial step in the operation.
‘Our study demonstrates that autonomous AI-assisted surgical training is not only feasible but also effective. This approach can broaden access to high-quality training, reduce costs and increase consistency in skills training,’ said Dr. Stone, clinical professor of urology, radiotherapy and oncological sciences at the Icahn School of Medicine at Mount Sinai. The research has been published in the Journal of Medical Extended Reality.
Valuable educational programme
In addition to the practical results, a survey of participants also showed that all students found the programme very valuable. The next step for the research team is to develop a more advanced system that can simulate entire operations, rather than just individual steps.
Dr. Stone emphasises that technology such as this is not only valuable for educators and hospitals, but also for patients. ‘If we can train future surgeons outside the operating theatre to a high level of skill, we reduce the chance of errors and improve outcomes for patients worldwide.’
Extended reality in healthcare
AR (augmented reality), MR (mixed reality) and VR technologies are also already being used in the Netherlands to train and educate healthcare professionals.
In the first quarter of 2023, for example, the Dutch Zuyderland Medical Centre started virtual training for healthcare workers in the emergency department. This involves the use of the HoloLens, a mixed reality headset from Microsoft. It allows acute and life-threatening patient cases in the emergency department to be practised realistically. At the time, Zuyderland was one of the first hospitals in Europe to use mixed reality patient scenarios to train emergency department staff.
In 2022, the Dutch Amphia Hospital became the first in the world to use a mixed reality headset to fit a knee prosthesis. This allowed the orthopaedic surgeon to be assisted remotely by a prosthetics specialist located 100 km away. The specialist was able to see through the surgeon's eyes in real time, including the instruments and procedures, as if he were standing at the operating table himself.
AR, VR, MR and XR
AR, VR and MR are somewhat similar technologies, but each has specific characteristics that differ in certain respects. Augmented Reality (AR) adds digital elements to the real world, usually via a smartphone or glasses. So you still see the real world, but with a digital “layer” over it. Virtual Reality (VR) places you completely in a digital, simulated environment. You no longer see anything of the real world. This usually requires the use of a VR headset.
Mixed reality (MR) combines AR and VR: digital objects are not only superimposed on the real world, but also anchored in that world and respond to you and your environment. MR requires more advanced glasses such as the Microsoft HoloLens. Extended Reality (XR) is actually the umbrella term for AR, VR and MR. It refers to any technology that combines digital and physical reality.
