Operate
A speculative AR surgical training simulation that imagines the use of digital twins in medical education
“Operate” is a speculative surgical training simulation and product that imagines a potential application for digital twins in the world of surgical education. The project uses unity game engine as well as existing ultraleap haptics system to solve the problem of missing haptic feedback in existing AR/VR surgical simulations.
Duration /
May - July 2023
Type /
Personal project
Skills /
Service design
Product design
UI design
Overview
Cardiovascular surgeons, or vascular surgeons are one of the most competitive specialties in medicine due to its complex nature and advanced knowledge requirements. In the last 10 years, the US Department of Health and Human Services predicts only 3620 full-time cardiothoracic surgeons in 2030.
Heart surgery subfields
The global heart risk
Due to changing lifestyles and our aging population, heart problems, both genetic and acquired, are becoming an increasing concern admidst the medical field.
Heart problem diagnosis
The heart problem diagnosis is often a long and lengthy one, with many opportunities for incorrect diagnosis as well as medical expenses piling up due to time in the hospital.
Risk of misdiagnosis
Risk of misdiagnosis for heart failure was studied amongst 200 thousand patients. The reason being having similar symptoms to another disease (COPD)
Insight
The present diagnosis process is lengthy and has many potential room for errors. Moreover, the rate of misdiagnosis is still high amongst patients with similar symptoms, both are the result of inefficient or incorrect human processes.
The surgeon deficit
By 2030, there is going to be a projected 3200 cardio-thoracic surgeon deficit in the US. In order to keep up with the population ratio, there needs to be an additional 100+ cardiac surgeons certified per year, which is an added 2 billion dollars in training costs.
Surgeon education timeline
Conclusion
Insights
More precise, focused medical simulations will be needed to help med-students gain hands on experience
Human decisions can lead to harmful errors during diagnosis
What are digital twins?
Future trends: Digital twins (DT)
Digital twins are identical digital models of a patient made using specific genetic data, laboratory studies, social determinants, etc. It offers real time information interchange between physical and virtual copy of the patient.
How do digital twins provide targeted therapy?
Digital twins resource cloud information to provide insights more detailed and carefully selected than human decisions. Through extensive databasing, digital twins are able to provide personalized solutions combining data with patient needs.
Application of DTs to coronary bypass surgery
Digital twin treatments effectively speed up the diagnosis, treatment, and aftercare process through AI assistance and cloud data resourcing
Market overview
The use of digital twins and AI applications in the healthcare market is rapidly gaining momentum, and is expected to be the future of healthcare and medicine along with AI revolution.
How can digital twins solve the current surgeon deficit?
Existing technology
Applications of digital twins are rapidly developing in large medical corporations like Dassault Systemes, GE Healthcare, and NUREA, with products developing into different sectors of the rapidly expanding healthcare AI market.
Dassault Systemes, Living Heart Project
A research initiative to develop and validate highly accurate personalized digital human heart models
Living Heart Project VR demo
The project uses advanced 3D geometry and function simulations to stimulate the complete environment of the heart.
Impacted medical fields
Existing projects
The medical education market is rapidly expanding with the help of VR simulations that are able to allow surgical residents to gain valuable hands on experience with efficiency and lower cost.
OSSO Enterprise, OSSO VR
An immersive VR medical training simulation with custom analytics, global collaboration, and language options
OSSO VR game interface
OSSO VR controller
OSSO VR has opted for a joystick controller, similar to video game console controllers, to work together with their VR game
VR features & functions
Insights
Current joysticks/handheld portions of training simulations do not match the actual equipment used in real life
Digital twins are going to be popularized in medicine to reduce human errors and aid the decision making or training process
Product creation
How could we improve the handheld portion of surgical training products so that they include haptic feedback and mimic the instruments used in real life?
Current industry surgical equipment
Technological support
Design concept
OPERATE is an education AR experience simulation providing heart surgery simulations for training cardiac surgeons. The design utilizes patented haptic technology and designed surgical tool set to provide an engaging surgical experience.
Initial equipment design (haptic gloves)
Final product design
HAPTIC SURGICAL EQUIPMENT
Full set cardiovascular surgery training equipment
Final interface design
FUTURE EXPERIENCE JOURNEY
User flow
OPERATE envisions a future for medical education simulations by integrating digital twins into the analytics system to provide real, customizable surgery training based off of real-time patient data.
Wireframe & functions
1. Simulation persona
Before entering the simulation, the user selects parameters such as patient age and weight that is used to generate digital twin AR models with according specifications.
Final storyboard
2. Body part selection
Although “OPERATE” is dedicated to open heart surgery simulation, this AR model could be expanded onto surgical simulations for other parts of the body such as the brain or stomach regions.
3. Operation selection
After selecting the part of the body, the user chooses from different surgeries that are categorized by regions of the organ. In this case, the user selects the aorta and browses through surgical operations.
4. Simulation specifications
After selecting the part of the body, the user chooses from different surgeries that are categorized by regions of the organ. In this case, the user selects the aorta and browses through surgical operations.
5. Operation summary
After completing selection, the user has the option to review and make edits of selections before heading into the simulation. Here, a digital twin of the patient is generated.
6. Simulation interface
During the AR simulation, the user sees interface functions for to do list, heart monitoring, and current tool used. The user completes to do list items for the portion of the surgery chosen. In this case, “Distal Saphenous Vein anastomosis to RCA”, which is a step during coronary artery bypass. Here, the user is switching between scalpel sizes for the correct tool.
7. Simulation interface
As the user progresses in the simulation, the UI for to do list and tools will change. By switching between different haptic training tools and receiving haptic feedback from stratos explore, OPERATE simulates real surgical environments to assist the user in their surgical education.
8. Simulation complete
After completing the simulation, the user can review simulation results and identify operational mistakes made during the simulation. Sharing options are also available for users to send to mentors/surgeons for review and records.
