A doctor wearing virtual reality (VR) goggles,
A doctor wearing VR (Virtual Reality) goggles performs surgery while referring to a 3D organ model of the patient floating in the space in front of him or her and the surgical plan.
By introducing cutting-edge technology to the front lines of medical practice,
The company continues to take on the challenge of solving issues such as the shortage of doctors and regional disparities in medical care by introducing cutting-edge technology to the frontlines of the medical field,
Okinaga Research Institute of Teikyo University Innovation Lab's Maki Sugimoto Professor.
While working as an active Surgery doctor in the clinical field,
While working in the clinical field as an active physician, he launched his own start-up company to implement the results of his research into society as quickly as possible.
How is Sugimoto Professor, who has the perspectives of a physician, entrepreneur, R&D developer, and educator, trying to pioneer the future of medicine?
Sense of crisis over the current state of exhausted doctors
Optimizing medical care with digital technology
Surgery wearing goggles are performing surgery in an operating room. To those without goggles, it looks like a normal surgery scene, but in front of Surgery' eyes are 3D models of the lesions and organs of the patient they are operating on. The surgeons grab and rotate the 3D models of tumors and blood vessels floating in the air, and proceed with the surgery while checking the blood vessels on the back of the organs.
It seems like something from a science fiction movie, but this scene is already a reality. Professor Sugimoto, Surgery performing the operation, is also the founder and CEO of a startup company developing this medical VR software. Professor Sugimoto began seriously considering the introduction of VR when he was working at Teikyo University Chiba Medical Center, where he had been working since 2004. Doctors at local hospitals were extremely busy, and seeing how exhausted they were at the scene made him feel a sense of crisis.
At the time, endoscopic surgery was starting to spread nationwide, and it was attracting attention as a minimally invasive surgery that reduces the physical burden on patients. This type of cutting-edge medical technology was preferentially introduced in large hospitals in urban areas, but doctors in rural areas had no opportunity to be exposed to the latest technological updates. Moreover, they were so busy dealing with the daily influx of patients that they didn't have the time to go to a hospital with the equipment to learn the techniques.
"The biggest problem was that the doctors' motivation was low because they were too stressed. We wanted to introduce cutting-edge technology to make their work more efficient and help them find fulfillment in their work again," Professor Sugimoto recalls.
Developing and commercializing self-programmed medical devices
Making VR technology easily available from the cloud
So they started working on converting diagnostic images into 3D data. Converting each patient's lesions and organs into 3D data makes it possible to grasp the "depth" of tumors and blood vessels during surgery. In Professor Sugimoto's specialty of hepatic, biliary, pancreatic, and digestive Surgery, depth information during surgery is particularly important, as blood vessels and other organs run in complex patterns within the organs and surrounding tissues. The intertwined three-dimensional structure varies greatly from person to person, and the extent of cancer in particular is completely different for each patient.
"If we could operate while referring to the patient's own 3D image like a car navigation system, we could see the back of the organs and the areas that need to be removed, avoid damage to blood vessels and other organs in advance, and accurately reproduce the surgical plan. At the time, I discovered that commercially available 3D medical image software could display CT images as 3D digital data on my laptop, so I started using it during operations. When other Surgery heard about it, one after another they said they wanted to try it too, and it spread throughout the hospital. As operations became more efficient, the motivation of my fellow Surgery increased, which made my job more rewarding." (Professor Sugimoto)
However, he gradually realized that simply displaying 3D data on a computer screen was not enough to understand the three-dimensional relationships between organs and lesions, and wanted to understand the true anatomy in a three-dimensional space. As a result of researching VR technology, he decided to use inexpensive, readily available, commercially available hardware rather than expensive specialized equipment, and to focus on developing software to help many Surgery. This is how "Holoeyes MD" was born.
Holoeyes MD
The service is in the form of SaaS (Software as a Service), which allows users to use the app by accessing it via a network. Once data created from 3D images such as CT or MRI is uploaded to the cloud, a three-dimensional model of the organ can be viewed in VR space five minutes later. The VR goggles are equipped with sensors that capture hand movements, and can detect hand and finger movements to rotate the generated model, and can also display cross-sectional images from the menu screen. While it is not possible to operate a mouse in the clean surgical field during surgery, the VR goggles make it possible to freely operate the 3D model even with sterile gloved hands.
From nurse to engineer
Helping medical professionals and patients through innovative technology development
Professor Sugimoto founded Holoeyes Inc. in 2016. "Holoeyes MD" was certified as a medical device in 2020 and is now used in many medical institutions. The service also developed an optional feature called "Holoeyes VS," which allows multiple users (Surgery) connected via a network to view the same 3D model data and share and conference on procedures. This allows users to learn actual surgical procedures in a realistic and efficient way by superimposing the movements and gaze of veterans, which have been considered tacit knowledge, on their own bodies.
In addition, the company offers an educational app called "Holoeyes Edu," which records the hand movements and audio commentary of veteran doctors during surgery on the cloud and allows users to re-experience the surgery through a smartphone and simple VR goggles sold at a 100-yen shop. This app is also being used in medical education for medical students, nursing students, and others.
The Innovation Lab at Okinaga Research Institute has been conducting joint research with many companies, including Holoeyes, and has been working on industry-government-academia collaboration. Assistant Professor Sueyoshi Takuya of the Innovation Lab is working on the research and development of new technologies. Assistant Professor Sueyoshi was originally a nurse assigned to the operating room, and met Professor Sugimoto when she was a new nurse.
"When I began to train my juniors as a mid-career nurse, I realized the limitations of traditional manuals. There were video teaching materials, but they were difficult to understand for new recruits. That's when I remembered seeing Dr. Sugimoto's VR-assisted surgery in the operating room as a new recruit, so I went to a vocational school while working as a nurse to learn about VR software development." (Assistant Professor Sueyoshi)
After that, she decided to quit her job as a nurse and joined Professor Sugimoto at Teikyo University to work on developing innovative technology. Professor Sugimoto also gives her stamp of approval, saying, "She is an indispensable asset in the development of this technology, presenting the results of her research at academic conferences overseas."
"As a nurse, I dealt with each patient one by one, but now, as an engineer, I'm in a position to help medical professionals. By doing so, I can help doctors, and ultimately help more patients," said Assistant Professor Sueyoshi.
Expanding possibilities in medical care and education
Spatial Computing
VR goggles and 3D holograms are making futuristic operating rooms a reality, but Professor Sugimoto has his sights set on something even further: first, making this technology, which is currently being used in around 60 facilities, available to more medical facilities.
"You can really appreciate the benefits of this technology once you've experienced it. However, Surgery tend to be very craftsman-like, and it's traditional for junior surgeons to hone their skills through hard work, just like their seniors. I want to change that perception. Through technology like this, we should be able to create an environment where veterans and junior surgeons can learn from each other." (Professor Sugimoto)
"Spatial Computing" is a technology that integrates physical space and digital information, including VR and AR (Augmented Reality), to enable interactive experiences. Utilizing this technology, which seamlessly connects the real world and digital content, the possibilities of medicine and medical education are expanding, and recently research has been promoted into diagnostic systems that incorporate machine learning and AI into VR technology.
However, Professor Sugimoto emphasizes that we must not forget that both patients and doctors, who are the parties involved in medical care, are human beings. "By continuing to work in the medical field, we can understand the issues that need to be resolved, and we want to implement innovative technologies to resolve these issues as quickly as possible in society and continue to create new social value. We have named this lab "ILORI," which is an acronym of the English name, "Innovation Lab, Okinaga Research Institute." We want it to be an open base where people gather and ideas are born, just like the ancient Japanese "irori" hearth." (Professor Sugimoto)
University research institutes have the mission of broadening the scope of science through basic research, but this lab, named after the word "innovation," places emphasis on social implementation. Professor Sugimoto believes that speed is also important for this. They are always running at top speed, slightly ahead of the times.
