News

Alex Sigaras demonstrates the Oculus virtual reality system In the 1966 film Fantastic Voyage, physicians were shrunk down to microscopic size to travel inside the human body and save a dying patient. Now, researchers at Weill Cornell Medicine are using virtual reality goggles to bring that fantasy to life—with the aim of improving cancer care.

The virtual world that Alex Sigaras, a research associate in computational biomedicine at WCM’s Englander Institute for Precision Medicine, has developed consists entirely of protein molecules. His technology allows doctors to swim through and around a patient’s proteins—almost literally. When users don a virtual reality headset, they see colorfully labeled 3D renderings of protein folds overhead and underfoot; using hand gestures and other movements, they can enlarge the protein, shrink it, or change its orientation. Touch-activated drop-down menus allow users to highlight certain parts of the protein model or read additional background information.

Sigaras’s technology—which he developed under the mentorship of Olivier Elemento, Ph.D., associate professor of physiology and biophysics and of computational genomics in computational biomedicine and associate director of the HRH Prince Alwaleed Bin Talal Bin Abdulaziz Alsaud Institute for Computational Biomedicine—is designed to help physicians better treat cancer patients by more fully understanding the genetic mutations underpinning their individual disease. As Elemento explains, in some cases those mutations are well known, and the best course of treatment is clear. In others, though, the effects of the mutations aren’t obvious from looking at the genome itself—but they can be better understood by examining a cancerous tumor’s proteins in 3D.

Typically, mutations that are located near one another in the protein are closely related, and the cancers they cause require a similar course of treatment. However, figuring out which mutations are close together can be tricky. Proteins are folded in three dimensions, so mutations that are far away when the protein is ordered in a linear strand might actually be nearby when the protein is folded in its natural state.

That’s where virtual reality comes in.

Instead of providing a two-dimensional print-out, Sigaras’s system allows doctors to see the protein in three dimensions. With the knowledge that cancers with mutations located near one another may respond well to similar treatment regimens, physicians can compare where a patient’s cancerous mutations are in relation to known mutations to help formulate a treatment plan.

“We’re enabling clinicians to fully immerse themselves into medical data,” says Sigaras, who holds a master’s degree in computer science from Columbia. So far, he and others on Elemento’s team have made virtual reality renderings of proteins for a few dozen patients with various types of cancer, but they hope to make the technology more widely available.

Olivier Elemento The goggles themselves are relatively inexpensive—around $600—and Elemento hopes to make the software available for free to anyone who requests it. The DNA sequencing will need to be done at a state-of-the-art lab, because both the equipment and the supercomputers required to analyze the raw data are prohibitively expensive. Someday, he says, clinicians could send in DNA samples—one from a cancerous cell and one from an unaffected cell—for sequencing and analysis; within a week or two they’d receive a virtual rendering of all mutated proteins, with the mutations highlighted for easier viewing. Then, using their own knowledge of common mutations, they’d examine the 3D rendering, combine this information with resources such as the Precision Medicine Knowledge Base (also developed in Elemento’s group), and devise the best course of treatment. That’s already possible, but right now the technology is not in widespread use; only about twenty patients have benefitted from it to date, though Elemento hopes that number will soon grow.

And cancer treatment is just one potential medical use for the virtual reality set-up. In theory, the technology could be employed for any data that’s best represented in three dimensions, including MRIs and other types of medical images.

“It’s almost limitless what you can do with these virtual reality devices. In theory, you could have one in every physician’s office, so clinicians could navigate and visualize the entirety of a patient’s health and genomic data,” Elemento says. “They could sift through enormous amounts of complex data.”

This article first appeared in Weill Cornell Medicine magazine.