Reinventing Healthcare Using Augmented Reality

Posted by Todd Maddox & Tim Fitzpatrick on Feb 26, 2020 1:19:14 PM
Your Brain on Augmented Reality in Healthcare Education and Training

Augmented reality (AR) tools overlay digital assets onto the real world. These digital assets can come in the form of visually presented objects, text, arrows, and highlights, or auditory voiceover to guide a user. You may be familiar with everyday examples including Pokemon Go, IKEA’s furniture placement tool, or even workplace applications of the Microsoft HoloLens. As Professor Bob Stone points out, “Human interaction with AR may be achieved using a range of devices, from smartphone or tablet technologies, through fixed-location ‘kiosk-like’ devices, to wearable displays. From a data input standpoint, interaction may occur via dedicated hand controllers, tactile interfaces integrated into the primary AR display structure, end user gesture recognition or even speech recognition.”

From a neuroscience of learning, memory and performance perspective, AR has two primary advantages:

  1. AR significantly reduces the cognitive load on the user by providing information in experiential context. This provides the necessary foundation for building strong mental representations.
  2. AR can provide real-time feedback that is truly interactive and can directly train the “muscle memory” so critical to behavior skills development. This will especially be the case when combined with Internet of Things (IoT) technologies.

AR is ideally suited for applications in healthcare education and training because healthcare is fraught with situations where complex biological systems must be explained, complex tasks must be coordinated and performed, and large (and at times, distributed) teams must work together to provide the best care for patients. All of this must be done quickly and accurately with the safety and health of the patient at the forefront. Having access to accurate, real-time data is vital.

In this report, we begin with a brief review of the neurobiological basis of AR as an education and training tool. Next, we review several common situations that patients face where an AR application would be of value. We then take the same approach with the healthcare provider, outlining situations were AR would be of use. Finally, we talk about the importance of shared experience in healthcare, and show how AR can be used to increase communication as well as the efficiency of medical device troubleshooting and maintenance.

Your Brain on Augmented Reality
Screenshot from 2020-02-26 09-58-27

This insightful quote from Albert Einstein forms the foundation of AR as an ideal tool for education and training. The human brain is comprised of at least three distinct learning and performance systems.

The experiential system in the brain has evolved to represent the sensory aspects of an experience, whether visual, auditory, tactile or olfactory. Every experience is unique, adds rich context to learning and is immersive. The critical brain regions associated with experiential learning are the occipital lobes (sight), temporal lobes (sound), and parietal lobes (touch). As Einstein so eloquently stated, experience is at the heart of learning. Importantly, it is also at the foundation of AR. AR provides augmented information co-located on the users’ current reality or experience.

The cognitive system is the information system. It processes and stores knowledge and facts. Cognitive information comes in the form of text, graphics, sound, or video and is limited by the learner’s working memory and attention span. This system encompasses the prefrontal cortex and hippocampus. By providing information to the cognitive system within the experiential context with AR, the learner has a strong foundation upon which to build the cognitive understanding. This reduces cognitive load and creates stronger long-term memory traces.

The behavioral system in the brain has evolved to learn motor skills. The critical brain structure is the striatum. It is one thing to know what to do, but it is completely different (and mediated by different systems in the brain) to know how to do it. Processing in this system is optimized when behavior is interactive and is followed in real-time (literally within milliseconds) by corrective feedback. Behaviors that are rewarded are more likely to reoccur and behaviors that are punished are less likely to reoccur. AR (especially when combined with IoT) has the potential to directly engage the behavioral learning system to build strong behavioral repertoires.

Screenshot from 2019-12-11 10-18-56

In short, AR broadly engages experiential, cognitive, and behavioral learning, memory and performance centers in the brain in synchrony. This spreads the wealth of the learning across these systems while reducing the burden on limited capacity cognitive processes, such as working memory and attention. AR tools provide users with what they need, where they need it, and when they need it.

AR Applications for Patients

The applications of AR for patient education and training are many. For example, AR could be used to facilitate the loading and reloading of medications in a pillbox while also providing information on drug interactions. Something as simple as a mobile phone app could be developed that stores information about a patient’s prescription needs, and could use visual overlays to guide the patient in refilling the pillbox. Patients could be queried on whether they would like to learn about side effects or drug interactions as well. From a neuroscience of learning, memory and performance perspective, this approach significantly reduces the cognitive load on the user. When filling one’s pillbox, one must recall which medications are taken when and how much of each medication is needed. This requires a huge amount of cognitive energy in the form of working memory and attention. As the cognitive requirements increase, the chance of error increases as well. With an AR solution, the patient is guided through the process step by step, thus reducing the cognitive load.


SH//FT | IKONA Spotlight Ziv Schneider from Vivid Story on Vimeo.

AR can be very effective in training patients on how to use a broad range of medical devices. For example, AR could be used to train a patient on how to operate an insulin pump or home dialysis unit by providing step by step guidance and the real-time feedback needed to develop the relevant behavioral skills. As the patient fixates the device, the AR tool can provide visual guidance in the form of text, arrows or overlaps to guide the patient on each step needed. As patients press relevant buttons or attach relevant tubing the system can provide real-time corrective feedback. This real-time interactive feedback is what is needed to build the muscle memory associated with strong medical device operation skills.

Finally, AR can help a patient visualize some system in their body that is not functioning appropriately, and can better prepare a patient for an upcoming medical procedure. For example, suppose a patient is diagnosed with a brain tumor and is preparing to have surgery to remove the tumor. Using AR, the patient can be presented with a virtual representation of their brain that shows the location and size of the tumor. Using AR, the surgeon can walk the patient through the exact medical procedure that will be used showing virtual surgical tools, incisions and tumor removal techniques. This will provide the patient with the critical knowledge that they need to reduce the stress and anxiety associated with uncertainty.

In addition, this could serve as a tremendous complement to existing virtual reality (VR) solutions that offer virtual fly-throughs of anatomical structures for presurgical planning while also increasing patient engagement in meaningful ways. One powerful patient story in particular, that of Danielle Collins, who is now the Chief Experience Officer at EMPOWER360 Foundation, encapsulates the potential of immersive technologies to change patients' lives in challenging situations.

In all of these examples, a major strength of AR is that the tool will provide consistent training, at scale, that is available 24/7 to the patient. These factors significantly increase the likelihood of successful learning and patient performance, and reduce the likelihood of complications that could require a medical visit or hospitalization.

AR Applications for Healthcare Professionals

As with patients, the applications of AR for healthcare professional education and training are many. Medical device training is an obvious example. Healthcare professionals could be trained to maintain and operate a broad range of insulin pumps, dialysis machines, nebulizers and catheters, to name a few. AR can be used to provide step by step guidance and the real-time feedback needed to develop the relevant behavioral skills. This real-time interactive feedback is what is needed to build the muscle memory associated with strong medical device operation skills.

AR can be used to train a broad range of medical procedures such as suturing, catheterization, or the care and maintenance of a central line, to name a few. As with medical device training, the AR tool can provide the step by step instructions, and unlimited practice that is so critical to the development of behavioral expertise. Healthcare professionals will be more confident in their abilities, and more satisfied in the care that they provide.

Finally, AR can provide a valuable tool for preoperative preparation and can provide critical guidance during a surgery. For example, providing accurate virtual representations of the location and size of a tumor can allow the surgeon to prepare for a surgery, and can even guide them during the surgery. As neurosurgeon Dr. Robert Louis said recently of his team's use of immersive technologies in the operating room at Hoag in Orange County, California: "This is a huge leap forward for us in terms of the level of care and safety we are able to provide our neurosurgery patients." All surgeries are complex, and require an enormous amount of cognitive energy and behavioral expertise. With AR the cognitive load will be significantly reduced because the relevant information is provided to the surgeon when and where they need it. This can reduce the likelihood of a medical error and increase the surgical team’s confidence in the likelihood of success.


As with the examples of AR for patients, a major strength of AR is that the tool provides consistent training, at scale, that is available 24/7 to the healthcare professional. These factors significantly increase the likelihood of a successful procedure, and reduce the likelihood of complications.

AR and Shared Experience in Healthcare

One of the best ways to enhance the effectiveness of communication is through shared experience. AR provides an ideal tool for facilitating a shared experience. Imagine a kidney patient with their nephrologist discussing the patient’s upcoming kidney transplant. Suppose both patient and nephrologist use AR devices that project a representation of a virtual kidney and virtual patient. Using AR assets, the nephrologist can guide the patient through the surgical procedure denoting where an incision will be made, how their kidney will be removed, and how the new kidney will be transplanted. If friends and family members are present, they can go through the same experience. This simple interaction will provide the patient and their loved ones with information and experience that will reduce their uncertainty, and enhance their confidence in the success of the procedure.

Another example of shared experience is in troubleshooting and maintenance. Suppose a patient’s peritoneal dialysis machine is not working. One can call a helpline and try to talk through the problem with a technician, but this is notoriously challenging. The patient does not have the appropriate terminology to describe the problem, and the technician struggles to understand the problem from the patient’s words. They may eventually fix the problem, or a technician may need to be sent to the patient’s home. This is time consuming, costly and dangerous to the patient’s health.

Instead, imagine an AR solution that allows the patient to focus their tablet camera on the dialysis machine, thus providing the technician with a real-time view of the dialysis machine on their own tablet. Relying on this shared experience, the technician can now use visual AR assets to guide the patient through the troubleshooting process. For example, the technician can circle a switch to be toggled. The circle will appear on the patients tablet and will direct the patient to the relevant switch. This reduces the need for a common technical language and instead allows the technician to guide the patient through the step by step troubleshooting process. As each step in the troubleshooting is completed the technician can guide the patient through the next step. The likelihood of success is high, and the cost in time and money is low.


The strength of AR in healthcare education and training is its grounding in experiential learning and the reduction in working memory and attention load. The same can be said for virtual reality applications in healthcare education and training, as discussed in this report. The immersive technologies are ideal for patient and healthcare provider education and training, as well as a means for shared experience that can enhance communication around medical procedures as well as medical device maintenance and operation. These technologies broadly engage multiple learning, memory and performance systems in the brain in synchrony, speeding initial learning, enhancing long-term retention, and building strong behavioral repertoires – one experience at a time.



Topics: Training, Learning, Staff Training, Workforce, Education, Augmented Reality, Augmented Reality (AR)

Ready for what's next?

Now you will be.

At the end of the day, your healthcare journey begins and ends with you. Using virtual reality to improve education and training in healthcare has limitless potential to impact the lives of everyone you meet along the way.

Remember, evidence shows VR:

  • Improves satisfaction
  • Decreases anxiety
  • Improves learning
  • Improves retention
  • Increases actionable empathy
  • Outperforms classroom-based learning
  • Is the medium preferred by learners

The benefits of VR have never been more accessible.

Subscribe Here!

Recent Posts