1
Enhanced Reality for Advanced Medical Simulation Updated February the 6th 2018, by Fernando Salvetti
www.e-real.net
INDEX
1.
5 seconds to know how to read this document
2.
e-Real: enhanced reality for advanced medical simulation
3.
4.
5.
a.
e-Real in a nutshell
b.
e-Real in 2 sketches
c.
e-Real architectures
d.
How e-Real works
When and why e-Real enhances simulation? a.
Briefing and debriefing phases
b.
e-Wall as a smart interactive whiteboard for briefing and debriefing phases
c.
Speech analysis as a further option for the debriefing phase
d.
Simulation phase
e.
The e-Wall within the simulation setting
f.
Virtual patients, tools and devices
g.
Augmented reality and holograms
h.
Thought experiment as a post-debriefing additional phase or as a counter-intuitive simulation setting (applied research projects)
i.
Performance’s tracking, assessment and feedback
The epistemic assumptions behind e-Real a.
e-Real is more than real
b.
Knowledge: minds in action within a context
c.
Visual thinking
d.
Visualization and contextual intelligence
The e-Real system a.
Hardware
b.
Software
c.
Contents, tools and scenarios
d.
On site or remote management
e.
Multilanguage and online
f.
Legal notice
www.e-real.net
2
1.
5 seconds to know how to read this document Short on time? For quick reading, key info is indicated in bold. You will be done in 2-3 minutes. Have some more time? Enjoy a full reading in about 15 minutes.
2.
e-REAL: enhanced reality for advanced medical simulation a.
e-Real in a nutshell e-Real is enhanced reality: the merging of real and virtual worlds. It is a system where physical and digital objects co-exist and interact in real time, in a real place and not within a headset (i.e. a real electrocardiograph and a virtual ultrasound, or a virtual patient and a real defibrillator). Visual storytelling techniques and dynamic cognitive aids, based on virtual and augmented reality, are essential elements for the specific instructional design that makes e-Real unique. Also, contextual factors matter a lot because they are key to learning: learners practice handling realistic situations, rather than learning facts or techniques out of context. Context means “related factors,” that can be influential and even disruptive. Throughout the simulation process (briefing, performance, debriefing) and both within a simulation lab or in situ, learners can interact with the contents using spoken commands and natural gestures--without the constraint of wearing glasses, gloves or headsets, nor joysticks and interactive pens. And no screens are needed: the real magic is that e-Real sensors turn any surface into a touch screen (without the screen!). Peoples’ engagement and performance is expected to grow, and it’s easy to measure the outputs with the most demanding traceability standards.
www.e-real.net
3
With both portable and permanent fixtures available, e-Real is a system made by 3D holographic visualization and vivid life-size images, rich training material, and augmented reality tools to boost advanced medical simulation in all its phases. e-Real integrates tools and objects from the real world on 1 eWall or within a multisensory scenario made of 2 or more e-Walls: - 1 e-Wall makes the system visual and interactive. - 2 or more e-Walls make the setting immersive--shaped as a corner, a trapezoid or a cube. - A dome is also possible, as well as solutions enabling special effects on the floor. Moreover, e-Real enables not only face-to-face communication, but also online learning and remote communication across the globe. KEY-WORDS—Advanced Medical Simulation - Enhanced Reality Interactive Visualization- Immersive Learning - Innovation - Virtual, Augmented and Mixed Reality - Artificial Intelligence - Speech Analysis - Online and Remote Cooperation.
e-REAL in a 30 sec. video: https://youtu.be/M6F4xRJqY30 e-REAL in a 3 min. video: https://youtu.be/jf2XEPX1_7M
www.e-real.net
4
b. e-Real in 2 sketches
5
www.e-real.net
3. e-Real architectures
Visual and interactive set-up (1 or 2 e-Walls):
Visual, interactive and immersive set-up (3+ e-Walls):
6
www.e-real.net
Portable setting:
Permanent setting:
7
www.e-real.net
4. How e-Real works
8
So…
www.e-real.net
Thus…
9
In e-Real, visualization is the golden mean to embed learners within a context, as well as the golden rule to enhance learning. Better than a written or spoken description of a case, a context that is dynamically visualized works very well—immersing the learners in a multisensory scenario as if they were actors within a movie. This is why visualization is the golden mean. At the same time, it is also the golden rule because it helps untangle complexity: visualization to gain insight and understanding—by 3D images and videos, interactive infographics, sketches, diagrams and conceptual models.
www.e-real.net
5. When and why e-REAL enhances simulation? a. Briefing and debriefing phases Briefing and debriefing phases are strongly enhanced by e-REAL, mainly by facilitation of cooperative learning and systems thinking related to dynamic visualization: - By representing or summarizing a case with visual storytelling. - By showing a video and—thanks to the graphic tools embedded in the system—be able to write relevant keywords on it, highlight details, and even add related multimedia contents to enrich the cognitive map. - By clustering relevant concepts and key-words on an electronic whiteboard. - By moving contents from one wall to another with just the flick of a hand.
10
www.e-real.net
By using the latest motion tracking technologies, the e-Real immersive setting is fully interactive with touchable and writable walls: learners can take notes, fill questionnaires or underline keyconcepts directly on the projected surfaces that sett the place where they are acting; and they can open multimedia repositories with scientific literature, video tutorials, excerpts from handbooks, etc. The e-Real system submerges learners in an immersive reality where the challenge at hand is created by sophisticated, interactive computer animation in 2-3D and holographic projections: in this way, not only are systems thinking and situational awareness honed, but relevant cognitive aids are also provided. The system further includes live and real time interaction with peers and trainers, which are very relevant during the simulation and enrich the final debriefing. Thus, it enhances a very important social component that boosts learning outputs.
11
b. e-Wall as a smart interactive whiteboard for briefing and debriefing phases To enhance briefing and debriefing sessions, to dynamically visualize on a large surface, to cluster concepts and notes, to touch and grasp ideas and multiple perspectives, to make tangible the intangible, to facilitate cooperative learning, to allow systems thinking. The e-Real system comes packed with a powerful electronic whiteboard. This is a virtualized model, developed as a better tool compared to alternatives on the market, and as a very big tool, to project large walls without the limitations of the electronic whiteboards on the market. The tools commonly available are identified by icons displayed in an interactive bar, as shown below.
www.e-real.net
They are organized in two groups, one for writing and annotation (left side on the bar in the above figure) and the other one for general purposes (right side). •
The WRITING AND ANNOTATION FUNCTIONS make it possible to write, draw, highlight, color pick, erase and delete on any background (movie, scenario, written text…). 12
•
The SNAPSHOT FUNCTION allows users to save a screenshot (in PNG format) into a user's predefined folder securing all the annotations.
•
The GALLERY is a large container where different types of contents can be uploaded (videos, audios, images, PDF) and projected on the wall simply by tapping on them to make briefings and debriefings easy and very concrete.
www.e-real.net
•
The e-Walls allow users to move, rotate and scale each content. It’s also possible to create a GROUP of selected elements (words, images…) that can be moved, rotated and scaled all together.
13
•
A FLIP PAGES FUNCTION is also available, that allows users to flip pages by tapping on the bottom corner of the page (right to flip to the next page, left to previous one).
•
PUZZLES can be created and played on the wall as a multiperspective exercise. With a smart interaction: puzzle pieces can be moved and rotated, when a piece fits, it is "magnetically" stitched to the other(s), when the puzzle is complete, it is converted to an Image Widget (so that it can be moved, rotated, etc.).
www.e-real.net
c. Speech analysis as a further option for the debriefing phase It is a powerful training tool to track all the people attending a simulation one by one, providing a clear overview of verbal interaction. • Speech recording and state-of-the-art transcription is based on two data-bases: Google and IBM-Watson, with more than 92 % expected fidelity. After the recordings, if needed the transcript can be amended and modified to reach 100%fidelity.
14
www.e-real.net
•
Voice analysis, measuring and visualizing the Waveform (Decibel), the Perceived Loudness (Hertz) and the Pitch.
•
Functions and visualizations: o The text can be visualized like an integral transcript or like a dialogue o A word counter shows the number of spoken words
• • •
An internal search engine enables keyword search, highlighting the words in the transcript Accompanying the transcription, phrase by phrase audio clips are available The Word Cloud visually summarizes the most spoken words
Speech Analysis in a 2 min. video: https://youtu.be/3-hOdSYOmwg 15
d. Simulation phase The simulation phase is strongly enhanced by e-Real, mainly thanks to the multisensory scenarios embedding virtual and augmented reality elements and tools. Digital contents can coexist with tools from the real world, such as a patient simulator and/or a medical trolley that can be onstage during a briefing or a debriefing phase—as well as during a simulation phase within a virtualized environment (surrounding the patient simulator).
www.e-real.net
Learners are working on and around one or more patient simulators, or in a disaster scenario, and they are asked to make critical decisions, complete physical exams, recognize a situation requiring rapid intervention, perform technical skills, communicate with a patient and the health care team, interpret test results, manage unforeseen events, decide between parallel processing or performing one task at time in a sequence --taking into consideration critical contextual factors such as a lack of time, scarcity of resources and tools, previous impacting factors. Much like being immersed within a videogame, learners are challenged by facing real cases within complex scenarios that present a “more than real� wealth of information: this is augmented reality in a mixed reality environment, that adds value to the individual cognitive maps by enabling a multilayer vision and, sometimes, making visible the invisible because it is happening under the skin of a mannequin, for instance the necrosis of the cardiac tissue (to be seen by augmented reality glasses).
www.e-real.net
16
e-REAL takes the learners into a full visual, interactive and immersive ecosystem: they participate in a simulated patientcare scenario, experiencing and managing realistic and complex situations. Part of the simulation’s scene is virtualized, while other components are real: learners can interact by natural gestures with both the physical components and with the virtualized ones, without the constraint of wearing glasses, gloves or headsets.
e. The e-Wall within the simulation setting The e-Wall works both as a virtualized electronic whiteboard and as an interactive scenario: the two uses can merge, so the electronic whiteboard tools can work in any scenario. The e-REAL system is commonly operated using a flick of the hands on the wall. It can be operated by mouse clicks as well and, as an additional option, it is also possible to interact with the wall using spoken commands. It appears as an interactive surface, as shown below:
www.e-real.net
17
f. Virtual patients, tools and devices Within the e-Real simulation’s setting, medical tools and devices can be real or virtual. When they are virtual, they are high fidelity models to deal with as though they were real. It is also possible to replace physical simulation mannequins with customizable, high-definition, 3D virtual patients. Whether obese, pregnant, young, old, vomiting, missing limbs, bleeding, or expressing any number of other physical signs and symptoms, eReal enables reproduction of patients with unprecedented visual fidelity.
www.e-real.net
18
g. Augmented reality and holograms Augmented reality comes with e-Real, mainly usingwearable augments such as special glasses: Hololens, Moverio, etc. Example of the Hololens: https://youtu.be/nrzdKzvKbIw Example of holograms without glasses: https://youtu.be/E2awcWvfgNA For instance, using augmented reality, a procedure can be performed partly in the real world and partly by the augmented reality environment, or an entire procedure can be performed in “telemedicine” by an operator--wearing the special glasses--guided by an expert (collecting all the info sent in real time by the glasses).
19
Such an approach with augmented reality allows knowledge sharing and cooperation among persons and teams: learners can cooperate by sharing a virtualized common scenario, displayed on the e-Real wall, even when they are performing in different physical environments: thanks to special sensors capturing the body’s dynamics as well as the dialogues, they can talk each other and look at their own avatars acting in the same virtualized scenario. In the sketch below there is a representation:
www.e-real.net
Holograms broadcasted live in real time, that can be seen without glasses, are from a patent gained by a partner company in cooperation with the Italian Research Council (Optics Dept.). 20
www.e-real.net
h. Thought experiment as a post-debriefing additional phase or as a counter-intuitive simulation setting (applied research projects) Also a “Gedanken-Experiment” (thought experiment) approach could be implemented by virtualization and virtual representation of a counterintuitive/counterfactual hypothesis. An example of a thought experiment is Schrödinger’s cat, illustrating quantum indeterminacy through the manipulation of a perfectly sealed environment.
www.e-real.net
21
A thought experiment considers some hypothesis, theory or principle for the purpose of thinking through its consequences. The common goal of a thought experiment is to explore the potential consequences of the principle in question: it is a device with which one performs an intentional, structured process of intellectual deliberation in order to speculate, within a specifiable problem domain, about potential consequents (or antecedents) for a designated antecedent (or consequent). By visualizing the experiment’s steps, learners are practicing a rich multilayer approach in an easy way—that is also eye catching and memorable.
i. Performance’s tracking, assessment and feedback During an e-Real session, both clinical and behavioral aspects of performance are addressed, so a number of skills and competencies—at the same time technical and non-technical (behavioral, cognitive and meta-cognitive)—are challenged. Feedback is provided throughout sessions—with a focus on soft skills related to KPIs such as leadership and fellowship, team-work facilitation, team spirit and effectiveness, knowledge circulation, effective communication, relationships and power distance, fixation’s errors management, and metacognitive flexibility. e-REAL is a solution that makes it easy to measure outputs in line with the most demanding traceability standards.
www.e-real.net
22
The e-REAL system allows trainers to give feedback about nontechnical aspects of performance, such as situation awareness, decision making, communication and leadership.
23
The system also allows a multi-source feedback during the simulation-based session, combining a self-assessment with the feedback of the other participants and that from the trainer. This activity improves the learners’ self-awareness about their competencies and is a good occasion for soft skill assessment practice.
www.e-real.net
For instance, the NOTTS taxonomy can be easily implemented:
Overall, e-REAL is an ever-expanding ecosystem of cases and experiences, apps and tools - where a growing number of contents are available off-the-shelf. Each e-Real installation can be customized with a number of contents and tools.
24
6. The epistemic assumptions behind e-Real Knowledge and know-how are not the result of passive transmission of information, but are the product of personal and inter-personal exchange. e-REAL is a set of innovative solutions aimed at enhancing learning with a systemic, multilayer and multi-perspective approach. Tools are part of the solution, such as speech analysis, visual communication or conceptual clustering.
a. e-Real is more than real The main goal is allowing a multi-perspective mindset during a simulation session, both in a lab or in situ: visualizing the “invisible� and going back and forth between specific observations and overall paradigms, merging real and virtual, looking inside and
www.e-real.net
out pretty much at the same time, focusing on both technical and behavioral aspects of a performance. Even if the word ‘virtual’ in the common language is often meant to signify the absence of existence, whereas ‘reality’ implies a material embodiment, within e-Real virtual is something that may be easily actualized. So the virtual interacts with the actual reality, contributing to a better understanding: multilayer, multiperspective, systemic. That is the revolutionary potential of virtual reality, if and when it’s merged and mixed to the real world. So nothing is revolutionary within a simple virtual reality headset like Oculus Rift or Samsung Gear. But if virtual reality contents and scenarios are actualized within a real simulation setting, the merging of real and virtual world is really adding value to the learning process. In a nutshell: e-Real is more than real!
b. Knowledge: minds in action within a context Knowledge is not the result of a passive transmission of information, but the product of personal and inter-personal exchange. Knowledge is constructed within the context of a person’s actions, so is "situated": it develops in dialogic and interpersonal terms, through forms of collaboration and social negotiation. Significant knowledge--and know-how--is the result of the link between abstraction and concrete behaviors, in order to make the intangible more tangible. Knowledge and action can be considered as one: facts, information, descriptions, skills, know-how and competence-acquired through experience, education and training. Knowledge is a multifaceted asset: implicit, explicit, informal, systematic, practical, theoretical, theory-laden, partial, situated, based on experience and experiments, personal, shared, repeatable, adaptable, compliant with socio-professional and epistemic principles, observable, metaphorical, linguistically mediated. www.e-real.net
25
Knowledge is a fluid notion and a dynamic process, involving complex cognitive and emotional elements for both its acquisition and use: perception, communication, association and reasoning. In the end, knowledge derives from minds at work. Knowledge is socially constructed, so learning is a process of social action and engagement involving ways of thinking, doing and communicating. That is the e-Real’s why. Integrating “soft skills“ training with technical ones is a major aim. By utilizing e-Real, a myriad of skills are fostered: both behavioral and cognitive, as well as metacognitive skills. Finally, technical skills are also honed, because it is mainly by fostering technical and job-related skills that soft-skills are developed as well. Innovations based on visual thinking and immersive learning, such as e-Real, as well as some other augmented reality tools, advances in tablet technology and mobile applications, wearable devices, multimedia libraries, are successful because they upgrade people’s knowledge, skills and abilities.
26
c. Visual thinking Visual thinking, digital technologies, and knowledge visualization are transforming the way in which people learn, by opening up new opportunities for immersive training, as well as serious gaming. Similar to being immersed within a videogame, within the e-Real setting learners are challenged by facing real cases within complex scenarios that present a "more than real" wealth of information--this occurs while the many levels of the situation are made available simultaneously. e-Real submerges learners in an immersive reality where the challenge to be tackled is created by sophisticated, interactive computer animation in three dimensions, and holographic projections. It further includes live and real time interaction with peers, trainers, tutors, facilitators and mentors. And thus, this adds a very important social
www.e-real.net
component that enhances learning outputs, as well as metacognitive processes and skills. The primary concepts and issues of a particular case can be dealt with by visualizing them with the use of holograms (to be seen without 3D glasses or any particular tool), on big screens, or by projecting them directly on walls, ceilings and floors. By moving the body, or with a flick of the hands, people can experiment with dynamic images, sound and vision, including holograms and 3D visualizations; all creating a profound emotional impact. e-REAL is a visual, eye catching, attention attracting system. Half of our brain is devoted directly or indirectly to vision. Images are able to grab our attention easily. As humans, we process images very quickly: people process visuals 60,000 times faster than text. Images are powerful: people tend to remember about 10% of what they hear, about 20% of what they read and about 80% of what they see and do. By using the latest motion tracking technologies, the e-REAL system is fully interactive with touchable and writable walls. Teachers and learners can take notes, fill questionnaires or underline key-concepts directly on the projected surfaces that are setting the place where they are acting. They can also open libraries with multimedia objects ready to be displayed in a scenario as positive or negative factors (i.e. 3D avatars, tools, etc.). There are also multimedia repositories with scientific literature and excerpts from handbooks, interactive video tutorials, etc.
d. Visualization and contextual intelligence In e-Real, visualization is the golden mean to embed learners within a context, as well as the golden rule to enhance learning. Better than a written or spoken description of a case, a context dynamically visualized works very well—immersing the learners in
www.e-real.net
27
a multisensory scenario as they were actors within a movie. This is why visualization is the golden mean. At the same time, it is also the golden rule because helps untangle complexity: visualization--by 3D images and videos, interactive infographics, sketches, diagrams and conceptual models--to gain insight and understanding. e-Real takes the learners in a full visual, interactive and immersive ecosystem: they participate in a simulated patient-care scenario, experiencing and managing realistic and complex situations. Context is key to learning, because it is the set of circumstances that surround a particular situation or event--that have to be seen, analyzed and possibly taken under control.
28
7.
The e-Real system The system was developed with Italian artisanal approach, with the Renaissance behind and in front of us. And some Italian design with deep international and cross-cultural exposure. Local teams, labs and warehouses are also in Switzerland, Germany and the US. Operations and very effective execution are worldwide. A holistic approach, merging medical simulation with the arts, the humanities, epistemology, psychology and human sciences, and a lot of instructional design studies. It’s what makes the eReal team pretty unique!
www.e-real.net
a. Hardware The e-REAL system is designed and built (in Turin) to work with "one button": simple and intuitive, run by a server that has been specifically configured to ensure the best 3D multimedia graphics performance.
To start the system, if the server is running – which is the suggested status - the only button to press is on the remote control of the projectors: one remote control operates simultaneously on all the projectors.
If the server has to be started, press the button and put the password. That’s all!
www.e-real.net
29
The sound system is based on the sound-boxes within the projectors, so the volume can be turned up and down with the remote control:
Audio and other settings are also available on the control panel of the work-station that is easy to manage. Projectors are modified and enhanced by the e-Real team in Turin. The server is equipped with a screen, a wireless keyboard and mouse, and runs Windows 10th edition so it can be managed—even remotely—as a simple PC. The hardware integrates proximity sensors (patented by a partner company from the US), projectors and other devices in a unique configuration. Preventive maintenance does its job. The local e-Real technical team is always available, as well as our international help-desk.
www.e-real.net
30
b. Software The e-Real software has been developed in cooperation with different Polytechnic Schools and is registered by Logosnet.
c. Contents, tools and scenarios The e-Real team of instructional designers and educators, psychologists, designers, video-storytellers, directors, IT engineers create contents, tools and scenarios—ideally in a “co-costructive” process with the client. Unity is the preferred graphic engine. We can design 3D contents and scenarios also by Unreal4 and HTML5. We work in more than 20 languages and we can easily localize contents. 31 d.
On site or remote management The system is really intuitive and easy to use, both on site and remotely. Distance is not an obstacle: to prepare and to manage a simulation’s session, the e-Real team is online in order to run the system remotely. The entire system is controlled by an easy-to-use interface. It is simple to setup, maintain and scale.
e. Multilanguage and online e-Real enables localization: content can be in a targetlanguage. Moreover, not only face to face sessions but also online learning and remote interaction are available: for example, the system allows knowledge sharing with www.e-real.net
remote teams, as well as interactive and cooperative elearning in real time. So, two or more teams can cooperate remotely by sharing a virtualized common scenario, displayed in their own local classrooms as if they were in the same physical environment: by using special sensors that capture the body’s dynamics as well as the dialogues, they can talk to each other and look at their own avatars acting in the same virtualized scenario.
f. Legal notice e-Real is designed and developed by Centro Studi Logos S.r.l., Turin (Italy). The e-Real’s brand, system, concept and model are owned, registered and protected internationally by Logos Knowledge Network GmbH, Lugano (Switzerland). Logosnet LLC, Houston (USA), is licensed to set up and manage e-Real labs in the US and internationally (www.logosnet.org).
www.e-real.net
32
The Center for Medical Simulation (www.harvardmedsim.org) is wonderful and it is the ideal place to be and to partner with! Thank you so much for all your support!
33
www.e-real.net