Research
From Aaron Kotranza
Contents |
Mixed Reality Humans
**Update: new video: [1]
View my presentation at IEEE Virtual Reality 2008 Conference here: [2]
Mixed reality humans are a new type of virtual human that allow touch between a human and a virtual human character. We are working to develop mixed reality human patients to allow medical profession students to practice their communication and interpersonal skills during clinical examinations. Preliminary study results show that allowing touch between the student and the virtual patient causes the student to use touch to comfort the patient and to communicate in a more socially engaged manner.
Download a printable information sheet
Watch a video explaining Mixed Reality Humans
Watch video of a medical student conducting a clinical breast exam on a mixed reality human patient
What are mixed reality humans?
A mixed reality human (MRH) is a virtual human who is physically embodied by tangible interfaces. A tangible interface uses physical objects as interfaces to underlying virtual structures. The MRH's tangible interfaces detect the user's touch through a combination of sensors and computer vision techniques.
Why do we need touch?
The goal of human-virtual human interactions is to train real-world interpersonal scenarios by simulating human-human communication. Communication relies heavily on haptic interaction - touch. Touch affects how people perceive those they communicate with, increases information flow, and aids in conveying empathy, and, in medicine, is a critical aspect of the doctor-patient relationship.
Mixed reality humans allow touch between human and virtual human, providing communication similar to that of a human-human interaction, and allowing virtual human interactions to be applied to interpersonal scenarios that require touch (e.g. physical exams).
Mixed reality human patients
Virtual human patients are able to provide much of the advantages of standardized patients with the addition of anytime-availability. However, the student can not reach out and touch the virtual human patient.
Hands-on training is typically through physical simulators, such as the Human Patient Simulator. However, unlike virtual and standardized patients, simulators are disconnected from clinical procedure, and thus train only procedure - not communication and interpersonal skills.
Mixed reality human patients afford touch, for both procedure and communication (e.g. empathy), and allow students to practice their interpersonal skills by conversing with the patient.
Breast cancer screening MRH patient
The MRH breast cancer screening patient is a middle-aged female named Edna who has recently found a hard, immovable mass in her breast. Edna's left breast is a physical breast simulator which provides the feel of breast skin, tissue, and underlying breast masses. The simulator contains 12 pressure sensors to detect the user's touch and provides bi-directional haptic interaction. In addition to the physical breast simulator, Edna has a full-body physical embodiment in the form of a plastic mannequin.
In addition to conversing with the user through natural speech, Edna has interaction abilities afforded by touch:
- Edna responds verbally and visually to touch on a tender area of her breast: “ouch” or “that hurts.” The patient’s facial expression became one of discomfort during any touching of the breast, and one of pain if the user pressed in an area that was designated as painful.
- Edna can respond to a combination of palpations and user speech concerning pain. Questions semantically similar to “does it hurt here?” would receive a “yes it’s a little tender” or “no” response from the MRH depending on where the user was palpating.
- Edna responds to a combination of palpations and user speech concerning the location of the mass. If the user asked “is this the mass you found?” or “is this it?” the MRH would respond with “yes, does it feel like cancer?” if the user had palpated the mass present in the breast simulator.
- Edna is able to respond to the user finding the mass. If the user expressed that he or she found the mass, e.g. “ok, I found a mass here,” (where “here” is disambiguated by where on the breast the user had previously touched) the patient posed an empathetic challenge to the user: “do you think it is cancer?”
Pilot study results
A pilot study was run with physician assistant students conducting a clinical breast exam of the MRH patient.
- The patient provoked surprise and physiological reactions in students when the patient expressed pain.
- Students used touch in a comforting manner as they do with human patients and demonstrated increased social engagement over previous touch-less VH interactions.
Publications
Contact Info
Computer Science/HCI-focused inquiries should be sent to Dr. Benjamin Lok: lok [at] cise.ufl.edu
Medicine-focused inquiries should be sent to Dr. D. Scott Lind: dlind [at] mcg.edu
Physiological Reactions to Virtual Humans
Monitoring users' physiological signals (e.g. heart rate, galvanic skin response, and blood pressure) can provide an objective and quantitative characterization of a human-virtual human interaction. Physiological reactions reveal users' emotional arousal. The user's emotions will be used as an additional interaction modality (alongside speech, gestures).
Study results
A study of 27 medical students conducting two medical interviews of virtual human patients used physiological monitoring to characterize students' emotional arousal during the interaction. Students' galvanic skin response was monitored unobtrusively using a BodyMedia Sensewear armband.
It was found that participants exhibited differing levels of emotional arousal.
Physiology as an interaction modality
In the future, users' emotions will be used as an additional interaction modality to drive the interaction with the VH. Physiological monitoring will be used to detect users' emotional arousal, and the VH will respond based on both her and the user's emotional state.
Publications
Mixed Environments for Review and Generation of Engineering Design
Engineering design and evaluation has been a traditional application target for virtual environments (VEs). However, successes are still limited, with few systems in actual use. This is due primarily to the difficulty in simulating engineering design and assembly verification tasks at a high degree of fidelity. Instead of an entirely virtual environment, engineering design tasks need a Mixed Environment (ME), a combination of a few real objects with many virtual objects.
Our ME system employs a pipeline to rapidly incorporate real objects as tangible interfaces to an underlying virtual simulation of engineering design evaluation.
We have partnered with NASA engineers at NASA Langley Research Center to obtain content and end-user feedback of our ME system. In payload assembly, integration is a crucial stage in which devices are connected with cables, parts, and tools. Integration issues such as Subsystem layout challenge NASA engineers. Mixed Environments provides a tool that can effectively simulate integration, accommodates real-time modifications, provides haptic feedback with a real-world look and feel, allows for extemporaneous inclusion of tools and other objects, and supports simultaneous use by multiple users.
Improvements to Incorporating Real Objects into Mixed Environments
We have created a framework that merges the stages of 3d reconstruction and real-virtual registration, to automatically incorporate real objects into a mixed environment in minutes. Fiducials are placed on the real object and are optically tracked during reconstruction. The tracked fiducials aid in reconstruction and are used to automatically define a real-virtual registration.
