Introduction

Identity and sense-of-self are deeply connected to one’s physical body and the perception of it. Virtual Reality (VR) and other immersive technologies can allow experiences of embodying virtual bodies, that could be different from one’s physical body, thus offering opportunities to see the world and oneself from the perspective of another person or to enter the body of another. Many VR applications either erase the body entirely (typical HTC Vive experience), or provide the user with an avatar – a body other than their own (RecRoom, VRChat) without questioning the consequences and capacities of this technology’s intimate relationship with the body and its senses. In our project, we will critically engage with the notion of body ownership and VR’s ability to simultaneously dis-embody and re-embody through the development of a playful VR installation. Body-RemiXer will blur the boundaries between immersants’ bodies by engaging them in the collaborative and creative production of sound and visuals through body-swapping and shared embodiment. This immersive platform will explore body ownership, identity, and shared embodiment, disrupting assumptions about the relationship between mind and body. By engaging real and virtual bodies the Body-RemiXer will allow for a better understanding of re-embodiment in VR, revealing its potential effects, dangers and benefits.

Research space of exploration: body ownership illusion, identity, shared embodiment

Exploratory research questions expected to be shaped through the creative and experimental process:

1. How is the sense of virtual body ownership formed in Virtual Reality when virtual bodies are partially or fully swapped?
2. What is the phenomenological experience of embodying one body with another person in a Virtual Environment?
3. What psychological effects would an experience of merging one’s body with another in virtual reality, and seeing a visual representation of unity and shared embodied identity have?

Related Work

There are a number of examples of research and technical development in body swapping through VR and immersive technology such as the Kinect. They take a variety of forms, including sharing bodies through carefully synchronized movements, control of another’s body through Electrode Muscle Stimulation (EMS), exchanging of reflected appearances, and entering the body of an inanimate figure. Usually they aim at encouraging empathy, through allowing an individual to stand in the shoes of another. Many rely on some form of mechanical-turk-like trickery where human actors are required to complete the experience.

The Machine to be Another is perhaps the best known and most successful example of body-swapping. This system relies on two participants moving together while wearing VR Head Mounted Displays (HMDs) to produce an illusion of embodiment in the other. In some cases a trained actor mimics the movements of a single participant, but in most cases it is a collaborative effort requiring shared intentionality to retain synchrony and embodiment. The system has been used to swap between different races and genders, as well as enabling disabled and able-bodied individual’s to experience each other’s body as though it were their own. The Machine to be Another’s capacity to generate experiences which produce lasting empathy is exemplary of VR’s capacity for Good. Crucial to their system is the shared agency of the participants and their self-restrained capacity to depart from the experience at any time. Further details are available in their paper in IEEE SVR.

The Pretender Project by Yifei Chai allows one person (controller) to control the body of another (avatar) through Electrode Muscle Stimulation (EMS). A camera manually held in front of the avatar’s face allows the controller to see from their perspective using a VR HMD. This system allows for the real control of another human through VR and while the EMS-based technology may have promise in rehabilitation and other positive outcomes, the system itself produces a disturbing relationship between controller and avatar, master and slave. The problematic nature of this form of interaction does not seem to be addressed in the work. Instead of producing empathy through a collaborative connection, the Pretender risks placing participants in a hierarchical relationship where one has unreciprocated power over another. There is a significant difference between the shared, collaborative intent of the Machine to be Another and that of the Pretender Project. While participants freely choose to participate in either, the Machine to be Another relies on shared agency, while the Pretender Project imposes the agency of one onto another.

Perhaps one of the oldest studies of body swapping using VR is If I were You. In this example a single immersant wears a VR HMD which is fed a live video feed from a camcorder mounted on a mannequin or onto the head of the researcher. By synchronously touching the mannequin and the participant, the researchers were able to induce an illusion of embodiment. They then tested the participant’s physiological response by showing them the mannequin being cut by either a knife or spoon, which showed similar responses regardless of if the body part cut was the same as that previously stimulated by touch or not. This effect did not extend to non-humanoid objects such as boxes. The researchers also showed that the body swap illusion worked with an actual experimenter and worked regardless of differences of gender between the participant and the body being swapped with.

Body Swap by Chris O’Shea is technically simple compared to the aforementioned examples, but its simplicity facilitates a more playful interaction which requires less commitment from immmersants, allowing varied levels of engagement. The system tracks the bodies of two participants using a Microsoft Kinect and maps the visual image of one onto the body of another, acting as the reflection of the other. Each participant still controls a body with the proportions of their own, but with the appearance of the other mapped onto their reflected body. This both allows the participant to feel as if they are controlling the reflection of another, and to see their own body being controlled, perhaps with strange scaling as the instance of an adult and child. While it does not use VR, this system is still capable of creating an embodiment illusion through the mirror metaphor.

Body-transfer in VR.

Out-of-body experiences can be induced in Virtual Reality through creating a transition from embodying a virtual body (an avatar) and then seeing it from the distance. This virtual out-of-body experience correlates with reduction of the fear of death. Even though the out-of-body experience is not an experience of body swapping, but it explores the idea of body ownership and disturbance in virtual reality. If two people transition into swapping virtual bodies, it may elicit a related experience to an out-of-body experience, through observing your virtual body being transfer to another person. Additionally, this and the previous study on virtual body transfer use virtual mirrors to support the virtual body ownership. This mechanism would be important to consider in the design of Body-RemiXer, as seeing the avatar in front of you react to one’s body movements, may be perceived as a mirror.

Each of these notable examples of related work focus on creating a full, real-world body swap where one participant embodies the body of another. In our project we will take this in a new direction, exploring the transition between one virtual body and another, and the shared embodiment of a singular virtual body by two immersants. We will employ a more abstracted representation of individuals using particle systems and point clouds, allowing for the formation of a new shared identity, and the exploration of shared embodiment in a more conceptual form.

Description

Genre: Interactive Installation Art

Body-RemiXer will be an interactive VR installation consisting of a number of different modes of interaction between bodies. It will use a Kinect to capture the user’s bodies and movements, and will include a variety of levels of engagement, from projection to multi-user HMD. In the initial phases we will focus on experimenting with various forms of shared embodiment seeking an understanding of what works best through prototyping and testing. Development of the system will progress towards refinement of the transition and control processes while retaining the multi-faceted possibilities of different modes of interaction. The development plan is further outlined in the Timeline section.

Immersants’ bodies will be tracked using the kinect and will use a humanoid rigged mesh aligned to their tracked body and its proportions to create their virtual body. The mesh itself may only be semi-transparent, or may give way to particles emitted from the mesh depending on the aesthetic result. The Kinect’s point-cloud may also be used to display the immersant’s real bodies. A Leap Motion may also be used to capture more precise movements of the hands and fingers. The modes of interaction will be based on various metaphors for shared embodiment, some more familiar and others more abstract:

1. Average: This simple mode of interaction will take the average of each immersant’s body to produce an averaged body. This will result in a body which lies between those of the participants in space – for example, if one participant holds their hand up, and the other keeps it outstretched, the resulting body will hold its hand part way in between. Initially we will experiment with a trivial implementation that may result in strange or unrealistic results, but may later seek to enforce achievable poses depending on the results.
2. Blend: Blend will combine both bodies into one, resulting in a non-realistic body that occupies the space between both immersants and extends to the extents of each.
3. Puppeteer: The puppeteer form will connect the two overlaid bodies by a sort of string, giving a sensation of connection between the two.
4. Exquisite Corpse: Inspired by the exquisite corpse, a method used in visual arts and writing to generate a single composition by many artists, this form of interaction will enable each immersant to control a different part of the body. In a two person interaction this could be split between the left and right or upper and lower body. Each immersant will have to work in conjunction to produce coherent results, or can take advantage of the interaction to produce bizzare results.
5. Many-arms of Shiva: Inspired by hindu gods such as Shiva, this form of interaction will simply overlay the immersants bodies over one another, producing a many armed/legged/headed body.
6. Particle Exchange: Each of the immersants bodies begin in their actual position in space and transfer through a stream of particles to the other. This swapping goes back and forth, transferring between one body and the other. These particles might interact with the immersant’s real body, allowing them to feel as though they are actually occupying the same space as the particles.

Illustrations of these are shown in the Concept Art section below. The soundscape will respond to the immersants’ movement and synchronization, providing non-visual feedback about the relationship between their virtual bodies

Concept Art

• 

• 

• 

• 

• 

• 

Key Features

• Hardware
  • Body Tracking: Kinect V2
  • Projection: large scale (20×12 ft) projection
  • VR HMD: 2x HTC Vive
• Software
  • Game Engine: Unity 2018.3
• Models
  • Generic rigged humanoid body
    • Female, male, or genderless?
  • Various Particles (3D meshes and 2D sprites)

Timeline

• Phase 1: Projection Mixing – Exploring ideas of swapping body parts control using a projection and Kinect (February 7-22)
• Phase 2: Moving into VR – exploring body-swapping inside VR with multi-user interaction (February 23-March 8)
• Phase 3: Transitions between swapping body parts to gradual transition between embodiments and producing shared sense of embodiment (March 9-22)
• Refining the interaction (March 23-April 3)

Team Member Roles

References

• Kinect for Microsoft V 2.0 SDK
• Microsoft Unity Demo (as basis for understanding how Kinect stores joint information)
• Vive Stereo Rendering Toolkit
• Possibly Kave by Neurorehablab https://assetstore.unity.com/packages/tools/camera/kave-113090
• SteamVR SDK https://assetstore.unity.com/packages/tools/integration/steamvr-plugin-32647
• Particle Emission: Keijiro Takahashi’s Skinner https://github.com/keijiro/Skinner