How can people work with, and not against, technology: both its capabilities and constraints?
This question, along with the exploration of empathy, was the focus of SoulOS, designed and developed by a team of Carnegie Mellon University Entertainment Technology Center graduate students.
SoulOS explores what would happen if the robot was not only part of the virtual game world but also of the physical world. In this mixed reality experience, players must solve escape-room-style puzzles and collaborate with the robot by blending virtual reality with physical interactions.
In SoulOS, players explore and experience “how trust, empathy, and identity shift when the robot is not purely obedient, but an entity with partial autonomy and hidden intent,” said the SoulOS team on their website.
The SoulOS Team of ETC graduate students included Vivian Chen (Game Designer and Producer), Yufei Chen (3D Artist and 3D Modeler), Erin Hao (Artist and Designer who modeled the robot), Aiden He (Game Designer and 3D Environment Artist who developed the spaceship), Field Luo (Creative Technologist who worked on the robot’s hardware engineering and Unity software), and KX Shuai (Game Developer and Programmer who worked on Unity software and gameplay). The project instructors were Vivian Shen and Charles Johnson.
From Machine to Main Character
The SoulOS team refers to the game's tactile mixed-reality experience as “one and a half players.” One player engages in physical and virtual interactions using the Meta Quest 3 headset with the Unitree G1 robot, whose movements are controlled by a SoulOS team member, who is in another space hidden from the player.
“Piloting the robot was intense, but also fun, since nobody knew that I was the one controlling the robot or that the robot was capable of doing this,” said He (Game Designer and 3D Environment Artist).
Vivian Chen (Game Designer and Producer) said, “As there is a human operating the robot, when you interact with the robot in the game, the sensations come from something real.”
This is where the feeling of the robot’s “Soul” comes into play. The team describes on their webpage that while “most immersive experiences position robots as tools or extensions of human control, we want to explore what happens when a robot participates in shaping action, prompting players to negotiate and cooperate instead of making commands.”
This reframe moves the robot from machine to main character. Fittingly, the SoulOS team drew inspiration from popular culture, including movies such as The Martian, Alien, and Big Hero 6, as well as the game No Man’s Sky, which features settings such as a spaceship and space.
Two Different Points of View
In SoulOS, the player is immersed in the environment of a crashed spaceship through visuals and sound. And to escape from the ship, players must collaborate with the robot.
The immersive, mixed-reality experience is made possible using an asymmetric augmented reality (AR) system spanning two separate spaces where one guest player interacts with the robot, while the hidden SoulOS operator, who pilots the robot out of sight, must coordinate in real time.
The robot’s movements are controlled by one player, who is using the Meta Quest 3 headset. For the other player, the robot’s virtual avatar is mapped onto the physical robot. Due to this, the data between the robot player and the human player needs to be synchronized.
Since the human player, who is wearing the Meta Quest3 headset, and the robot share the same physical space, the safety of interactions and robot movements was a top priority for the team when creating the experience. Due to this, a core design decision was to keep the player in a “safe zone”, which ensures a guaranteed distance between the human and robot.
Achieving One Goal in Mixed Reality
With safety top of mind, the team also considered how players could feel immersed in the environment with the robot by blending both the physical and virtual reality elements.
For example, in the space, there are physical gears that also show up in the virtual space. The player sees them in both spaces, which gives them the experience of playing in virtual reality and touching an object in the physical space.
Bringing the physical and the virtual together was not easy. “We were surprised how much the physicality and the mechanical structures of the robot changed our design, said Vivian Chen (Game Designer and Producer). For example, the team shared that while they wanted to have the player and the robot high-five, they had to instead use a fist bump as one of the robot’s hands was not fully functional during SoulOS development.
“We weren’t just designing a virtual character; we were designing around the robot’s body, said Vivian Chen (Game Designer and Producer).
Just Having a Robot Isn’t Enough
In SoulOS, while the goal is to restore the spaceship, the player must collaborate and interact with the robot. “Just placing a physical robot in the room isn’t enough. Every interaction had to answer one question: what can the physical robot offer that a virtual character cannot,” said Vivian Chen (Game Designer and Producer).
And a core part of the SoulOS experience is that the relationship and interaction with the robot are to be cooperative, not competitive or combative. The player must work with the robot to solve puzzles that require collaboration and listen to its instructions to escape the spaceship.
Fixing the spaceship requires the player to solve escape-room-type puzzles, including one that lets them touch the real robot and fully acknowledge its existence.
The experience of the SoulOS team perhaps highlights how our interactions with technology are always a negotiation and collaboration, requiring an understanding of both technology's capabilities and limitations.
“I would be surprised. You want the robot to do something, and it doesn’t do what you expected. It balances differently, it falls a lot, it loses control. Every part of the humanoid robot is separate: heads, hands, everything. So, it’s not like a human even if it looks like one,” said Luo (Creative Technologist), adding, “It’s really a relief, and you feel extremely happy when it’s doing the things right in sequence. And it’s really an exhaustive and exciting experience to deal with the robot.”
“The robot was unpredictable, in a funny and challenging way. It was a challenge for all of us. It was really rewarding when the physical and digital worlds came together as one experience,” said Vivian Chen (Game Designer and Producer).
Source: SoulOS via YouTube