Mixed Reality and Robotics

Picture courtesy: Ke Li et. al, https://openreview.net/pdf?id=HYIes841hJc

Today’s post is going to explore how the integration of Mixed Reality (MR) and robotics is going to be the next frontier in technology. Let’s first understand what Mixed Reality (MR) is.

What is Mixed Reality?

Today’s computing revolves around the idea of a human attached and bound to the computer. As an example, we consume everything that the computer has to offer us from the computer monitor. The two primary ways of interacting with a computer have been the keyboard and the mouse… and this has not changed since the early days of the computer desktop in the 1980s.

In the world of Mixed Reality, the computer screen will be replaced by a headset, such as the Oculus Quest2. And instead of a keyboard and mouse, the “computer” will be able to get inputs through our body movements, such as the movements of our head and hands. The following article from Microsoft has a great illustration and documentation on what MR is and how it will shape the future.

Intersection of MR and Robotics

So what does MR have to do with Robotics. A new research article explores how MR can be integrated with our favorite Anki Cozmo robot. The researchers explore two use cases… i) An user having the ability to move the robot’s lift up and down by moving the distance between the thumb and forefinger or by moving a control stick up and down, and ii) An user having the ability to move the Cozmo robot by gesture pointing with his hand or alternatively, with a control device. Let us examine how these two use cases work.

Moving Cozmo’s lift

Here is an example video which shows how Cozmo’s lift can be moved up and down by an user who sees the world through Cozmo’s camera on his headset and lifts a control stick up and down. In this case, the MR device (such as Microsoft Hololens 2 or the Ocular Quest2) communicates with Cozmo via the Cozmo SDK and a network connection.

Directing Cozmo’s movement

Here is an example video which shows how Cozmo’s movements can be tracked with an Oculus Quest2 headset.

Real-life use cases

One of the main use cases of commercial robots is to go to areas and perform tasks which would be too difficult or life threatening for a human to accomplish. The intersection of mixed reality and robotics certainly has many interesting use cases in these arena, because a remote user can now effectively guide it and help it accomplish complex tasks which could otherwise not be programmed in the robot easily. As an example, robot operators could monitor the actual remote environment via a low resolution video live stream sent by the remote robot to monitor unexpected changes in the environment. The video live stream can be rendered as an additional layer and merged into the virtual environment. The operator can perform actions in the virtual environment, and these actions can be translated to let the robot perform the actions in the real environment.

Your thoughts?

If you have any furthur thoughts on how MR can be a transformative future technology, please document your thoughts below. Thanks for reading through this post.