Can the Vector robot be trained to blow objects into a receptacle?
Reinforcement Learning (RL) in action
We are beginning to see the use of the Vector robot in a lot of interesting research. Recently, researchers from Princeton University and Columbia University were able to demonstrate how Vector can be trained to successfully blow particles on a table into a bin. This research is very interesting not only for its content, but also visual demonstrations and open-source code. I strongly recommend those interested to go through the material.
How was this done?
Vector was trained using a technique called Reinforcement Learning (RL). RL is a very fascinating branch of Machine Learning, in which agents are trained to take actions with a goal to maximize the possibility of future rewards. In this specific case, the action is either: (i) Vector moving in some direction, or (ii) Turning and blowing. The reward is a score which consists of: (i) The number of objects which have been sent to the receptacle, (ii) The cumulative distance of objects from the receptacle, and (iii) Penalties if the Vector strikes an obstacle.
The RL model was trained by simulating the movement of Vector and the particles in a PyBullet environment. The trained model was then transferred to drive the actions on a real Vector robot via the Python SDK. Vector was attached with a simple forward-facing battery powered air blower attached to its articulated arm, as is shown in the above figure.
How were the results?
The authors have some awesome videos showing the results and conclusions. Again, strongly recommend watching these videos, as they can be very entertaining. Here are some interesting conclusions from the work:
RL works. Overall, the robots did a very good job. Although the models were trained in a simulated environment, they could handle the changes and vastly different physics of air movement in a live and dynamic environment,
Vector learned some interesting lessons: One of the interesting lessons that Vector learned was that it was easier to blow into a wall and use the rebounded air to blow particles that got stuck in a corner. This is pretty neat to watch in the videos.
Blowing is better than pushing: Although this might sound obvious, it isn’t an easy conclusion to make, because the objects can easily be blown in a wrong direction. The authors have a great side by side comparison video showing how well blowing works.
Generalization to other objects: The authors successfully demonstrate that the trained robots could work equally effectively with larger objects, as well as with irregular objects such as maple leaves. Again, the side by side comparison videos are simply very interesting to watch.
Conclusions:
This work shows how robots can be trained effectively to perform mundane tasks. To me, an impressive contribution from the authors is to show how a policy trained in a successful simulated environment can be transferred into a real-life environment. This work has many practical implications: I look forward to the day when a robot will be able to blow the leaves in my garden to the compost bin. I am also very happy to see the use of Vector in research coming out of premier institutions.
If you found the work interesting, let’s discuss it in the comments below.