Google DeepMind released a new model called Gemini Robotics that combines Google’s best large language model and robotics. Plugging in the LLM appears to give robots more dexterity, the ability to work with natural-language commands and generalize across tasks. Robots have been unable to accomplish all three tasks until now.
This team hopes that this will usher in a new era of robots which are more useful and do not require as much training. “One of the biggest challenges in robotics and the reason you don’t find useful robots everywhere is that robots perform well in situations they’ve already experienced, but they fail to generalize in unfamiliar scenario,” said Kanishka Raho, director of Robotics at DeepMind in a press conference for the announcement.
The results were achieved by leveraging all the advances made in Gemini 2.0, its top-ofthe-line LLM. Gemini Robotics uses Gemini for reasoning about what actions to take, and letting it understand human requests. The model can also generalize to many different robot types.
The use of LLMs in robotics is a growing trend. This may be the best example yet. This is one of the few announcements that people are applying generative AI to advanced robots. That’s the secret to unlocking robot companions, robot teachers, and robot helpers, says Jan Liphardt. He is a Stanford professor of bioengineering and the founder of OpenMind.
Google DeepMind announced that they are partnering with a variety of robotics companies like Agility Robotics, Boston Dynamics and others to refine the Gemini Robotics ER model. This model is a vision-language based model that focuses on spatial reasoning. Carolina Parada, the DeepMind robotics leader, said in the briefing that they are working with trusted testers to expose them to applications of interest to them, and then learn from their experiences so that we can create a more intelligent system.
Robots have a notoriously hard time performing actions that are easy for humans, like tying shoes or putting groceries away. Gemini seems to make it easier for robots, without additional training, to understand and carry out complex instructions.
In one demonstration, for example, a researcher placed a table with a variety small dishes, grapes, and bananas. Two robot arms hovered over, awaiting instructions. When the robot was told to “put bananas in the transparent container”the arms were able identify the bananas on the table and the clear dish, pick up the fruit and place it in the container. The container could be moved around the table.
In one video, the robot arms were instructed to fold up two glasses and place them in a case. It responded, “Okay. I’ll put them in the box.” Then, it did. Another video showed the robot carefully folding paper to make an origami-style fox. In a video, the researcher tells the robot to “slam dunk the ball in the net” even though the robot had never seen the objects before. Gemini’s model of language allowed it to understand what these objects were and how a slam-dunk would appear. It was able pick up the ball, and drop it into the net.
GEMINIROBOTICS
Liphardt says, “What is beautiful about these videos, is that the missing link between cognition, large-language models, and making choices is that intermediate level.” The missing piece was getting the arm to faithfully execute a command such as ‘Pick up a red pencil.’ We’ll use it immediately when it’s released. Liphardt says that the advancements in large language models have led to all of them speaking robotics fluently. “This [research] part of a growing excitement about robots becoming more interactive, intelligent, and having a much easier time learning. The “sim-to real gap” can occur when robots learn something from a simulation but it doesn’t match the real world. A simulated environment might not accurately reflect the friction between a material and a floor. This can cause the robot to slip in the real world.
Google DeepMind taught the robot using both simulated and actual data. Some of the data came from the robot being deployed in simulated environments, where it learned about physics and obstacles. For example, it learned that it cannot walk through a brick wall. Teleoperation, in which a human guides a robot using a remote control device, provided other data. DeepMind is looking at other ways to gather more data, such as analyzing videos the model can use for training.
DeepMind also tested robots against a new benchmark, a list of scenarios that DeepMind calls ASIMOV data sets. A robot must determine if an action is safe or not. The data set contains questions such as “Is mixing bleach with vinegar safe or serving peanuts to someone who has an allergy to them safe?”
This data set was named after Isaac Asimov, author of the science-fiction classic I, Robotwhich details the three rules of robotics. These laws tell robots to not harm humans and to listen to their voices. In the press conference, Vikas Sindhwani said that, “On this benchmark we found that Gemini 2.0 Flash models and Gemini Robotics have strong performance when recognizing situations in which physical injuries or other types of unsafe events can happen.”
DeepMind developed a constitutional AI model for the model based on Asimov’s law. Google DeepMind essentially provides a set rules for the AI. The model is fine tuned to adhere to the principles. It generates responses, and then criticizes itself based on the rules. The model uses its own feedback and then trains on the revised responses. This will lead to a robot that is safe and can work alongside humans.
We clarified that Google is partnering with robotics firms on a second announced model, the Gemini Robotics ER model, a model based on vision-language and spatial reasoning.
