Pantograph: Building a preschool for robots

Summary

Pantograph: Building a Preschool for Robots In order to solve robotics' data problem, we're building a preschool for robots. The areas of deep learning that have seen the fastest progress in the past decade are those where data is abundant: language models and image generators can train on the entire internet; game-playing models like AlphaGo can generate data by playing against themselves. These datasets don't exist for robotics, so we need to create them from scratch. At Pantograph, we're creating systems that are capable of unsupervised data gathering in the real world. Our models build representations of the world as they go, gradually learning about the world around them and about how they can influence it. Like language models, they are trained on enough diverse data to be able to generalize to new, unseen tasks. Like AlphaGo, our models learn from experience, continuously improving as they interact with the world. Exploration in the Real World What would the ideal real-world robotics dataset look like? Scale is important, as is diversity. The internet has an abundance of videos, so the most important real-world data to collect is about things that are difficult to infer from video. We need data about the properties of materials: texture, viscosity, density, what it feels like to bend something, to rub something against something else. This first phase of data collection will look something like a robot preschool: thousands of small, inexpensive robots, touching everything they can get their hands on, tossing things around, finding the exact balancing point of two wooden blocks, bending, rubbing, scraping, building up a model of the world around them. This data will be the foundation upon which we will train increasingly capable models. The robots will not only learn about the world around them, but also about themselves. The resulting models will be native to the robot's hardware, better able to exploit its capabilities and idiosyncrasies than any human opera...