Enter the Objaverse: 800,000 Virtual Props for AIs to Play With

If AI is to move out of the chat box and into our living rooms, it will need to better understand spaces and objects. To further this work, the Allen Institute for AI has created a massive and diverse database of 3D models of everyday objects, so that simulations for AI models can be much closer to reality.

Simulators are essentially 3D environments intended to represent real places that a robot or AI might have to navigate or understand. But unlike, say, a modern console game, training simulators are far from photorealistic and often lack detail, variation, or interactivity.

Objaverse, as it goes by a goofy but nice name, aims to improve on that with its collection of over 800,000 (and growing) 3D models with all sorts of metadata. The objects depicted range from types of food to tables and chairs to household appliances and gadgets. Any relatively ordinary object that you might expect to see in a home, office, or restaurant is represented here.

It is intended to replace aging object libraries like ShapeNet, an old backup database with around 50,000 less detailed models. If the only “lamp” your AI has ever seen is a generic lamp with no pattern or color, how can you expect it to recognize a funky cut-glass lamp or one with a totally different shape? Objaverse includes variations on common objects so the model can learn what defines them despite their differences.

Of course, your AI assistant probably won’t need to identify a library as “medieval” or not, but it certainly should know the difference between a peeled and unpeeled banana. But you never know what might matter.

The use of photorealistic images (captured by photogrammetry, that is clear) also brings a level of diversity and realism that is evident in retrospect. Sure, all beds look pretty much the same, but what about unmade beds? All different!

Having objects that also animate to do their “main thing” if you want is also useful. Knowing what a fridge, cabinet, book, laptop or garage door looks like when closed is one thing and when open is another, but how does it get from A to B? It sounds simplistic, but if the AI ​​models aren’t given this information, they’re not likely to make it up or guess it.

You can read more about the features and details of this huge dataset in the AI2 article describing it. And if you’re a researcher, you can start using it for free through Hugging Face.