Instant videos could represent the next leap in AI technology

Ian Sansavera, a software architect at a New York startup called Runway AI, typed out a short description of what he wanted to see in a video. “A quiet river in the forest,” he wrote.

Less than two minutes later, an experimental internet service generated a short video of a lazy river in a forest. The running water of the river glistened in the sun as it cut between trees and ferns, turned a corner and gently splashed against the rocks.

Runway, which plans to open its service to a small group of testers this week, is one of many companies developing artificial intelligence technology that will soon allow people to generate videos simply by typing multiple words into a box. on a computer screen.

They represent the next stage in an industry race – which includes giants like Microsoft and Google as well as much smaller start-ups – to create new kinds of artificial intelligence systems that some say could be the next big thing. technological advance, as important as web browsers or the iPhone.

New video generation systems could speed up the work of filmmakers and other digital artists, while becoming a new and fast way to create hard-to-detect online misinformation, making it even harder to tell what’s real on the internet. .

The systems are examples of so-called generative AI, which can instantly create text, images and sounds. Another example is ChatGPT, the online chatbot created by San Francisco startup OpenAI, which stunned the tech industry with its capabilities late last year.

Google and Meta, the parent company of Facebook, unveiled the first video generation systems last year, but did not share them with the public because they feared the systems could eventually be used to spread disinformation with rapidity and speed. found efficiency.

But Runway chief executive Cris Valenzuela said he thought the technology was too important to keep in a research lab, despite its risks. “It’s one of the most impressive technologies we’ve built in the last hundred years,” he said. “You have to have people actually using it.”

The ability to edit and manipulate movies and videos is of course nothing new. Filmmakers have been doing it for over a century. In recent years, researchers and digital artists have used various AI technologies and software to create and edit videos often referred to as deepfake videos.

But systems like the one Runway has created could eventually replace editing skills with the press of a button.

Runway’s technology generates videos from any short description. To get started, just enter a description like you would for a quick note.

This works best if the scene has action – but not too much action – something like “a rainy day in the big city” or “a dog with a cell phone in the park”. Hit enter and the system generates a video in a minute or two.

The technology can reproduce common images, such as a cat sleeping on a rug. Or it can combine disparate concepts to generate weirdly funny videos, like a cow at a birthday party.

The videos are only four seconds long and the video is jerky and blurry if you watch closely. Sometimes the images are weird, distorted and disturbing. The system has a way of fusing animals like dogs and cats with inanimate objects like balls and cell phones. But given the right prompt, it produces videos that show where the technology is going.

“At this point, if I see a high-resolution video, I’m probably going to trust it,” said Phillip Isola, a Massachusetts Institute of Technology professor who specializes in AI. “But that’s going to change pretty quickly.”

Like other generative AI technologies, Runaway’s system learns by analyzing digital data — in this case, photos, videos, and captions describing what those images contain. By training this kind of technology on ever-increasing amounts of data, researchers are confident that they can quickly improve and extend its skills. Soon, experts say, they will be generating professional-looking mini-movies, complete with music and dialogue.

It is difficult to define what the system currently creates. It’s not a photo. It’s not a cartoon. It is a collection of lots of pixels mixed together to create realistic video. The company plans to offer its technology alongside other tools that it says will speed up the work of professional artists.

Last month, social media services were abuzz with images of Pope Francis in a white Balenciaga puffer jacket – a surprisingly trendy outfit for an 86-year-old pontiff. But the pictures weren’t real. A 31-year-old construction worker from Chicago had created a viral sensation using a popular AI tool called Midjourney.

Dr. Isola has spent years building and testing this kind of technology, first as a researcher at the University of California, Berkeley and OpenAI, then as a professor at MIT. completely false images of Pope Francis.

“There was a time when people would post deepfakes, and they wouldn’t fool me, because they were so weird or not very realistic,” he said. “Now we can’t take any of the images we see on the internet at face value.”

Midjourney is one of many services that can generate realistic still images from a short prompt. Others include Stable Diffusion and DALL-E, an OpenAI technology that started this wave of photo generators when it was unveiled a year ago.

Midjourney relies on a neural network, which learns its skills by analyzing massive amounts of data. It searches for patterns by sifting through millions of digital images along with text captions that describe what each image represents.

When someone describes an image for the system, they generate a list of features that the image can include. One feature might be the curve at the top of a dog’s ear. Another could be the edge of a cellphone. Then, a second neural network, called the diffusion model, creates the image and generates the pixels needed for the features. It ultimately transforms the pixels into a coherent image.

Companies like Runway, which has around 40 employees and has raised $95.5 million, use this technique to generate moving images. By analyzing thousands of videos, their technology can learn to string together many still images in the same consistent fashion.

“A video is just a series of images – still images – that are combined in a way that gives the illusion of movement,” Valenzuela said. “The trick is to train a model that understands the relationship and consistency between each image.”

Like early versions of tools like DALL-E and Midjourney, the technology sometimes combines concepts and images in curious ways. If you ask for a basketball-playing teddy bear, it might be some sort of mutant stuffed animal with a basketball for a hand. If you ask for a dog with a cellphone in the park, it might get you a cellphone-wielding pup with a weirdly human body.

But experts think they can eliminate the flaws by training their systems on more and more data. They believe technology will make video creation as easy as writing a sentence.

“Before, to do something like this remotely, you had to have a camera. You had to have props. You had to have a location. You had to have permission. You had to have money,” Susan Bonser said. , a Pennsylvania-based author and editor who experimented with early incarnations of generative video technology, “You don’t have to have all of this now. You can just sit back and imagine it.”