Reddit wants to get paid to help teach big AI systems

Reddit has long been a hot spot for conversations on the internet. About 57 million people visit the site every day to discuss topics as varied as makeup, video games and pointers for power wash aisles.

In recent years, Reddit’s range of cats has also been a free teaching aid for companies like Google, OpenAI, and Microsoft. These companies are using Reddit conversations in the development of behemoth artificial intelligence systems that many Silicon Valley residents believe are poised to be the next big thing in the tech industry.

Now Reddit wants to get paid for it. The company said on Tuesday it plans to start charging businesses for access to its application programming interface, or API, the method by which outside entities can download and process the vast selection of person-to-person conversations. of the social network.

“The Reddit body of data is truly valuable,” Reddit Founder and CEO Steve Huffman said in an interview. “But we don’t need to give away all that value to some of the biggest companies in the world.”

The move marks one of the first significant examples of a social network charging for access to conversations it hosts for the purpose of developing AI systems like ChatGPT, OpenAI’s popular program. These new AI systems could one day lead to big businesses, but they’re not likely to help many companies like Reddit. In fact, they could be used to create competitors – automated duplicates of Reddit conversations.

Reddit’s move also comes as it prepares for a possible initial public offering on Wall Street later this year. The company, which was founded in 2005, makes most of its money from advertising and e-commerce transactions on its platform. Reddit said it is still working out the details of what it will charge for API access and will announce pricing in the coming weeks.

Reddit chats — or subreddits, as the company calls them — have become valuable commodities as large language models, or LLMs, have become an essential part of building new AI technologies.

LLMs are basically sophisticated algorithms developed by companies like Google and OpenAI, which is a close partner of Microsoft. For algorithms, Reddit conversations are data, and they are part of the vast pool of material fed into LLMs to develop them.

The underlying algorithm that helped build Bard, Google’s conversational AI service, is partially trained on data from Reddit. OpenAI’s GPT Chat cites Reddit data as one of the sources of information it was trained on.

Other companies are also starting to see value in the conversations and images they host. Shutterstock, the image hosting service, also sold image data to OpenAI to help build DALL-E, the generative AI program that creates vivid new graphical images with only a text prompt required.

Last month, Twitter owner Elon Musk said he was cracking down on the use of Twitter’s API, which is used by thousands of outside companies and independent developers to track the millions of conversations happening. produce on the network. Although he did not cite LLMs as the reason for the change, the new fees could run into the tens or even hundreds of thousands of dollars.

To keep improving their models, AI makers need two important things: a huge amount of computing power and a huge amount of data. Some of the biggest AI developers have a lot of computing power, but are still looking outside their own networks for the data needed to improve their algorithms. This includes sources like Wikipedia, millions of digitized books, academic articles, and Reddit.

Reddit has long had a symbiotic relationship with search engines from companies like Google and Microsoft. Search engines “crawl” Reddit’s web pages to index information and make it available for search results. This crawling, or “scraping”, is not always welcomed by all sites on the Internet. But Reddit benefited by appearing higher in search results.

The dynamic is different with LLMs – they gobble up as much data as possible to create new AI systems like chatbots.

Reddit thinks its data is particularly valuable because it is continually updated. This novelty and relevance, Huffman said, is what great language modeling algorithms need to produce the best results.

“More than any other place on the internet, Reddit is a place for authentic conversation,” Huffman said. “There’s a lot of stuff on the site that you would never say except in therapy, or AA, or ever at all.”

Huffman said Reddit’s API will always be free for developers who want to build apps that help people use Reddit. They could use the tools to create a bot that automatically tracks whether user comments meet a subreddit’s rules, for example. Researchers who wish to study Reddit data for academic or non-commercial purposes will continue to access it for free.

Reddit also hopes to integrate more so-called machine learning into the operation of the site itself. It could be used, for example, to identify the use of AI-generated text on Reddit and add a tag that alerts users that the comment is from a bot.

The company also promised to improve software tools that can be used by moderators – users who volunteer their time to keep the site’s forums running smoothly and improve conversations between users. And third-party bots that help moderators monitor forums will continue to be supported.

But for AI makers, it’s time to pay.

“Crawling Reddit, generating value and not giving anything back to our users is something we have trouble with,” Huffman said. “Now is a good time for us to tighten things up.”

“We think it’s fair,” he added.