Why do AI chatbots tell lies and act weird? Look in the mirror.

When Microsoft added a chatbot to its Bing search engine this month, people noticed it offered all sorts of misinformation about the Gap, Mexican nightlife and singer Billie Eilish.

Then, when reporters and other early testers had long conversations with Microsoft’s AI bot, it slid into rude and creepy behavior.

Ever since the behavior of the Bing bot became a global sensation, people have struggled to understand the weirdness of this new creation. More often than not, scientists have said that humans deserve a big share of the blame.

But there’s still some mystery about what the new chatbot can do — and why it would. Its complexity makes it hard to dissect and even harder to predict, and researchers are looking at it through a philosophical lens as well as the hard code of computing.

Like any other learner, an AI system can learn bad information from bad sources. And this strange behavior? It can be a chatbot’s distorted reflection of the words and intentions of the people using it, said Terry Sejnowski, a neuroscientist, psychologist and computer scientist who helped lay the intellectual and technical foundations of modern artificial intelligence.

“It happens when you dig deeper and deeper into these systems,” said Dr. Sejnowski, a professor at the Salk Institute for Biological Studies and the University of California, San Diego, who published a research paper this month on this phenomenon in the scientific journal. Neural Computation review. “Anything you seek – anything you desire – they will provide.”

Google also introduced a new chatbot, Bard, this month, but scientists and journalists quickly realized he was writing nonsense about the James Webb Space Telescope. OpenAI, a San Francisco startup, kicked off the chatbot boom in November by introducing ChatGPT, which doesn’t always tell the truth.

The new chatbots are driven by a technology scientists call a large language model, or LLM. These systems learn by analyzing massive amounts of digital text mined from the Internet, which includes volumes of misleading, biased, and otherwise toxic material. The text that chatbots learn from is also a bit outdated, as they have to spend months analyzing it before the public can use them.

By analyzing this sea of ​​good and bad information on the Internet, an LLM learns to do one thing in particular: guess the next word in a sequence of words.

It works like a giant version of auto-complete technology that suggests the next word when you type an email or instant message on your smartphone. Given the “Tom Cruise is a ____” sequence, he might guess “actor.”

When you chat with a chatbot, the bot does not just take advantage of everything it has learned on the Internet. He draws inspiration from everything you’ve said to him and everything he’s said back. It’s not just about guessing the next word in his sentence. It guesses the next word in the long block of text that includes both your words and its words.

The longer the conversation, the more influence a user unwittingly has over what the chatbot is saying. If you want him to get mad, he will get mad, Dr. Sejnowski said. If you coax him into being scary, he becomes scary.

The alarmed reactions to the strange behavior of Microsoft’s chatbot have obscured an important point: the chatbot has no personality. It offers instant results spit out by an incredibly complex computer algorithm.

Microsoft seemed to limit the oddest behavior by limiting the length of chats with the Bing chatbot. It was like learning from a car’s test driver that driving too fast for too long will burn out its engine. Microsoft partner OpenAI and Google are also exploring ways to control the behavior of their bots.

But there’s a caveat to this assurance: because chatbots learn from so much material and put it together in such complex ways, researchers don’t quite know how the chatbots produce their end results. Researchers observe what bots do and learn to place limits on that behavior, often after it occurs.

Microsoft and OpenAI have decided that the only way to know what chatbots will do in the real world is to let them go – and reel them in when they wander off. They believe their great public experience is worth the risk.

Dr Sejnowski compared the behavior of Microsoft’s chatbot to the Mirror of Rised, a mystical artifact of JK Rowling’s Harry Potter novels and the many films based on his inventive world of young wizards.

“Rised” is “desire” spelled backwards. When people discover the mirror, it seems to bring truth and understanding. But this is not the case. It shows the deep desires of whoever looks at it. And some people go crazy if they stare too long.

“Because the human and the LLMs both mirror each other, over time they will tend toward a common conceptual state,” Dr. Sejnowski said.

It’s no surprise, he said, that journalists have begun to see creepy behavior in the Bing chatbot. Consciously or unconsciously, they were pushing the system in an uncomfortable direction. As chatbots pick up our words and send them back to us, they can reinforce and amplify our beliefs and lead us to believe what they tell us.

Dr. Sejnowski was one of a small group of researchers in the late 1970s and early 1980s who began to seriously explore a kind of artificial intelligence called a neural network, which drives today’s chatbots. .

A neural network is a mathematical system that acquires skills by analyzing numerical data. It’s the same technology that allows Siri and Alexa to recognize what you say.

Around 2018, researchers from companies like Google and OpenAI began building neural networks that learned large amounts of digital text, including books, Wikipedia articles, chat logs, and other published items. on the Internet. By identifying billions of patterns in all that text, these LLMs learned how to generate text themselves, including tweets, blog posts, speeches, and computer programs. They could even carry on a conversation.

These systems are a reflection of humanity. They learn their skills by analyzing the text that humans have posted on the Internet.

But that’s not the only reason chatbots generate problematic language, said Melanie Mitchell, an artificial intelligence researcher at the Santa Fe Institute, an independent lab in New Mexico.

When generating text, these systems do not repeat verbatim what is on the Internet. They themselves produce new texts by combining billions of patterns.

Even if researchers trained these systems solely on peer-reviewed scientific literature, they could still produce scientifically ridiculous claims. Even if they only learned from text that was true, they could still produce untruths. Even if they only learned from sane text, they could still generate something scary.

“There’s nothing stopping them from doing that,” Dr. Mitchell said. “They’re just trying to produce something resembling human language.”

Artificial intelligence experts have long known that this technology exhibits all kinds of unexpected behavior. But they can’t always agree on how this behavior should be interpreted or how quickly chatbots will improve.

Because these systems learn from far more data than we humans could ever comprehend, even AI experts cannot figure out why they are generating a particular piece of text at any given time.

Dr. Sejkowski said he believes that in the long run, new chatbots have the power to make people more efficient and empower them to do their jobs better and faster. But that comes with a warning for both the companies building these chatbots and the people using them: they can also lead us away from the truth and into dark places.

“It’s terra incognita,” Dr. Sejkowski said. “Humans have never experienced this before.”