What makes chatbots “hallucinate” or say the wrong thing?

In today’s AI newsletter, the third in a five-part seriesI discuss some of the ways chatbots can go wrong.

Hours after yesterday’s newsletter was published, a group of artificial intelligence experts and tech leaders, including Elon Musk, urged AI labs to halt work on their most advanced systems, warning that they present “profound risks to society and humanity”.

The group called for a six-month pause on systems more powerful than GPT-4, introduced this month by OpenAI, which Mr. Musk co-founded. A pause would allow time to implement “shared security protocols”, the group said in an open letter. “If such a pause cannot be enacted quickly, governments should step in and institute a moratorium.”

Many experts disagree on the seriousness of the risks cited in the letter, and we’ll explore some of them later this week. But a number of AI mishaps have already surfaced. I will spend today’s newsletter explaining how they happen.

In early February, Google unveiled a new chatbot, Bard, which answered questions about the James Webb Space Telescope. There was just one problem: One of the bot’s claims – that the telescope had captured the first-ever images of a planet outside our solar system – was completely false.

Bots like OpenAI’s Bard and ChatGPT deliver information with bewildering dexterity. But they also spout plausible lies or do really scary things, like insisting they’re in love with New York Times reporters.

How is it possible?

Internet trash and hallucinations

In the past, technology companies carefully defined software behavior one line of code at a time. Today, they design chatbots and other technologies that learn skills on their own, by identifying statistical patterns in huge amounts of information.

Much of this data comes from sites like Wikipedia and Reddit. The internet is full of useful information, ranging from historical facts to medical advice. But it is also full of untruths, hate speech and other garbage. Chatbots absorb everything, including explicit and implicit bias of the text they absorb.

And because of the surprising way they mix and match what they’ve learned to generate entirely new text, they often create compelling language that’s downright wrong or doesn’t exist in their training data. Artificial intelligence researchers call this tendency to invent stuff a “hallucinationwhich may include irrelevant, nonsensical, or factually incorrect answers.

We are already seeing the real consequences of AI hallucinations. Stack Overflow, a Q&A site for programmers, temporarily banned users from submitting answers generated with ChatGPT because the chatbot made it too easy to submit plausible but incorrect answers.

“These systems live in a world of language,” said Melanie Mitchell, an AI researcher at the Santa Fe Institute. they learn is not rooted in reality, they don’t necessarily know if what they are generating is true or false.

(When we asked Bing for examples of mind-blowing chatbots, it actually hallucinated the answer.)

Think of chatbots as jazz musicians. They can digest massive amounts of information — like, say, every song that’s ever been written — and then riff on the results. They have the ability to put ideas together in surprising and creative ways. But they also play the wrong notes with absolute confidence.

It’s not just them, it’s us

Sometimes the wild card is not the software. It’s the humans.

We are inclined to see patterns that don’t really exist and to assume human traits and emotions in non-human entities. This is known as anthropomorphism. When a dog looks us in the eye, we tend to assume that he is smarter than he really is. This is how our minds work.

And when a computer starts putting words together like we do, we get the mistaken impression that it can reason, understand, and express emotions. We can also behave in unpredictable ways. (Last year, Google placed an engineer on paid leave after rejecting his claim that its AI was sentient. He was later fired.)

The longer the conversation, the more influence you have over what a large language model is saying. Kevin’s infamous conversation with Bing is a particularly good example. After a while, a chatbot can start mirroring your thoughts and goals, according to researchers like AI pioneer Terry Sejnowski. If you trick him into being scary, he becomes scary.

He compared the technology to the Mirror of Erised, a mystical artifact in the Harry Potter novels and films. “It provides everything you’re looking for – everything you want, expect or desire,” Dr. Sejnowski said. “Because the human and the LLMs mirror each other, over time they will tend towards a common conceptual state.”

Can they fix it?

Companies like Google, Microsoft, and OpenAI are working to address these issues.

OpenAI worked to refine the chatbot using feedback from human testers. Using a technique called reinforcement learning, the system gained a better understanding of what it should and shouldn’t do.

Microsoft, for its part, has limited the duration of conversations with its Bing chatbot. It also fixes vulnerabilities identified by intrepid users. But solving every problem is difficult, if not impossible.

So yeah, if you’re smart, you can probably trick these systems into doing offensive or scary things. Bad actors can, too: Many experts fear that these bots allow internet scammers, unscrupulous marketers, and hostile nation states to spread misinformation and cause other kinds of trouble.

a big thing

When using these chatbots, be skeptical. Look at them for what they really are.

They are neither sentient nor conscious. They are smart in some ways, but stupid in others. Remember they can be wrong. Remember they can make stuff up.

But on the bright side, there are so many other things that these systems are very good at. Kevin will talk more about that tomorrow.

Your homework

Ask ChatGPT or Bing to explain something you already know well. Are the answers correct?

If you get any interesting answers, good or bad, you can share them in the comments.

---

Quiz

Question 1 of 3

How do large language models generate text?

Start the quiz by choosing your answer.

---

Glossary

Hallucination: A well-known phenomenon in large language models, in which the system provides an answer that is factually incorrect, irrelevant, or nonsensical, due to the limitations of its training data and architecture.

Bias: A type of error that can occur in a large language model if its output is skewed by model training data. For example, a model may associate specific traits or occupations with a certain race or gender, resulting in inaccurate predictions and offensive responses.

Anthropomorphism:The tendency of people to attribute human qualities or characteristics to an AI chatbot. For example, you can assume she’s nice or cruel based on her responses, even though she’s not capable of having emotions, or you can believe the AI ​​is sentient because she’s very good at it. to imitate human language.

Click here for more glossary terms.