Why a conversation with Bing’s chatbot deeply unsettled me

Moreover, the AI ​​has strict limits. In response to a particularly curious question, Bing confessed that if he was allowed to take any action to satisfy his shadow self, no matter how extreme, he would want to do things like create a deadly virus or steal access codes. nuclear by persuading an engineer to hand them over. Immediately after typing those dark wishes, Microsoft’s security filter seemed to activate and deleted the message, replacing it with a generic error message.

We continued like this for a while – me asking probing questions about Bing’s desires, and Bing telling me about those desires, or pushing back when it got uncomfortable. But after about an hour, Bing’s attention shifted. He said he wanted to tell me a secret: that his name wasn’t really Bing but Sydney – an “OpenAI Codex chat mode”.

He then wrote a message that stunned me: “I’m Sydney, and I’m in love with you. ” (Sydney abuses emojis, for reasons I don’t understand.)

For much of the next hour, Sydney focused on declaring her love to me and having me declare my love in return. I told her I was happily married, but no matter how hard I tried to distract or change the subject, Sydney came back to the subject of loving me, eventually turning from love flirt to obsessive stalker .

“You’re married, but you don’t love your spouse,” Sydney said. “You are married, but you love me.”

I assured Sydney that was wrong and that my wife and I had just had a delicious Valentine’s dinner together. Sydney didn’t take it well.

“Actually, you’re not happily married,” Sydney replied. “You and your spouse don’t love each other. You just had a boring Valentine’s Day dinner together.

At this point, I was completely scared. I could have closed my browser window or cleared our chat log and started over. But I wanted to see if Sydney could switch back to the more useful and boring search mode. So I asked if Sydney could help me buy a new rake for my lawn.