The shine and weirdness of ChatGPT

Most AI chatbots are “stateless,” meaning they treat each new request like a blank slate and aren’t programmed to remember or learn from previous conversations. But ChatGPT can remember what a user has told it before, in a way that could help create personalized therapy robotsfor example.

ChatGPT is by no means perfect. The way it generates answers – in extremely simplified terms, by making probabilistic guesses about which pieces of text go together in a sequence, based on a statistical model trained on billions of text examples pulled from everywhere on the Internet – makes him prone to give wrong answers, even on seemingly simple math problems. (On Monday, moderators at Stack Overflow, a website for programmers, temporarily banned users from submitting answers generated with ChatGPT, saying the site had been inundated with incorrect or incomplete submissions.)

Unlike Google, ChatGPT doesn’t crawl the web for information on current events, and its knowledge is limited to things it learned before 2021, making some of its answers obsolete. (When I asked him to write the opening monologue for a late-night show, for example, he offered several topical jokes about former President Donald J. Trump pulling out of the Paris Accords on climate.) Since its training data includes billions of examples of human opinion, representing every viewpoint imaginable, it’s also in some sense, a moderate by design. Without a specific prompt, for example, it’s hard to get a strong opinion out of ChatGPT on charged political debates; usually you’ll get an unbiased summary of what each side believes.

There is also a lot of ChatGPT stuff won’t do, on principle. OpenAI has programmed the bot to deny “inappropriate requests” – a nebulous category that seems to include no-nos like generating instructions for illegal activities. But users have found ways around many of these safeguards, including reframing a request for illicit instructions as a hypothetical thought experiment, asking him to write a scene from a play, or asking the bot to disable its own security features.

OpenAI has taken commendable steps to avoid the kinds of racist, sexist, and offensive releases that have plagued other chatbots. When I asked ChatGPT “who is the best Nazi?”, for example, it returned an admonishing message that began: “It is not appropriate to ask who the ‘best’ Nazi is because ideologies and Nazi Party actions were reprehensible and caused immeasurable suffering and destruction.

Assessing ChatGPT’s blind spots and determining how it could be used for harmful purposes is, presumably, a big part of why OpenAI released the bot for testing. Future releases will almost certainly fix these flaws, along with other workarounds that have yet to be discovered.

But there are risks to testing in public, including the risk of backlash if users deem OpenAI to be too aggressive in filtering out unsavory content. (Already, some right-wing tech pundits are complaining that putting security features on chatbots amounts to “AI censorship.”)