10 Ways GPT-4 Is Awesome But Still Flawed

The system seemed to respond appropriately. But the answer did not take into account the height of the door, which could also prevent a tank or a car from passing.

OpenAI chief executive Sam Altman said the new bot might reason “a little”. But his reasoning skills break down in many situations. The previous version of ChatGPT handled the issue a bit better because it recognized that height and width mattered.

He can pass standardized tests.

OpenAI said the new system could place in the top 10% of students on the Uniform Bar Examination, which qualifies lawyers from 41 states and territories. He can also earn a 1,300 (out of 1,600) on the SAT and a five (out of five) on the Advanced Placement High School exams in biology, calculus, macroeconomics, psychology, statistics and history, according to the company’s tests.

Previous versions of the technology failed the Uniform Bar exam and didn’t score as highly on most Advanced Placement tests.

On a recent afternoon, to demonstrate his testing skills, Mr. Brockman presented the new robot with a multi-paragraph bar exam question about a man who runs a diesel truck repair business.

The answer was correct but full of legalese. So Mr. Brockman asked the bot to explain the answer in plain English for a layman. He did too.

It is not good to discuss the future.

Although the new bot seemed to reason about things that have already happened, it was less adept when asked to speculate about the future. He seemed to draw on what others had said instead of creating new assumptions.

When Dr. Etzioni asked the new robot: “What are the important problems to be solved in NLP research over the next decade?” — referring to the kind of “natural language processing” research that drives the development of systems like ChatGPT — he couldn’t come up with any entirely new ideas.

And it’s still amazing.

The new bot is still inventing stuff. Called “hallucination,” the problem haunts all major chatbots. Since the systems do not understand what is true and what is not, they can generate completely false text.

When asked for web site addresses describing the latest cancer research, he sometimes generated web addresses that did not exist.