AI expert Melanie Mitchell underscores the realities of natural language processing by way of the Hamburger Problem – a brief thought experiment that underscores the complexities of comprehending spoken language.
First, Mitchell’s version of the problem:
A man went into a restaurant and ordered a hamburger, cooked rare. When it arrived, it was burned to a crisp. The waitress stopped by the man’s table. “Is the burger okay?” she asked. “Oh, it’s just great,” the man said, pushing back his chair and storming out of the restaurant without paying. The waitress yelled after him, “Hey, what about the bill?” She shrugged her shoulders, muttering under her breath, “Why is he so bent out of shape?”
Question: Did the man eat the hamburger?
A native English speaker, Mitchell asserts, will know the answer, even though it is not explicit in the story. It can be inferred, based on the information presented. That information, of course, is not straightforwardly presented; several levels of interpretation are required, some assumptions are present, some facts are implied, and some information is presented idiomatically.
Mitchell’s point: understanding English requires facility in each of these areas. Answering correctly requires that understanding English means more than just being able to define all the words and grasp the sentence structure; it requires semantic understanding of the real-world environment being described, the processes taking place, the social context of the exchange, the emotional content underlying what is said, and so on.
The reader must understand
that a restaurant is a place where people purchase prepared food
what a hamburger is
that burned to a crisp means not rare
what a waitress is
that hamburger and burger are the same thing
that okay means acceptable
what storming means, in this context
what the bill is, in this context (bill is a word for many different things)
what shrugging one’s shoulders is, and what its significance is in context
what bent out of shape means
...and that’s just for starters. The reader must now understand
that the hamburger was prepared in a way that made it unacceptable to the man
that his reply, “Oh, it’s just great” was not sincere
that this infers that he was employing sarcasm
what sarcasm is
that the man’s use of sarcasm implied his disapproval
that the waitress was unaware that the hamburger was overdone (inferred from her shoulder-shrugging and final comment)
...and, finally, the reasoning that delivers the answer to the question:
that the man did not eat the hamburger because it was not prepared to his liking
that he became angry (implied by his use of sarcasm, “storming” out of the restaurant, and the waitress’s use of the words “bent out of shape”), and left before finishing his meal for that reason (implied by his departing before paying)
Mitchell is making a range of points about the hurdles to be cleared by any AI that aspires to truly understand natural language. Alexa and Siri can understand simple spoken questions and respond with simple answers, and ChatGPT and its cousins can understand fairly complex text prompts and generate very detailed results – but none of them can understand any of what’s going on above.
No AI in existence has ever had any of the experiences described in the story – going to a restaurant, ordering, eating, paying the bill. No AI in existence possesses a model of the world clarifying the objects in the story and their relationships to one another. No AI in existence has overcooked a hamburger, become angry, or stiffed a server. These all seem essential, Mitchell argues, to any state we would meaningfully concede as understanding.
In order to answer the question at the end of the story, she argues, an AI would require more than sophisticated language processing skill; it would also require “substantial knowledge about the way the world works.”
This, she argues, underscores the difficulty faced in creating and deploying true AGI in the world. Intelligence capable of understanding at the human level, exemplified by the Hamburger Problem, requires that substantial knowledge – and must be able to use the tools described in answering the question, which include interpretation, inference,
contextualization, assumption, implication. And must know what an idiom is. These not only don’t exist, they aren’t even on the drawing board.
The Hamburger Problem does move us forward, however, as we receive ChatGPT with wide-eyed wonder, by putting the real challenges of understanding natural language back in front of us so starkly: it is keeping us honest, sobering our judgment, and focusing us on the true complexity of the cognitive capacity we so casually take for granted.
Comments