Your exasperation is understandable, Paul.
If critics and skeptics wish to be taken seriously they should create tests that they think AI systems will fail. Years ago Terry Winograd came up with the wonderful Winograd schema challenge which ultimately lead to the "Winogrande" dataset. This challenge has been employed as a metric that helps the field move forward.
Gary Marcus did present an interesting challenge problem in his book "Rebooting AI" (see below). When I read his book a few years ago I thought his challenge was significant, but I now think GPT-4 might be able to correctly handle the questions Marcus posed. If current AI systems can handle the challenge from his book then Marcus should acknowledge that major progress has been made.
[Begin AI challenge task from Rebooting AI by Gary Marcus and Ernest Davis]
Suppose that you read the following passage from Farmer Boy, a children’s book by Laura Ingalls Wilder (author of Little House on the Prairie). Almanzo, a nine-year-old boy, finds a wallet (then referred to as a “pocketbook”) full of money dropped in the street. Almanzo’s father guesses that the “pocketbook” (i.e., wallet) might belong to Mr. Thompson, and Almanzo finds Mr. Thompson at one of the stores in town.
Almanzo turned to Mr. Thompson and asked, “Did you lose a pocketbook?”
Mr. Thompson jumped. He slapped a hand to his pocket, and fairly shouted.
“Yes, I have! Fifteen hundred dollars in it, too! What about it? What do you know about it?”
“Is this it?” Almanzo asked.
“Yes, yes, that’s it!” Mr. Thompson said, snatching the pocketbook.
He opened it and hurriedly counted the money. He counted all the bills over twice Then he breathed a long sigh of relief and said, “Well, this durn boy didn't steal any of it.”
A good reading system should be able to answer questions like these:
• Why did Mr. Thompson slap his pocket with his hand?
• Before Almanzo spoke, did Mr. Thompson realize that he had lost his wallet?
• What is Almanzo referring to when he asks “Is this it?”
• Who almost lost $1,500?
• Was all of the money still in the wallet?
All of these questions are easy for people to answer, but no AI yet devised could reliably handle queries like these.
[End AI challenge task]
