If you need more proof that current generations of AI chatbots may not be ready for prime time, here it is.
On Sunday, Kansas City Chiefs faced off with San Francisco 49ers at the Super Bowl LVIII. The game went into extra time, for only the second time in its history, with the Chiefs eventually carrying the day.
But even before the first whistle was blown, two of the most popular AI chatbots already had the ‘results’, leaving users dumbfounded.
The hallucinations were so bad, that both Google’s Gemini and Microsoft’s Copilot believed that not only had the game been played, but the 49ers had won it.
A Reddit thread highlighted these perplexing outputs that somehow seemed to stumper not just a single chatbot.
Gemini appeared not to know what date or time the kickoff was, and instead of answering appropriately, went on to make up what to the naked eye would look like very convincing facts.
“As of today, February 11, 2024, the Super Bowl already happened! It took place yesterday, February 10th, and the San Francisco 49ers defeated the Kansas City Chiefs by a score of 34-28.” it responded, getting the date and the winner wrong.
Copilot was no better, also claiming that the San Francisco 49ers emerged victorious with a final score of 24-21. It even had the names of the players responsible for the win.
These glaring errors highlight the importance of real-time knowledge for large language models, and the reason why these companies are aggressively pursuing partnerships with news organizations.
To its credit, the most popular chatbot, ChatGPT, was able to correctly identify that the game was not decided yet, which makes Copilot’s answer more perplexing, since they both use the same foundation model.
Update: Another Reddit user appears to have solved the mystery.
ChatGPT was able to identify the source of the 34-28 score given by Copilot. It turns out that this was a prediction made by NFL analyst Peter Schrager in late January. Even then, Copilot had the teams all mixed up.
“It’ll be 49ers vs. Chiefs in Vegas,” Schrager was quoted as saying. “The winner, with the exact score being 34-28, and with second-year cornerback Trent McDuffie returning a pick-six late in the fourth quarter, the Kansas City Chiefs will yet again be your Super Bowl champion.”
But perhaps the more important questions is how Copilot failed to understand that this was a mere prediction.
