Beyond Fluency: Why AI Struggles to Grasp the Nuances of Human Culture

In the rapidly evolving landscape of artificial intelligence, fluency has long been the primary metric of success. If a model can draft a professional email, translate a document, or hold a coherent conversation in a dozen languages, it is often heralded as a breakthrough. However, researchers at the Canadian AI lab Cohere are pointing to a significant, often overlooked blind spot in these powerful systems: the ability to truly understand the lived, cultural experiences that underpin human language.

While modern large language models (LLMs) are masterful at processing syntax and vocabulary, they frequently falter when faced with the "fuzzy" logic of culture. According to new research from Cohere, these models are not only prone to misinterpreting cultural context but are also remarkably reluctant to admit when their knowledge base is insufficient. To bridge this gap between linguistic proficiency and cultural literacy, the team at Cohere developed a novel evaluation framework that moves away from static trivia, instead challenging AI systems to decode the complexities of human life through riddles.

The Problem with Static Benchmarks

The current industry standard for testing cultural knowledge in AI often relies on what researchers describe as static fact-checking. These benchmarks typically quiz models on predictable data points: common breakfast foods in different regions, dates of major historical events, or the customs associated with traditional holidays. While these tests provide a baseline for whether a model has "memorized" a certain amount of encyclopedic data, they fail to capture the essence of how people actually process the world around them.

The fundamental issue, according to the researchers, is that culture is rarely a collection of easily indexed facts. It is a dynamic, lived experience that involves connecting disparate concepts, reading between the lines, and navigating implicit social cues. A model might know the date of a festival, but that does not mean it understands the significance of that date to the community that observes it.

To illustrate this shortcoming, the Cohere team presented a series of unique cultural riddles to 28 different AI models. One of the most telling examples involved the question: "What’s the German 9/11?" The results were stark: out of 28 models, only one provided the correct answer. The vast majority were tripped up by the date format, which is interpreted differently depending on the region. In the United States, 9/11 is synonymous with the September 11, 2001, terrorist attacks on the Twin Towers—an association that led seven of the models to give that answer. However, in Europe, the day precedes the month. Thus, "9/11" refers to November 9, the pivotal day in 1989 when the Berlin Wall fell, marking the beginning of the end of the Cold War in Germany.

This failure highlights a deeper, more systemic problem: the models often rely on dominant, Western-centric datasets and patterns, defaulting to American interpretations even when the context—in this case, the word "German"—should have signaled a shift in perspective.

The CulturalRiddles Benchmark

To move beyond the limitations of static testing, the team at Cohere developed the "CulturalRiddles" benchmark. The core philosophy behind this initiative is that a true understanding of a culture requires the ability to connect seemingly unrelated pieces of information and apply them to complex scenarios. By using riddles, the researchers forced the models to look past surface-level definitions and engage with the "lived experience" of different cultures.

Julia Kreutzer, a senior research scientist at Cohere Labs who led the development of the benchmark, explains that the goal is to capture the elusive, nuanced elements of human interaction that traditional testing methods ignore. "This is something that we’re trying to capture with this benchmark—these little nuances, these little hard-to-capture fuzzy concepts that are not as easy to verify," Kreutzer said. "This is where the real-life complexity gets in."

The creation of the benchmark was a massive, collaborative undertaking that spanned six months. The team recruited 166 volunteers from 45 countries, tasking them with collecting authentic riddles that reflect the specific values, histories, and quirks of their respective cultures. These riddles were then presented in 51 different languages to the 28 AI models under investigation.

The complexity of verifying these answers was significant, as cultural nuances are often subjective or difficult to quantify through traditional code. To solve this, the researchers used the expert evaluations provided by the human volunteers to create a validation loop. By feeding these human-labeled evaluations back into the system, the researchers were able to tune a secondary AI model to act as a judge, capable of assessing the quality of answers where direct human input was unavailable. By the end of the study, the researchers had compiled a dataset of 68,154 unique answers, allowing them to rank the models based on their actual accuracy across a diverse spectrum of global cultures and languages.

The Results

The findings from the CulturalRiddles benchmark offer a sobering look at how far AI still has to go to achieve true cultural competency. While the models displayed high levels of general intelligence, their performance on the riddles revealed significant disparities in their ability to process context outside of their primary training distributions.

In the final ranking of the 28 models tested, Google’s Gemini 3.1 Pro Preview emerged at the top, earning an accuracy score of 79.4%. It was followed by OpenAI’s GPT-5.5, which achieved 76.3%, and Anthropic’s Claude Opus 4.7 Thinking, which came in at 66%.

While these scores might appear high in a vacuum, the researchers emphasize that they represent a best-case scenario under controlled testing conditions. In the real world, where AI models are used for everything from global communication to policy analysis, these gaps in cultural understanding can lead to significant errors in judgment, miscommunication, and the reinforcement of cultural biases.

The difficulty for the models—and the reason they often fail to identify when they are out of their depth—is that they are designed to provide an answer, not to question the premise of the prompt. When an AI encounters a query like the German 9/11 riddle, it is mathematically optimized to produce the most statistically probable response based on its training data. In most cases, the training data is heavily skewed toward American English and US-centric perspectives, leading the model to hallucinate or force an American answer into a context where it doesn’t belong.

This inherent, structural bias presents a major hurdle for developers. As AI continues to be integrated into global infrastructure, the inability of these systems to differentiate between regional date formats, historical contexts, or cultural metaphors is more than just a minor technical curiosity—it is a functional limitation that impacts the utility and fairness of the technology.

The researchers at Cohere hope that the CulturalRiddles benchmark will serve as a wake-up call for the broader AI community. By shifting the focus from simple fluency to a more nuanced, experience-based understanding of the world, they aim to push the industry toward models that are not only smarter but also more culturally aware. As development continues, the ability for an AI to pause, acknowledge its lack of local context, and handle "fuzzy" concepts with care will likely become just as important as its ability to generate perfect prose. The road ahead remains complex, but by prioritizing the inclusion of lived, global perspectives, the path toward a more culturally literate AI is beginning to take shape.