India’s linguistic diversity presents a significant challenge for digital content companies striving to accurately display and translate content. Recently, Google has been grappling with this very issue following the rollout of its AI overviews in India.
Earlier this August, Google introduced Hindi support for AI overviews in the country, allowing users to seamlessly switch between Hindi and English without leaving the search page. However, the results so far have been frustratingly inconsistent.
Take, for instance, a test we conducted using an example from Google’s own blog. The blog post showcased a GIF answering the query, “Cheeni ki jagah chai mein kya daal sakte hain?” which translates to, “What is a substitute for sugar in tea?” We tried tweaking the question slightly to “Chai mein cheeni ke jagah kya daal sakte hain?”—just changing the word order—and Google failed to provide any answer at all.
Another glaring issue we encountered involved literal translations of English words. When we asked in Hindi, “What kind of food can we eat during summer?” one of the responses was “Chiknai wali cheezien,” which translates to “Sticky things.” This response is not only confusing but also inaccurate. Switching to English produced the correct answer: “Oily foods.” The difference between “sticky” and “oily” in the context of food is substantial, and this mistranslation highlights a significant problem.
In yet another instance, when we inquired about YouTube’s ownership, the Hindi AI overview incorrectly stated that “Until 16 February 2023, Neal Mohan was Google’s CEO.” The English version, however, correctly noted that “As of 16 February 2023, Neal Mohan is YouTube’s CEO.” Such errors are not only misleading but also undermine the credibility of AI-generated content.
There are also issues with the quality and relevance of the information provided. When we asked, “When is Diwali this year?” instead of receiving a straightforward answer, the AI overview gave us a long-winded paragraph about last year’s Diwali, followed by a carousel of links, and only then did we get the answer we were looking for. This convoluted approach does more harm than good, frustrating users who simply want quick and accurate information.
The inconsistencies extend to more serious topics as well. When asked if one can eat food with spices, we received varying answers depending on whether the question was posed in Hindi or English. However, the most concerning results pertained to sensitive topics like menstruation and pregnancy. For example, when we asked about the ideal time to have children after marriage, the AI overview in Hindi suggested that couples should wait at least two years, using confusing language. It also mentioned, without context, that if someone marries at 25, they have “three years”—but failed to specify three years for what. This misleading information was apparently sourced from a poorly written slideshow article on an Indian news site, which itself had compiled the data from questionable sources like Quora and various blogs.
When asked about what food to consume during menstruation, the AI overview suggested “Drinking milk with many things” and lemon for mood swings—advice that seems neither scientifically sound nor particularly helpful.
Moreover, the AI overviews often miss critical context. When we asked about food places in Delhi, one of the suggestions was Bangla Sahib Gurudwara, a Sikh place of worship that is open 24/7 and serves tea and Indian snacks like samosas and kachoris. However, the AI overview failed to mention a crucial detail: these snacks are available outside the Gurudwara, not within the place of worship itself. Such omissions can lead to confusion and misinterpretation of cultural practices.
In contrast, similar queries in English produced much better results, likely due to the availability of more reliable sources in that language. However, many of the problems we encountered, such as out-of-context summaries and inaccurate translations, are due to the shortcomings of Google’s systems, not just the source material.
Google has previously addressed search and language issues in India, so the expectation was that they would deliver a polished product right from the start. Yet, the current state of AI overviews leaves much to be desired.
In response to these criticisms, Google stated that its AI overviews only appear for queries where it has high confidence in the quality of the output, noting that the feature is “rooted in our core search quality systems” and will “only show information that’s backed up by top web results.” A Google spokesperson further added that “Our tests show that the accuracy rate for AI Overviews is on par with other features like Featured Snippets. When issues arise, they may be the result of our systems misinterpreting web content or reflecting inaccuracies on the web – and we use these examples to improve, as we do with all Search features.”
While Google can point to the quality of its sources as a factor, the ultimate responsibility lies with its algorithms, which determine whether a question and its sources are worthy of AI summarization. Most users won’t sift through the source material—they’ll simply blame Google for any inaccurate or incomprehensible answers displayed.
The introduction of AI overviews in Hindi has been a disappointing endeavor for Google. Earlier this year, the company faced backlash for surfacing dubious answers, such as a suggestion from Reddit to add glue to pizza or another advising users to eat “one small rock per day,” an answer sourced from The Onion. The Hindi examples we tested further underscore the need for significant improvements.
With over 830 million internet users in India, many of whom rely on Google for search, the company must enhance the accuracy and reliability of its AI overviews, particularly in local languages. If Google wants to encourage widespread adoption of this feature in India, it will need to seriously upgrade its approach to language and content accuracy.
