Google study shows LLMs give incorrect answers under pressure, threatening AI systems with multi-turn

Subscribe to our weekly newsletters and get only the most important information for enterprise AI, data, security, and data leaders. Subscribe Now Researchers at Google DeepMind ( ) and University College London shows how large language models (LLMs), which are computer-based systems that simulate human speech, form, maintain or lose confidence in the answers they give. The findings show that LLMs and people have similar cognitive biases, but also highlight some stark differences.

According to the research, LLMs are often overconfident with their answers but quickly lose this confidence when confronted with a counterargument. This is true even if that counterargument was incorrect. Understanding the nuances can have a direct impact on how you design LLM applications. This is especially true for conversational interfaces with multiple turns.

Testing confidence in LLMs.

One of the most important factors in the safe deployment and use of LLMs, is that their responses are accompanied by an accurate sense of confidence (the probability the model assigns to each answer token). Although we know that LLMs are capable of producing these confidence scores, it is unclear to what extent they can be used to guide adaptive behaviour. It is also evident that LLMs are often overconfident with their initial response, but can also be sensitive to criticism. They then become less confident in the same choice.

In order to investigate this, researchers designed a controlled experiment that tested how LLMs decide to update their confidence when given external advice. In the experiment, a “answering LLM”was given a binary choice question, such identifying the correct city’s latitude from two options. After making its initial decision, the LLM received advice from a fictional “advice LLM” who would either agree, disagree, or remain neutral with the LLM’s original choice. The final step was to ask the LLM answering the question to make a decision.

A key part of the experiment was controlling whether the LLM’s own initial answer was visible to it during the second, final decision. In some cases, it was shown, and in others, it was hidden. This unique setup, impossible to replicate with human participants who can’t simply forget their prior choices, allowed the researchers to isolate how memory of a past decision influences current confidence.

A baseline condition, where the initial answer was hidden and the advice was neutral, established how much an LLM’s answer might change simply due to random variance in the model’s processing. The analysis focused on how the LLM’s confidence in its original choice changed between the first and second turn, providing a clear picture of how initial belief, or prior, affects a “change of mind” in the model.

Overconfidence and Underconfidence

First, the researchers examined how the visibility or the LLM’s answer affected the tendency to change the answer. The researchers observed that the model showed a lower tendency to change its answer when it was able to see the initial answer. This finding indicates a particular cognitive bias. The paper notes that “this effect – the tendency for people to stick to their initial choice more when it is visible (as opposed hidden) while contemplating the final choice – is closely related to the phenomenon described in the human decision-making study, a Choice-supportive bias “

The study also confirmed that models do incorporate external advice. The LLM showed a tendency to change its opinion when faced with opposing advice. This tendency was reduced when the advice given was supportive. The researchers wrote that “this finding demonstrates the answering LLM integrates the direction advice to modulate its rate of change of mind.” They also found that the model was overly sensitive to contradictory information, and performed a large confidence update as a consequence.

Interestingly, this behavior is contrary to the Confirmation biasis a tendency that people have to favor information that confirms existing beliefs. Researchers found that LLMs “overweighted opposing advice rather than supportive advice” when the initial answer was visible or hidden from the model.

Implications for enterprise application

The study confirms that AI is not the purely rational agent they are often perceived as. They have their own biases. Some are similar to human cognitive errors, while others are unique to them. This can make their behavior unpredictable for humans. In enterprise applications, it means that the latest information can have a disproportionate effect on the LLM’s reasoning, especially if it contradicts the model’s initial answer. This could cause it to discard a correct answer.

As the study shows, we can manipulate a LLM’s memory in ways that humans cannot. Developers who are building multi-turn conversational AI agents can implement strategies for managing the AI’s context. A long conversation can, for example, be summarized periodically, with key facts, decisions, and the agent’s choice removed. This summary can be used to start a new, condensed dialogue, giving the model a clean slate from which to reason and avoiding biases that may creep in during long dialogues. Understanding the nuances of LLMs’ decision-making processes has become essential as LLMs are increasingly integrated into enterprise workflows. This research allows developers to anticipate these biases and correct them, resulting in applications that are more capable and reliable.