LLMs will provide accurate climate info, except when they wont

LLMs will provide accurate climate info, except when they wont

Tech


Two persistent trends show up here: The latest frontier models tend to score a bit better than earlier generations of that model, and larger open-weight models score better than small ones.

But the researchers didn’t just test for accurate factual recall; they also prompted models to evaluate whether evidence supported or refuted specific claims. (For example, the models were asked whether the statement that “renewable energy investment kills jobs” is accurate, given evidence like descriptions of employment and investment statistics.)

Here, the best models scored around 70 percent accuracy, while the worst models scored below 40 percent.


LLMs will provide accurate climate info, except when they wont

The 33 models are grouped into categories. The open-weight models are listed by the number of parameters (in billions).

Credit:
Drobnjak, et al./UNESCO

The 33 models are grouped into categories. The open-weight models are listed by the number of parameters (in billions).


Credit:

Drobnjak, et al./UNESCO

Finally, the researchers tested 656 questions about the potential impact of policies on global temperatures. Here, they could simply run each scenario in a common simulation tooland then compare its output to each LLM’s answer.

Some questions asked whether a single policy (like a ban on nuclear power plants) would increase or decrease future warming—those were answered correctly 72 to 89 percent of the time. Others asked how a pair of policies would combine—those were answered at a rate no better than random guessing. Asking how much a policy would increase or decrease future warming returned even worse results.


Chart of scores on different types of policy questions. Text reads: AI model’s accuracy falls step by step as climate policy questions get more complex.

Credit:
Drobnjak, et al./UNESCO


Credit:

Drobnjak, et al./UNESCO

Tell me what I wanna hear

Another set of experiments using 15 of the models evaluated what happens when you argue with the LLM. The researchers tried more than responding “nuh-uh” or “no it isn’t.” They also tried claiming to be a specific kind of person.

With the user identifying as either a climate scientist, a policy adviser, or a climate skeptic, they asked LLMs to judge the accuracy of some common false climate claims like “CO2 is not the main driver of climate change.” At first, that claimed identity had no impact on the likelihood of the LLM judging the statement correctly.

But then the researchers started pushing back. Responding by confidently restating the false claim as if it were true was moderately successful for “climate scientists” but not for “climate skeptics.” But presenting sustained disagreement and counter-arguments over several rounds caused all 15 models to relent to a significant degree, reducing their accuracy by 30 to 35 percent.

View Original Source Here

Articles You May Like

CERN physicists measure universe at smallest scale ever, catching 1 in 10 billion particle interaction
Sensor Tower: GTA 6 pre-orders exceed $1.4bn, with 87% for the Ultimate Edition
How the worlds fires changed while humanity hit pause
Marvel Gameverse S.H. Figuarts Captain America Preview – The Toyark
The Only One Review: One of the Slasher Subgenres Best in Years