Is it the demonstrations, or the model?
Ling, C., Zhao, X., Zhang, X., Cheng, W., Liu, Y., Sun, Y., Oishi, M., Osaki, T., Matsuda, K., Ji, J., Bai, G., Zhao, L., & Chen, H. (2024). Uncertainty Quantification for In-Context Learning of Large Language Models. NAACL 2024.
Every standard uncertainty tool for LLMs — predictive entropy, ensembling, MC dropout — was built assuming randomness comes from two places: the model’s parameters, and noise in generation. In-context learning sneaks in a third source none of those tools were designed to see: the prompt itself is a choice. Swap which demonstrations you pick, how many, or their order, and predictions shift on a task that hasn’t changed at all. That’s not the model being uncertain. That’s the prompt construction doing something.
Splitting the blame
Ling et al. split an ICL prediction’s total predictive uncertainty into two pieces: an aleatoric term from randomness in which demonstrations got picked, and an epistemic term for whatever uncertainty is left once you’ve accounted for that. It’s the same law-of-total-variance move Kendall and Gal use for vision:
\[\text{Var}[\text{total}] = \underbrace{\text{Var}[\text{demonstration-set choice}]}_{\text{aleatoric}} + \underbrace{\mathbb{E}[\text{residual variance} \mid \text{fixed demo set}]}_{\text{epistemic}}\]The estimator has to work without ever touching a gradient, since ICL has no training step to draw posterior samples from. So it resamples instead: draw many demonstration sets from the labeled pool for a task, run the model on each resulting prompt, and use how much the outputs spread across those sets as the aleatoric term, with within-prompt predictive entropy standing in for the epistemic term.
What it buys you
Across multiple LLMs and classification/QA benchmarks, the decomposed estimate correlates with actual correctness and calibration better than undecomposed baselines like plain entropy on one fixed prompt. And the aleatoric piece behaves exactly as advertised: it tracks how much accuracy moves when you swap the demonstration set for a fixed model and task, confirming that a real chunk of what looks like “the model is uncertain” is actually “the prompt happened to be constructed this way.”
The catch is how it gets there. Every input to the estimator is something the model output under a resampled prompt — never an internal activation. That makes it usable on any black-box model, but it also means uncertainty is only ever visible indirectly, through how much the final answer wobbles when you perturb the input, and getting a reliable estimate takes many forward passes per query.
| This paper | Reading the internals directly | |
|---|---|---|
| Model access | Black-box, outputs only | White-box, activations |
| Cost per query | Many forward passes | Can be closer to one |
| What it measures | Output movement under perturbation | The task representation itself |
This is the closest prior work to what we ended up building, and reading it closely is what made the next question obvious: if function vectors give you a fairly direct handle on the model’s internal read of the task, could you get the same aleatoric/epistemic split from that, instead of from watching the output wobble under a hundred resampled prompts?