012026
LLM tooling
LLM Reasoning Debugger
A local web app that opens up a language model's reasoning token by token — confidence, entropy, attention weights, and hallucination risk in a terminal-style interface.
- Served Qwen2.5-1.5B-Instruct behind a FastAPI backend, capturing per-token logits and attention.
- Built a 28-question benchmark and scored it with AUROC and expected calibration error.
- Found the model's stated confidence was badly calibrated (~32% ECE) against its own token probabilities (~15% ECE).
- ~32%
- ECE, stated confidence
- ~15%
- ECE, token probability
- Python
- FastAPI
- Qwen2.5-1.5B