Confabulation is a real term, borrowed from psychology and neurology. It describes what happens when someone — often a patient with a memory disorder — fills a gap in their recollection with a fabricated detail, and states it with total sincerity. Not a lie. Lying requires knowing you’re wrong. Confabulation is something else: the part of the brain that would normally flag “I’m not actually sure about this” just doesn’t fire. The person believes it completely, and it shows.
That’s not a psychology term I picked up reading a journal. That’s what Llama did to me on a Tuesday night.
I’d started a 14-day plan to get hands-on with local AI — Ollama, Open WebUI, LM Studio, running models on my own machine instead of in the cloud. Day 3 was supposed to be simple: a model accuracy check. I gave the same mortgage question — calculating qualifying overtime income — to three local models and compared the answers.
Qwen punted. Gave me a vague non-answer instead of real methodology, which is at least honest about its limits.
Mistral got close. It used a straightforward average, directionally reasonable, though not quite how Fannie Mae actually wants it split.
Llama was the one that got me. In a fabricated problem it created to show the calculation, it walked through the math step by step, clean and confident, like a worked example in a textbook: base salary $6,000/month, 20 overtime hours a week, times 4 weeks — 80 monthly overtime hours. Then it divided those 80 hours by 12 months and called the result $6,667 a year, roughly $556 a month, in overtime income. Read that again slowly. It divided a number of hours by a number of months and reported the answer in dollars. The units don’t mean anything. But it was formatted so cleanly, and stated so plainly, that you’d have to already know the math to catch that it wasn’t math at all — just numbers arranged to look like math.
That was the lesson from Day 3: the most fluent-sounding model was the most dangerously wrong one. Confidence and correctness turned out to have nothing to do with each other.
I should have carried that lesson forward more carefully than I did, because a few days later it came back around in a form that actually mattered.
I was trying to build a small script to extract data from a W-2 — one of my own, an old one, the kind of document that should never leave my machine regardless of whose it is. I asked a local model to write the extraction code. It handed me something that looked completely legitimate: clean Python, sensible structure, a request to an API endpoint. I went to run it in PowerShell before I’d even loaded a real document, just to see if it would execute.
It failed immediately — a DNS error, couldn’t resolve the address. Good thing, because when I looked at what it was actually trying to call, the model had invented an external API out of thin air. api.inferencehub.io. Doesn’t exist. The model needed a plausible-looking endpoint to complete the script, so it manufactured one — with the same total confidence it brought to the overtime math. No hedge, no “I’m not sure this service exists.” Just a URL, formatted like fact.
Nothing was exposed. The script died before it ever sent anything, and I hadn’t fed it a real document yet anyway. But if I’d been one step further along — document loaded, script looking clean, no reason yet to doubt it — that’s sensitive financial data routed to a server nobody owns, built and run by me, on my own machine, because I trusted output that sounded right.
Two different failure shapes. One invented a number, one invented a destination. Same root cause: a local model will hand you something fluent and complete, and fluent has nothing to do with true. It doesn’t feel uncertain internally, so it doesn’t perform uncertainty externally. It just answers.
The fix wasn’t “get a smarter model,” though that’s the instinct.
It was tightening what I asked for. Once I scoped the overtime question to the actual method — Fannie Mae’s B3-3.1 guidance, specifically — results got better across the board. Precision in the ask bought me precision in the answer. It didn’t buy me a model that knows when to doubt itself. That part’s still on me.
So that’s the boundary I run now, plainly: sensitive documents — W-2s, tax returns, paystubs, bank statements, always my own old ones when I’m testing something new — stay local, and I check the actual output before I trust it, every time. Reasoning, drafting, the stuff where I want a second set of eyes — that goes to Claude. Not because local models are bad. Because they’ll confabulate with a straight face, and so, it turns out, will I, if I don’t stay honest about what “looks right” actually means.