I’ve spent 16 years in mortgage operations. W-2s, paystubs, bank statements — I don’t have to think about what I’m looking at anymore, I just know. So when it came time to point AI at something closer to the actual work, this is where I started: teach a pipeline to do the part of the job that’s pure pattern recognition, and see how far that actually gets you.
Two weekends, three sessions. A Make.com scenario watches a folder, sends each new document to Claude, gets back a document type and the key fields as structured data, and files it — or holds it back for a person to look at if it’s not confident. Just a note: every document in my test set is fabricated top to bottom. Names, numbers, employers, all of it made up.
I ran 20 documents through the finished pipeline — five each of W-2, paystub, bank statement, and non-mortgage junk, with some deliberately broken: a bad scan here, a smudged field there. I wanted a test that could actually fail. A test that can’t fail doesn’t tell you anything.
It didn’t fail. Classification: 20 for 20. Field extraction: 85 for 85 gradable fields, including three documents I built specifically to have unreadable data — and in every one of those, the pipeline came back with a null instead of making something up. That mattered more to me than the perfect score. A system that admits it doesn’t know something is more trustworthy than one that’s always confident.
The perfect score comes with an asterisk, though. My “degraded” documents were still clean, machine-readable text — the scan problems were described as literal text markers, not the actual visual noise of a real scanner or a phone photo in bad light. This hasn’t been tested against what a document actually looks like coming in rough. That’s the honest next test, not something this run already answered.
I also found something I didn’t expect. The confidence score didn’t track difficulty the way I assumed it would. Documents with real problems — missing data, damaged sections — still scored 0.95 confidence in most cases, same as a clean document. Only one of my broken test docs actually tripped the review threshold. If I were running this for real, I wouldn’t trust that number to tell me something’s wrong. That’s a finding, not a footnote.
And there was a plain build bug. Partway through scoring, the extracted field data — employer names, wage figures, account balances — wasn’t showing up in my log. No errors anywhere. Everything upstream looked fine. Turned out I’d wired the field mappings into the wrong branch of the pipeline — the low-confidence path, not the one my real test documents were actually flowing through. Nothing broke. I’d just built the right thing in the wrong place. Running clean and running correctly aren’t the same thing. You have to check what came out the other end, not just whether anything threw an error.
While I was writing this up, I had “build an income calculator” listed as the next step — pull the wage and YTD data this pipeline already extracts, project a monthly income figure, route it for sign-off. Then Make.com shipped a new AI feature, and I used it to build that exact tool in a few seconds. It writes to a second tab in my log and annualizes income straight off the YTD numbers.
I want to be straight about what that is and isn’t. The math is deliberately simple — a straight YTD annualization, nothing more — because the point wasn’t to nail underwriting-grade income calculation in one sitting. The point was proving the capability exists end to end: extracted data flowing into a calculation, landing somewhere useful, with no new pipeline built by hand. The real logic underwriting needs — averaging periods, handling variable or overtime pay, knowing when a YTD figure is a fair basis to annualize — is a rules problem, not an AI problem. The ceiling here was never the tool. It’s the formulas I tell it to run.
I didn’t learn that AI can replace document review in mortgage lending. I learned something narrower: a well-scoped piece of that problem is buildable in a weekend, testable honestly, and — when the eval comes back at 100% — worth being suspicious of your own test set instead of just taking the win.