madikz

Founder building document-automation tools for tax firms here — the hardest part is definitely the verification step. For us it's not whether the agent can navigate a UI, but whether the output it produced (a reconciliation report, a tax form) is actually correct when every screen says "success."

We ended up building a secondary rule-based checker that runs independently over the agent's output — it catches things like missing schedules, transposed account numbers, and date mismatches with prior-year filings. The agents handle the execution path, the rules handle the truth.

Curious how you're thinking about this for workflows where a wrong but "successful" action has real downstream cost — do you rely on human review, or do you have a layered verification approach built-in?

Great observation on the "commits that don't commit" failure mode. We ran into the exact same class of bug building document extraction for regulated financial workflows — turns out vision-only verification has a blind spot that shows up whether you're parsing a screen or a PDF.

In tax preparation, documents have a lot of "the screen lies" equivalents: a PDF renderer can display perfectly aligned columns while the underlying text layer has misaligned fields. We learned this the hard way when an OCR pipeline extracted "123 Main St" beautifully from a W-2 image, but the XML metadata had a completely different address. The screen looked right, the data was wrong.

Our approach was a second verification layer structurally independent from extraction: deterministic rules that reconcile AI output against the source document's known structure. For a W-2, that means cross-walking every box number against the IRS's published schema and flagging deviations — same principle as your "downstream tells" but formalized into a rules engine.

The key insight: in regulated domains, the verification layer must consume something structurally different from what extraction consumed. If both use the same pixel-based representation, they share the same failure modes. Using DOM state, API call logs, or document metadata schemas as the verification source catches the cases where "pixel says done, payload says null."

Defining explicit "verification invariants" (similar to Coasty's approach) has been the most practical defense — things like "total across all line items must equal reported total" or "taxpayer name must match across all documents in the return." Simple arithmetic checks that don't need AI but catch the expensive failure modes that vision alone misses.

Curious how Coasty handles multi-source reconciliation — e.g., when the data on screen needs to be verified against a separate document (like a PDF guidance doc) rather than just against itself?

Perplexity Is a Bullshit Machine here is the link: https://www.wired.com/story/perplexity-is-a-bullshit-machine...

very good!