nkov47

That’s exactly the failure mode we’re most worried about. A UI saying “success” is weak evidence that the underlying work is correct.

We’re using a layered approach rather than relying on the computer-use model alone. The execution agent handles navigation, but workflows can also define independent checks against source data and expected invariants,for example, matching patient IDs, totals, dates, procedure codes, or confirming that a record actually changed in the destination system. For higher-risk actions, we can pause before submission or route the completed output to human review.

We’re also trying to keep verification logically separate from execution where possible, so the same agent isn’t simply grading its own work. In some workflows that means deterministic rules; in others it may mean rereading the destination, comparing against the original document, or using a separate model/checker.

Your point about the rules handling truth is a useful way to frame it. I’d be curious how much of your checker is reusable across tax firms versus custom to each firm’s process.

Exactly. We’ve found that for real production workflows, “did the task finish?” isn’t enough. You need to know what the agent saw, why it acted, what changed, and where a human approved something consequential.

The checkpoints and event log are really about making failures inspectable instead of mysterious.

I mean we've spent a year of our life on this(ironing out infra for more than half of the time), so i wouldn't say super low quality but again YC accepted us based on how our business is growing! Thanks for the feedback though!

This is a great read of it, and honestly we probably did undersell that part.

The handoff works through checkpoints. The deterministic workflow defines the state it expects before resuming, and the agent’s recovery job is to get the UI back into one of those known states, not just “fix whatever happened” and keep going.

If it can reconcile back to a checkpoint, the script resumes deterministically from there. If it can’t, the run pauses rather than pretending it knows where it is.

That reconciliation layer is definitely where a lot of the hard engineering lives.

Yep, this is a real failure mode, and screenshot-after-action alone does not solve it. A field can look populated while the application never commits the underlying value.

We try to define verification around the actual outcome of the task rather than the immediately preceding UI state. Depending on the workflow, that can mean checking the confirmation page, reopening the submitted record, verifying a status change elsewhere in the application, matching a generated reference ID, or confirming that a downstream artifact was created.

We can also verify against separate inputs such as a PDF, CSV, or another screen for example, confirming that the values entered into a portal match the source document and then checking the resulting record after submission.

Today, our core agent is screen-driven, so we do not claim that pixels can prove every hidden state transition. For high-consequence commits, the workflow should use explicit downstream invariants and, where those are unavailable, a human approval or review step. Longer term, we think verification should be able to consume whatever independent evidence the environment exposes, including network or application-level signals, rather than treating vision as the only source of truth but it will be quite slower.

Thanks!!! completely agree that this is where trust is won or lost.

We support explicit human approval gates in workflows, so you can have the agent prepare everything, then pause before steps like submitting a form, sending a message, or confirming a payment. The workflow moves to `awaiting_human`, and the caller approves or rejects it before execution continues.

Autonomous runs can also pause and hand over the live machine when the agent encounters something requiring human input. For teams that want stricter control, our lower-level API exposes each predicted action so the caller can inspect or approve actions before executing them.

Today, the irreversible boundary is primarily defined by the developer rather than us trying to universally infer what is consequential. We think explicit policy is safer because “irreversible” varies a lot by application.

We've hit 82.8% on OSWorld Verfied(on the official page) and with our latest internal testing we're hitting 85.6% which we've posted (https://github.com/coasty-ai/coasty-osworld).

We're SOC2 and HIPAA compliant and have zero data retention policies through our enterprise platform where it's all covered and we don't keep any data that you don't want us to!

Yes, you have access to all your logs and screenshots and playback!

With our API, you can also create a very customizable blend of deterministic workflows and AI calls for CUA, so for example, Coasty for recovery can kick in when something unexpected like a dialog box or popup happens during your deterministic workflow runs.

We don't just provide end-to-end API, we're a SOTA harness where you can bring in your own OpenAI or Claude keys and run it on and also we're the only modular API in the market - we give you the ability to control any part of CUA and charge by the type of call as you can see in https://coasty.ai/docs.