HN user

jonasnelle

231 karma
Posts3
Comments64
View on HN

Yes, Autotab can reason over the state of applications and the data it is seeing. You can also teach it to do certain steps only in specific cases.

If you wanted Autotab to reconcile payments you would teach it to go to wherever the payments are listed eg a banking app. There you would have it iterate through the unreconciled payments. For each payment you’d have Autotab go to the invoicing tool and look up any details from the payment (eg IBAN, information from the reference number, amount, etc) to find the matching customer and invoice. This is where most of the reasoning happens - you can teach Autotab what counts as sufficiently close to be a match with prompts and examples. Then you can have Autotab mark the invoice as paid and go back to the payment app and mark the payment with the invoice number it grabbed from the matched payment.

If by pixel level you mean vision-first understanding and control of the UI then you’ve misunderstood my comment - Autotab primarily uses vision to reason about screens and take action.

You can also use Anthropic’s Computer Use model directly in Autotab via the instruct feature - our users find it most helpful for handling specific subtasks that are complex to spell out, like picking a date in a calendar.

Quoting my cofounder from another thread:

For 2FA, different users take different approaches. Everything from teaching Autotab to pull auth codes from their email, to setting intervention requests at the top of their skills, to enterprise integrations that we support with SSO and dedicated machine accounts.

Autotab also has the ability to securely sync session data from your local app to cloud instances. This usually removes the need for doing 2FA again for sites with “remember this device” functionality.

We can enable captcha solving for select customers, but don’t allow that in the public app to prevent abuse.

Assuming the bank’s websites look totally different from one another, you’d need open ended exploration to data extraction. We’ve focused more on reliability for repetitive tasks over flexibility for open ended tasks historically, but models are getting good enough that this tradeoff is diminishing. Expect updates from us on this front soon.

You can schedule skills in Autotab to run at arbitrary frequency.

Indeed! A little secret: Internally we call the skills/workflows in Autotab macros :)

Currently there is a bit of a learning curve for training Autotab to be really reliable in hard cases. We expect we’ll be able to decrease significantly in the next few months, as we get models to do more of the thinking about how to best codify a given task solution/workflow. As an intuition pump for why we expect such rapid progress: in the scenario you described you’d just have a model write the VBA code for you.

Thanks! Our most common use cases are repetitive tasks people have at work, think updating Hubspot with analytics data from an internal tool or reconciling payments between an invoicing system, a payment system and a CRM.

Haven’t done a lot with Scribe-like documentation cases. Given the pace at which this technology is developing we’re focused on making Autotab really good at the most economically valuable tasks.

Depends on how different the flows are for different restaurants. If they're just different names but use the same booking system you'd typically use an input and have Autotab find the correct restaurant first. If they're totally different booking systems you can try the instruct (open ended agentic) step but my guess is that will be too slow and unreliable for now, so you'd probably want to record different skills for each.

Docs are here with sample code: https://docs.autotab.com/api-reference

Thanks! We think of the browser as an authoring tool where you create, test and refine skills.

After you've done that, the API is great for cases where you want to incorporate Autotab into a larger data flow or product.

For instance, say Company A has taught Autotab to migrate their customers' data - so their customers just see a sync button in the Company A product, which kicks off a Autotab run via API. Same for restaurant booking, if you'd want that to happen programatically.

Data from external systems can be provided to Autotab in the form of CSV files or string inputs, which can be passed to the API to parametrize skills. However, in most cases, ingesting data into Autotab is easiest by just having Autotab navigate to the website where the data is present.

Autotab has a structured type system underlying the workflows, so any data processed in the course of an automation can be referenced in later steps. It's a bit like a fuzzy programming language for automation, and the model generates schemas to ensure data flows reliably through the series of steps.

For example, users often start by collecting information in one system (using an extract step as you mentioned), then cross reference it in another and then submit some data by having Autotab type it into a third system. In Autotab, you can just type @ to reference a variable, each step has access to data from previous steps.

At the end, you can get a dump of all of Autotab's data from a run as a JSON file, or turn specific arrays of data into CSV files using a table step.

Autotab is definitely a hybrid approach, because when it comes to deciding where on the page to take an action, Autotab has to be fast & cheap (humans are both of those) while also being robust to changes. The solution we use is a "ladder of compute" where Autotab uses everything from really fast heuristics and local models up to the biggest frontier models, depending on how difficult the task is.

For instance, if Autotab is trying to click the "submit" button on a sparse page that looks like previous versions of that page, that click might take a few hundred milliseconds. But if the page is very noisy, and Autotab has to scroll, and the button says "next" on it because the flow has an additional step added to it, Autotab will probably escalate to a bigger model to help it find the right answer with enough certainty to proceed.

There is a certain cutoff in that hierarchy of compute that we decided to call "self-healing" because latency is high enough that we wanted to let users know it might take a bit longer for Autotab to proceed to the next step.

I meant that nobody feels like they "have to" come in in the sense that everybody wants to come in on Saturdays.

I don't feel like I "have to" work 6 days a week - I do it because I want to. You may say that this is just because I'm a cofounder, but that's not true - I did the same thing at past startups where I was an early hire, but not a cofounder. Because I enjoyed the work, found it meaningful and loved learning. This is the kind of person we're looking for.

If you just want a job that pays the bills, I respect your decision, but you should not apply.

The hardest and most important problems won't be solved working 9-5, 5 days a week.

Nobody feels like they "have to" come in on Saturdays - we find purpose in our work, and see it as the main way we can have a meaningful impact. We enjoy working, learning and spending time together.

We would much rather spend 70 hours a week doing great work than clocking in for 40 hours a week.

Plus we do take Sundays off.

Autotab (YC S23) | Founding Engineer | NYC | 6 days a week in the office

If AGI is going to 100x GDP, it will need to be able to use a computer.

Autotab is the first digital robot that controls a computer like a human. It excels at the highly repetitive tasks that make up the worst parts of modern knowledge work. If we succeed, we will provide the platform that unlocks the next 100x productivity increase, and frees humans from trillions of hours of soul-crushing work.

We are second time founders backed by OpenAI and YCombinator. We are live in production and are working with some of the biggest companies you use every day to handle critical tasks that humans used to do. We’re rapidly scaling up to meet demand, and building a 5-person founding team where everyone builds product and talks to users.

If you are technical and have founded or want to start a company we should talk!

More info: https://www.ycombinator.com/companies/autotab/jobs/V5V8saO-f...