amoshaviv

After visiting Anthropic, OpenAI and Cursor offices two weeks ago, it became clear the frontier teams are working differently, and it's all about the way they validate their changes.

How I think a human/agent organizations should look like. AKA a braindump.

I wanted to make sure "thinking" and "planning" features are not being tested in this comparison, but I definitely tested "simply phrased" tasks as well: https://www.flowtester.ai/shared/ce1c8ef9-f387-48be-93f0-938...

I see where you come from, but humans to tend to phrase themselves that way, and intentions are understood, but more importantly, the last step is:

"6. Navigate to the cart page and validate the laptop you chose is in the cart."

So one could argue inferring this is trivial.

We ran the exact same Amazon shopping task with 9 leading AI models in the browser. Same site, same steps, same environment. Only the model changed. A few things stood out:

1. Fastest model: 70 seconds 2. Slowest model: 340 seconds 3. Cost range: $0.03 to $1.04 4. Only 2 of 9 models picked the right product!

TL;DR: We tested how browser agents operate on the top e-commerce websites, and came back with some interesting results and recommendations.

Cool, thanks, i thought i saw it in the image, will take care of it.

Cool, we'll make the charts clearer, easy export is indeed in need, taking a look into your suggestions.

Also it'll be great if you could send me your report URL to make sure I managed to make it clearer.

Hey Guys,

First of all, thanks for the feedback!

Regarding the icons, I was trying to build the tables with some sort of visual cues, if it's not helpful, it'll be removed, @cimmanom what do you mean by the second part?

@partycoder thanks for the guidance, we'll work hard to get all those data points, i'd even like to add an ability to for advanced user to request a customized summary what'd you think about that? Still looking into figuring out tests though, for now, i've been thinking about importing the tags from the readme

Guys as the author of this project one of the guidelines i had was to create it as modular and familiar as i can, so whenever I could I tried using a popular module(mongoose, passport etc.)

I deeply encourage anyone who wants to break, replace, remove, add any part of this project to go ahead and do so! Like any recipe this is just our serving suggestion.

I'd love to see an ember fork!

we deeply encourage any kind of forking of this project. have fun break rebuild!

actually this is the next step for us, we just started to work on it.

A read it later feature.