OmniParser for Pure Vision Based GUI Agent

https://microsoft.github.io/OmniParser/

by fzliu • 2 years ago

145 34 2 years ago

microsoft.github.io

I have a little bit of a vice of enjoying some "idle" games. I have intended to do some very basic manual screen carving & ocr & computer vision to try to "read" my state in these games, & have multi-actor "play" models for them, just for fun really & to decrease time sunk gaming (by spending significant time coding/learning).

This certainly seems like it has a lot of promise to make that much much much easier. Game UI's are less uniform so maybe this might be harder or not easily be applicable, but hopefully

_adamb• 2 years ago

As someone who has done this to many games over a few decades, I can definitively say: 100% of the time, it ruins the fun of the game.

I can't say exactly why. Maybe you feel like you haven't earned it. Maybe it's the idle nature of farming that we really enjoy...

fragmede• 2 years ago

Depends what you consider fun, and how far you take it. Some people enjoy programming more than repetitive clicking in a GUI. For a clicker game, writing a bot lets you iterate on strategies easier - is it faster to get to level 2 if I buy the upgrade for A or B first? For Trackmania, it lets you get a world record and a YouTube video with 14M views.

https://youtu.be/Dw3BZ6O_8LY

nmstoker5• 2 years ago

You might want to look at Serpent AI: granted the repo is now in an archived state, but it did similar things to those you mention.

https://github.com/SerpentAI/SerpentAI

trq_• 2 years ago

This is awesome, can't wait for evals against Claude Computer Use!

amelius4• 2 years ago

Can we first test this with basic sysadmin work in a simple shell?

Can't wait to replace "apt get install" by "gpt get install" and then have it solve all the dependency errors by itself.

anonym29• 2 years ago

Threat actors can't wait for you to start doing this either.

ErikBjare• 2 years ago

This had been possible for a year already. My project gptme does it just fine (like many other tools), especially now with Claude 3.5.

asdev• 2 years ago

how can you write metrics against something that's non deterministic?

Smaug123• 2 years ago

To a considerable extent, we are stuck in the world we live in; but I am reminded of a quote by Guillaume Allais:

My entire job seems to be repeating variations of "never start by forgetting the user's stated intent only to then attempt to guess it".

akshayKMR• 2 years ago

Does it also tell the coordinates (x,y) of the annotated box w.r.t. the screenshot dimensions?

s3tt3mbr1n1• 2 years ago

Has anyone gotten this to work?

Copying the repo and downloading the models through HuggingFace or manually does not seem to work, you get errors indicating missing files.

suriya-ganesh• 2 years ago

I tried as well. Seems like it is a proprietary model.

dartos2• 2 years ago

FOSS-washing?

bbor• 2 years ago

:( I was so, so excited to try this when I found it yesterday, it has like 3 star emojis in my list of models. I’ll post here if I get it working tomorrow, I guess. I doubt they’d release a model on HF without intending to make it useable

EDIT; surely it’s just broken, the repo does include .safetensor weights. Maybe the problem is the “suspicious”-flagged PyTorch weight for “icon detection”, whatever that means?

amelius4• 2 years ago

Can it detect ads and mask them out?

bee_rider• 2 years ago

If these sorts of tools kill the ad business, it would be so incredibly cool, and justify Nvidia’s half-of-the-economy-or-whatever market cap.

dartos2• 2 years ago

Only like 6%

amelius4• 2 years ago

Let's hope so! But now that I'm thinking about it more: nvidia might go into the advertisement business themselves :(

dymk• 2 years ago

I’m reminded of Permutation City where your personal AI intercepts ads sent to you, but ad companies of course have their own AI for tricking your AI, so of course you have a countermeasure AI to intercept that, and so on and so forth

patrickhogan12• 2 years ago

Since this is a research paper with promising ideas but non-functional code, what are people using as the best-in-class agents for computer automation? For example:

1. Claude for computer use

2. Various startup offerings—if you have recommendations, please list them

3. Established tools like Playwright, Selenium, and WebDriver, combined with screenshots and LLM-based guidance

What tools or approaches are actually working for building useful automation solutions?

nmstoker5• 2 years ago

Are you sure about the non-working code point?

I've yet to try it but my understanding is the repo here has got working code along with installation instructions:

https://github.com/microsoft/OmniParser

nmstoker5• 2 years ago

I confirm it works: I got the gradio demo working locally and it's pretty reasonable.

Slight rough edges (to be expected) and you do need to read the README with attention but it's all par for the course. I had to install einops which wasn't in the requirements.txt and even though I had downloaded the HF models they released, it still needed to pull in another model when I first ran the demo.

patrickhogan12• 2 years ago

Thanks for the tip, will try again.

tomatohs• 2 years ago

our agent is available via NPM: http://testdriver.ai

anonzzzies• 2 years ago

Computer Use, Agent.exe and so on, but nothing actually is useful yet. It's all very terrible. And then to think we had perfection already (and Claude is good at it); emacs... No need for any of this; everything can be scripted.

deergomoo• 2 years ago

One thing I have found very odd about the current wave of AI tools is that there seems to be an unspoken element of giving up and admitting failure in other areas of computing.

Programming copilots are often sold on how they can automate drudgery and boilerplate, which implies we are incapable of or uninterested in designing programming languages, tools, and patterns which do not require boilerplate or drudgery.

Teaching models to use traditional GUI apps implies we have given up on or are not even bothering to create proper hooks for an automation system to utilise.

Something about it feels wrong to me, because it bakes existing inefficiencies into the system. Can we really not solve the inefficiencies instead of pouring unfathomable amounts of compute into working around them?

Fulgen• 2 years ago

Focusing on the GUI applications: There have been a few GUI automation solutions over the years - since the post's software is from MS, I'll take UI Automation as an example. Works well with Win32 controls, not sure how well it works with the XAML-based toolkits.

But not all software is written with those UI frameworks. Some use different widget frameworks, some immediate GUIs, others just render a webpage and either use HTML or fully render the controls themselves. And without everybody using the same standard, the only standard we have for parsing their output is the pixels they render to.

mountainriver• 2 years ago

Computer based agents have no limits, that’s the advantage. Sure a proper automation hook is better if its available, but a lot of the time it isn’t, either due to lack of resources or monopolistic behavior

franga2000• 2 years ago

This is not a computer problem, it's a human one. It's not that we don't have APIs and hooks because they're so difficult to implement - we don't have them because software producers don't want or care for us us to have them.

Enabling automation will never be zero effort and anything more than zero effort for something with such a low ROI is a no-go by default. But increasingly, automation is actually seen as a danger to their business models and companies sometimes even go out of their way to prevent it.

Looking at the screen the same way a user does is the only way to win.

OmniParser for Pure Vision Based GUI Agent

Related Stories