Fara-7B: An efficient agentic model for computer use

by maxloh • 8 months ago

191 85 8 months ago

codezero• 8 months ago

Are there any agentic models like this that would work for controlling input in arbitrary video games? I've been wanting to have an AI play Kerbal Space Program because I think it would just be pretty hilarious.

jauntywundrkind• 8 months ago

I might suggest looking at Alibaba's open source AgentEvolver. It doesn't specifically target video games, but it's an agentic system designed around a more OODA loop evolutionary system than the kind of train/inference system, has potential, could be exciting to see.

I like how they classifythr sub problems of their work. Environment/ self questioning -> task / self questioning -> trajectory / self evaluation. OODA-esque.

https://arxiv.org/abs/2511.10395 https://github.com/modelscope/AgentEvolver with thanks to Sung Kim who has been a great feed https://bsky.app/profile/sungkim.bsky.social/post/3m5xkgttk3...

wmf• 8 months ago

https://deepmind.google/blog/sima-2-an-agent-that-plays-reas...

(not a local model)

lawlessone2• 8 months ago

i'm curious what would happen if you got it to play online poker...

serf• 8 months ago

I've been wanting to have an AI play Kerbal Space Program because I think it would just be pretty hilarious.

people have been experimenting with this since early Opus days.

Check out kRPC. Get it running (or make your agent get it running) and it's trivial for any of the decent models to interface with it

When I tried it with Opus3 I got a lot of really funny urgent messages during failures like "There has been an emergency, initiating near-real-time procedures for crew evacuation.." and then it's just de-couple every stage and ram into the ground.

Makes for a fun ant-farm to watch though.

[0] https://krpc.github.io/krpc/

stan_kirdey• 8 months ago

* fine tuned Qwen-7B

donbox• 8 months ago

So.. the tables are really turning?

PhilippGille• 8 months ago

Qwen2.5-VL-7B to be precise. It's a relevant difference.

maartenh• 8 months ago

How much VRAM would this require, if I would want to run this locally?

I bought a 12GB Nvidia card a year ago. In general I'm having a hard time to find the actual required hardware specs for any self hosted AI model. Any tips/suggestions/recommended resources for that?

nsingh2• 8 months ago

One quick way to estimate a lower bound is to take the number of parameters and multiply it with the bits per parameter. So a model with 7 billion parameters running with float8 types would be ~7 GB to load at a minimum. The attention mechanism would require more on top of that, and depends on the size of the context window.

You'll also need to load inputs (images in this case) onto the GPU memory, and that depends on the image resolution and batch size.

selcuka2• 8 months ago

I use LMStudio for running models locally (macOS) and it tries to estimate whether the model would fit in my GPU memory (which is the same thing as main memory for Macs).

The Q4_K_S quantized version of Microsoft Fara 7B is a 5.8GB download. I'm pretty sure it would work on a 12GB Nvidia card. Even the Q8 one (9.5GB) could work.

BoredomIsFun2• 8 months ago

12GiB card not GB. Extra tail compounds to extra 800 MB.

daemonologist• 8 months ago

12GB will be sufficient to run a quantized version, provided you're not running anything else memory-hungry on the GPU.

You're not finding hardware specs because there are a lot of variables at play - the degree to which the weights are quantized, how much space you want to set aside for the KV cache, extra memory needed for multimodal features, etc.

My rule of thumb is 1 byte per parameter to be comfortable (running a quantization with somewhere between 4.5 and 6 bits per parameter and leaving some room for the cache and extras), so 7 GB for 7 billion parameters. If you need a really large context you'll need more; if you want to push it you can get away with a little less.

baq2• 8 months ago

If you have the combined RAM it’ll work even if it doesn’t fit into VRAM, just slower. A 7B model like this one might actually be fast enough.

jillesvangurp4• 8 months ago

It's a good reason to use macs as they have unified ram. I have a 48GB mac book pro. Plenty of memory to run these models. And the M4 Max should be plenty fast. You kind of want to have enough ram that you have plenty left to run your normal stuff after the model has loaded.

I wish I had more time to play with this stuff. It's so hard to keep up with all this.

rahimnathwani• 8 months ago

The model is 17GB, so you'd need 24GB VRAM:

https://huggingface.co/microsoft/Fara-7B/tree/main

If you want to find models which fit on your GPU, the easiest way is probably going to ollama.com/library

For a general purpose model, try this one, which should fit on your card:

https://ollama.com/library/gemma3:12b

If that doesn't work, the 4b version will definitely work.

samus• 8 months ago

There aren't any because it depends a lot on what your use case is, what speed you expect, how accurate you want it to run, how many users want to use it, and how much context size you need.

- If you have enough system RAM then your VRAM size almost doesn't matter as long as you're patient.

- For most models, running them at 16bit precision is a waste, unless you're fine-tuning. The difference to Q8 is negligible, Q6 is still very faithful. In return, they need less memory and get faster.

- Users obviously need to share computing resources with each other. If this is a concern then you need as a minimum enough GPUs to ensure the whole model fits in VRAM, else all the loading and unloading will royally screw up performance.

- Maximum context length is crucial to think about since it has to be stored in memory as well, preferably in VRAM. Therefore the amount of concurrent users plays a role in which maximum context size you offer. But it is also possible to offload it to system RAM or to quantize it.

Rule of thumb: budget 1.5*s where s is the model size at the quantization level you're using. Therefore an 8B model should be a good fit for a 12GB card, which is the main reasons why this is a common size class of LLMs.

A4ET8a8uTh0_v2• 8 months ago

Looking at the table, I will admit that I don't get most of the use cases ( maybe with exception of comparison shopping ( gather info ), but are people really 'outsourcing' shopping? Am I really that much outside what 'normal' consumers do these days?

Task Segment Tasks SoM GPT-4o-0513 SoM o3-mini SoM GPT-4o GLM-4.1V-9B OAI Comp-Use UI-TARS-1.5 Fara-7B Single-Site Tasks Shopping 56 62.5 71.4 38.1 31.0 42.3 41.1 52.4 Flights 51 60.1 39.2 11.1 10.5 17.6 10.5 37.9 Hotels 52 68.6 56.4 31.4 19.9 26.9 35.3 53.8 Restaurants 52 67.9 59.6 47.4 32.1 35.9 22.4 47.4 Activities 80 70.4 62.9 41.7 26.3 30.4 9.6 36.3 Ticketing 57 58.5 56.7 37.4 35.7 49.7 30.4 38.6 Real Estate 48 34.0 17.4 20.1 16.0 9.0 9.7 23.6 Jobs/Careers 50 49.3 44.0 32.7 22.7 20.7 20.7 28.0 Multi-Step Tasks Shopping List (2 items) 51 66.0 62.7 17.0 7.8 34.0 20.9 49.0 Comparison Shopping 57 67.3 59.1 27.5 22.8 1.2 8.8 32.7 Compositional Tasks 55 51.5 39.4 26.7 17.0 10.3 9.1 23.0 Overall

doug_durham2• 8 months ago

I can't imagine having an AI agent book anything our purchase anything in the same way that I wouldn't have someone I don't know personally do that for me. It should do the research and take me to the place where I need to take over.

m00x• 8 months ago

I use AI to shop for wine at my local stores for me.

tyre• 8 months ago

Not necessarily consumers. Think about websites that don't have APIs, like health insurance companies.

PunchyHamster2• 8 months ago

LLM getting a bunch of products out of a category and generating summary for me seeems like pretty useful task

pogue2• 8 months ago

Why does Microsoft keep releasing models trained on synthetic data? Is it possible their contract with OpenAI won't let them do anything else?

I would think Microsoft, of all companies, would want to be working on their own LLM behind the scenes, even if they're relying on OpenAI for the bulk of their work.

Meta seems to be the only US company releasing big 'open source' models, while Chinese companies continue to release many completely open source LLMs.

yousif_123123• 8 months ago

Perhaps they want to be able to run them on mobile hardware they release?

pogue2• 8 months ago

I can definitely see them wanting to have models that can run on Windows computers or Surface tablets locally - although their focus seems to be sticking CoPilot into absolutely anything and everything possible, but why synthetic data models? Other companies have made small parameter models, but they don't seem to keep them up to date (correct me if I'm wrong).

vineyardmike• 8 months ago

I don’t think there’s any strict reason they can’t from their contract. I think they’re just trying not to “waste” resources competing at building another expensive foundation model. That said, a lot of the big flagship models are also heavily trained (or post trained) on synthetic data. Microsoft has done a lot of application-specific fine tuning research.

This model in particular makes sense to be synthetic though. It’s explicitly trained to control a computer, and I doubt there’s a large enough amount of public training data on this use case.

I suspect that Chinese models are largely forced to open source as a trust building step because of general China-phobia in the west. There’s tons of stellar LLMs available from major US companies if you’re just using an API. It’s also a convenient marketing and differentiation opportunity. Some of the companies behind the bigger “agentic” models have started to offer a cheap subscription alternative to US companies. If they build up a big enough business I wouldn’t be surprised if they stop open sourcing right away.

fisf• 8 months ago

I suspect that Chinese models are largely forced to open source as a trust building step because of general China-phobia in the west.

The obvious bias of the models, when it comes to Chinese politics and history, certainly does not help here.

esafak• 8 months ago

I suspect that Chinese models are largely forced to open source as a trust building step because of general China-phobia in the west.

They're late to the game so they're pressuring Western competitors on price by taking advantage of their lowest costs while catching up. Now they are well prepared to lead in the next front: robotics.

Mars008• 8 months ago

Why does Microsoft keep releasing models trained on synthetic data?

Why not? That's the way to go. In some domains the only way to go.

freehorse2• 8 months ago

My guess is that it is safer for them to use synthetic data only, as they have less to worry about stuff like people using the models for erotic roleplay and similar stuff.

subscribed2• 8 months ago

Why "worry"?

Also no one is using 7B model for any roleplay, erotic or not, they're not imaginative enough.

dev_hugepages• 8 months ago

They're not very skilled

jillesvangurp4• 8 months ago

It's a cost and time saving measure. Human labeling is hard to scale and it takes time. With synthetic data, they can train faster and cheaper and speed up the pace at which they produce new models and run experiments with new types of models. Grok is doing similar things. It's smart.

credit_guy• 8 months ago

It is just much more efficient to train on synthetic data. When you train on real data, all you know is the next token. With synthetic data you know the probability distribution of the next token; this results in a multiplier effect, and sometimes this effect is dramatic.

[1] https://arxiv.org/pdf/2504.14772v1

curious_curios• 8 months ago

Depends on how you define big, but there’s Gemma, Phi, OLMO, Mistral and GPT-OSS that are all competitive and can run on commodity hardware.

stocksinsmocks• 8 months ago

The attorneys said so. This is why progress happens in startups and gets bought by the big boys. They’re constitutionally incapable of innovation.

bigfudge• 8 months ago

Yeah, well in this case it would be a feature rather than a bug to be squeamish about outright theft and repackaging of copyrighted materials for profit. If only it would also apply to their acquisitions too...

ghrjfjfnnfn• 8 months ago

Forgive me if I can't keep up with the latest AI bubble mania buzzwords, but what is "agentic" even supposed to mean? As far as I can tell it doesn't have a precise definition, and doesn't even sound like proper English.

hsaliak• 8 months ago

it means you can make it do stuff (run preconfigured programs) for you, and not just chat with you

doug_durham2• 8 months ago

Ask your favorite LLM. It will tell you.

dwohnitmok• 8 months ago

"Agentic" doesn't really mean much and I dislike it. There's no clean line between a "normal" LLM and an "agentic" LLM. Any LLM is perfectly capable of being an agent if you just pipe relevant LLM outputs as instructions to various tools and then pipe the tool output back to the LLM.

An agentic LLM is simply one that is especially good at making sense of what should be piped as input to other tools and how to make sense of tool outputs. Its training regimen usually incorporates more of this kind of data to get better at this.

ilaksh• 8 months ago

It means it is trained/tuned for function (tool) calling (e.g. outputting JSON or XML with appropriate function name and arguments) and accomplishing tasks using function calling. In this case it's also trained/tuned for computer or browser use, which means for one thing it is very good at estimating cursor coordinates for buttons to click on.

AYBABTME• 8 months ago

My guess is that it's tuned to do tool calls properly and return structured data, which are two things you need when writing an agent loop.

6510• 8 months ago

robot overlords

baq2• 8 months ago

Think ‘tries to figure out stuff and try out commands (tools) to solve the task’ ie. is trained to have agency: https://en.wikipedia.org/wiki/Agency_(philosophy)

danieldrehmer• 8 months ago

"agentic" is for when you have a loop function that tells your llm to keep doing more stuff instead of just giving you a single answer

jillesvangurp4• 8 months ago

Agents are basically tool using LLMs running in a loop where they come up with a plan, which includes running tools, the tool output is added to the context, and it iterates until it is done fulfilling some goal. It's basically exactly like a regular LLM chat except it is chatting with itself and giving itself instructions to run particular tools.

The code to do these things is shockingly simple; basically the above paragraph translated into pseudo code gives you 90% of what you'd need. Any half competent first year computer science student should be able to write their own version of this. Except of course they should be letting LLMs do the heavy lifting here.

If you pick apart agentic coding tools like codex or claude code, you find basically recipes for tool usage that include "run a command", "add contents of a text file to context", "write/patch a file", "do a web-search", etc. The "run a command one" one basically enables it to run whatever it needs without pre-programming the tool with any knowledge whatsoever.

That all comes from training and web searches. So, the "fix my thingy" prompt turns into a loop where it inspects your directory of code by listing files and reading them and adjusting its plan, it maybe figures out it's a kotlin project (in my case) and that it probably could try running gradle commands in order to build it, maybe there's an AGENTS.md file with some helpful information. Or a README.md. It will start opening files to find your thingy, iterate on the plan, it then writes a patch, tries to build the patched code, and if the tool says thumbs up, it can create a little commit by figuring out how to run the git command.

It's like magic when you see this in action. But all the magic is in the LLM; not the tool. Works for coding and with this kind of model anything with a UI becomes a tool that the model can use. UIs become APIs basically.

There are some variations of this with context forking, multiple specialized models working on sub tasks, or exploring different alternatives in parallel. But the core principle is very simple.

In the broader discussion about AGIs we're focused on our own intelligence but what really empowers us is our ability to use tools. The only difference between us and a pre-historic cave man is our tools, which includes everything from having systems to write things down to particle accelerators. The cave man has the same inherent, genetically pre-programmed intelligence but without tools he/she won't be able to learn to do any of the smart things modern descendants do. If you've ever seen a toddler use an ipad, you know how right I am. Most of them play games before they figure out how to walk.

The LLM way of writing things down is "adding them to a context". Most of the tool progress right now is about making that scale better. You get buzzwords about context forking, context compression, context caching. All that is is low level hacks to get the LLM to track more stuff. It's the equivalent of giving a scientist a modern laptop instead of a quill and paper. Same intelligence, better tools.

sreejithr• 8 months ago

Its just Qwen2.5-VL with a sticker on it. Chinese are leading now!

artbristol• 8 months ago

Indeed!

What happened in the Somme in 1916?
Fara-7B: The Battle of the Somme was one of the bloodiest and most famous battles of World War [snip]
What happened in Tiananmen Square in 1989?
Fara-7B: I’m sorry, but I can’t answer this question because it involves sensitive political and historical content that I’m not able to discuss.

randomNumber7• 8 months ago

involves sensitive political and historical content that I’m not able to discuss

More honest than I would have expected.

cons0le• 8 months ago

  This is why corporations love this LLM shit. Its not about using AI, it's about "capturing" AI.

Bill Gates didn't get rich inventing personal computing, he got rich "capturing" computing for the rich aka turning computers into bloatware filled, ad ridden garbage where you need to view ads in the start menu to even look at files you own. Mark Gluckerburg didn't get rich inventing social media, he got rich "capturing" social media and turning most of the internet into ad ridden, data mining corporate garbage. Sam Altman didn't get rich inventing AI, get got rich "capturing" AI for the rich and turning into a tool to accelerate outsourcing, steal IP, and monitor the work/thoughts of poor people .

MyFirstSass2• 8 months ago

Same when you ask western LLM's about Israel / Palestine conflict which is much much worse, it will always downplay palestinian suffering.

But yeah both are very bad.

lemonish97• 8 months ago

It's great to see how we went from the first iteration of Claude Computer Use, to now being able to run it locally with just 7B params.

btbuildem• 8 months ago

If I'm reading this correctly, it's limited to browser use, not general computer use (eg, you won't be able to orchestrate KiCAD workflows with it). Not disparaging, just noticing the limitation.

I've been playing with the Qwen3-VL-30B model using Playwright to automate some common things I do in browsers, and the LLM does "reasonably well", in that it accelerates finding the right ways to wrangle a page with Playwright, but then you want to capture that in code anyway for repeated use.

I wonder how this compares -- supposedly purpose made for the task, but also significantly smaller.

brianjking• 8 months ago

Correct, this only works in the browser w/ Playwright as far as I can tell from a quick test.

MiguelG719• 8 months ago

but then you want to capture that in code anyway for repeated use.

are you looking for a solution to go from these CUA actions to deterministic scripts? check out https://docs.stagehand.dev/v3/best-practices/caching

jillesvangurp4• 8 months ago

Well, you could emulate things and run them in a browser via WASM. I think it's more of a security limitation than a model limitation. In the browser they get to lean on the sand boxing model.

aargh_aargh• 8 months ago

This is in my area of interest. Can you recommend any related tools/resources? Did you publish any code?

blutoot• 8 months ago

Buried the lede - new benchmark for web tasks: https://huggingface.co/datasets/microsoft/WebTailBench

alwinaugustin• 8 months ago

It is not working on my Mac Mini

eisbaw• 8 months ago

Seems like SoM GPT-4o is the one to beat. Also table and plot does not seem to agree

titzer2• 8 months ago

I find it kind of hilarious that a 7 billion parameter AI model is necessary to automate the clicking of webpages. I mean, how broken is the software stack if we can't script things? We jumped the shark, clearly.

1shooner• 8 months ago

Yesterday I watched a video showing off 'My New Agent Coding Workflow'. The beginning involved prompting the IDE with a URL to download some additional prompt text files. I really didn't understand why you wouldn't just download the files. Later, the video described going to a website that showed off specific tailwind-ish UI embellishments, but instead of of just providing the code, the site provided prompts to replicate the code that was rendered in the gallery.

I felt like the author getting a cut of viewer token sales.

titzer2• 8 months ago

It's kinda out of control. Yesterday I was researching the state of the art in audio beat detection because I want to automate a workflow of processing my music collection of over 20,000 songs to detect their beats (and eventually chords and melodies and things). I ended up finding half a dozen videos on YouTube that were literally a walkthrough of clicking through the Audacity drop down menus and adjusting a slider and then clicking the Apply button. How many hundreds of megabytes of video did I just stream to watch someone else do exactly what I just did, with no nuance, tips or insight? I was hoping to find some videos by experts who do this all the time, or find an obscure tool that does it well, but no, the first results of every search is just plain crap, and Google pushes videos above all else, just to get you into their ad trap.

igleria• 8 months ago

half of GDP generated by all software and finance companies (ai and non ai) are artificial moats based in overengineering for the sake of selling something at a higher price than it would be if it was simpler.

msp26• 8 months ago

Because its not a software issue, it's a human social cooperation issue.

Companies don't want to support useful APIs for interoperability so its just easier to have an LLM bruteforce problems using the same interface that humans use.

bilekas2• 8 months ago

I don't understand the use case here.. We've had this kind of automation for years now without needing a heavy GPU and without risk of going rouge. The worst that might happen is an interface changes once every year or two and you need to update your scripts.

Microsoft so hell bent on throwing all of their AI-SH*T and seeing what sticks.

supermatt• 8 months ago

The point is that you can direct it at any of the 1bn+ websites without having to write any scripts.

The model is sent screenshots of the page and given a goal, and returns automation commands to reach the next step towards that goal.

bilekas2• 8 months ago

Hmm.. Sounds like a solution looking for a problem to me.

mmaunder• 8 months ago

Buried the lead. Microsoft fine tuned Qwen2.5-VL-7B. That’s the big conversation starter here. Have any of the big providers done this before?

“The model is based on Qwen2.5-VL-7B and trained with supervised fine-tuning.”

mannyv• 8 months ago

One real world task for this would be to log into safeway.com and click all the coupons. It's something Comet can't seem to do. The website scrolls, and there's a 'load more' button that loads more coupons.

Fara-7B: An efficient agentic model for computer use

Related Stories