kirtivr

Deepseek has always been better than people gave it credit for.

You have to be careful with the inference provider though, Chinese providers are subject to laws that mandate data sharing with their government.

One big reason I can think of is to avoid vendor lock-in. This is why most enterprises use a multi-cloud setup, despite the increased complexity.

Here is a counterexample to whatever counterexample you have in mind -

Most popular models on OpenRouter right now: https://openrouter.ai/models?categories=programming&order=mo...

Top 7 are all open models.

FixBugs, an agent that ingests the rich context surrounding production bugs to reproduce them in a sandbox and generate verified fixes. It's available in the form of a self-hosted VSCode extension and as a Github app.

VSCode Extension:

  https://fixbugs.ai/go/vscode-extension

  - full code and data privacy.
  - zero data retention models opted out of training.

  https://fixbugs.ai/go/github-app
  - we do access your code temporarily.
  - pick a repo to install FixBugs on.

  - Reproduce the bug.

  - Identify the root cause.

  - Generate a fix.

  - Verify the fix.

  - Review the generated code using multiple AI models to help catch potential regressions.

to be fair, coding agent harnesses have been becoming more and more complex.

it's not an llm in a loop with tools anymore (as claude code was rumoured to be on HN).

A few different ways:

- Copilot may be better at implementing features. We're better at investigating bugs and fixing them.

- We handle huge context very easily. We specialize towards investigating large amounts of logs/metrics and traces.

- We do a lot of work to generate a non-trivial reproduction test case in a sandbox, which allows us to verify bug fixes. Ship confidently, not just based on the best available hypothesis.

- You can apply and remove code changes with the click of a button. We have an in-built VCS that allows you to add or remove sets of changes from a long coding session.

- Our pricing model is different. While with the standard Copilot developer plan you get $10 of AI usage (nothing if you're using Claude or GPT), we allow you unlimited triages on a fixed number of bugs. It's not about AI credits, it's about investigating and fixing complex bugs.

This is on our roadmap!

At this time we are focussing on evaluation benchmarks like SWE-bench (verified). This is a simpler benchmark and does not really map well to investigating alerts that have a huge amount of context. But its a start.

I am wondering if we can improve upon foundation models with our reproduction <-> hypothesis loop approach.

Foundation models tend to be precision first, and context limited, so they can get sidetracked by various things.

This is an interesting problem to be working on right now!

Thank you!

are you using any AI tools to debug productions issues at this time?

Would love to have you join our Discord- https://discord.gg/XNXVD34P8

Hey satyamtiwary thanks so much for trying the product!

We'd love to have you join our Discord: https://discord.gg/XNXVD34P8

Thanks. As teams move faster with AI generated code, the focus on quality and reliability will have to increase as well.

And we will have to build better tools if we want to accelerate shipping velocity.

Thank you!

are you using any AI tools to debug productions bugs at this time?

Nginx, Caddy, Flink and Firefox are some applications where we we've managed to consistently fix and reproduce reported bugs.

Of course, we did not send the PRs to the repos seeing how they're already overloaded with them.

Nginx for example has 181 open PRs right now, but they only merge 2 or 3 in a day.

We either mock or fake out the interfaces.

I much prefer faking to mocking, because that still preserves a lot of the real-world behavior relevant to prod bugs.

A full prod reproduction would be a holy grail, but probably only attainable for complex distributed systems if we have access to a prebuilt staging-like environment.

We're traversing the bridge between mocks, fakes and staging at this time.

War story time? xD

On how does the mocking work, that's a really interesting question.

We do a lot of AST parsing - for both code and build configuration languages. Even then, we still have to rely on the LLM to figure out a lot of the details.

Making this work reliably for non-frontier models and codebases that don't have existing test harnesses is where a lot of the design work goes in.

It's really nice to hear that.

FixBugs was built because while investigating prod incidents, I had an epiphany.

SWEs build tools to solve all types of problems, but we ourselves use the flakiest tools.

While working at Google for example I was surprised GDB support for any type of binary debugging was almost non-existent.

On Linux, we rely on a chroot-ed workspace at this time - although we are working on a prototype using the new landlock kernel interface (https://github.com/Zouuup/landrun).

OSX is the best, we use the in-built (seatbelt) sandbox via sandbox-exec.

For Windows, we use WSL containers when available.

By default, if a safe sandbox environment is not available, we inform the user that a repro is not possible in the current conditions.

Thank you so much.

are you using any AI tools to debug productions issues at this time?

Thank you so much.

are you using any AI tools to debug productions issues at this time?

Oh the repro runs in an isolated sandbox, and all interactions outside the sandbox (with lets say other services or databases) are mocked. The repro harness doesn't have access outside of it.

This also allows us to inject various types of faults, which is helpful with debugging more complex systems.

Thanks a lot!

I think FixBugs is most useful during high volume bug triage. This is where having a low false-positive async debugging agent is most helpful.

A different approach that we took to root causing bugs that you may find interesting is that we first try to reproduce the bug before coming up with a fix for it.

This is essentially a (RCA <-> Repro test case) loop until we're recreated the bug. If our attempts are not converging and we’re on the wrong track, we ask for human input.

Yeah, the problem reduces to trying to restrict a motivated model which is trying to exfiltrate data.

That's a problem we are just now wrapping our minds around.

It's not as simple as prompt sanitization. The model is the interpreter, and we don't yet have the right tools to guide it.

TIL. Thanks for that!

Is this an admission that prompt injection attacks can indeed not be blocked by an analysis based technique?

If so many tools are straight up blocked, I would be very sceptical of the quality of the results.

C++ is one of the languages less suited to the strengths of coding agents.

The language which still supports C-style pointers, arbitrary datatype conversions, and inherits architecture-specific undefined behavior gives you too many ways to fail at solving a problem.

As a programmer, I love coding in C++ because I know what I'm doing. I'd hate reviewing C++ code though.

Yeah, when you had multiple agents working on the same machine, branch isolation was no longer sufficient.

A repository folder can only be on one branch at a time.

A worktree is basically equivalent to a cp -R + git branch, which allows this new workflow to occur.

I loved this particular historical insight as to why `git worktree` was added in 2015:

Before worktrees, kernel devs faced a major inconvenience when switching contexts, e.g., stopping feature work to fix an urgent bug on a release branch.

Running git stash and switching branches alters timestamps on thousands of files.

This forces `make` to perform a full re-compile, which can take up to an hour on large kernels.

Some super relevant benchmarks like Humanity's Last Exam, Long context reasoning (MRCR 128K-256K) are not included.

Overall this seems to be a strong agent-oriented model. What are the benchmarks that most closely track model coding performance in the real world?

They got 1K tok/s with Deepseek v4 Pro. That's kinda cool..

I can think of real time video, shader generation, real time worldbuilding type problems could require such a high token throughput.

For instant code generatio, 400-500 tok/s should be sufficient, though most frontier models give us closer to 70 tok/s.