MarcoDewey

you are right that a traditional fuzzer (especially a grey-box one like AFL or a white-box one) is superior in speed, cost per execution, and comprehensiveness.

The argument for using an LLM to generate a curated set of fuzz inputs isn't to replace traditional fuzzers, but to complement them by targeting a different class of bugs that traditional fuzzers are often poor at finding.

The goal of this tool is two fold.

1. give LLMs the ability to make use of traditional software testing tools

2. enhance some of the shortcomings in traditional software testing tools by selectively using LLMs (specifically their ability to understand the larger context the code is written in)

Finally, someone who gets that SSR is overkill. I just want a simple static documentation page.

I think that diffusion models might be a better way to generate code than autoregressive left-to-right generation

The middle class is dead. It is almost impossible to buy a home and raise a family in your 20s now.

Here is a list of a few tools that do something like that: jazzberry ai, greptile, coderabbit, codeant (lots of code), cubic, qodo.

GitLab may also have a code review tool available now (I know that GitHub does)

Yes, we have caught some navigation bugs on the front end.

There are some other really cool tools built explicitly for this, like QualGent and Operative.sh

We are working daily on making Jazzberry better for larger teams and repositories. Our goal is to find deep, difficult bugs in large repos

I also wish that I had more real world experience. It would help me a ton if I had 25 years of software testing experience.

It sounds like you do have experience, and I would love to learn from you. It would be awesome if you could help us build a tool that is truly useful for you and your work.

That is one of the obvious use cases. There are many others, you are welcome to install the bot and play around with it. I would love to hear your feedback.

The bugs shown in the "real bugs" section are real output from the tool. Are you referring to looking at the full table of bugs that we return? Sometimes we only find one bug in the PR, sometimes our clients don't want us to share other bugs that could expose their work.

Correct, what is unique about LLMs is their ability to match an existing tool or practice to a unique problem.

I definitely agree that there's a lot of research happening in this space, and the false positive issue is a significant hurdle. From my own research and experimentation, I have also seen how challenging it is to get LLM-powered tools to consistently find real.

Our approach with Jazzberry is specifically focused on the dynamic execution aspect within the PR context. I am seeing that by actually running the code with the specific changes, we can get a clearer signal about functional errors. We're very aware of the need to demonstrate our ability to find those high-severity/exploitable bugs you mentioned, and that's a key metric for us as we continue to develop it.

Given your background, I'd be really interested to hear if you have any thoughts on what approaches you think might be most promising for moving beyond the false positive problem in AI-driven bug finding. Any insights from your work at MIT would be incredibly valuable.

You are right that static linters are incredibly fast and efficient for catching certain classes of issues.

Our focus with the dynamic sandbox execution is aimed at finding bugs that are much harder for static analysis to detect. These are bugs like logical flaws in specific execution paths and unexpected interactions between code changes.

What if - some of us - just like to use excessive - dashes when writing -- ?

What are some of your best prompting practices to get AI to write reliable code?

I have been experimenting with forcing the AI to write the test cases first and then build software that passes all those tests.

I love that you all are doing real old school machine learning and not just LLM transformer based work!

I believe that the unprecedented scale of LLM-generated code will demand a novel approach to software review and testing. Human review may not be able to keep up (or will it become the bottleneck?)

I think this is a seriously excellent point.

The bet that I am making is that the system reduces its error rate by splitting a broad task into two more focused tasks.

However, it is possible that generating meaningful test cases is a harder problem (with a higher error rate) than producing code. If this is the case, then this idea I am presenting would compound the error rate.

improving code generation would be awesome :)

I believe that I have unintentionally misled you. When I say "unbiased oracle" I am talking specifically about the test oracle being unbiased by how the software was implemented. ie. Black Box testing.

I don't think I made the point very clear in the blog (I will rectify that), but I am saying that because LLMs are so easily biased by their prompting that they sometimes perform better when doing black box testing tasks than they do when performing white box testing.

Yes, this test-driven approach will likely increase generation time upfront. However, the payoff is more reliable code being generated. This will lead to less debugging and fewer reprompts overall, which saves time in the long run.

Also agree on the specification formality. Even a less formal spec provides a clearer boundary for the LLM during code generation, which should improve code generation results.

You are correct that the notion of LLMs being completely unbiased or neutral does not make sense due to how they are trained. Perhaps my title is even misleading if taken at face value.

When I talk about "unbiased oracles" I am speaking in the context of black box testing. I'm not suggesting they are free from all forms of bias. Instead, the key distinction I'm trying to draw is their lack of implementation-level bias towards the specific code they are testing.