AST-Driven Python Testing

https://jazzberry.ai/blog/ast-driven-python-testing

by MarcoDewey • 12 months ago

10 2 12 months ago

Jtsummers• 12 months ago

What's the argument for having the LLM generate 20 fuzz tests instead of using a fuzzer that will likely be more comprehensive? Especially paired with grey-box and white-box techniques. It'll be cheaper and faster to execute.

MarcoDewey• 12 months ago

you are right that a traditional fuzzer (especially a grey-box one like AFL or a white-box one) is superior in speed, cost per execution, and comprehensiveness.

The argument for using an LLM to generate a curated set of fuzz inputs isn't to replace traditional fuzzers, but to complement them by targeting a different class of bugs that traditional fuzzers are often poor at finding.

The goal of this tool is two fold.

1. give LLMs the ability to make use of traditional software testing tools

2. enhance some of the shortcomings in traditional software testing tools by selectively using LLMs (specifically their ability to understand the larger context the code is written in)

AST-Driven Python Testing

Related Stories