sanketsaurav

FWIW Claude Code Opus 4.5 ranks ~71% accuracy on the OpenSSF CVE Benchmark that we ran against DeepSource (https://deepsource.com/benchmarks).

We have a different approach, in that we're using SAST as a fast first pass on the code (also helps ground the agent, more effective than just asking the model to "act like a security researcher"). Then, we're using pre-computer static analysis artifacts about the code (like data flow graphs, control flow graphs, dependency graphs, taint sources/sinks) as "data sources" accessible to the agent when the LLM review kicks in. As a result, we're seeing higher accuracy than others.

Haven't gotten access to this new feature yet, but when we do we'd update our benchmarks.

$8/100k tokens strikes me as potentially a TON

It's $8/100K lines of code. Since we're using a mix of models across our main agent and sub-agents, this normalizes our cost.

I could easily see hitting 10k+ LOC on routine tickets if this is being run on each checkpoint. I have some tickets that require moving some files around, am I being charged on LOC for those files? Deleted files? Newly created test files that have 1k+ lines?

We basically look at the files changed that need to be reviewed + the additional context that is required to make a decision for the review (which is cached internally, so you'd not be double-charged).

That said, we're of course open to revising the pricing based on feedback. But if it's helpful, when we ran the benchmarks on 165 pull requests [1], the cost was as follows:

- Autofix Bot: $21.24 - Claude Code: $48.86 - Cursor Bugbot: $40/mo (with a limit of 200 PRs per month)

We have several optimization ideas in mind, and we expect pricing to become more affordable in the future.

[1] https://github.com/ossf-cve-benchmark/ossf-cve-benchmark

We haven't included Gemini Code Assist or Gemini CLI's code review mode in our benchmarks[1] (we should do that), but functionally, it'll do the same thing as any other AI reviewer. Our differentiator is that since we're using static analysis for grounding, you'll see more issues with lower false positives.

We also do secrets detection out of the box, and OSS scanning is coming soon.

[1] https://autofix.bot/benchmarks/

One of the main benefits of Semgrep is its unified DSL that works across all supported languages.

People can disagree, but I'm not sure that tree-sitter S-expressions as an upgrade over a DSL.

100% agree — a DSL is a better user experience for sure. But this is a deliberate choice we made of not inventing a new DSL and using tree-sitter natively. We've directly addressed this and agree that the S-expressions are gnarly; but we're optimizing for a scenario that you wouldn't need to write this by hand anyway.

It's a trade-off. We don't want to spend time inventing a DSL and port every language's idiosyncrasies to that DSL — we'd rather improve our runtime and add support for things that other tools don't support, or support only on a paid tier (like cross-file analysis — which you can do on Globstar today).

Thanks!

I'd love to hear how this project differs from Bearer, which is also written in Go and based on tree-sitter? https://github.com/Bearer/bearer

The primary difference is that we're optimizing for users to write their custom rules easily. We do plan to ship built-in checkers [1] so we cover at least OWASP Top 10 across all major programming languages. We're also truly open-source using the MIT license.

Regardless, considering there is a large existing open-source collection of Semgrep rules, is there a way they can be adapted or transpiled to tree-sitter S-expressions so that they may be reused with Globstar?

I'm pretty sure there should be a way to make that work. We believe writing checkers (and having a long list of built-in checkers) will be a commodity in a world where AI can generate S-expressions (or tree-sitter node queries in Go) for any language with very high accuracy (which is where we have an advantage as compared to tools that use a custom DSL). To that extent, we're focused on improving the runtime itself so we can support complex use cases from our YAML and Go interfaces. If the community can help us port rules from other sources to our built-in checkers, we'd love that!

[1] https://github.com/DeepSourceCorp/globstar/pulls

Thanks! We still have a long way to go and a pretty extensive roadmap.

Is there a general way to apply/remove/act on taint in Go checkers? I may not be digging deeply enough but it seems like the example just uses some `unsafeVars` map that is made with a magic `isUserInputSource` method. It's hard for me to immediately tell what the capabilities there are, I bet I'm missing a bit.

Assuming you're looking at the guide [1], the `isUserInputSource` is just a partial example and not a magic method (we probably should have used a better example there).

The AST for each node along with the context are exposed in the `analysis.Pass` object [2]. We don't have an example for taint analysis, but here's an example [3] of state tracking that can be used to achieve this. This is a little tedious at the moment and you'll have to do the heavy-lifting in the Go code — but this is on our roadmap to improve. We want to expose a lot more helpers to make doing things like taint analysis easily.

Here's another idea [4] we're exploring to make the YAML interface more powerful: adding support for utilities (like entropy calculation) that you can call and perform a comparison.

[1] https://globstar.dev/guides/writing-go-checker#_1-complex-pa...

[2] https://globstar.dev/reference/checker-go#analysis-function

[3] https://globstar.dev/reference/checker-go#state-tracking

[4] https://github.com/DeepSourceCorp/globstar/issues/27

Not at the moment, but we'll put something up soon.

We're focused on keeping globstar light-weight, so a hosted runtime is not in the roadmap (although we'll add support for running Globstar checkers natively on our commercial product DeepSource). You should be able to write any checkers in Globstar that you can write in the other tools you've listed.

Our goal is to make it very easy to write these checkers — so we'd be optimizing the runtime and our Go API for that.

Supporting C / C++ is in our roadmap. It needs some additional work to handle preprocessor directives [1] [2], which is why we didn't focus on it for the initial release.

[1] https://github.com/tree-sitter/tree-sitter-c/issues/13

[2] https://github.com/tree-sitter/tree-sitter-c/issues/108

Not yet, but this is in our roadmap: https://github.com/DeepSourceCorp/globstar/issues/135

We're planning to implement a `skipcq` mute word.

Congrats on the launch! I tried this on a migration project I'm working on (which involves a lot of rote refactoring) and it worked very well. I think you've nailed the ergonomics for terminal-based operations on the codebase.

I've been using Zed editor as my primary workhorse, and I can see codebuff as a helper CLI when I need to work. I'm not sure if a CLI-only interface outside my editor is the right UX for me to generate/edit code — but this is perfect for refactors.

This is impressive! We use Metabase and I've been wanting this exact user experience for quite some time. So far, I've been dumping our Postgres schema into a Claude project and asking it to generate queries. This works surprisingly well, save for the tedious copy-paste between the two tabs. The Chrome extension workflow makes perfect sense.

Is there a way to select which model is being used? Anecdotally, I've found that Claude 3.5 Sonnet works incredibly well with even the most complex queries in one shot, which is not something I've seen with GPT-4o.

aside: if you're on Arc browser, I made a boost that adds some styling to Dan's website to make the reading experience just a little better: https://arc.net/boost/80CE9A49-4D0A-48C6-9C53-13BF02696009

There's a lot of value in making information buried in videos accessible in an organization. What tools like Notion did to documentation, I think there's a big opportunity to do that for videos. Congratulations on the launch!

This is very useful! Thanks for building this.

I found a minor bug though: the `credentials not found` error message doesn't go away after you've put in the correct credentials.

Including free storage in the free plan is a good strategy. Aside from small projects, I bet it'll make it much easier for people looking to host their production workloads to try before buy. Love this!

The new website and branding looks great. Congratulations!

This is a great product critique. I went through a similar thing with HEY — subscribed in curiosity (and awe of Jason's product demo skills, also the Twitter hype in part). I cancelled my subscription after 2 months because of similar reasons mentioned in this post and went back to Gmail.

ShellCheck is a great tool! If you're looking for an easy way to run static analysis on .sh files continuously, you might want to look at DeepSource. ShellCheck is supported. https://deepsource.io/gh/deepsourcelabs/demo-shell/issues/

I've been excited about Mighty ever since it was first announced — I was the ideal customer: I use all the apps mentioned on the landing page all day long and was super pissed at how slow everything was on my MacBook Pro.

And then I upgraded to the new M1 MacBook Pro. It's been a week, and this one's so smooth I can never go back to my old computer. I just realized I get most of the advantages mentioned on Mighty's landing page (more tabs, fast performance, no fan noise) already. I don't think I need anything beyond my local Chrome.

Question to suhail: Do you think people who are on M1 (and in future, those who're on more advanced Apple Silicon) are your target customers? Is there a benchmark for Mighty's performance vs M1s?

Thanks for your feedback. I've updated the blog to make the product plug non-intrusive.

A few differentiators:

* More issue coverage — for Python, we detect 520+ issues. We also enable you to run things like type checking (if you're using type hints) just by enabling it in the config.

* Custom issues — we have an analyzer team that keeps adding new, novel checkers to the analyzer for common bugs and anti-patterns.

* Fewer false positives — we've optimized our analyzers for reporting less than 5% false positives. On the lowest level, we write augmentations to each checker to remove known false-positives and noise. On the application level, we enable users to very easily ignore issues (for a file, all test files, some file patterns), and also report a false positive. We monitor all false-positive reports and proactively improve our analyzers to resolve them.

* Autofix — we just released this, which allows you to automatically fix some commonly occurring issues directly from DeepSource. In future, we will add more autofixers for issues, so at least 70% issues that we detect can be reliably autofixed.

For our analyzers, we actually do use existing static analysis behind the scenes in addition to our custom checkers that we write by hand. So our Ruby analyzer, which is in beta at the moment, does use Rubocop behind the scenes. We’re working towards the stable release of Ruby analyzer which uses augmentations to remove false positives and decrease the noise — since guaranteeing less than 5% false positives is one of the primary values that DeepSource adds. As the anlayzer moves towards stable, we'll add custom issues to it.

The general categorization of anti-patterns is based on the consensus of the community around the language, and also some obvious things based on objective reasons. Although we understand that everyone has their own flavor of conventions — so it is very easy to triage and ignore specific issues in DeepSource.

Ruby is already in beta, stable release in the next 3-4 weeks. Next up is JavaScript. Rust, Java, and PHP are further down the line.

A few issues from our Python analyzer:

* Dangerous mutable default argument passed in functions

``` def some_func(arg=[1,2,3]):

    ...

which should be

``` def some_func(arg=None):

    if arg is None: arg = [1,2,3]
    ...

* `yield` used inside a comprehension (which breaks code in Python 3.8)

* file opened with the "r" flag, but a write is attempted on the file

* i/o detected on a closed file descriptor

* providing an unexpected keyword argument in a function call

Author here. Great point! Would love to take a shot at writing a follow-up focused on this.

Hi, author of the post here.

Thank you for your feedback. The framework that you put down, of code reviews being of these 2 levels, is valid. The goal of this blog post, however, was to touch on the basic hygiene which makes the process more effective -- including both these two levels of review.

I would take a shot at writing a follow-up post to this, going deeper into the subjective aspects of peer code review. Thanks for reading!

YMMV, but when I'm working on a feature branch that can take some time, I try to break it into pieces to merge into master often. One way to think about this is breaking away from the compulsion to commit the entire thing together in a branch. So I have `feat-x-part-1`, `feat-x-part-2` and so on.

It's important to resist the urge to do push things only on perfection. It works for me.

If you look at it, the main problem here seems to be that the way evaluation essentially works (even on the first level of the funnel) has not evolved over the years. People ask algorithm-based questions for assessing developers because there's simply no other tool that allows you to evaluate on these actionable skills. For example, how do you evaluate someone on their understanding of JavaScript's prototypical inheritance? Sure, a text I/O based problem (which HackerRank and others) use cannot help you do this. So you resort to MCQs. Asking a JavaScript guy to solve a problem using dynamic programming simply doesn't make sense.

Technical evaluation needs to evolve.

It looks counter-intuitive, though -- even if it is the spec. Especially for beginners, who might feel completely out of place. As other people have pointed out here, it's better to be implemented as a step of the build process if you really want to save on those bytes. Counter-intuitive patterns are nightmares for devs.

Wow, thanks. This is probably the best advice a first-time founder going through a pretty rough patch like me can get. Puts things in perspective.