wesleyyue

If anyone would like to try the new Sonnet in VSCode. I just updated https://double.bot to the new Sonnet. (disclaimer: I am the cofounder/creator)

---

Some thoughts:

* Will be interesting to see what we can build in terms of automatic development loops with the new computer use capabilities.

* I wonder if they are not releasing Opus because it's not done or because they don't have enough inference compute to go around, and Sonnet is close enough to state of the art?

Yep you're right, thanks for catching (sorry for the ninja edit!)

Ah you're right I totally misread that!

Interesting observations:

* Llama 3.2 multimodal actually still ranks below Molmo from ai2 released this morning.

* AI2D: 92.3 (3.2 90B) vs 96.3 (of Molmo 72B)

* Llama 3.2 1B and 3B is pruned from 3.1 8B so no leapfrogging unlike 3 -> 3.1.

* Notably no code benchmarks. Deliberate exclusion of code data in distillation to maximize mobile on-device use cases?

Was hoping there would be some interesting models I can add to https://double.bot but doesn't seem like any improvements to frontier performance on coding.

If you've tried others, would love to understand if there's anything you didn't like specifically (I'm one of the creators for https://double.bot)

Ah interesting! I'm building https://double.bot (ai assistant vscode extension) and someone asked about VSCodium but I didn't realize there's a open marketplace for that specifically.

Maybe something's timing out with the longer o1 response times?

Let me look into this – one issue is that OpenAI doesn't expose a streaming endpoint via the API for o1 models. It's possible there's an HTTP timeout occurring in the stack. Thanks for the report

Just added o1 to https://double.bot if anyone would like to try it for coding.

---

Some thoughts:

* The performance is really good. I have a private set of questions I note down whenever gpt-4o/sonnet fails. o1 solved everything so far.

* It really is quite slow

* It's interesting that the chain of thought is hidden. This is I think the first time where OpenAI can improve their models without it being immediately distilled by open models. It'll be interesting to see how quickly the oss field can catch up technique-wise as there's already been a lot of inference time compute papers recently [1,2]

* Notably it's not clear whether o1-preview as it's available now is doing tree search or just single shoting a cot that is distilled from better/more detailed trajectories in the training distribution.

[1](https://arxiv.org/abs/2407.21787)

[2](https://arxiv.org/abs/2408.03314)

If you're open to trying new AI coding assistants, would love if you can give https://double.bot a try! (note: I'm one of the creators) The main philosophical differences is that we are more expensive and are trying to build the best copilot with the technology possible at any given time. For example, we serve a larger, more accurate, and more modern autocomplete model, but it does cost more to serve. We also do a lot of somewhat novel work in getting the details right, like improving the autocomplete model to never screw up closing brackets, and always auto-close them as if you typed them.

Yes, I built the extension. I actually also just went to send another message asking what the first msg was just to double check I didn't have a bug and it does know what the first msg was.

definitely not an articulate request, but the point of using these tools is to speed me up. The less the user has to articulate and the more it can infer correctly, the more helpful it is. Other frontier models don't have this problem.

Llama 405B response would be exactly what I expect

https://x.com/WesleyYue/status/1816157147413278811

Yes, there's a few things wrong: 1. If it assumes typescript, it should do `as const` in the first msg 2. If it is python, it should be something like https://x.com/WesleyYue/status/1816157147413278811 which is what I wanted but I didn't want to bother with the typing.

I'm building a ai coding assistant (https://double.bot) so I've tried pretty much all the frontier models. I added it this morning to play around with it and it's probably the worst model I've ever played with. Less coherent than 8B models. Worst case of benchmark hacking I've ever seen.

example: https://x.com/WesleyYue/status/1816153964934750691

Just added Llama 3.1 405B/70B/8B to https://double.bot (VSCode coding assistant) if anyone would like to try it.

---

Some observations:

* The model is much better at trajectory correcting and putting out a chain of tangential thoughts than other frontier models like Sonnet or GPT-4o. Usually, these models are limited to outputting "one thought", no matter how verbose that thought might be.

* I remember in Dec of 2022 telling famous "tier 1" VCs that frontier models would eventually be like databases: extremely hard to build, but the best ones will eventually be open and win as it's too important to too many large players. I remember the confidence in their ridicule at the time but it seems increasingly more likely that this will be true.

I've been using it in https://double.bot (vscode coding copilot I'm hacking on) and it's been pretty great so far. We have support for all the top models (gpt4o, deep seek, opus, etc) so I regularly like to switch between them especially if I see a challenging problem. So far, sonnet 3.5 has the best quality and is the one I use by default.

If anyone would like to try it for coding in VSCode, I just added it to http://double.bot on v93 (AI coding assistant). Feels quite strong so far and got a few prompts that I know failed with gpt4o.

fyi for anyone testing this in their product, their docs are wrong, it's claude-3-5-sonnet-20240620, not claude-3.5-sonnet-20240620.

Surprised no comments are pointing out that the analysis is pretty far off simply due to the fact that the author runs with batch size of 1. The cost being 100x - 1000x what API providers are charging should be a hint that something is seriously off, even if you expect some of these APIs to be subsidized.

You can acknowledge how customers feel while shielding your customer facing staff from verbal abuse. If you've ever worked a customer facing job you'll understand.

If anyone wants to try it for coding, I just added support for GPT4o in Double (https://double.bot)

In my tests:

* I have a private set of coding/reasoning tests and it's been able to ace all of them so far, beating Opus, GPT4-Turbo, and Llama 3 70b. I'll need to find even more challenging tests now...

* It's definitely significantly faster, but we'll see how much of this is due to model improvements vs over provisioned capacity. GPT4-Turbo was also significantly faster at launch.

humaneval is generally a very poor benchmark imo and I hate that it's become the default "code" benchmark in any model release. I find it more useful to just look at MMLU as a ballmark of model ability and then just vibe checking it myself on code.

source: I'm hacking on a high performance coding copilot (https://double.bot/) and play with a lot of different models for coding. Also adding Qwen 110b now so I can vibe check it. :)

Looks interesting! I feel like Qwen has always been one of the most underrated model families that doesn't get as much attention as other peers for whatever reason. (maybe b/c it's from Alibaba?)

I've been working on https://double.bot (high performance coding copilot) and the number of high quality OS models coming out lately has been very exciting.

Adding this to Double now to so I can try it for coding. Should have it done in an hour or two if anyone else is interested. Will come back and report on the experience.

Hope they'll release API access soon so I can add it to https://double.bot

They've been talking about an API since June and it's now 9 months later with no updates on API access.

for future readers, the setting is now shipped in >v0.49. The default is now back to GPT-4 as it has lower latency but you can manually change it to Claude 3 in settings if you wish to try out Anthropic's new model.

Thank you for the report! We're using Mintlify for the docs (which that URL links to). Let me report it upstream to see if they can fix.

Thanks for the feedback – what search terms did you use? Let me make sure those keywords are on the page :P

more early impressions on performance: besides the endpoint erroring out at a higher rate than openai, time-to-first-token is also much slower :(

p50: 2.14s p95: 3.02s

And these aren't super long prompts either. vs gpt4 ttft:

p50: 0.63s p95: 1.47s

I think the tldr would be that they have more products (for example, their agent to write git commit messages). In the products we do have (autocomplete, chat), we spend a lot of time to get the details right. For example for autocomplete:

* we always close any brackets opened by autocomplete (and never extra brackets, which is the most annoying thing about github copilot)

* we automatically add import statements for libraries that autocomplete used

* mid-line completions

* we turn off autocomplete when you're writing a comment to avoid disrupting your train of thought

You can read more about these small details here: https://docs.double.bot/copilot

As you noted we don't have a vim integration yet, but it is on our roadmap!

It's default to claude 3 right now so I could get it out quick, but working on a toggle for the front-end now to switch between the two.

Thanks for the feedback. I was actually reworking the default shortcuts and the onboarding process when I got pre-empted by claude. I was planning to change the main actions to alt-j, alt-k to minimize conflicts.

Are you asking because it conflicts with an existing shortcut on your setup? Or another reason?

it's an interesting idea. We asked our users this as well but at least for those we talked to, running their own model wasn't a big priority. What actually mattered to them is being able to try different (but high performance) models, privacy (their code not being trained on), and latency. We have some optimizations around time-to-first-token latency that would be difficult to do if we didn't have information about the model and their servers.