itamarcode

Unlike most SWE bench submissions, Qodo Command one uses the product directly.

I think that the next step is getting an official "checked" mark by the SWE bench team

AI code review will very probably have signal-to-noise problems. It is good to see practical solutions aimed at addressing this. I wonder if fine-tuning models would help - this isn't addressed in the blog.

So protecting models behind API isn't working, ha?

While Sonnet-3.5 excels in accuracy-token, o1 excels in self-reflection in small isolated tasks. With AlphaCodium, o1 tasks are broken to small isolated tasks, while the flow introduced in AlphaCodium actually guides the steps and overall decision making framework.

We will see more of these frameworks for different use cases

Did you get any response from users detecting this automation?

Hey, co-creator here, I agree with the sentiment that code coverage may be a proxy and even sometimes a vanity metric but at the same time, IMO unit regression tests are necessary for a maintainable production codebase. I personally don’t feel confident making changes to production code that isn’t tested.

Specifically for generating unit regression tests the Cover-Agent tool already works quite well in the wild for some projects, especially isolated projects (as opposed to complex enterprise-level code). You can see in the few (somewhat cherry-picked) examples we posted [0] that it generates working tests that increase coverage (they were cherry-picked in the sense that these are examples we like to work with often internally at CodiumAI).

I believe that it’s possible to generate additional meaningful tests including end-to-end tests by creating a more sophisticated flow that uses prompting techniques like reflection on the code and existing tests, and generates the tests iteratively, feeding errors and failures back to the LLM to let it fix them. Just as an example. This is somewhat similar to the approach we used with AlphaCodium [1] which hit 54% on the CodeContests benchmark (DeepMind’s AlphaCode 2 hit 43% [2] with the equivalent amount of LLM calls).

If like me you think tests are important but hate writing them, please consider contributing to the open source to help make it work better for more use cases. https://github.com/Codium-ai/cover-agent

[0] https://www.youtube.com/@Codium-AI/videos [1] https://github.com/Codium-ai/AlphaCodium [2] https://storage.googleapis.com/deepmind-media/AlphaCode2/Alp...

This is interesting. Are there specific use cases for which it works really well?

Hey, one of the creators here. As mentioned in the post, TestGen-LLM (by Meta) focused on Kotlin, and the prompts were very Kotlin-oriented. In Cover-Agent (by CodiumAI) we tried to reimplement Meta's work, and stay mostly similar to the original implementation, although we did a bit improved the prompts. But it isn't generic enough. We believe we know how to improve generality, as we did with our PR-Agent, and here is a rough plan: https://github.com/Codium-ai/cover-agent/issues/13

Can it consider an existing code base? rather be inspired by it or even integrate with it?

The user can choose the relevant context as follows: go to the relevant code part/snippet/files, select, then Ctrl+shift+E, one by one.

In the near future, the agent will suggest these for you, after first indexing your code base

We will support other, I promise

SaaS uses best of bread from OpenAI, whole enterprise get to choose also from our own model for on prem deployment

Well,

To put it simple: Like copilot, it does have an auto-complete and also chat interface. Different than copilot, it focuses on generating a full code task plan, then having the auto completion work acrroding to your plan, and it check the of quality code.

Like agents, it tries to help you complete a full task, yet,does that in tandem with you, as you work inside tour favorite IDE writing the code with you. In addition, there is focus on code quality, testing, and fetching relevant context from your codebase.

Hey HN, CodiumAI co-founder here.

We’ve come up with a bit of a different concept for what a coding agent should be. We believe it should work in tandem with a developer inside the IDE. Over time as the tech improves, it will get more and more autonomy.

We’ve been using our coding agent internally and see a 5-10x boost on some tasks.

The agent is available now to Codiumate VS Code users. We want to hear what kind of tasks it works well on and improve it over time to expand the task set. Would love to get feedback.

https://marketplace.visualstudio.com/items?itemName=Codium.c...

Seeing a good demo, and a credible influencer actually trying the product

Oh, this looks very useful. Do you have a success story?

Hey, this is one of the makers here. Feel free to ask me anything.

Paper: https://arxiv.org/abs/2401.08500 Blog: https://www.codium.ai/blog/alphacodium-state-of-the-art-code...

Paper: https://arxiv.org/abs/2401.08500 Blob: https://www.codium.ai/blog/alphacodium-state-of-the-art-code...

What are the main topics you think you will cover?

There are tools focused on Code Generation, and those that are focused on Code Analysis or Integrity. Their UX/UI might be very different as well as the underlying tech.

I wonder if you have tried tools that are dedicated to Code Integrity, e.g. generating tests?

I wrote a blog about it: https://www.codium.ai/blog/code-integrity-supercharges-code-...

disclaimer: I'm the co-maker of PR-Agent and CodiumAI

GPT-4 Vs. AlphaCode: Comparing Two Leading Code Generation Tools

The benchmark - Codeforces programming contest.

GPT-4 Codeforces Rating is 392 points, improving GPT-3.5’s 260 points.

AlphaCode by DeepMind achieves 1,238 points!

Those who tried Bard or Codey will very likely agree that Google’s models and solutions are not better than OpenAI ones. So, what is going on here?

The secret sauce? AlphaCode is composed of two components, not one. A Code Generation component, and a Code Integrity component (that includes test generation, filtering and clustering according to tests runs).

Will we very soon see some high quality locally hosted alternatives to GitHub Copilot?

Considering the latest news, like llama.cpp, new code generators, etc... maybe it is doable?

>> ... transformative concepts that have the potential to redefine software development practices. By integrating automatically generated tests and code reviews into the development process, organizations can significantly improve code integrity, accelerate delivery, and stay ahead in the competitive software landscape.

it is impressive that tests are being generated as part of the process, well done! can you explain how does it work and when it won't work?

so basically, you think that AI tools can help with unit/component level testing and documentation?

>> In the fast-paced world of software development, busy developers often find themselves grappling with the challenges of creating effective tests.

Can AI tools help in generating meaningful tests? In the linked post, Ankur Tyagi reviews CodiumAI tools and vision.

>> CodiumAI’s vision is simple; an AI coding assistant/agent to assist developers in reaching zero bugs. >> Codium AI automatically generates Happy Paths, Edge Cases, and Other test suites when you finish writing your code and saving it.

Can AI-empowered tools really understand develop intent and analyze the code to generate edge cases?

i wrote about AI Agents & SW 3.0 on March 2022!

some selected quotes from the blog

1) "In SW 3.0, the business logic, I/O control, and the data pre/post-processing code are partially or even entirely created by the AI agent during the optimization process"

the `optimization process` for the entire chain of actions is still immature

2) "It is likely that any setting where the program is not obvious but one can repeatedly evaluate its performance (e.g. - programming competitions?) will be subject to this transition, because the optimization can probably find much better code than what an average human can write"

3) "It is likely that testing [automation] will become even more crucial in the SW 3.0 stack for a number of possible reasons. To name two: 1) programmers would be able to do more, including less experienced ones, 2) programs will be much more dynamic"

4) "In the coming years, we will see immense research efforts to improve the setup and learning schemes used to create AI agents that generate programs"

4) "Bottom-up: Functions, classes, and capabilities will be generated with newly introduced IDEs, code management, testing, code generation and analysis of SW 3.0 stack. Top-down: graphical and programable application interfaces will be generated with newly introduced no/low-code SW 3.0 based platforms"

did these age well?

do you think that the vision of AI Agents will be realized before 2025?

curious, would you consider `ChatGPT + Code interpreter` as an agent?

Is it like JasperAI but dedicated to Technical Writers? What about team collaboration around MD editing?

Quote> Looking further ahead, Cohere plans to build models that can take action and “do work” for customers, like book a flight, schedule a meeting or file an expense report on a person’s behalf. In that way, it’s chasing after competitors like Adept, Inflection and OpenAI, all of which are building — albeit using different approaches — systems to connect AI with third-party apps, services and products.

nice! https://news.ycombinator.com/item?id=36317978 have you open sourced it or made it available somehow?