Show HN: We are building an open-source IDE powered by AI

by mlejva • 3 years ago

369 175 3 years ago

Etheryte2• 3 years ago

While you mention that you can bring your own model, prompt, etc, the current main use case seems to be integrating with OpenAI. How, if at all, do you plan to address the current shortcoming that the code generated by it often doesn't work at all without numerous revisions?

anonzzzies5• 3 years ago

it often doesn't work at all without numerous revisions?

It doesn’t so for many/most humans either. I hope they have a revisions prompting, but I did not try it yet.

I noticed adding in a feedback/review loop often fixes it, but the you still need someone saying ‘this is it’ as it doesn’t know the cut off point.

ValentaTomas4• 3 years ago

Revisions prompting is not there yet but it is one of things that we are experimenting with.

The feedback/review loop is spot on - a lot of the problems can be fixed automatically in a few steps but you actually need the outputs/errors.

Etheryte2• 3 years ago

I think there's a big gap here you might be missing. Most developers beyond juniors can generally write code that at least compiles on the first pass, even if it isn't functionally correct. Current AI models often generate code that doesn't even compile.

mlejva11• 3 years ago

Good point and feedback, thank you. We'll update readme.

A lot of UX, UI, DX work related to LLMs is completely new. We ourselves have a lot of new realizations.

While you mention that you can bring your own model, prompt, etc, the current main use case seems to be integrating with OpenAI

You're right. This is because we started with OpenAI and to be fair it's easiest to use from the DX point of view. We want to make it more open very soon. We probably need more user feedback to learn what would be the best way how this integration should look like.

How, if at all, do you plan to address the current shortcoming that the code generated by it often doesn't work at all without numerous revisions?

The AI agents work in a self-repairing loop. For example they write code, get back type errors or a stack trace and then try to fix the code. This works pretty well and often a bigger problem is short context window.

We don't think this will replace developers, rather we need to figure out the right feedback loop between the developer and these AI agents. So we expect that developers most likely will edit some code.

mlejva11• 3 years ago

Hey everyone, I quite frankly really didn't expect our project getting on HN front page in 2 minutes after posting. I'm one of the creators of this project.

It's pretty nascent and a lot of work needs to be done so bear with us please. I'm traveling in 30 minutes but I'm happy to answer your questions.

A little about my co-founder and myself: We've been building devtools for years now. We're really passionate about the field. The most recent thing we built is https://usedevbook.com. The goal was to build a simple framework/UI for companies to demo their APIs. Something like https://gradio.app but for API companies. It's been used for example by Prisma - we helped them build their playground with it - https://playground.prisma.io/

Our new project - e2b - is using some of the technology we built for Devbook in the past. Specifically the secure sandbox environments where the AI agents run are our custom Firecracker VMs that we run on Nomad.

If you want to follow the progress you can do follow the repo on GH or you can follow my co-founder, me, and the project on Twitter:

- https://twitter.com/e2b_dev

- https://twitter.com/t_valenta

- https://twitter.com/mlejva

And we have a community Discord server - https://discord.gg/U7KEcGErtQ

peteradio• 3 years ago

Maybe this is a bit naive but is this AI batteries included? Or what level of work needs to be done to integrate with an AI?

ValentaTomas4• 3 years ago

Right now you just need the GPT-4 API key to run it but we plan to support custom models in the future.

Thiebaut• 3 years ago

It’s a very interesting idea, i haven’t tested the app, maybe integration with vscode ?

nico4• 3 years ago

Definitely would be great to use it directly in vscode

mlejva11• 3 years ago

I don't think the integration with vscode would work well here. We're ditching a lot of existing workflows for a completely new interaction between the developer and the agent. It's not like copilot at all. E2b is more like having a virtual developer at your disposal 24/7. You write a technical spec and e2b codes builds the app for you

celeritascelery• 3 years ago

This looks quite interesting. So I have question for these AI powered editors: what advantage would a dedicated editor like this have over just using an AI plugin for VsCode? How do you fundamentally build the editor differently if you are thinking about AI from the ground up?

rco8786• 3 years ago

How do you fundamentally build the editor differently if you are thinking about AI from the ground up?

Great question. I would love to hear the devs thoughts here. This is one of those questions where my intuition tells me there may be a really great "first principles" type of answer, but I don't know it myself.

prepend• 3 years ago

I suspect that if the goal is to make money, you want your own ide and don’t want to rely on vscode.

swyx• 3 years ago

i mean the obvious answer is just being able to do things that the VS Code extension api does not let you do.

per the end of the readme:

- one step to go from a spec to writing code, run code, debugging itself, install packages, and deploying the code

- creates "ephemeral UI" on demand

ValentaTomas4• 3 years ago

In essence when working with code stops being the major thing you do (you abstract that away) and start managing the agents working on code and writing the spec, you need new tools and working environment to support that.

Swizec• 3 years ago

and start managing the agents working on code ... you need new tools

Jira?

Only slightly joking. It really sounds like we're moving in the direction of engineers being a more precise technical version of a PM, but then engineers could just learn to speak business and we don't need PMs.

namaria3• 3 years ago

So you lose precision and control?

mlejva11• 3 years ago

Hey! Thanks for the question.

Our editor isn't a regular coding editor. You don't actually write code with e2b. You write technical specs and then collaborate with an AI agent. Imagine it more like having a virtual developer at your disposal 24/7. It's built on a completely new paradigms enabled by LLMs. This enables to unlock a lot of new use cases and features but at the same time there's a lot that needs to be built.

walleeee• 3 years ago

I'm not sure I follow this answer. What are the entirely new paradigms? Writing is still the initial step. If text editing remains a core part of the workflow, why restrict the user's ability to edit code?

mayank• 3 years ago

You don't actually write code with e2b. You write technical specs and then collaborate with an AI agent.

If I want to change 1 character of a generated source file, can I just go do that or will I have to figure out how to prompt the change in natural language?

eikenberry2• 3 years ago

Writing technical specs a fancy way to say coding. This reads to me like you're writing a new programming language tightly integrated with an IDE that targets a new AI based compiler.

stinkbutt3• 3 years ago

by technical specs you mean some kind of prompt mechanism, no?

barbariangrunge• 3 years ago

If you could use it without submitting data to some ai company, or if it came with a non-disgusting terms of service, that would be a killer feature for me.

For example, the last ai company installer I just clicked "decline" to (a few minutes ago) says that you give it permission to download malware, including viruses and trojans, onto your computer and that you agree to pay the company if you infect other people and tarnish their reputation because of it. Literally. It was a very popular service too. I didn't even get to the IP section

edit: those terms aren't on their website, so I can't link to them. They are hidden in that tiny, impossible to read box during setup for the desktop installer

stinkbutt3• 3 years ago

can you name this company please?

UberFly• 3 years ago

Sounds like fantastic software. Would be nice to know who you're referencing here.

campbel• 3 years ago

I built this https://github.com/campbel/aieditor to test the idea of programming directly with the AI in control. Long story short, VS Code plugin is better IMO.

la647102• 3 years ago

All these tools starts with grand proclamation of “open” and then the first thing you notice is the field to add your OPENAI_KEY. My humble suggestion is that if you are building something truly open please use some other models like LLAMA or BERT as default example and keep options for adding other models as needed.

mlejva11• 3 years ago

Hey, thank you for the feedback. I understand that having a requirement for OpenAI key isn’t good for folks. The project is 4 weeks old and OpenAI is what we started with as a LLM. We want to create interface for people to bring and connect their own models though. We will do a better work of explaining this in our readme

anonzzzies5• 3 years ago

I wouldn’t worry too much about it; there will be more and more models and fine-tuning services and fine-tuned downloads. Different models to mix and match.

la647102• 3 years ago

Thanks as you might imagine in many scenarios, lock in to any particular hosted model is not desirable.

bgirard2• 3 years ago

Is there examples / case studies of more complex apps being built by LLMs? I've seen some interesting examples but they were all small and simple examples. I'd love to see more case studies of how well these tools perform in more complex scenarios.

My gut feeling is we're still a few LLMs generations away from this being really usable but I'd love to hear how the authors are thinking about this.

ryanSrich2• 3 years ago

Can you give an example of complex? I’ve used ChatGPT to help me build an app that authenticates a user using Oauth. That information creates a user in the backend (Rails). That user can then import issues tagged with specific information from a 3rd party task management tool (Linear). The title for these issues are then listed in the UI. From there, the user can create automatic release notes from those issues. They can provide a release version, description, tone, audience, etc.

All of that (issues list, version, tone, etc) is then formulated into a GPT prompt. The prompt is structured such that it returns written release notes. That note is then stored and the user can edit it using a rich text editor.

Once the first note is created the system can help the user write future notes by predicting release version, etc.

This isn’t that complex imo, but I’m curious to see if this is what people consider complex.

ChatGPT wrote 90% of the code for this.

mritchie712• 3 years ago

I'd consider that non-trivial (e.g. not a todo example).

How long did it take?

bgirard2• 3 years ago

I don't have a specific definition of complex in mind. Seeing more examples of this with the prompts used + output and the overall steps is exactly what I'm asking for. I'm particularly interested in how the success rate changes as the code base evolves. Are LLMs effective in empty repos? are they effective on large repos? Can prompts be tweaked to work on larger repos?

ehutch794• 3 years ago

A Line Of Business app. With questionable specs. Where inputs are cross dependant and need to be filtered. Some fields being foreign keys to other models.

mypalmike2• 3 years ago

How about a 2 million line legacy app spanning 5 languages including one created by a guy who left the company 14 years ago which has a hand-rolled parser and is buggy.

ivan888• 3 years ago

My biggest concern with tools like this is reproducibility and maintainability. How deterministically can we go from the 'source' (natural language prompts) to the 'target' (source code)? Assuming we can't reasonably rebuild from source alone, how can we maintain a link between the source and target so that refactoring can occur without breaking our public interface?

ValentaTomas4• 3 years ago

This is a valid concern and we are still experimenting with how to do this right. A combination of preserving the reasoning history, having the generated code, and using tests to enforce the public interface (and fix it if anything breaks) looks promising.

I think the crucial part is indeed not being able to deterministically go from NL to code but to take an existing state of the codebase and spec and "continue the work".

larsonnn• 3 years ago

Pretty simple it’s just like any abstraction. This AI will not work when nobody would deliver the answer beforehand. LLMs are given inputs of existing code. When you abstract that you better hope you have good code in it.

So my question would be, what is the use case?

I guess it’s more like planing software and not implementing it.

You can pretty well plan your software with ChatGPT. But it will just help you not really doing the job.

Mali-• 3 years ago

We're building ourselves out of a job in real time.

jkettle• 3 years ago

We are all Biz Devs now. :(

Manjuuu2• 3 years ago

Or freeing up time for more complex parts of the job.

stinkbutt3• 3 years ago

damn you got some top shelf, high grade copium. let me get a hit

jfengel• 3 years ago

I've been expecting to be replaced by a much, much cheaper developer in another country since I graduated college ... three decades ago. I'm still not 100% certain why that hasn't happened.

I suspect it has to do with the equivalent of prompt engineering: it's too difficult to cross the cultural and linguistic barriers, as well as the barrier of space that could have mitigated the other two. By the time you've directed somebody to do the work with sufficient precision, you could have just done it yourself.

And it's part of the reason we keep looking for that 10x superdeveloper. Not just that they produce more code per dollar spent, but that there is less communication overhead. The number of meetings scale with the square of the number of people involved, so getting 5 people for the same price as me doesn't actually save you time or money.

I have no idea what that means for AI coding. Thus far it looks a lot like that overseas developer who really knows their stuff and speaks perfect English, but doesn't actually know the domain and can't learn it quickly. (Not because they aren't smart, but because of the various human factors I mentioned.)

I'd be thrilled to be completely wrong about that -- in part because I've been mentally prepared for it for so long. I hope that younger developers get a chance to spin that into a whole new career, hopefully of a kind I can't even imagine.

halfjoking2• 3 years ago

Mark my words... custom AI IDEs are the new javascript frameworks.

I think everyone had this idea and is building something similar. I know I am.

mlejva11• 3 years ago

Would love to learn more about what you're building! Do you have anything to share?

halfjoking2• 3 years ago

Nothing yet - so far it’s just a basic electron app that you select what files to send as reference for the feature you want to add, then streamlines the process of applying edits ChatGPT sends back.

I’m not really planning on turning it into a product. It sounds like this guy is a lot farther along than me if you’re looking for a competitor - I think you’re going to have plenty. https://mobile.twitter.com/codewandai

gumballindie3• 3 years ago

Good observation. I did notice that a lot of the types that jumped from framework to framework are now jumping onto ai. Please keep them there, at least that way js will stand a chance at becoming sane.

namaria3• 3 years ago

What do you think these fuzzy interpreters are gonna be used for? Machine code running on metal? It's gonna be all scripted servers and web apps for saas startups slurping that VC or UE money

Jupe2• 3 years ago

This is awesome, scary and very interesting. But, for me, it comes with a personal concern:

For some time I've been giving serious thought about an automated web service generator. Given a data model and information about the data (relationships, intents, groupings, etc.) output a fully deployable service. From unit tests through container definitions, and everything I can think of in-between (docs, OpenAPI spec, log forwarder, etc.)

So far, while my investment hasn't been very large, I have to ask myself: "Is it worth it?"

Watching this AI code generation stuff closely, I've been telling myself the story that the AI-generated code is not "provable". A deterministic system (like I've been imagining) would be "provable". Bugs or other unintended consequences would be directly traceable to the code generator itself. With AI code generation, there's no real way to know for sure (currently).

Some leading questions (for me) come down to:

1. Are the sources used by the AI's learning phase trustworthy? (e.g. When will models be sophisticated enough to be trained to avoid some potentially problematic solutions?)

2. How would an AI-generated solution be maintained over time? (e.g. When can AI prompt + context be saved and re-used later?)

3. How is my (potentially proprietary) solution protected? (e.g. When can my company host a viable trained model in a proprietary environment?)"

I want to say that my idea is worth it because the answers to these questions are (currently) not great (IMO) for the AI-generated world.

But, the world is not static. At some point, AI code generators will be 10x or 100x more powerful. I'm confident that, at some point, these code generators will easily surpass my 20+ years of experience. And, company-hosted, trained AI models will most likely happen. And context storage and re-use will (by demand) find a solution. And trust will eventually be accomplished by "proof is in the pudding" logic.

Basically, barring laws governing AI, my project doesn't stand a cold chance in hell. I knew this would happen at some point, but I was thinking more like a 5-10 year timeframe. Now, I realize, it could be 5-10 months.

sebzim45007• 3 years ago

Not OP but I've been playing with similar technology as a hobby.

1. Are the sources used by the AI's learning phase trustworthy? (e.g. When will models be sophisticated enough to be trained to avoid some potentially problematic solutions?)

Probably not, but for most domains reviewing the code should be faster than writing it.

2. How would an AI-generated solution be maintained over time?

I would imagine you don't save the original prompts. Rather, when you want to make changes you just give the AI the current project and a list of changes to make. Copilot can do this to some extent already. You'd have to do some creative prompting to get around context size limitations, maybe giving it a skeleton of the entire project and then giving actual code only on demand.

When can my company host a viable trained model in a proprietary environment?

Hopefully soon. Finetuned LLaMA would not be far off GPT-3.5, but nowhere close to GPT-4. And even then there are licencing concerns.

Jupe2• 3 years ago

Ok, a couple of derivative "fears" around this...

1> Relying on code reviews has concerns, IMO. For example, how many engineers actually review the code in their dependencies? (But, I guess it wouldn't take that much to develop an adversarial "code review" AI?)

2> Yes, agreed, that would work. Provided the original solution had viable tests, the 2nd (or additional) rounds would have something to keep the changes grounded. In fact, perhaps the existing tests are enough? Making the next AI version of the solution truly "agile"?

3> So, at my age (yes, getting older) I'm led to a single, tongue-in-cheek / greedy question: How to invest in these AI-trained data sets?

mlejva11• 3 years ago

Thanks for the questions. I’ll reply just a bit later. I’m in a train and reading/writing makes my stomach sick

8note• 3 years ago

Given a data model and information about the data (relationships, intents, groupings, etc.) output a fully deployable service. From unit tests through container definitions, and everything I can think of in-between (docs, OpenAPI spec, log forwarder, etc.)

AWS roughly has one of these in Amplify. The data mapping parts are pretty great, though lots of the rest of it suck. The question I'd ask is if those parts suck by nature of the setup, or is it just amplify that has weird assumptions

nbzso2• 3 years ago

Maybe until we all have a local LLM's and custom models with full control, this level of abstraction(prompting) is not useful. I refuse to contribute to the"Open"AI scheme. Let marketing and teens to give them data.:)

exhaze3• 3 years ago

I agree with your point about self hosting as long term strategy. However at current stage, it’s still a balance between capability and control. Greater control means you lag behind on bleeding edge features (gap seems to be constantly narrowing here though for LLMs thanks to tons of OSS efforts).

Disagree with your fears about “give them data”.

Here’s their data policy: https://openai.com/policies/api-data-usage-policies

Literally states it won’t use your data. Ofc there’s non trivial risk that this policy will change over time. Still, I don’t feel like there’s any huge lock-in risk with OpenAI right now, so advantage of using it outweighs the risk for most.

nbzso2• 3 years ago

I simply have no trust in Microsoft. Data is the new petrol.:)

mov_eax_ecx• 3 years ago

This is like the third post today on code generator using a react application, in twitter there are dozens of them.

Copilot already exists and copilot X already packs the features this package promises AND much more, why use this application over Copilot?.

jumpCastle• 3 years ago

One reason might be that some people value open source.

fabrice_d• 3 years ago

Looking at the license of this project (Business Source License 1.1), this is not an open source project: https://github.com/e2b-dev/e2b/blob/master/LICENSE#L16

mlejva11• 3 years ago

e2b isn't like copilot. You don't really write code in our "IDE". Currently, it works in a way that you write a technical spec and then collaborate with an AI agent that builds the software for you. It's more like having a virtual developer available for you.

maxilevi• 3 years ago

Do you use it for developing?

teddyh• 3 years ago

The current idea is to offer the base cloud version for free while having some features for individuals behind a subscription. We'll share more on pricing for companies and enterprises in the future.

What happens if you use the README.md and associated documentation as a prompt to re-implement this whole thing?

mfts0• 3 years ago

Huge fan since Devbook and really happy you found a killer feature of Devbook‘s tooling combined with AI

mlejva11• 3 years ago

Thank you for the kind words:)

sebzim45007• 3 years ago

Can it produce projects which are longer than the context length of the LLM? (8k for us plebs)

M4v3R2• 3 years ago

It's perfectly possible to assemble a project by writing individual files one at a time, so you would basically get 8k tokens per file. Or you could even write out the files in parts.

sebzim45007• 3 years ago

Sure but in that case how do it keep track of what it has already output? What stops it 'hallucinating' functions in files that it has already produced.

exhaze3• 3 years ago

Aren’t most devs working on existing apps?

Don’t most devs tend to be extremely sticky to their preferred dev env (last big migration was ST -> VS Code back in 2015-2017)

Read all the comments in here. Still not getting why this isn’t a VS Code plugin.

Distribution almost always beats product.

px43• 3 years ago

Maybe to spite Mircosoft? I would love to completely remove Microsoft from my workflow. The way telemetry creeps into everything bugs the hell out of me. I also really don't like the security model of vscode where it's assumed that any code you look at comes from a trusted source, and might execute code on your local system. That's a ridiculous security assumption for a text editor, but not surprising considering that these people also made MS Office.

exhaze3• 3 years ago

When I run VS Code on Mac and open some new non trusted directory it literally forces me to click “yes” in dialog that says something like “do you trust stuff in this directory to be able to run code?” Is this a Mac only thing and maybe you use windows or linux? Cuz I found your security concern addressed to an almost obnoxious extent…

nullandvoid• 3 years ago

I dislike telemetry as much as the next guy, but it is a little funny to complain about it on the post for this editor, which instead sends your entire codebase to OpenAI.

Also it's a somewhat recent update, but VSCode asks you if you trust the code author now when you open a project https://code.visualstudio.com/docs/editor/workspace-trust

graypegg• 3 years ago

If this truly works as the pitch describes:

mlejva: Our editor isn't a regular coding editor. You don't actually write code with e2b.

then what licensing problems arise from its use? In theory, if you only prompt the AI to write the software, is the software even your intellectual property?

It seems like this is a public domain software printing machine if you really aren’t meant to edit the output.

euroderf2• 3 years ago

Could I feed this a decent set of software requirements and get back something that reflects them ?

mlejva11• 3 years ago

Yes. That’s one of the goals with e2b

euroderf2• 3 years ago

Good to hear. To hell with prompt engineering. Gimme some old-time requirements engineering!

sjm• 3 years ago

IMO Emacs is a perfect candidate for this kind of thing, or maybe something akin to LSP so you can bring your own editor. New GPT extensions are coming out daily for Emacs, e.g. https://github.com/xenodium/chatgpt-shell

cube22223• 3 years ago

Based on my experiences having spent a bunch of time experimenting with LLMs for writing code, I think the hardest part that I haven't yet seen a solution to is modifying existing code bases.

Sure, if you're doing greenfield, just ask it to write a new file. If you only have a simple script, you can ask it to rewrite the whole file. The tricky bit however, is figuring out how to edit relevant parts of the codebase, keeping in mind that the context window of LLMs is very limited.

You basically want it to navigate to the right part of the codebase, then scope it to just part of a file that is relevant, and then let it rewrite just that small bit. I'm not saying it's impossible (maybe with proper indexing with embeddings, and splitting files up by i.e. functions you can make it work), but I think it's very non-trivial.

Anyway, good luck! I hope you'll share your learnings once you figure this out. I think the idea of putting LLMs into Firecracker VMs to contain them is a very cool approach.

chatmasta2• 3 years ago

I think this will get easier as the supported context sizes get larger. Of course the tooling needs to take care of most of the drudgery, but I'm not sure there's any underlying limitation of an LLM that makes refactoring existing code any different from creating new code. It's just a matter of feeding it the layout of the repo and the code itself in a way that the LLM can retrieve and focus on the relevant files when you ask it for help with a task.

cube22223• 3 years ago

Oh yeah, I agree with the refactoring bit. As I said, rewriting a file (section) works great if your file fits into the context window.

But context windows are far from being large enough to fit entire repos, nor even entire files (if they're big). I'm not sure how hard just scaling up the context window is, from the current early access of OpenAI GPT-4.

in a way that the LLM can retrieve and focus on the relevant files

This I think is something we haven't really figured out yet, esp. if a feature requires working on multiple files. I wouldn't be surprised if approaches based on the semantic level (actually understanding the code and the relationships between it's parts; not the textual representation of it) won't be needed in the end here.

M4v3R2• 3 years ago

This is actually possible, I made a proof of concept agent that does exactly this. The agent has access to several commands that get executed on demand - ListFiles, ReadFile and WriteFile. You give it a prompt - for example "add a heading to the homepage and make it yellow" and it will use ListFiles to locate the correct file, then use ReadFile to read it and then finally use WriteFile to write the new contents of that file.

It was just a PoC so it wasn't bulletproof and I only ever made it work on a small Next.js codebase (with every file being intentionally small so it fits the context window) but it did work correctly.

dakiol• 3 years ago

Shouldn't it be a plugin for a more used open-source IDE?

MacsHeadroom2• 3 years ago

Microsoft who owns CoPilot and OpenAI makes the most used Open Source IDE.

You think improving their produce for them while submitting yourself to the authority of their plugin market is a good idea?

mnd999• 3 years ago

Because developers love writing documentation.

wruza4• 3 years ago

I find it surprising that many developers find using ai prompts easier than an actual programming language (not the first time I hear about it, but now it seems to be serious).

When I tried to “talk” to gpts, it was always hard to formulate my requests. Basically, I want to code. To tell something structured, rigorous, not vague and blah-blah-ey. To check and understand every step*. Idk why I feel like this, maybe for me the next ten years will be rougher than planned. Maybe because I’m not a manager type at all? Trust issues? I wonder if someone else feels the same.

Otoh I understand the appeal partially. It roots in the fact that our editors and “IDEs” suck at helping you to navigate the knowledge quickly. Some do it to a great extent, but it’s not universal. Maybe a new editor (plugin) will appear that will look at what you’re doing in code and show the contextual knowledge, relevant locations, article summaries, etc in another pane. Like a buddy who sits nearby with a beer, sees what you do and is seasoned enough to drop a comment or a story about what’s on your screen any time.

Programming in a natural language and trusting the results I just can’t stand. Hell, in practice we often can’t negotiate or align ourselves in basic things even.

* and I’m not from a slow category of developers

ramraj072• 3 years ago

I used to think like you until I actually tried copilot. I don’t even write much boilerplate code (it’s python, and the framework is already setup). It is still an effective and useful tool. It saves mental bandwidth, and the code it writes is exactly what I was intending to write, so it turns me into a reviewer than a coder and thus makes it easier to get the job done (I’m still confirming the code does what I want).

This is just gpt-3. With chatGPT i finally managed to get unit test backbones for the most annoying methods in our code base that stuck our coverage at 92%. Once we get full copilot gpt-4 integration we will likely get to 98% coverage in a days time. That’s not nothing.

wruza4• 3 years ago

I tried copilot in some toy projects back then but wasn’t left impressed. Maybe should try gpt-4 later. Thanks for sharing your experience!

sunshinerag• 3 years ago

Can I know which IDE did you try copilot with?

arketyp• 3 years ago

Totally. All I want is an easier time navigating code; not getting lost in tabs and page scrolling, nested class structures and function references, interrupted by context-switching text searches. It feels like I spend 75% of my brain capacity managing my short term memory. Your buddy mentor is a great formulation of the kind of AI aid I need. As for generating code, that's the least of my concerns. I'm actually happy when I get the opportunity to just hammer down some brain dead lines for a change.

ivanjermakov• 3 years ago

Code navigations can be improved in a plenty of ways. I use JetBrains' IDEs daily, but most points can be applied to other IDEs/editors.

Some things that helped me to become a lot more efficient:

- Vim. Since you mostly edit code, not write it, it is reasonable to be in the editing mode by default, with a full keyboard of shortcuts you can use, not only ctrl/alt combinations. I use IdeaVim[1] plugin for Vim emulation.

- Never use your mouse. Keyboard actions are faster and more consistent once you get used to it. IDEs and editors are highly customizable and should allow to bind almost any action.

- Get rid of tabs, keep 1-3 active buffers in your vicinity. And then use ctrl+tab or ctrl+e to switch between them.

- Learn tricks for code navigation: jump to definition/usage/implementation, search file/class/symbol.

- Learn tricks for text navigation: pretty much any decent tutorial on Vim navigation will do.

[1] https://github.com/JetBrains/ideavim

firatsarlar• 3 years ago

We're in same page. Me too constantly jump over code sections. I realized in many solutions to this, most effective could be meta thinking. Self awareness. I'm quick - like most of us - on editor, I'm quick at coding why ? It is kind of conditioning with time management. Quick thinking - assume - make us jumps, just like a battle with many fronts, keep your eye on that, then that . In reality, the code part we wrote in an one hour, probably done in one spark of brain act. Rest is coordination.

88j88• 3 years ago

A large part of software development is documentation. It is often overlooked or not kept up to date. I think the great advancement of AI is that we can now more closely link and validate one with the other. You can easily summarize some code with chatGPT, and also provide some structure of code based on documentation (as an outline, or first cut).

However, this is the state of the art today. In the future, the training set will be based on the prompt-result output and refinement process, leading me to believe that next generation tools will be much better at prompting the user to provide the details. I've already seen this enhancement in gpt4 recently, I think this is a common and interesting use case.

Overall, these tools will become more and more advanced and useful for developers. Now is a great time to become proficient and start learning how the future will require you to adapt your workflow in order to best leverage chatGPT for development.

This response was written by a human.

vanjajaja1• 3 years ago

I think it has to do with how invested you are in the appearance of the final code. If you're very focussed on getting a result and no more, then GPT is probably very powerful. But if you also care about the "how" of the implementation then you'll probably get frustrated micromanaging GPT. I know I do, but I do find its pretty useful when my memory of how to achieve something is blurry. I know what looks wrong but I don't remember how to make it right.

thequadehunter• 3 years ago

As an infra dude it saves me loads of time. Most of the code I write is stuff where I already know what needs to happen, I already know how the process looks un-automated, and I'm not going to learn a whole lot writing more boilerplate, or learning the names of all the endpoints for a dumb api and parsing the text.

It's still mental work. You still have to read the code over and edit stuff.

It helps me save lots of time so I can do quality things like comment on hn during work..

yunwal• 3 years ago

Infra as code is really where copilot shines. GPT/codex isn’t great as a state machine, but it’s great for solving the “what’s that variable called?” or “how’s that yaml supposed to be structured?” problems. I think the architect and infra eng distinction will disappear soon as writing IAC becomes nearly instantaneous

johntash• 3 years ago

I'm not super impressed yet either, but I do think there's a lot of value. I tried a few times to get it to write some simple python scripts, and usually it'd get some things wrong. But it did help to get the basics of the script started.

If you haven't already, look into trying out Copilot (I don't think there's a free version anymore?) or Codeium (free and has extensions for a lot of editors, just be careful with what code you're okay sending to a third party). Using AI prompts as a fancy auto complete is what has been giving me the most benefit. Something simple like a comment "# Test the empty_bank_account method" in an existing file using pytest, I go to the next line and then hit tab to auto-complete a 10-15 line unit test. It's not always right, but it definitely helps speed things up compared to either typing it all out or copy/pasting.

My biggest annoyance so far, at least with Codeium, is that sometimes it guesses wrong. With LSP-based auto-complete, it only gives me options for real existing functions/variables/etc. But Codeium/copilot like to "guess" what exists and can get it wrong.

Like a buddy who sits nearby with a beer, sees what you do and is seasoned enough to drop a comment or a story about what’s on your screen any time.

I agree with you, this is probably where it would be most useful (to me). I don't need someone to write the code for me, but I'd love an automated pair-programming buddy that doesn't require human interaction.

hyperthesis• 3 years ago

Everyone projects their difficulties on to you, so here's mine:

A specification can be rigorous, structured, precise and formal: a declarative description of what is to be done - not how. It is a very different perspective from designing algorithms - even one that proves the thing to be done, is done.

But I think trusting the results is misplaced. It's more like being a partner in a law firm, where the articled clerks do all the work - but you are responsible. So you'd better check the work. And doing that efficiently is an additional skill set.

amelius• 3 years ago

and I’m not from a slow category of developers

You will be if you don't embrace AI ;)

cube22223• 3 years ago

I feel like the real value is when working with tools you're not proficient with.

I.e. for a backend dev that hardly knows react and needs to write some, who usually gets by doing small modifications and copy-paste-driven development there, ChatGPT / Copilot is basically a 10x productivity multiplier.

I agree that I've not been feeling that much value yet when working in areas I'm proficient in.

all27• 3 years ago

I've been leaning on ChatGPT about 1/4 of the time for programming personal projects. It's a great way to get boilerplate out of the way, as well as quickly introduce me to things I didn't know about. Once I get started, though, I stopped using the AI. I'd use it for code reviews, but it isn't smart enough to thoughtfully make comments about how data is pipelined through a complete application (it lacks in other ways, too, but that's the big thing I noticed).

rileyphone• 3 years ago

it isn't smart enough to thoughtfully make comments about how data is pipelined through a complete application

My take is that data-centric programming requires too much context for GPT, and we're going to see a move back to doing things in a more OO way, hopefully with better languages than exist now. The ability to reason locally about objects helps both human and AI developers, so we can build larger systems. Data-centric/functional programs are akin to hand-crafted, artisanal goods that were slowly overtaken by standardized parts/division of labor production methods in the industrial revolution, and we are in the middle of one for software [0]. By the ended of it, software engineering may no longer be an oxymoron.

0. https://www.researchgate.net/profile/Brad-Cox-2/publication/...

8n4vidtmkvmk• 3 years ago

funny enough i finally decided to add some unit tests to my project after 8 years because i didn't trust the code that the ai was generating. hah.

cwp• 3 years ago

I've found Copilot to be much better than GPT-4.

GPT seems really smart and you can give it very high-level prompts, which generate good quality code. But it's limited by the chat interface, and if it doesn't get it right the first time, it's tricky to get a tweaked version. For example, I asked it to generate some code for a login endpoint, and gave it all the details about language, libraries, database etc. It produced some pretty good code. But then I asked it to do a version didn't store passwords in plaintext. It did that, but also made other changes including using a different library than one I had specified. So then I had to try to fix that, which led to more weirdness. I ended up just using the code it generated as inspiration and writing it myself.

Copilot, with its integration into VSCode is quite different. It's *awesome*. It makes suggestions in inline, as you type. It seems to have a lot of context, and generates very good code, apparently based on information contained in other files in the same project. It matches style and naming conventions, and even when it's not quite right, I find it easy to just accept the suggestion, and tweak it. It's a huge help.

So my current practice is to rely on Copilot to generate code and use GPT as sort of a pairing partner that I can talk to about the code. It's pretty good at suggesting alternatives, developing ideas, explaining error messages and so on. This is a very, very early attempt at coding with the help of AI, and I haven't actually seen a productivity improvement, because I don't know how to use these tools well. But it's fun and exciting!

Turing_Machine• 3 years ago

I find it surprising that many developers find using ai prompts easier than an actual programming language (not the first time I hear about it, but now it seems to be serious).

I, too, was skeptical until...

I've making the switch to VS Code (not by choice...my previous daily driver, Atom, has been EOLed since December). Accordingly, I needed to port an Atom extension I wrote a long time ago to VS Code. What I didn't want to do was spend hours or days getting up to speed on how to write VS Code extensions, especially since this is the only one I'm ever likely to write (it's the only one I ever wrote for Atom :-)).

I used GPT-4 and it worked like a charm -- almost like magic. It wrote almost all the boilerplate and bullshittery for me. Was it perfect? No, the regex it came up with for recognizing the code pattern to which the extension applies (Scheme atoms or pairs) was crap. That's okay -- I know how to write regexes (in fact, I pretty much just dropped in my state machine parser from the old extension). But not having to spend hours and hours groveling through Microsoft docs and Googling to learn stuff I'm not likely to ever need again? Man, that was great.

Programming in a natural language and trusting the results I just can’t stand.

This would worry me, too. But in this case, I was already well-familiar with Javascript and Typescript, so it was no problem for me to read the resulting code and give it a sanity check, test it, etc. I just didn't have to write it.

neets• 3 years ago

You know you live in the future when

addmin123• 3 years ago

Is it using the copilot of microsoft?

mlejva11• 3 years ago

No it's not. Even the UX is different from copilot.

armchairhacker• 3 years ago

I would expect the opposite: you write the code and have the AI write documentation, tests, and examples. Especially documentation.

WalterSear• 3 years ago

It's good for both.

throwaway426079• 3 years ago

As an analyst, could I write an analysis plan and have this tool implement the plan in, say python pandas?

qaq• 3 years ago

So what prof is everyone is planning to switch to once this takes off?

skor• 3 years ago

ive tried gpt4 for simple things and ended up doing it myself. Simple regex for matching whether an url is a link to the root of the domain or is some page. Told it several times to try again and it failed.

How are you folks looking beyond something that is so simple and should be correct always to be useful? How does it save you time if any small detail may be a potential problem?

all27• 3 years ago

I suspect that regex is very much like mathematics for these models: they're just bad at it. If you provided the model failure cases for each failed attempt, you might get better output. I've also seen folks say that requiring reasoning makes answers more robust as well.

zaphar• 3 years ago

I am imagining the kind of documentation most developers write and I just can't see this actually working. Maybe it's just because I'm old.

Myrmornis• 3 years ago

The name implies it will always be restricted to English. Is that something you have an opinion about at this stage?

victorsup• 3 years ago

what's the difference between your product and MS VScode AI copilot?

meatjuice• 3 years ago

Isn’t it better to make lsp or something to provide ai code support for existing editors than building their own ide from scratch?

Show HN: We are building an open-source IDE powered by AI

Related Stories