chrisnolet

Worth checking out Sculptor from Imbue as well: https://imbue.com/sculptor-announce

I made a post about this the other day! I think it's a sliding scale – you can use agentic coding in a more-professional or less-professional way.

Post as follows:

—

I have a take on vibe-coding for professionals. TL;DR: If you’re not loving AI editors, you’re probably doing it wrong.

Vibe-coding isn’t just for mock-ups.

It pains me when I hear high-level SWEs say that LLMs are only good for projects ‘up until a point,’ before it all supposedly falls apart.

—

I use Cline. It’s the dark horse of AI code editing. It’s an open source VS Code plugin and you should absolutely try it.

The best part about Cline is that it forces you to think about PLANNING and ACTING as two distinct phases. When I meet people who don’t love AI coding, it’s usually because they stay in the equivalent of ACT mode the whole time.

(Other IDEs are capable of doing this, too, to varying degrees. Cline just makes it very explicit.)

If you’re not spending at least 4x the amount of time PLANNING vs. ACTING in your AI code editor, you’re probably doing it wrong!

The best engineers will spend more than 20 minutes planning before they unleash AI to make any changes whatsoever.

If you’re one-shotting absolutely everything, you’re absolutely doing it wrong. When you try to one-shot complex changes, the AI is going to mess them up. And it’s going to take forever for you to have the opportunity to intervene.

This is where people get really frustrated: ‘I have to wait forever, and then the code is completely wrong anyway.’

The better approach is to treat Cline like a thought-partner. ‘Let’s add this class.’ And then review the plan before it submits one line of code. It’ll say, ‘I’m thinking of doing it this way,’ or ‘I’ll use this library.’ It’s often wrong – but that’s okay, it’s a quick turn-around to nudge it back on track and nothing’s set in stone in PLAN mode!

That’s when you have the opportunity to coax the AI into doing something more salient. And after three or four back-and-forth exchanges, you’ll have a really great plan for how Cline should execute.

Then you’ll press ACT, and go make a coffee. Most of the time, when you come back, it’ll have finished with near-perfect edits across many files.

And that’s it! If you do that, you should be able to leverage your years of experience, and your AI coding partner will do all the busy work, while you’re thinking through edge cases and architecture.

*chef’s kiss*

Thanks for trying it! It's not open source, but it is a single-page app. You could download the source and run it locally.

The web app is calling Jina Reader under the hood. If you needed something a little different, it wouldn't be too hard to write a CLI tool to call Jina and achieve similar results.

It's cheap to host, so I'll be leaving this up for the foreseeable future.

Right – I had the same concern! I thought that the notifications were going to be fictitious.

It sounds like that’s not the intention, though. Might be a good idea to update the copy!

Update to add: https://youtube.com/shorts/D8-sdTm8bd4

Thanks for the nudge!

Thank you so much for this question, and for your thoughtful post below. It's really easy to put privacy and security to one side when you're launching a startup. And lots of users don't mind privacy when they're signing up for products. But it's something that's personally very close to my heart, and I put a tremendous effort into privacy and security because I knew I wouldn't be able to sleep at night if I cut any corners.

I worked at Apple for many years and their approach to privacy really left a mark on me. I strongly believe that preserving privacy is a moral obligation. (Especially when you're handling people's emails.)

Now, while the beta is running, when you log in to Pocket, there is a big blue switch above the fold under the title 'Privacy.' It says: 'Share recordings with our team.' If you leave it on, that's really helpful for me! But it does exactly what it says, and if you have anything sensitive you don't want to share with me, turn it off.

For your questions:

- The voice data is routed through Retell and the transcripts are passed to OpenAI's API.

- Sensitive data is retained by Retell for 10 minutes (when sharing is off).

- Sensitive data is retained by OpenAI for 30 days 'to identify abuse.'

I'm working with OpenAI to get Zero Data Retention. As it stands, their commitment has been that they will not use API input or output to train models. (I personally trust that commitment, but I understand the skepticism and if that's a deal-breaker for you.)

Retell is HIPAA-compliant and SOC 2 Type II certified. They've been great to work with.

- Regarding the privacy policy: 'User data obtained through third-party APIs (will not be used) to develop, improve, or train generalized AI and/or ML models.' This language was actually required by Google. The use of the word 'generalized' here is actually less specific; it's not AGI, but includes any kind of foundation model. There might be a point in the future where we can fine-tune one model per user with a LoRA, but I agree that the risk of PII leaking from a shared model is far too great.

- The company is a Delaware C-corp and subject to U.S. and California laws.

I really appreciate the opportunity to discuss this. I want to put privacy and security first always, and make sure that's baked into the company culture. Thanks for advocating!

Footnote to add that it works with Google Workspace accounts, too!

Oooh, thanks for the report! It looks like Firefox doesn't have dithering on gradients. (There's a bug report, but it's been opened for 14 years!)

The gradient animation is super-subtle :) Do you think I should disable it for Firefox users, or do you still think 'cool background animation' in spite of the banding?

It's using OpenAI's API at the moment, actually. An offline model could _probably_ handle the conversation and tool calling, but it just needs to be really fast to keep up with conversational speeds. (And really, GPT-4o is a bit too slow for my liking in this current iteration. I'm hoping that GPT-4.5 will be faster.)

I'm writing up a full accounting of the stack for the post above, so check back for that and let me know if that doesn't answer your questions/concerns!

ps. Check out https://pocket.computer/dungeons if you just want to try the original ‘D&D meets Siri’ voice stack. It doesn’t require a login.

It’s coming! The screenshot on the right of the homepage is a placeholder for the impending video. (It won’t be fancy, but I want to at least give people a sense of what they’re signing up for.)

Thanks for the note and for checking out the page!

Me, too! I have pocketcomputer.com as well, which I use for email. (I learned a long time ago how confusing it is to read out a special TLD over the phone!)

I had to pass the Google CASA audit and implement a ton of security procedures. Basically everything is encrypted, we don’t store your emails, verified best-practice for session tokens and so on. I probably went a little overboard to be honest, but it’s people’s emails and I need to respect the gravity of that.

Nice! I’d love to check it out when you’re ready to share it. I’ll likely release the source code for mine if/when I publish the DSL.

In the meantime, since the original link has changed, feel free to try it out at: https://pocket.computer/dungeons. Happy to chat more if you want to know how parts of it are done!

Thanks for trying it!

You can draft emails and reply to threads as well, actually! And if you're unsure of what to say, you can throw some hints at the agent and it'll generate a draft in your tone of voice. (The agent analyzes your past emails to match your style.)

For your question: The providers (Vapi, Retell) handle the big pieces well. My framework/DSL sits on top, helping developers manage the conversation in TypeScript.

Quick example... When you start your first session, we spin up a 'worker agent' to figure out your name, then we say something nice, and display a personalized welcome message:

  dataStore.userName = await aside(z.string(), this.emailText, `What is the user's first name?`);

  prompt(`Say something nice about ${dataStore.userName}.`);
  display(`Welcome to _Pocket_, ${dataStore.userName}.`);

It's very cool and I hope to share more in the future!

Oh cool, this is great! I love the aesthetic.

For this demo, the app architecture really depends on users sticking (more or less) to the scripted options – if they want to progress with the story. I’ve included something in the prompt to encourage that.

There are also some ‘hidden’ choices, though. For example, you can attack the merchant and the blacksmith. Those options aren’t enumerated by the GPT when it describes the scene, but they’re equally valid paths in the backend. (That gives me an opportunity to script some of the more popular transgressions.)

How did you set up Spellbound? Do you have one longer prompt, or did you split it up?

Hahah, alright Merlin! That is so far from the narrative arc I had planned, I don’t even know what to say, lol.

Do you ever tell players that they flat-out can’t do something? Do players realize when they’re going off-script? (Do you let them know with a sigh or a long stare, or do you just roll with it and encourage the playful creativity?)

I like your three-part planning a lot! That’s a great framework.

So firstly, I think there’s a sliding scale here, between structured narrative and hands-off emergent storytelling.

If we assume something in the middle:

While we might not be able to predict that you’ll encourage the farmer to (try to) overthrow the king, we might predict that you’ll try something of that sort. Maybe you enlist a guard for an inside job, or perhaps you do it yourself! But if we want to plan for it, we can plant the seeds, and then nudge the GPT to weave the threads of our story together when it happens.

Interestingly, because GPT requires so little prompting for relatively intricate story lines, (yes, really), you can probably add an ungodly number of these semi-scripted moments. I think they would be absolutely magical.

I think all of that would add up to a world that feels alive, with deep world building and many ‘surprising but inevitable’ moments – and the potential for some great emergent storytelling along the way.

I’m very excited for it!

Hahah, yep – I’ve had a few of those moments myself!

I think adding an option to change the voice is the #1 most frequent request that I’ve gotten. Time to dig through ElevenLabs and see what else I can find! :)

100%. Putting on flying shoes to escape any situation isn’t actually all that fun, (after the first time).

The trick is to figure out how to put appropriate guardrails and narrative structure on an otherwise too open-ended LLM. If you just leave GPT to its own devices, it allows everything and goes everywhere.

Wow, this is great! From a quick read through, the structure discussed in the paper is very similar: dialogue trees with fuzzy matching, and actions that materially alter the state and the available options going forward. I have to admit, my tree is a lot shallower right now, though!

(If you want to see what I mean: the alleyway option isn’t available until after you attack the merchant, for example.)

Sorry – the servers went down! They should be back online if you'd like to try again. Thanks for letting me know!

Sorry, hug of death! Trying to get it fixed now.

Thank you – short and sweet! But this actually means a lot. Thanks for playing!

Ahh, yes! I’d love to add an undo. (There is a rollback system on the backend, but I use it for when the AI gets ahead of itself and starts speaking too early. I could repurpose that for undo.) But otherwise, adding a text field shouldn’t be too hard.

You can interrupt and ask the AI to do what you wanted to do in the first place, also! Depending on what the action was, it will often just correct the dialogue and continue on.

Thanks for trying it!

Awesome, thanks for trying it out! This is great feedback.

I'd love to connect this up to Flux.1 and have auto-generated hero images at the top! And getting the sound right will be a huge part of it, since it's basically an audio-first experience. I'm wondering if it would work to change voices for the dialogue when you speak to different people in the world...

I've noted that save games are essential! Thanks for playing it through long enough to think about that :) I'm glad you enjoyed it enough to keep going!

Thanks! Some good insights here. The quest is based on A Wild Sheep Chase. I can relax the guardrails a little bit, but part of what is being shown is that the world is controllable by the developer. It's nascent, but this is a demonstration of an AI that is being nudged along a well-planned narrative.

It sounds like you might have hit a blocker by trying to move to a part of the town that isn't on the map. I've forbidden those actions for now, to make sure users don't go too far off the beaten path.

You should be able to attack the townsfolk, though, (although you may need to insist). I've just added a formal option for you to attack the merchant. There's a 50/50 chance that it succeeds at each round. After that, you can escape down the alleyway and the AI will occasionally begin to speak like a pirate.

This is just one proof-of-concept for one possible domain for controllable voice AI – which I think shows potential. I appreciate that you disagree, and that’s fine!

But I can absolutely see a world where people play AI video games because they want the unbounded complexity and subtlety of a TTRPG with a human-level DM, powered by AI. I hope to have the opportunity to convince you with another play through!

I think all of these modalities have their place. Sure, you can hang out with friends and play poker, sports or D&D IRL. But we still play video games.

Hey, glad you enjoyed it! And I agree – the sudden freedom to shape the story and really form relationships with the NPCs is mind-blowing. It's definitely a glimpse into the future of gaming and storytelling.

Also, having the text appear in sync with the voice is a great idea. I'll experiment and see what feels best, but even just having the words fade in one-by-one at a speaking rate could be good. Thanks for the suggestions!