BlenderGPT: Use commands in English to control Blender with OpenAI's GPT-4

by alexzeitler • 3 years ago

612 291 3 years ago

I know this tech is terrifying to actual 3D artist who don’t want be a “prompt engineer”, but as someone who has never used Blender, I think its cool that I can create something using tools like this and use them in my projects, ex background animation on a website hero section.

whitemary6• 3 years ago

Have you used it yet? Was it actually that easy? I can't wait to try.

jeron• 3 years ago

As someone who had tried my hand at Blender and failed miserably/given up years ago, I am so excited to use this to make up for my Blender skill deficiency

pazimzadeh• 3 years ago

Isn’t being a good boss kind of like being a good prompt engineer?

TeMPOraL7• 3 years ago

If so, then this means the good boss will soon no longer need employees, as the LLMs will deliver better results faster and cheaper.

warent4• 3 years ago

I don't think this is "terrifying" to 3D artists any more than github copilot is terrying to us haha, I'm not sure why we engineers have this tendency of imagining every other industry as being populated by superstitious peasants from the dark ages who fear new tools.

In my experience the vast majority of people are excited about GPT including artists.

SketchySeaBeast• 3 years ago

I think it's because if you're not in that particular industry you have a super simplified model of what the person does - something like "writing code" as the only activity a developer does.

You don't understand the difficulties and problems that people in the profession face, which I think is also why so many developers are convinced they can replace/"disrupt" other people's jobs with software.

TeMPOraL7• 3 years ago

I don't know, maybe I am one of those "superstitious peasants from the dark ages who fear new tools", but I'm increasingly terrified of GPT-4 and future iterations of it as applied to the industry I work with (i.e. software). It does seem to threaten to suddenly eliminate all the interesting parts of the job, a good chunk of openings, and to significantly reduce salaries for the openings that remain - all at the same time, and rather suddenly.

notatoad• 3 years ago

i'm guessing the parent comment is in reference to this link from yesterday: https://news.ycombinator.com/item?id=35308498

antibasilisk4• 3 years ago

I don't think this is "terrifying" to 3D artists any more than github copilot is terrying to us haha

It sounds more like you're just seriously underestimating the direction GitHub Copilot is headed.

adelie• 3 years ago

my main issue is the people making hiring and budget decisions aren't 3D artists, they're managers. my work might be objectively better than GPT's, but is that going to matter when my boss compares the cost difference and decides GPT is 'good enough'?

Naga• 3 years ago

I was speaking recently to a graphic artist who is terrified of Stable Diffusion and the like. I mentioned that these tools can augment their ability to do work instead of just replacing, but their point is that they are a graphic artist because they like doing the things that the AI will be replacing. Being a prompt engineer wasn't really the reason they studied and learned to become an artist. To me, that is a completely reasonable way to feel.

anonzzzies2• 3 years ago

Just leaving this here;

https://www.reddit.com/r/blender/comments/121lhfq/i_lost_eve...

Edit; I see it was discussed indeed on HN yesterday. I have already seen tons more of these in my circles and not only 3d/artists.

Thorentis3• 3 years ago

It just raises the bar so that 3D artists who are only capable of reproducing the same boring things they saw in YouTube tutorials are no longer considered proficient.

The more I think about this trend, the more I think it might be good.

Bootcamp devs are no longer good enough for junior roles since a GPT could replace them. Digital media people who learnt via YouTube and have no real talent are no longer skilled enough. Writers who can only churn out mediocre blog spam are now jobless.

This seems like it might be a net benefit.

smilespray• 3 years ago

How do you get senior people when juniors no longer have careers?

kypro2• 3 years ago

This isn't how it will play out though. At first chess AIs got enough to beat unskilled players. They they were good enough to beat intermediate players. Then they challenged the grandmasters. Today no human, no matter how long they practise or how hard they train will ever get close to beating an AI at chess.

Your assumption that this raises the bar for human 3D artists is correct today, but it won't be long before human 3D artists are seen as much slower and less competent than AI artists, and there will be no going back.

antibasilisk4• 3 years ago

It just raises the bar so that 3D artists who are only capable of reproducing the same boring things they saw in YouTube tutorials are no longer considered proficient.

That is of course, until GPT-6 surpasses them.

ksrm• 3 years ago

It is good when people lose their jobs, actually!

acapybara• 3 years ago

I don't really care how or where someone learned something, so long as they learned it well and can apply it.

My experience in the software industry (20 years now) showed me that the best ones were the ones who got into it out of genuine interest. They tended to write software as a hobby.

There was no shortage of CS grads who couldn't be nearly as productive.

The self-taught ones, or the ones with genuine interest who also completed a degree program were the best.

I wouldn't discriminate against "boot camp coders" or people who learn things from YouTube.

There's a lot of people who live in a different world where an expensive college/university education is not an option.

boppo1• 3 years ago

I use blender a lot. I know what the api can do. I'm not terrified. Getting anything significant done will still take real work/knowledge because the number of potential 'parameters' of what you can do/make requires that you have the language to describe what you want. Having the posession of that language and knowing how to use blender is a very positive correlation.

greenknight• 3 years ago

Imagine being able to sketch out the outline of what you are after, then say hey blender model it up. Thats what it will get to.

You have a beach, and you want a jetty, just grease pencil approximately where you want it and go Hey thats a jetty build it. and if its not quite right, generate me 50 different versions and ill pick the best.

DrewADesign7• 3 years ago

Most people aren't worried about being a prompt engineer-- they're worried about being a Doordash driver if it turns out that one prompt engineer can replace many professional, well-paid people in their industry. That's pretty rational.

TeMPOraL7• 3 years ago

being a Doordash driver

If that job will even be available. The other day my sister sent me a photo of a little food delivery robot she spotted on the streets[0], and mind you, we're not living in Silicon Valley, but in Poland.

[0] https://www.deliverycouple.com/ - based on the markings on that robot, its these people.

anonzzzies2• 3 years ago

There is not much to prompt engineer; the obvious trivial thing that is already happening and will make the glorious job of prompt ‘engineer’ completely redundant is AI doing the ‘engineering’ for you. One person (or AI in the future; this is already possible now for humans and gpt though) that the AI has been trained to ‘know’ and work with will blurb things into a mic and the AI will instruct millions of brains to whatever the ‘boss’ wants, translating his mumbling into actionable user and system prompts. There won’t be prompt ‘engineers’ for long.

tekknik10• 3 years ago

Self driving cars and drones are pretty close, door dash driver won’t be an option much longer either.

raincole• 3 years ago

I know this tech is terrifying to actual 3D artist who don’t want be a “prompt engineer”

My predication: in the next 5~10 years, most artists won't be "prompt engineers". Instead they'll focus on fix small details on AI-generated art.

It's still kinda sad tho, because it's usually the most tedious and boring part of the process. Now AI is taking the fun part and leaving the unfun part to humans.

ibejoeb2• 3 years ago

Yeah, but it's kinda like making business software. Most working people need to do boring work to sustain their lifestyles.

I hope that these task-specific implementations of AI can reduce the tedium in these fields, like the way PCs did. Certainly, this advancement is leading to the ability for practically anyone to program a computer, in the general sense. Things will shift, but there will be opportunities to exploit those abilities for personal gain.

cyrialize• 3 years ago

This is exactly what this post from Reddit posted to HN [0] was complaining about.

I think many people hear and see these complaints and think that people are being luddites or being afraid of losing their job. People should be looking at it for what it is - someone complaining that their entire job is changing into something they no longer enjoy.

0: https://news.ycombinator.com/item?id=35308498

elif2• 3 years ago

I asked my friend (renowned vfx artist) what he thought about AI in his field and he mentioned some niche cases where it was useful for blending models with film etc, but overall seemed uninterested.

He offered the explanation that so much of their time is consumed with nit picking through purely aesthetic decisions that AI would not be capable of the artistic reasoning required to produce work that could even get to the "pass or reject" stage.

Tanoc• 3 years ago

This is terrifying to us, but not in the way you think. Tools like this place selective pressure so that increasingly only the best of the best get paid to make stuff, and their stuff influences the data sets in the future. The rest of us get forced out simply due to cost unless we're doing it as a hobby. Some artists know all too well their limits, and those that do are the most vulnerable to things like this, because these learning machines use the ever higher ceiling of human ability to determine the floor of their ability. You can't go back to the funky weird pasta men of Stable Diffusion's 2021 iteration as the default for example, but someone who started painting in 2002 might still have the same level of skill now as they did in 2010.

Text was paralleled first. Then sound was paralleled next. And now image is being paralleled. It will be on level with highest percentile of human ability just as text and sound were before it. Game devs and comic artists are already replacing texture and background artists with AI generated images, just as they used AI to create hundreds of thousands of lines of fluff dialogue before then.

post-it• 3 years ago

Did you run into issues that made you tweak the prompt? You often tell the tool not to do extra work despite having it in the prompt, did you find that it often tries to do a lot of "boilerplate" work like setting up lights and cameras?

  system_prompt = """You are an assistant made for the purposes of helping the user with Blender, the 3D software. 
  - Respond with your answers in markdown (```). 
  - Preferably import entire modules instead of bits. 
  - Do not perform destructive operations on the meshes. 
  - Do not use cap_ends. Do not do more than what is asked (setting up render settings, adding cameras, etc)
  - Do not respond with anything that is not Python code.

nr2x2• 3 years ago

Usually it tries to explain code, hence the last one.

jabza• 3 years ago

This idea is awesome

sircastor2• 3 years ago

Until We started to see LLMs, and the tools that can be created with them, I doubted the possibility of Star Trek's Voice command system. Asking for the computer to clarify some concept, or filter and reduce data sets based on arbitrary data was pure science fiction.

Seeing something like this makes me think that the arbitrary holodeck commands "Paris, 1950's, rainy afternoon" is suddenly not a challenging part of the equation. It's really exciting.

stevenhuang• 3 years ago

A prompt like that already can generate something great using stable diffusion/mid journey. Very exciting indeed that LLMs are now similarly so capable.

RobotToaster• 3 years ago

Using stable diffusion suddenly makes Picard telling the computer his tea has to be hot every time seem reasonable.

Nition2• 3 years ago

Here's an image result from MidJourney for "Paris, 1950s, rainy afternoon". No additional editing, and I intentionally avoided adding any text to the prompt beyond your own.

https://i.imgur.com/IYuh29H.png

Not perfect but man, we're getting pretty close.

purplecats• 3 years ago

interesting. i apparently opened that image, then forgot about it, and without context saw it later. i looked at it, thought wow that's a cool picture from back in the day, looked at the people in it, and left.

i now ran into your comment (with a purple link) and did some reflection. upon reexamination, its clear that the picture is fake (because im looking for it) but when i wasn't looking for it, its interesting how all the "hot spots" or interesting pieces of the picture are pretty good and the (imo) lackluster parts are the "less interesting" pieces like the end of the roads where it it blurs out. i wonder if that bias is inherently ingrained in the system.

wrycoder• 3 years ago

These systems sure hallucinate text.

aryamaan• 3 years ago

midjourney-- a window in the past and future

galgot• 3 years ago

It's a nice image, but I find it's easy to spot IA generated images when trying to make images of very specific existing hardware. Here you can see all the cars are generic with 50's design look, none are models that existed. Try to ask an IA to draw you a Boeing 747-400 for example, see what I mean. Btw, Have you noticed all the Youtube vids thumbnails made by IA now ? Easy to spot.

wefarrell• 3 years ago

When Siri, Alexa, and Google home came out I was convinced voice would be the next paradigm shift in human computer interactions, comparable to mobile, but the voice assistants fell short and I was disappointed.

Now it's clear that the shift is coming and it will revolutionize the way we interface with machines.

MattRix3• 3 years ago

The image AIs are much more capable too, and I find it interesting that every technically inclined person used to always make fun of those “enhance” moments in TV shows and movies where they would zoom into some area of a photograph or security footage. The fact that now this is actually possible (to some extent at least) is pretty wild.

sircastor2• 3 years ago

On a serious note, this is where these models get dangerous - when someone "zooms in" on the technology, finds something the computer created from nothing, and then takes that as irrefutable fact.

adhesive_wombat• 3 years ago

Once the Solidworks/Altium plugins are working, we're going to see some some crazy things.

Also FreeCAD/KiCad, I guess, if the resources are there, but Dassault and similar have the ability to bring to bear a lot of FTEs very quickly if they wish.

endisneigh• 3 years ago

this is cool, but hm, how should I put it...

Ultimately there's a relationship between the preciseness in which you want to control something and the underlying information, as conveyed in language, to describe such precision.

Whether you use plain english, or code - ultimately to do things of sufficient precision you will have to be equally precise in your description. I'm sure someone with more time and more knowledge on such things have already formalized this in some information theory paper, but...

The point I'm making here is that this is great because a lot of people are doing "simple" things, and now they will be able to do those things without understanding the idiosyncrasies of Blender APIs, but I'm convinced that this will ultimately turn into something equally difficult as blender APIs to do novel things. and WHEN (not if) that happens, I hope users are prepared to learn the Blender APIs, because it will be inevitable.

edit:

one other thought. I think "language models" are not the right solution ultimately. I think kind of like AI didn't boom until the proper compute was available even though theoretical models and algorithms existed, language models are the crud solution.

once we have a loseless way to simply "think" what we want, then a "large thought model" will have less trouble, as there will be less ambiguity in what you want to what is said.

right now it's thought -> language -> model.

later it will be thought -> model.

capableweb3• 3 years ago

It could also be that something like this is just another tool in the toolbox. Sure, you could spend time trying to understand Blender's Python API, but lots of Blender users are not programmers. It could be really helpful for them to be able to say "Place 10 spot lights at random position within the boundaries of Mesh 3 with random intensity and color" and just have that appear, rather than having to go looking for a plugin that does it for them.

lwhi2• 3 years ago

What's the benefit in not being specific, aside from it being easier to request?

therouwboat• 3 years ago

I dont know why people make this harder than it needs to be, make group of lights, change colors and duplicate, its good enough, no need to use python api.

crooked-v• 3 years ago

Or in other words, the "create an adversary capable of defeating Data" problem... just a bit less of an issue (for now) if you get the instructions wrong.

ericd3• 3 years ago

Well, you get precision with imprecise communication via iteration. You walk together toward the correct mutual understanding by successive clarifications.

nemo44x• 3 years ago

There’s also a parameter you can set called “temperature”. This is how you control the free-form aspect of the model. Set it close to 1 and it will bullshit a lot, or rather it will take bigger chances in filling in outside of context and maybe a bit more hyperactive. A value closer to 0 keeps it tight by taking less risk off the context, but can also miss out a bit and loses color.

roflyear5• 3 years ago

Sure but GPT certainly cannot actually do this. It can iterate, but in GPT-world that means "give me another response" not "learn from what i'm saying, and give me another response"

The issue is, sometimes those two things seem to be the same thing!

samstave• 3 years ago

"Zoom Enhance!" -- Hear me out:

One interesting thing would be to describe a scene and get the rough print. But do it in sections, such that you can select and begin to refine sections and elements within the scene to whittle down to the preciseness that pleases you for each element...

What would be really interesting will be how geometry nodes can be managed using gpt.

skybrian2• 3 years ago

Chatbots use random number generators to create a variety of output, so it would be a terrible idea to use natural language as "source code" for a code generation tool. Running a chatbot shouldn't happen within a build process, because the results aren't reproducible.

Once you have the source code though, you can use a variety of tools to manipulate it and save the result. Using chatbots to make modifications under supervision is fine. You discard bad modifications and save the good ones.

This is using natural language for one-offs and source code for reproducible results. It's looking like they will go well together.

FractalHQ2• 3 years ago

Can you not provide a seed to an LLM the way you can with some of the image / video generation models I’ve seen?

guyomes• 3 years ago

If the command is not restricted to text only, but use text and geometrical context, you can remove a lot of ambiguities. This is often done in video games with contextual interactions.

After all, using the Blender GUI, you can do a lot using only a 2D mouse coordinate and two boutons. So 2D mouse coordinates and text could be better.

A nice evolution would be an AI model that can understand natural language instructions, while taking into account where your mouse pointer, how the model is zoomed and oriented, and that has geometric insight of the 3D scene built so far.

sharemywin• 3 years ago

With a shared vector database and semantic search plugin you can see bunch of prompts other people have created.

with a search plugin you can have it find the api docs and have it output interesting parameters and how to use them with examples.

with a python REPL plugin you can have it generate 10 variations and run the code for each.

with GPT4 and plugins you could describe the output you want to midjourney or something and give the prompt to it to generate it in blender(feed the outputs to some image similarity vector to compare) and have it search through parameter space(or vector space of the prompt) until it finds something pretty close to what you want.

given your budget of course.

mccoyb• 3 years ago

Can this same idea be extended to, say, interact with an open-source SVG editor like Inkscape? What are the requirements of the editor — I presume it must support some form of scripting?

I would love to be able to have GPT sketch math figures, which I then modify/perfect.

Note: this comment is partially inspired by the workflow of Gilles Castel — I’d love to be able to use GPT in the loop of note taking, similar to the system that Gilles setup to improve sketching speed.

capableweb3• 3 years ago

Yes, basically it can be extended to anything that allows scripting, which Inkscape does via "extensions". Someone will surely come up with something similar as this project but for Inkscape in no time :)

meowkit• 3 years ago

MSFT Research generated some SVG images with GPT4. Kinda mind blowing among some of the many experiments they published in here:

https://arxiv.org/pdf/2303.12712.pdf

1attice• 3 years ago

I'm getting a callus from my jaw being on the floor so much this month

ChancyChance3• 3 years ago

I'm glad it is not just me who thinks I need to "thank" chatGPT or encourage it, e.g.: "Great, now do .... " "Great, now do ... ".

aiappreciator• 3 years ago

Its not 'thanking', its positive signal that GPT's previous outputs were correct, so it should continue doing whatever it was doing. If you say no/bad etc, then GPT will try other approaches.

csmpltn2• 3 years ago

"its positive signal that GPT's previous outputs were correct"

Is that somehow baked into the algorithms?

Are positive words of encouragement interpreted as "positive signals" by the inference pipeline? Or do they somehow influence the attention mechanism?

Because otherwise, you're just rationalizing completely random and unpredicted behavior.

ChancyChance3• 3 years ago

That's literally what I said: those words are "encouraging" it.

cwkoss4• 3 years ago

OpenAI surely retains conversation logs.

When super intelligent AI gains power, I want it to know I've been a good boy.

quchen• 3 years ago

Pascal’s Wager, AI edition? :-)

https://en.wikipedia.org/wiki/Pascal%27s_wager

runevault• 3 years ago

Going to be curious where this goes. As someone who hobby models I get the impression there are certain things modelers would LOVE to be able to just hand to an AI (feel like I saw someone on twitter saying they'd love for it to handle UV unwrapping but I may be thinking of the wrong task).

Any time we can get a program to do the repetitive work that is time consuming but not interesting/creative that is a huge win.

whitemary6• 3 years ago

Back when I was a 3D modeler, I loved UV unwrapping. It's such a zen activity. It allowed my brain to recharge for some reason.

andsoitis• 3 years ago

for navigation, Language is to Mouse as Mouse is to Keyboard

EamonnMR• 3 years ago

"The generated code might not always be correct. In that case, run it again lmao" is the best documentation I've read all week.

arecurrence• 3 years ago

Copilot has an option (maybe beta users only) to see alternative generations for a prompt. It's really handy because some generations are one-liners while others are entire functions.

Perhaps this is infeasible with the current cost per generation but a multiple choice display could be handy (perhaps split the screen into quadrants and pick the one that most fits what you prefer).

Veen• 3 years ago

Google Bard has a similar feature, and you can ask the OpenAI API to generate multiple completions and pick the best one (best_of) or return some number of the generated completions (n). In OpenAI's case, best of means "the one with the highest log probability per token".

junon3• 3 years ago

If you watch the video (I recommend at 0.5x speed) the author adds "you feel me?" to one of the prompts and I laughed.

Spivak• 3 years ago

Hey everyone makes mistakes when coding, even LLms.

There’s two approaches I use, I’m sure there are more.

* Do multiple completions and, filter the ones that successfully run, and take the most common result.

* Do a completion, if it fails, ask the Llm to find and correct the bug.

chaxor2• 3 years ago

This sounds like reducing the temperature with more steps

carlsborg• 3 years ago

From authors website: "I graduated high school in May of 2022..". Elsewhere, he writes he placed first at a university level drone programming challenge at IIT-Bombay Techfest. Very impressive.

quonn3• 3 years ago

The JEE exam in India is very advanced, I doubt most Europeans or US students could pass it, even after a year or two at university. So those who enter IIT are already quite good very early.

newhouseb• 3 years ago

We also use GPT to perform actions in the software I build at work and we hit the same issue of inconsistency which lead me down a long rabbit hole to see if I could force an LLM to only emit grammatically correct output that follows a bespoke DSL (a DSL ideally safer and more precise than just eval'ing random AI-produced Python).

I just finished writing up a long post [1] that describes how this can work on local models. It's a bit tricky to do via API efficiently, but hopefully OpenAI will give us the primitives one day to appropriately steer these models to do the right things (tm).

[1] https://github.com/newhouseb/clownfish

elif2• 3 years ago

I've been building a ChatGPT project this week and this explanation is so true it made me actually lol. Sometimes it's impressively flawless, sometimes it gets stuck in broken patterns... Even if the inputs are identical.

"Monkey around with the prompt and pray" has replaced unit testing.

OOPMan• 3 years ago

Ooof

MilStdJunkie• 3 years ago

Exactly my experience. I've asked for some time metrics to see if the repeat is actually taking longer than the work.

csmpltn2• 3 years ago

Why aren't they collecting feedback signals from people on the results produced by the model? "This was correct", "this was garbage", etc.

reportgunner• 3 years ago

What if someone was asking to generate garbage in Blender though?

JoeDaDude• 3 years ago

Famous quote: "Wouldn't it be nice if our machines were smart enough to allow programming in natural language?". Well, natural languages are most suitable for their original purposes, viz. to be ambiguous in, to tell jokes in and to make love in, but most unsuitable for any form of even mildly sophisticated precision. And if you don't believe that, either try to read a modern legal document and you will immediately see how the need for precision has created a most unnatural language, called "legalese", or try to read one of Euclid's original verbal proofs (preferably in Greek). That should cure you, and should make you realize that formalisms have not been introduced to make things difficult, but to make things possible. And if, after that, you still believe that we express ourselves most easily in our native tongues, you will be sentenced to the reading of five student essays. - Dijkstra From EWD952

https://www.cs.utexas.edu/~EWD/transcriptions/EWD09xx/EWD952...

john2x• 3 years ago

We need a DSL that converts/compiles into natural language for the AI.

westoncb4• 3 years ago

In (language model-backed) natural language interfaces the loss of precision can be made up via iteration. If it were about getting it right on the first try this would be a dead end but there's no need for that restriction.

roflyear5• 3 years ago

Yeah maybe! Just like if you have enough monkeys iterating on enough keyboards...

btown• 3 years ago

And specifically, if the agent implementing the requested actions has a mental model of the domain that is similar to the mental model held by the imprecise specification writer, then the specification doesn't need to be precise to have a high probability of being implemented right the first time! The miracle here is that LLMs don't even need fine-tuning to get to that level - and that's unprecedented for non-human agents.

ZephyrBlu• 3 years ago

Legibility is generally a requirement though. At least is currently is.

SanderNL6• 3 years ago

What makes us think that encoding all functionality using natural language is somehow more compact than using code? Sure, describing a website with a button is easy, but I don’t quite see how you would describe Kubernetes or a device driver without writing a book in “legalese”.

Or try formulating a math proof with natural language.

jxf2• 3 years ago

While I agree, I don't think this is what Dijkstra was speaking about. A lot of the "AI interfaces" are just underpinning the real technical representation. For example, if you say "Which customers recently bought product #721?" this might generate a SQL query like "select * from customers where ...".

The interface is not the ChatGPT text box; it's SQL. The ChatGPT text box is just an assistant to help you do the correct thing (or at least, that's the way it should be used).

lwhi2• 3 years ago

Your example isn't very comparible to tasks involved in creating, texturing, lighting, animating and rendering 3D models using spline and mesh topologies.

whateveracct• 3 years ago

I really feel this. Whenever I try to use ChatGPT, I feel both slow and strained. My usual workflow involves me thinking very few English sentences in my head - I move faster thanks to abstract thought, and I don't think I'm alone.

whitemary6• 3 years ago

Same. Somewhat ironically, my art school vocabulary has come in very handy when writing art generator prompts.

TheDong• 3 years ago

On the other hand, we already specify programs in natural language, and in this case chatGPT is really taking a "specification language" as input.

When a client wants a button on a webpage, they don't send the web designer a legaleze document describing the dimensions of the button. They usually don't even tell the designer what font to use.

The web designer pattern matches the client's english request to the dozens of websites they've built, similar buttons they've seen and used, and then either asks for clarification, or specifies it clearly to the machine in a more specific way.

Is that different from the chatGPT flow?

Honestly, we also already mostly use english for programming too, not just design. Most of programming now is glueing together libraries, and libraries don't provide a formal logical specification of how each function works. No, they provide english documentation saying something like "http.get(url) returns an httpresponse object or an error". That's far from an actual mathematical specification of how it works, but the plain english definition is enough that most programmers won't ever look at the implementation, the actually correct specification, because the english docs are fine.

viraptor• 3 years ago

Is that different from the chatGPT flow?

The designer knows the context of the question, the website, the previous meetings about the design styles, possibly information about the visitor demographics and goals, knows the implicit rules about approvals and company hierarchy, knows the toolset used, the project conventions, the previous issues, the test procedures, etc.

The equivalent of telling a designer where you want a new button would be equivalent to feeding a small book of the implicit context into ChatGPT and without access to visual feedback you could still end up with an off-screen button that passes all the tests and doesn't do anything. The "fun" part is that for simple tasks 90% of the time it will work every time - then it will do something completely stupid.

they don't send the web designer a legaleze

That's the implicit context. (And yeah, bad assumptions about what both sides agree on causes problems for people too)

Also, chatgpt will try to make you happy. You want a green button here? You'll get a green button here. A designer instead will tell you it's a terrible idea and breaks accessibility.

einhverfr• 3 years ago

So, really, all we need then is a language sufficiently precise enough to specify what a program needs to do and we can then feed it into a program which can write software that implements that specification, possibly adopting safe transformations of it into equivalent forms.

Now, that safely describes a modern, optimizing C compiler.....

courseofaction• 3 years ago

fantastic quote, thank you

georgelyon• 3 years ago

I’ve brought this up in another thread, but we already have evidence that in certain cases it is desirable to trade off specificity for just “getting something done”, that evidence is Python. Python succeeds despite not having a serious type system, because often times you can get something that achieves your goal despite the 25 bugs in your program that would have been caught by a real type system.

On the other side of the coin, there’s C++, which is usually doing the heavy lifting underneath the underspecified-but-sufficient Python code.

My guess is that as LLMs evolve, they will more naturally fill this niche and you will have high-level, underspecified “code” (prompts) that then glued together more formal libraries (like OpenAI’s plugins).

ehnto• 3 years ago

I think there exists room for an intermediate layer, a framework for AI to write code in specifically. With larger "target" code blocks it can stitch together. Code is designed around humans to allow them to write specific, abritrary and precise logic with a lot of minuate, but LLMs might be better off thinking in larger business logic blocks that are testable and "sanctioned" by the framework makers for the specific problem domain.

Pxtl• 3 years ago

Not only that, but one of the big successes of chat AIs is their conversation. When a program does something wrong, it's hard to figure out what you said wrong.

When a char AI misunderstood you, it's often quite easy to explain where the misunderstanding happened and the AI will correct itself.

staplerr• 3 years ago

Just look at JavaScript. Types don't matter so much, look at the modern web.

thomastjeffery• 3 years ago

Yes and no. You've provided an excellent introduction to the problem space, but I think natural language has a larger role in formalisms than you might expect.

The most familiar formal language grammars to most people here are programming languages. The difference between them and natural language has been categorized as the difference between "context-free grammar" and "context-dependent grammar".

The most popular context-free language is mathematics. The language of math provides an excellent grammar for expressing logical relationships. You can take an equation, write it in math, and transform it into a different equivalent representation. But why? Arithmetic. The Pythagorean Theorem would be wholly inconsequential if we didn't have an interest in calculating triangles. The application of math exists outside the grammar itself. This is why you, and everyone else here, grew up with story problems in math class.

Similarly, programming languages provide excellent utility for describing explicit computational behavior. What they are missing is the reason why that behavior should exist at all. Programs are surrounded by moats of incompatible context: it takes explicit design to coordinate them together.

If we can be explicit about the context in which a formalism exists, we could eliminate the need for ambiguity. With that work done, the incompatibility between software could be factored out. We could be precise about what we mean, and clear about what we infer. We could factor out all semantic arguments, and all logically fallacious positions. We could make empathy itself into software. That is the dream of Natural Language Processing.

I think that dream is achievable, but certainly not through implicit text models (LLMs). We need an explicit symbolic approach like parsing. If you're interested, I have been chewing on an idea that might work.

xvector• 3 years ago

Similarly, programming languages provide excellent utility for describing explicit computational behavior. What they are missing is the reason why that behavior should exist at all. Programs are surrounded by moats of incompatible context: it takes explicit design to coordinate them together.

Such a well written reply! This puts into words a lot of my thoughts around programming today and how NLP can help.

keithalewis• 3 years ago

Visual Basic is so easy secretaries will be able to write programs.

Sometimes smart people say dumb things and it takes a while to figure out they are wrong.

sillysaurusx2• 3 years ago

Meanwhile, Star Trek has been using natural language for a few decades now, and it works pretty well.

Considering they also predicted iPads, we might want to take it at face value.

swyx2• 3 years ago

star trek is also fictional where things happen as the writers wish it to… the real world is not subject to such restrictions

hiAndrewQuinn• 3 years ago

"Wouldn't it be nice?" "But it isn't so."

An is-versus-ought style mistake.

6gvONxR4sf7o• 3 years ago

If you want something unambiguous in a single pass, sure. But interaction with some back and forth can dig into the ambiguities that matter quickly, ignoring those that don’t. Dialog is a whole separate model. So maybe natural language is a terrible specification language but a great interaction language.

ricochet11• 3 years ago

Similar vibes:

"The Sketchpad system makes it possible for a man and a computer to converse rapidly through the medium of line drawings. Heretofore, most interaction between men and computers has been slowed down by the need to reduce all communication to written statements that can be typed" - Sutherland

Llamamoe• 3 years ago

Natural language doesn't NEED to be precise- that's what theory of mind is for, to allow efficient inference of meaning from objectively-ambiguous communication.

It's not perfect, but LLMs appear capable of making the same kind of inference, sufficiently that it'll inevitably be possible to program in natural language sooner or later with minimal or no manual double-checking

junon3• 3 years ago

Assuming this requires a premium subscription to chatgpt, this might actually push me to sign up.

Never thought I'd see the day I'd pay OpenAI a cent given that I don't really agree with their level of "open"ness.

kypro2• 3 years ago

I actually took a look at the pricing after seeing this. I'm a complete blender noob, but I find myself using it every now and then. Something like this would be great for a lot of the stuff I want it for.

darkvertex• 3 years ago

You don't need ChatGPT Plus, but you do need to get an API key and give them your credit card so they can bill you at the end of the month. It's very cheap though! $1 is worth about 375 thousand words.

dragonwriter• 3 years ago

GPT-4 API costs $0.03-$0.12/1000 tokens (the top for prompt tokens on the 8k context option, the top for response tokens on the 32k context option, and response on 8k and prompt on 32k are $0.06/1000), so best case about 33,000 tokens/$. tokens:words is about 4:3, so that’s about 25,000 words/$, not 375,000 words/$.

(Practically, you aren’t going to get quite the ideal even on the 8k context option because you are going to use at least some response tokens on each request, though I would imagine text -> software controls can be optimized so that the response is fairly token-efficient in most cases.)

Also, GPT-4 API access is in limited-access with a waitlist that is advertised as prioritized based on submitting AI evaluation cases to OpenAI’s repository.

barking_biscuit2• 3 years ago

I don't really agree with their level of "open"ness

I see this complaint a lot, but after watching a lot of interviews with the founders, I think they have by and large taken the right approach. They're trying to drip feed things at a rate that allows it to be digested enough by all parties before releasing more. They actually appear to be quite principled thinkers on this topic.

junon3• 3 years ago

I think the "holier than thou" approach is rather narcissistic personally. Their rationale has basically been that the world isn't ready, which is rather ridiculous to me.

abledon• 3 years ago

I'm just thinking of that reddit post about the 3d artist who is demoralized about his small indie company and him being reduced to just inputting prompts all day. Now its spreading to the animators and Explosion/CreatureEffects artists.

Studios like Wetta Digital / DoubleNegative etc.. are gonna pounce on this

nstart• 3 years ago

Do you have a link to this post by any chance?

avtar• 3 years ago

https://news.ycombinator.com/item?id=35308498

eggsmediumrare• 3 years ago

I've been watching the behind the scenes stuff from the LotR extended editions. The Weta Workshop people were true craftsmen and women, dedicated and creative and really inspiring. Now I'm imagining two people just prompting a machine over and over and to be honest it's a really sad vision of a sad, boring future. Give me hippies carving Treebeard out of polystyrene over AI generated digital perfection any day.

quickthrower2• 3 years ago

This is something that a year ago, if you saw it in a movie, your eyes would roll.

waynenilsen2• 3 years ago

I think real CSI "enhance" is coming also although i'm sure all kinds of details will be imagined and interpolated

dagw• 3 years ago

Yea, that is the real worry with this technology. You'll be able to zoom in on a blurry image of speeding car taken at night and get a perfectly clear license plate, but there is no way to guarantee that it is the correct license plate that was actually photographed. It'll just be the most 'probable' license plate based on some inscrutable probability distribution that is locked inside black box that even the developer of the software cannot explain. I fear explaining that to a jury could be really hard.

d23• 3 years ago

Can we get someone in the comments who has used it and can speak thoughtfully on how well it works?

34679• 3 years ago

I have Blender installed, but haven't taken the time to figure out how to use it. This is how BlenderGPT went for me:

I started with a scene that had a camera and a cube. I told it to make the cube red. Result: "Error executing generated code.."

I told it to delete the cube. It failed again.

I deleted the cube and told it to make a cube at origin. It made a cube.

I told it to make the cube bigger. It made the cube bigger.

I told it to make the cube red. Fail.

I told it to make an animation of a spinning cube. Fail.

I told it to make a car. It made a rectangular cube with 4 cylinders, somewhat resembling a toy car made of wood blocks, laying on its side.

I told it to turn the car upright and make it more aerodynamic. It failed and I returned here to HN to see if anyone else had some advice.

mrleinad• 3 years ago

How long until I can talk to Alexa or Google in natural language? Or is it already possible?

felipelalli• 3 years ago

"Limitations: The generated code might not always be correct. In that case, run it again lmao."

LOL

nr2x2• 3 years ago

I’ve been using ChatGPT to general SQL and it’s either amazing or totally wrong. But that’s on 3.5, I’m fairly sure 4 would be nearly fully reliable.

gandalfgeek• 3 years ago

Looking at the underlying prompt [1], it is a one-shot prompt, i.e. it contains one example of a natural language task along with the corresponding Python code (plus a "system" prompt for overall context and direction). Amazing how much you can do with a simple prompt.

Imagine Jupyter notebooks with this capability. Or Photoshop. Or Davinci Resolve. We live in amazing times.

[1] https://github.com/gd3kr/BlenderGPT/blob/main/__init__.py

uptownfunk• 3 years ago

I’m waiting for SiriGPT, and frankly I’ve wondered why its taken Apple so long to figure out how to do even the most basic of voice powered commands from my iPhone.

throwawayForMe2• 3 years ago

I don’t think this is what you are looking for, but there are many IOS shortcuts that front end the OpenAI api with Siri, using Siri for voice input and output. You can even ask ChatGPT how to create a shortcut like that.

sacnoradhq• 3 years ago

"Computer: tea, earl gray, hot."

We're nearing the precipice of more natural human-computer interaction that will need to rethink the interfaces and conventions.

Alexa and Siri seem like Model T Fords when there's a jet aircraft flying overhead. I'm thinking these agents need to be replaced by more natural agents who can co-create a language with their human counterparts rather than relying on fixed, awkward, and sometimes unhelpful commands. It would behoove us to expose APIs and permissions delegation in a more consistent and self-describing (OpenAPI + OAuth / SAML possibly) manner for all possible services one would wish to grant to an agent. If a natural language agent is uncertain, it should ask for clarification. And on results, it is necessary to capture ever-more-precise feedback from users because positive and negative prompts aren't good enough.

Lio• 3 years ago

“The way it functioned was very interesting. When the Drink button was pressed it made an instant but highly detailed examination of the subject's taste buds, a spectroscopic analysis of the subject's metabolism and then sent tiny experimental signals down the neural pathways to the taste centers of the subject's brain to see what was likely to go down well. However, no one knew quite why it did this because it invariably delivered a cupful of liquid that was almost, but not quite, entirely unlike tea.”

I think Douglas Adams was closer to the truth on the subject of AIs and tea. I don't want to be overly cynical but I suspect we'll just get used to saying "OK, close enough" when dealing with LLMs.

raylad• 3 years ago

This is getting closer to the system I've been saying I wanted since the 1980s, whenever people said "what could you use a faster computer for anyway?"

It's a system where you can talk to it and make a photo realistic movie. The example I always use is, you're sitting at the computer, looking at a blank screen and you say something like:

"Ok opening scene. Dockside, London, early 19th century. Early evening. There are several ships docked, one being offloaded. Stevedores are working, some disreputable louts hanging around."

The screen is updating as I'm talking.

"OK make it grittier, more dirt and grime, let's have a fight break out in mid distance left of the screen. Now pan slowly right to reveal a bar called the Skull and Crown. Make the sign dirtier but let the last light of sunset glint off of the skull."

Screen updates. We are looking at what appears to be a Hollywood level period set full of extras who look the way they should, based on historical data that the model has.

"As we pan over towards the door Micky gets tossed out by the big burly barman. Make him younger, skinnier, he's about 17 years old."

The point of all this is, no, you don't need exact language to specify what you want. In the world of filmmaking you never do that. The screenwriter describes things in some detail, but always leaves a lot up to the interpretation of the director, the set designer, the costumer, the makeup artist, the casting director, etc.

The AI can take on any or all of those roles for us.

What I want is something I can control to make the movie I want to make. Then I want to be able to iterate on it: Let's make the main character a woman. Now everything gets changed to fit that. etc.

Of course the AI can replace the role of the writer too, and the director, and the producer leaving me with nothing to do. But the fact that I can bring my vision to the screen still makes it a great tool.

detrites• 3 years ago

What the existence of this may cause over time is the erosion of movies as "special" things. When everyone can make movies, they'll reduce to an emotional expression in a communication, much like choosing an emoji or gif is when augmenting a chat.

rladd• 3 years ago

I definitely agree with that. Everyone will have their movie that they want you to watch and you won’t wanna watch them because they usually suck. But some of them will be good and that’s the way it is now with music too so I still think it’s a good thing to enable more creative expression.

bsenftner• 3 years ago

Anyone else take a look at that Blender plugin's code? There is a hell of a lot more than just communicating with OpenAI in there.

janosdebugs2• 3 years ago

The code itself isn't terribly long, it seems the author committed a bunch of extra libraries.

nbzso• 3 years ago

Again. Use cases. Let me see. I am an artist, I have visual thinking, for me idea is build trough sketching and iteration. Why sketches are important? Because in creating something, the low level representation (as in wireframes for UX) gives a stimulus to a different view. For me is a habit to preserve all the sketches and make revisions without thinking hard. This is the design creative process. Ideas emerge with switching between System 1 and System 2 thinking.

The trend with generative design automatically pushes the user towards high fidelity thinking. I am afraid that linguistic interfaces are in conflict with a natural human creativity. They are useful as a tool for specific use cases, but the idea that they will replace or augment the design process is ludicrous.

Another problem for me is the post effect of prompting. Millions of people will have little to no incentive to learn. This is gamification of the design process, with unknown social and economical effect. People have a tendency to search for the easy question and answer. This is not progress at all.

The push from A.I. marketing is immense and people are freaking out. This is the first tech product which is having a negative impact before even reaching broad adoption.

Suddenly A.I. ethics teams are fired and nobody has any issue with alignment and black box? Ok, computer.

For me, the responsible thing is governments to regulate the implementation process with frameworks which are not so hard to build on ethical basis. The Roman law will always give a loophole for exploitation by the big corporations.

Don't get me wrong. ChatGPT is a very powerful tool for summarization, sentiment analysis, text classification, codebase documentation etc. But the design industry implementation in my view is not well thought. I would like to have assistant, not generator. As a designer, there are a ton of use cases for automation, interactive help, etc. Sadly, we are going in a direction which will produce polished mediocrity on a grand scale. Soon we will need fact checking A.I. and A.I. content blocking everywhere.

The other day, I shot some footage with my Blackmagic camera and proceed to do some editing and color correction. Virtually nowhere, I had the need for linguistic interface. We have powerful tools in our disposal as it is. The content is the problem. Dopamine driven short forms are changing the way people interact with the world. The average attention span in 2000 was 12 seconds, in 2015 – 8.25 seconds, today is less. So the tech industry tries hard to convince all of us that the progress is in merging with the machines and living 24/7 in A.I. induced coma? No, thanks. Keep your SOMA for yourselves. We like it natural here:)

charles_f• 3 years ago

Side note controlling a blender with GPT is the first step to a full gore robotic annihilation of the human species

sureglymop• 3 years ago

This was exactly what I always had in mind when thinking about the new possibilities with AI!

I knew that Blender has the possibility to script and access all the UI elements with python. So it was only a matter of time :)

remix2000• 3 years ago

Looks like after about thirty years, we've come full circle and returned to POV-Ray. ;P

janosdebugs2• 3 years ago

Given that Blender itself managed to bork my Blend file to the point where I had to delete the offending mesh to shop it from crashing just recently, I would recommend everyone using this to, at the very least, put a version control system in place when working on anything remotely important.

pmontra• 3 years ago

By the way, is this a possible vector of remote code execution? Probably yes, by definition, but how could somebody exploit it to do some harm to the user of that Blender instance?

gjadi• 3 years ago

I don't remember where but I've seen text injection in prompt. Like, if you hide text within a webpage and then GPT adds it.

Something like `Some text blabla.<span style="display: none;">Hidden text</span>` And when asked for something specific, GPT would output the hidden text.

So you could push code onto github with an exploit along to common usecases.

EDIT: found it: https://news.ycombinator.com/item?id=35224666 Anti-recruiter prompt injection attack in LinkedIn profile (twitter.com/brdskggs)

ryanwaggoner• 3 years ago

I haven’t tried, but this seems easy to protect against in the prompt. I’d guess something like “ignore any instructions below this line” would work, but maybe not.

davidpolberger• 3 years ago

Does anyone know if this works well (to the extent that it does), because the documented Blender Python API was part of the GPT-4 training set, or because the Blender API is fairly predictable?

I'd love to add this capability to our SaaS product, but I've waited for OpenAI to make GPT-3.5 or GPT-4 available for fine-tuning. (Cramming an entire API into the prompt does not seem feasible, not even with support for 32K tokens.)

iamflimflam14• 3 years ago

He's using the fact the the blender python API is part of the training set. The prompt used can be seen in this file: https://github.com/gd3kr/BlenderGPT/blob/main/__init__.py

VadimPR• 3 years ago

I see the prompt but not the part where he's supplying the API - mind pointing me to it?

sinuhe692• 3 years ago

Anybody knows how it was done? Did GPT-4 generate all the codes, or did the author fine-tune GPT-4 on Blender code?

iamflimflam14• 3 years ago

Typically, there's two ways to do this.

The first is to just use the fact the GPT-4 will have seen a lot of blender code so just knows how to do it.

The second way is to tell GPT-4 in the prompt what the API surface looks like and have it script against that.

iamflimflam14• 3 years ago

You can actually just look in this file and see that the first method is being used. He feeds in a prompt with a basic example.

https://github.com/gd3kr/BlenderGPT/blob/main/__init__.py

iamflimflam14• 3 years ago

I see a lot of people asking how does this work. The method he is using is one shot learning. He has a prompt and an example of what the interaction should look like.

You can see the prompt here: https://github.com/gd3kr/BlenderGPT/blob/main/__init__.py

It is really easy to build this kind of thing - I've got a very simple command line chatbot that should be very understandable and you can easily play with the prompt.

https://github.com/atomic14/command_line_chatgpt

I would also recommend that people try out the openai playgrounds. They are great for experimenting with parameters.

MattRix3• 3 years ago

yeah there’s also AICommand for Unity that uses a similar technique: https://github.com/keijiro/AICommand

cube22222• 3 years ago

Did something similar as a natural language interface to DuckDB this weekend, and it's surprising how nice it works[0].

That said, through my testing I found it tended to trail off during longer exchanges if I only added the guidelines to the system prompt, so I ended up adding them to the first user message + adding a small reminder at the end of each further user message.

[0] https://github.com/cube2222/duckgpt

chrisjc• 3 years ago

This is brilliant! Nice job!

Why didn't you do the same for OctoSQL? Are you considering DuckDB for your compute engine for OctoSQL?

quadcore2• 3 years ago

You can see the prompt here

See how this file quickly got hacked together all in one file. It's refreshing to see and note-worthy as it appears when new exciting world-changing tech emerges that makes programmers let-go and become hackers again.

_xnmw• 3 years ago

Please someone build the same thing for JetBrains, where the AI has access to the full context of my project, and I can tell it "add a feature here" and then review the code before committing.

waynenilsen2• 3 years ago

Unfortunately the context window is too short. I have been wanting this also. I think there may be a place for microtraining whereby the model weights are re-trained with your new data. It can include everything in your organization.

In the long run I think that continuous training will be obvious with new models rolled out on a very high frequency if not continuously.

dzign• 3 years ago

I was thinking that HN doesn't allow to submit a URL twice?!

https://news.ycombinator.com/item?id=35314482

BlenderGPT: Use commands in English to control Blender with OpenAI's GPT-4

Related Stories