Visual programming is stuck on the form

https://interjectedfuture.com/visual-programming-is-stuck-on-the-form/

by iamwil • 10 months ago

66 31 10 months ago

For some reason while reading this, I thought of Erlang, a language I know nothing about, except that it's not a visual language and it's build out (many) processes, kind of like a tiny internet with server and client nodes that can be taken down anytime for repairs or updates.

Maybe there's an unobvious way to make visual programming actually useful?

Probably quite a few people have wondered that throughout the ages. I know I have.

In the meantime, this AI thing happened, emphasizing even more the use of text/voice as a mode of creative expression.

Towaway69• 10 months ago

Have a look at Erlang-Red[1] which uses Erlang for the basis of making a visual programming environment.

Erlang-Red is inspired by Node-RED which itself is inspired by flow based programming.

[1] https://github.com/gorenje/erlang-red

Disclaimer: I’m the author of Erlang-Red.

sublinear2• 10 months ago

Maybe there's an unobvious way to make visual programming actually useful?

I'd think it would be finding the optimal parameters for an algorithm that is probably better expressed in another language.

Even though they are all Turing complete, any programming paradigm is biased towards solving certain kinds of problems.

It seems "visual programming" is biased towards the computationally irreducable[1]. This is a class of problem very sensitive to initial conditions. The chaotic behavior may eventually settle towards a stable state. So, the image of that stable state then encodes the parameters you'd want to use on the algorithm you wrote in another language.

That's not meant to be harsh. This gets directly to the heart of why we may want to write the same ideas in different ways even if those writings seem logically equivalent. One way is just easier to understand than the other depending on what part of the problem you're trying to tackle. These multiple writings are not redundant, but the facets necessary to more thoroughly explain a problem. In fact, upon closer inspection you'd find that what seems like the same algorithm written two different ways is actually not the same because they are executed differently.

[1] https://en.wikipedia.org/wiki/Computational_irreducibility

boopity2025• 10 months ago

Feels like visual programming keeps trying to be “code, but with pictures” instead of asking what problems pictures are actually good at solving. We’ve basically been drawing the same boxes and arrows since the 80s and wondering why it still feels like wiring up a VCR.

If we really leaned into the visual cortex, maybe we’d get something where zooming out shows the big picture and zooming in shows the gritty details, like Google Maps for code. Until then, node‑and‑wire diagrams are just UML diagrams that decided to cosplay as circuit boards.

rwmj2• 10 months ago

Zooming UIs are a bit like that: https://en.wikipedia.org/wiki/Zooming_user_interface

I'm not aware of any that were very successful.

general14654• 10 months ago

zooming in shows the gritty details, like Google Maps for code

That's how KiCAD and other CADs for electronics make PCBs. You will put big parts (that's usually hardware design driven), then you will put smaller parts around it and then you are trying to connect each smaller part with correct signal without crossing each other.

spacebacon• 10 months ago

Fundamentally underestimates the chaotic, exploratory essence of innovation. I challenge you to write the argument for form over function for good measure. Form follows function inside the box.

aetherspawn2• 10 months ago

It would be awesome if someone could fund a free open source version of Simulink (as a VSCode plugin or something like that) so that we can build normal software using model-based design. Games, shaders, ML, reactive UI, applications like these…

Simulink is getting more expensive every year, and licensing is a huge barrier to open source models.

Edit: basically, we want Dyad (https://juliahub.com/products/dyad) except free.

general14654• 10 months ago

Like Open Modelica? https://openmodelica.org/

aetherspawn2• 10 months ago

Hm, maybe. I don’t think it has a code generator though.

general14654• 10 months ago

From what is practically used - you have things like CoDeSys where you can program PLC or Matlab Simulink which is heavily used in science and in automotive development or infamous LabView spaghetti the moment when program will get just a little bit more complex.

All are using blocks with input and output which are either having predefined function or you can define the block function with a code. Reading these programming schematics is sometimes great to understand the high level of how program is working with input data (left) towards output (right). However the moment you will introduce loops, visual programming will just fall apart like a glass hit by a hammer.

jayd162• 10 months ago

The tone of the article feels really preachy and full of itself without even fully understanding why Blueprints work well for games.

Killer features that come to mind are:

You can drag and drop asset references instead of relying on textual tokens.

You can build a framework where incorrect code does not fit together, or pins dont connect, or draggables do not allow for blocking. It's a much better feedback to a certain type of person than a textual error.

Then the article seems to go on and on about knowing how to make a great visual programming language despite not producing one.

dazzawazza• 10 months ago

I agree with you but somethings are missing from the BP experience.

I've implemented many VPLs in video games and I've used Blueprints extensively. I've probably made all the classic "mistakes" designing VPLs, many of which are mentioned in this article. I don't think I am very good at designing VPLs despite having done it on and off for 30 years.

I think BPs are the best example of a VPL out there at the moment. Certainly in video games. However it still falls short of the ideals of VPLs.

Essentially BPs trick people in to being programmers. They still have to understand the "ways" of programming (for loops, if then, etc). With a little context switch and training they would probably be more productive with a text based interface. So the abstraction BPs provide is very limited.

BPs are a general programming tool used for materials, game play logic, animation trees etc. Because of this there are few, if any, high level abstractions that relieve the user of the burden of programming. Don't get me wrong, this is hard, very hard, so I am not calling anyone out. It requires sitting down with a non-technical person and really understanding how they think and what they need. Turning that in to something that isn't node + wire is hard. The fact that the industry has created technical artists to fill the void says to me that BPs are failing to a certain extent (and TAs can just use text based programming and do in many studios).

Overall I agree that the field of VPLs is stuck at a local minima and the 10x productivity improvement for non programmers is still illusive.

hackable_sand• 10 months ago

Maybe the true abstraction layer is the BP market on fab.

Scaevolus• 10 months ago

UE BPs do have a few abstractions available-- macros, interfaces, classes, data-only. [1]

I think the greatest challenge of VPLs is how they don't have an inherent linear structure, so reading and understanding them for modification requires a nonlinear scan of a node soup. This puts them in the class of tools that struggle to make large programs readable, like Bash or Forth.

https://dev.epicgames.com/documentation/en-us/unreal-engine/...

jayd162• 10 months ago

Isn't this desire nonsensical? A visual programming language will indeed still have programming. You can't get away from the core principles of logic, event flow, etc. and that was never the point, imo. As I said, VPLs provide other advantages than that.

If you're looking for a gui configuration tool instead of a VPL... those exist too. They exist by being declarative data editors.

StateTree editors, Animation Blueprints, and discrete state machines are some possible examples of visual editors that change app functionality that strip away core programming concepts for simplicity. There's plenty of these floating around but I feel like they "don't count" while simultaneously being exactly what is asked for.

crq-yml• 10 months ago

There's a correspondence between where node-and-wires works best and where the problem statement prefers something like a circuit diagram because it's synchronization-heavy. Games need a lot of this, and so do audio apps. The problem is that as you do more compute on the workload and need more "plain old synchronous algorithms" to process it, that holds true relatively less often, so it's a solution disproportionate to what production tools actually need.

I did a survey of the popular VPL environments last year and found Blender's implementation notable for what it didn't do. It exposes node-and-wire UI elements and the scripting for them in Python; all further details are delegated to each subsystem. So shaders, video, etc. have no relationship within their programming model, just UI similarity.

Add to that the aspect of - why are programming languages always going for plaintext representation - and it's because of the tooling ecosystem. We have a huge investment in text editing and text processing, and in typing on standardized keyboard layouts, on speech-to-text and vice versa, and lately, on text generation language models. We have made that stuff really cheap and fast for everyone, in the same way that when Blender did node-and-wire, they elected to make the paradigm as a whole cheaper to reuse in a customized way, versus investing heavily in one particular implementation and trying to extend it to many scenarios.

I think part of the issue is that we design VPLs without taking any interest in draftsmanship - the traditional core skillset for working with "visual language". Instead we use mouse pointers and touchpads and treat it as a data entry problem that has to be structured right away instead of allowing sloppy syntax.

sublinear2• 10 months ago

That's how we get terrible interfaces where the end user needs to be aware of the data model to use it effectively, like Git.

This is only true if you prefer whimsical metaphor over concise description and cleaning up unexpected behavior instead of precisely defining what you want upfront.

cgio2• 10 months ago

I wonder why people always seem to miss the most successful visual programming interface, excel, in their analysis.

nlitened• 10 months ago

100%. Hundreds of millions of people have been successfully using this visual programming environment with little to no CS education to solve real business problems for decades. What’s most important — the “visual” part is what actually allows non-programmers to build the intuition and write correct programs, it’s a core feature that reinforces the success, not a gimmicky bolt-on.

Also important to reflect on this, I think: it’s a dynamic functional programming language with annoyingly weak and limited types, with inconsistent string parsing, and the most obtuse and uninformative error messages, and it still works wonderfully.

DonHopkins• 10 months ago

https://news.ycombinator.com/item?id=26668885

DonHopkins on April 2, 2021 | parent | context | favorite | on: Spreadsheet is a software development paradigm

Spreadsheet certainly are visual programming languages: by any measure, by far one of the most common most widely used types of visual programming languages in the world.

Taxonomies of Visual Programming (1990) [pdf] (cmu.edu)

https://news.ycombinator.com/item?id=26057530

https://www.cs.cmu.edu/~bam/papers/VLtax2-jvlc-1990.pdf

https://news.ycombinator.com/item?id=26061576

Brad Myers' paper answers the age-old argument about whether or not spreadsheets are visual programming languages!

https://news.ycombinator.com/item?id=20425821

DonHopkins on July 13, 2019 | on: I was wrong about spreadsheets (2017)
Google sheets (and other google docs) can be programmed in "serverless" JavaScript that runs in the cloud somewhere. It's hellishly slow making sheets API calls, though. Feels like some kind of remote procedure call. (Slower than driving Excel via OLE Automation even, and that's saying something!) Then it times out on a wall clock (not cpu time) limit, and breaks if you take too long.
A CS grad student friend of mine was in a programming language class, and the instructor was lecturing about visual programming languages, and claimed that there weren't any widely used visual programming languages. (This was in the late 80's, but some people are still under the same impression.)
He raised his hand and pointed out that spreadsheets qualified as visual programming languages, and were pretty darn common.
They're quite visual and popular because of their 2D spatial nature, relative and absolute 2D addressing modes, declarative functions and constraints, visual presentation of live directly manipulatable data, fonts, text attributes, background and foreground colors, lines, patterns, etc. Some even support procedural scripting languages whose statements are written in columns of cells.
Maybe "real programmers" would have accepted spreadsheets more readily had Lotus named their product "Lotus 012"? (But then normal people would have hated it!)

I Was Wrong About Spreadsheets And I'm Sorry:

https://www.reifyworks.com/writing/2017-01-25-i-was-wrong-ab...

HN Discussion:

https://news.ycombinator.com/item?id=20417967

Excerpt from "Taxonomies of Visual Programming and Program Visualization", by Brad A Myers, 1990/3/1, Journal of Visual Languages & Computing, Volume 1, Issue 1, pages 97-123:

Spreadsheets, such as those in VisiCalc or Lotus 1-2-3, were designed to help nonprogrammers manage finances. Spreadsheets incorporate programming features and can be made to do general purpose calculations [71] and therefore qualify as a very-high level Visual Programming Language. Some of the reasons that spreadsheets are so popular are (from [43] and [1]):

1. the graphics on the screen use familiar, concrete, and visible representation which directly maps to the user's natural model of the data,

2. they are nonmodal and interpretive and therefore provide immediate feedback,

3. they supply aggregate and high-level operations,

4. they avoid the notion of variables (all data is visible),

5. the inner world of computation is suppressed,

6. each cell typically has a single value throughout the computation,

7. they are nondeclarative and typeless,

8. consistency is automatically maintained, and

9. the order of evaluation (flow of control) is entirely derived from the declared cell dependencies.

The first point differentiates spreadsheets from many other Visual Programming Languages including flowcharts which are graphical representations derived from textual (linear) languages. With spreadsheets, the original representation in graphical and there is no natural textual language.

Action Graphics [41] uses ideas from spreadsheets to try to make it easier to program graphical animations. The 'Forms' system [43] uses a more conventional spreadsheet format, but adds sub-sheets (to provide procedural abstraction) which can have an unbounded size (to handle arbitrary parameters).

A different style of system is SIL-ICON [49], which allows the user to construct 'iconic sentences' consisting of graphics arranged in a meaningful two-dimensional fashion, as shown in Figure 5. The SIL-ICON interpreter then parses the picture to determine what it means. The interpreter itself is generated from a description of the legal pictures, in the same way that conventional compilers can be generated from BNF descriptions of the grammar.

10. Conclusions

Visual Programming and Program Visualization are interesting areas that show promise for improving the programming process, especially for non-programmers, but more work needs to be done. The success of spreadsheets demonstrates that if we find the appropriate paradigms, graphical techniques can revolutionize the way people interact with computers.

https://news.ycombinator.com/item?id=26057530

Taxonomies of Visual Programming (1990) [pdf] (cmu.edu) 107 points by mr_tyzik on Feb 7, 2021 | hide | past | favorite | 49 comments

BradAMyers on Feb 7, 2021 | next [–]

Thanks for your interest in my old article! That version is a draft update of this article: Brad A. Myers. "Visual Programming, Programming by Example, and Program Visualization; A Taxonomy," Proceedings SIGCHI '86: Human Factors in Computing Systems. Boston, MA. April 13-17, 1986. pp. 59-66.

http://www.cs.cmu.edu/~bam/papers/chi86vltax.pdf

rwmj2• 10 months ago

It's visual at the top layer (rows, columns) but soon gets very awkward when you're stuffing program statements into small boxes.

There was a big explosion of weird and wonderful visual programming languages in the 1980s, such as CASE tools, loads of commercial 8 bit game-making systems, HP's early 80s workstations, and dozens of BASIC variants (which weren't very visual, but let you draw things with extreme ease). I wonder if there are any under-explored ideas there.

viraptor2• 10 months ago

Excel is not really visual programming though. It's got visual presentation, but all the programming still happens in text and in the worst code editor there is. The visual programming parts that do exist are very limited/specialised.

cgio2• 10 months ago

Programming is not in text. Yes, formulas have textual representations but can be graphically defined if you are not an advanced user (e.g. click to select parameters including ranges). Also, other than formulas, there are e.g. iteration semantics in copying a formula down, applying it to a column on a table etc. There is even limited recursion. The model breaks down in some areas, e.g. using silver is not as well integrated.

mamcx• 10 months ago

Also, the next one is relational tables.

Tables are super visual (even in forms like CSV).

m0llusk• 10 months ago

Sort of related: It is interesting how even heavily symbolic programs can be translated into relatively plain language and from there into other languages. But from there complexity explodes as any real use must address essential challenges like typing and making use of existing interfaces which are harder to translate.

Here there is embrace of beauty and structure emerging from rethought low level expressions, but how does this work with typing, Windows, Mac Foundation Kit or whatever else?

dustingetz• 10 months ago

i think you’re looking for the term “computational structure”, as in, “what is the computational structure of a user interface” i.e. what typeclasses does the computation reify. To avoid impedance mismatches the authoring notation must align exactly to and emerge from the underlying computational structure. So as a starting point: modern applications have a frontend and a backend, they are reactive, async, and concurrent, they perform network IO and manage resources. Or at least they have components that do, which means the computational structure as a whole must encode these capabilities. if the computational structure is not uniform, i.e. the frontend and backend are treated separately, or the database is relational but the backend-for-frontend needs dynamic graph walking, then you will have seams between modules and arbitrary glue code cruft will accumulate and this will violate the alignment between authoring notation and underlying structure. Now your ORM needs XML configs to inject additional knowledge about the data access patterns that can’t be reflected from the notation.

emorning4• 10 months ago

>To avoid impedance mismatches the authoring notation must align exactly to and emerge from the underlying computational structure. <<

I thought that I was the only one that understood that the rules for transforming structures are emergent properties of the structure itself.

I'm not an academic, can you point me to where you encountered this idea of emergent notations??

Or anything regarding computational structures and web development. Such a thing seems like alien technology to me right now.

KaiMagnus• 10 months ago

Sometimes I wish VS Code had something like Code Bubbles, where it becomes much easier to see how pieces of code are related. I think it would make AI assisted coding much easier as well, since often the main challenge is having to piece together how changes across multiple files work. There has to be a lot of potential for better interfaces aside from a chat sidebar.

https://cs.brown.edu/~spr/codebubbles/

general14654• 10 months ago

Technically Doxygen provides something similar from generated code

https://www.doxygen.nl/#diagrams

Visual programming is stuck on the form

Related Stories