Graphical Programming: I Really Hope This Is the Future

http://pcmonk.wordpress.com/2013/10/14/graphical-programming-i-really-hope-this-is-the-future/

by pcmonk • 13 years ago

44 73 13 years ago

pcmonk.wordpress.com

danmaz743• 13 years ago

Shouldn't an article about graphical programming show something graphical, instead of just ascii characters?

Doublon• 13 years ago

That was exactly my thought. This article is just plain text. It feels like something is missing.

CmonDev8• 13 years ago

It's a "wouldn't it be nice" kind of article. Concept art would be nice even if there is no intent to make a prototype.

mathattack• 13 years ago

The precision needed for typing highlights why graphical programming is so tough.

pcmonk9• 13 years ago

I've been spending all my time writing the code. Concept art isn't worth the time investment for me right now. I could post screenshots, but I don't think it's quite ready for that.

danmaz743• 13 years ago

Well, it depends on what is your objective with the blog post. If it is to start creating an expectation and receiving feedback from people who would be interested, it would be very useful to spend some time for a couple of schematics or mock-ups, especially because your best target is quite likely very visual oriented.

If you just want to let out your thoughts, then, no problem!

rtb• 13 years ago

What's the difference between this effort and the countless Graphical Programming efforts that have come before (many with large teams as well as many other single-author efforts)?

Jare• 13 years ago

This is still at the "idea" or "vision" level and hasn't failed yet.

Sorry about the snark, but while I'd love someone to succeed in this type of quest, I believe this one is based more on ideology than rigor. Stuff like "WYSIWYG editors are basically evil" just doesn't give me confidence.

buro9• 13 years ago

Show, don't tell.

jdmitch• 13 years ago

Are you looking for others to test/collaborate on what you have done so far?

stinos2• 13 years ago

I doubt this is THE future, and I also hope it is not: still too limited. As far as I can predict, the future is a proper direct brain-machine interface. Controversial yes, avoidable: I seriously doubt it. (would I use it: maybe if it could be implemented using one way communication, ie similar to how it's done now by analyzing EEG). I guess it would also be by far the fastest way. At least I can think code (both concepts and actual implementation) way faster than I can output it in any way.

francis-• 13 years ago

Good to know someone out there 'gets it'.

Someone• 13 years ago

You only need one way; evolution has provided plenty of ways for reverse communication.

Houshalter2• 13 years ago

The rate of inputting information to the computer has never been the limiting factor for me. Displaying the information in more helpful format could be useful though.

voyou• 13 years ago

I have in front of me a system that uses electromagnetic signals to interface between my computer and my brain; part of it even works wirelessly - it's a high-tech system called "a screen and my eyes." In what way is your predicted "direct brain-machine interface" better than my actually-existing electromechanical brain-machine interface?

: I can think code (both concepts and actual implementation) way faster than I can output it in any way.

I doubt you can. You can have the psychological feeling of thinking you've developed the concepts much quicker, but in my experience it's only when I start doing the actual implementation that the full level of precision I need becomes clear, and its that level of precision that slows me down, not the need to physically press keys.

stinos2• 13 years ago

it's a high-tech system called "a screen and my eyes."

I meant the interface the other way around. From your brain to the computer, without any mechanical inert pieces in between. As for the prediction part, note that it exists already in a very early stage and it's extremely likely to only grow. Can't find the exact artical atm, but it was used to move a mouse on screen by someone who couldn't move anything below their head anymore.

I doubt you can

Maybe not on all levels, 100% sure I can for some. Simple example: C++, I need a basic iteration over a sequence. I just think 'ok gimme a for_each on container Foo, calling function Bar'. No matter how advanced the intellisense or whatnot is, it will take me more time to properly type the needed line.

RivieraKid• 13 years ago

I have been thinking about this too recently and I'm thinking about writing a simple IDE to demostrate these ideas. If anyone would like to discuss this over mail (or here of course), let me know.

vidarh• 13 years ago

People have tried this. Over and over and over. I love the idea, and I too have tried (and scrapped my effort), but it is an immensely difficult problem to tackle for a number of reasons:

- The moment you force a user to pick up the mouse, you have lost.

- It seems very compelling to augment the display with more information, but a lot of it is simply distracting, or reduce the density of relevant information that will fit on the page. There's a reason that even in most IDE's very little additional information beyond syntax highlighting have "stuck".

- Some weirdos (like me) won't touch tools that won't let me work over a dumb ssh connection and terminal when I need to, which is one of many reasons why...

- ... everything needs to have a sane textual format. Another reason is that it is essential to communicating about the code.

If you want to think about this, at least thoroughly review the long history of visual languages, as there are plenty of mistakes to repeat.

seanmcdirmid2• 13 years ago

There's a reason that even in most IDE's very little additional information beyond syntax highlighting have "stuck".

We are working on it; e.g. http://research.microsoft.com/en-us/people/smcdirm/liveprogr...

splawn• 13 years ago

I think visual programming works well if its geared towards a specific problem set. In the past I used Talend ETL to generate small java programs and it worked well. However I couldn't imagine trying to do something big with a general visual language.

ArbitraryLimits• 13 years ago

People have tried this. Over and over and over.

Seriously, it's the future and it has been for fifty years.

CmonDev8• 13 years ago

So it will probably be a part of final HTML5 specification. Worst case HTML5.1.

pcmonk9• 13 years ago

Oh, man, I wish I hadn't have posted this just before I went to bed.

At any rate, I understand your criticism. Regarding the mouse, I think you're almost right. In particular, you don't ever need to pick up the mouse to use my program -- there are no menus and no buttons. However, I do, on occasion, use the mouse to move around inside the tree structure. It's not necessary, but sometimes it can be the most natural way to interact. It's never a part of normal coding, but when I'm searching for a bug or surveying the structure of my code, it can be useful.

Regarding ssh, yeah, I know what you mean. I just hope that the benefits outweigh that unavoidable disadvantage.

lnanek2• 13 years ago

Everything's solvable and everyone else is wrong, but he has none of his own work to show. Very funny post. Maybe he's just a student? These constant failures at trying to implement visual programming often seem to come from academia where the goal is to write papers rather than actually accomplish anything.

TeeWEE• 13 years ago

I worked with numerous tools for graphical programming, mostly for academic work.

Such as graph-transformation based tools for software verification, in which your programs are graphs. http://wwwhome.ewi.utwente.nl/~rensink/papers/spin2006.pdf

Which is useful for software verification, but writing your actual program in it is cumbersome.

Software is a lot about abstractions, and there is no reason that graphical programs are better at representing abstraction that text (or maybe im wrong here?).

In theory, that what you can represent in graphical-form you can also represent in textual form.

However the information density with an 2D graphical form is much higher than textual form. And the more information you put into the same space, the more difficult it is to reason about it. Thats why textual programming is still very much popular.

super-serial• 13 years ago

I've always thought that a visual interface for programming would help with my procrastination problems.

It's so painless for me to open up a webpage and solve daily chess puzzles... on the other hand it seems more painful to open up an IDE and start solving code "puzzles." I wish I could visualize all the pieces of my software similar to chess.

Kurtz79• 13 years ago

I absolutely doubt that graphical programming, even in its best implementation, will be about solving "code" puzzles.

Regardless of the interface, programming is "work", linking up different interfaces, checking edge cases, conform to protocols...

It would feel more like glorified data entry than solving chess puzzles.

sesm• 13 years ago

Did some LabView programming as a student, didn't enjoy the experience.

Graphical programs are similar to circuits without memory, and thus are less powerful then Turing machines. As a result, graphical programs quickly grow in size, and growth happens in 2 dimensions, which makes it very hard to read them.

Houshalter2• 13 years ago

To be fair, text is merely a special case of graphical programming where the output is limited to a handful of characters and is completely linear. Is it impossible to improve upon that? Syntax highlighting and indentation were good improvements for example.

lmm• 13 years ago

It's possible to improve, but I think the key is progressive enhancement. Look at how IDEs tend to offer typing and implicit information in scala: it's there, it's very useful, but it doesn't stop you from seeing the plain text source code.

ansible• 13 years ago

Ugh, LabView. Do not want.

One of my main gripes is lack of version control integration. I want to use something like git, and I want to be able to easily and succinctly see what's been changed by who.

You have to look at the "code" only using their own GUI tool. Large functions involve lots of scrolling, and at least with Labview, I was not aware of any means by which to jump around to specific parts using hotkeys or such. You'd also have pages of logic stacked on top of each other, where you can only view one page at a time.

If you're implementing something that is going to closely resemble a circuit of some kind, I suppose it is OK, but when things start to get complicated, I found the system cumbersome to navigate. This impedes comprehension, and invites mistakes.

I've also used this other GUI-driven code generation system, and that had serious issues too. It has this hilarious graphical view of all the components, showing how they are connected to each other. However, due to the limited size of the window, it will just stack all the components almost on top of each other. If you started the laborious process to move them around to see what is going on, your layout would not be saved.

So, anyway, a modest proposal: Figure out how to display an existing programming language (maybe a simple one like Lua) with a graphical view. Allow me to easily navigate through the code, and let me see all the details easily.

pcmonk9• 13 years ago

See, that's exactly the problem with graphical programs nowadays. My program is essentially a layer on top of Racket, so it's much closer to textual programming than it looks like at first glance.

CmonDev8• 13 years ago

This is what you get when you lock yourself in the *nix Vim-niverse. Try Visual Studio 2012 Ultimate:

http://msdn.microsoft.com/en-us/library/vstudio/jj739835.asp...

http://msdn.microsoft.com/en-us/library/vstudio/dd409365.asp...

PS: pricing is the sad bit :(, but there is no competition in this area. I hope JetBrains will work harder.

lcedp2• 13 years ago

Diagrams look like a mess to me: virtually everything is connected to anything in unknown way (no method names or arguments specified). Text at the left is much clearer.

rubberband• 13 years ago

I sit in front of Visual Studio more hours in a day than I sleep, and by and large, I love it. But I can't for the life of me understand why the mapping diagram would help. I suppose in some unfamiliar code it may help me get up to speed faster, but this seems like a rehash of exporting UML to Visio?

I cannot afford Ultimate, just Professional, so I admit I have no hands on experience with the tool.

What JetBrains product is comparable?

CmonDev8• 13 years ago

There is no JetBrains product like that, I just wish they had, because they have a potential to build something lightweight. Best use cases for me personally are the visual dependency graph and method call diagrams when dealing with poor-quality code (huge classes with a lot of method dependencies). I would pay a hundred bucks for this, but not a thousand dollars.

dman• 13 years ago

If youre dependent on auto generated diagrams to understand the structure of your code you have bigger problems.

CmonDev8• 13 years ago

It's good for bad legacy code. Also good if you are so proud of your architecture that you want a print out on your wall :).

sparkie• 13 years ago

Diagrams may not be necessary, but we all depend on auto-generated visualizations, whether it's a "find all references" feature of some IDE that produces a hyperlinked list, or in the most primitive case, the use of grep to produce a list of locations in files for some identifier.

I'd argue that you need help if you're not using some auto-generated visualization to view relationships in code, because the alternative is to scan through files manually.

The choice of diagrams, lists, or some other visualization is irrelevant, it's the preference of the programmer.

lcedp2• 13 years ago

The future is what? Nothing's been proposed so far.

smoyer• 13 years ago

In the '80s I was working on a (never released) system I called FlowPro, but EGA adapters and 8088 processors weren't really capable of the kind of DnD I wanted. The idea was that flow-charts would be turned into code, with some meta data added if you wanted specific variable names etc. (instead of the words used in the diagram's box).

Then in the early-to-mid '90s, I came across a product called Prograph that ran on both Windows (I did a lot of programming for WFW with it) and Mac (OS 8 I believe). I loved that system, and realized it solved some of the syntax issues I had using flow-charts, but it still never really took off. The concept lives on with the Marten IDE though I'm not sure how many people are actively using it.

https://en.wikipedia.org/wiki/Prograph

http://andescotia.com/products/marten/

collyw• 13 years ago

Users will use Excel. That is the closest thing we have to graphical programming that works. They don't need to understand data structures, just columns and rows. Anything more than that, and you need to understand new concepts. By the time you understand these, drag and drop doesn't gain you much over typing.

pizo• 13 years ago

The idea of graphical programming seems in a lot of way like graphical mathematics.

Works in some cases, but not in all.

I used graphics to represent databases, and their relationships, because it's more easily understood than several hundred lines of code. It fails to tell the whole story though, and thus only serves as a thousand foot view of what is going on.

gshubert17• 13 years ago

Works in some cases, but not in all.

Delphi (for Windows, 1995-on) had a very nice graphical IDE which was good for developing graphical user interfaces in an object-oriented programming style. The rest of the IDE was fine for editing and debugging. But it was great for GUI development.

pcmonk9• 13 years ago

You're right about the similarity with graphical mathematics (and Bret Victor's Kill Math project was something of an inspiration for this (even though I have some fundamental disagreements with his methods), even though I didn't see it until after I'd started it).

You've identified a critical problem with graphical programming today: it fails to tell the whole story. This is exactly what I referred to in the post, and that's not inherent in the medium; rather, it's just that people have been trying to solve this problem in the wrong way.

svantana2• 13 years ago

Personally, I would prefer an IDE/debugger that allowed me to visualize my data really well. This ought to be a lot easier also - I've done some simple scripting with GnuPlot which has given me a lot of mileage. But imagine being able to put a breakpoint anywhere in your program and interactively browse your trees, dictionaries and arrays using OpenGL-accelerated real-time graphics. I don't understand why xCode and VS aren't putting more efforts into things like this.

jamii• 13 years ago

Sounds like this is what you want - https://github.com/Element84/vdd-core/wiki/What-is-Visualiza...

EDIT: Or even http://www.youtube.com/watch?v=L6iUm_Cqx2s

goetz• 13 years ago

A friend of mine is working precisely on this : a plugin for XCode for "visual debugging". What would you like to see in such a tool?

svantana2• 13 years ago

Sorry for the late reply. I'd love to be able to plot arrays of floats or ints as 2d or 3d graphs. If you've ever used Matlab, this is the sort of thing it excels at.

CmonDev8• 13 years ago

Well actually Visual Studio team put efforts to address exactly this need:

http://visualstudiogallery.msdn.microsoft.com/eedc48e7-5169-...

http://blogs.msdn.com/b/vcblog/archive/2012/07/12/10329460.a...

http://msdn.microsoft.com/en-us/library/zayyhzts.aspx

http://visualstudiogallery.msdn.microsoft.com/c7e02633-86d9-...

CmonDev8• 13 years ago

PS: if there is a need for something IDE-related it is either already done in VS, available as a third-party extension or there is an API to build it.

outworlder• 13 years ago

I can postulate that there is absolutely no way a purely graphical representation of code will be the future. The same way we do not talk using pictures.

A graphical language would be best used to display structure. Bonus points if it allowed changing. The rest, modules if you will, can be better described by text.

CmonDev8• 13 years ago

Don't want to argue with the spirit of your post, but Chinese hieroglyphics is derived from pictures and it's probably somewhat representative of their language.

erock• 13 years ago

I've spent the past year "working" within graphical programming, and I have to say it is a slow form of torture.

I have been "writing" IVR applications for a call center. The interface is far from intuitive, and it is jsut too easy to miss what is actually going on. The loops and if statements are all there, but you have to click through on the item to actually see what is going on.

Forget trying to find an easy way to locate where something is. at least in my case (VXML) I was forced to use grep to at least give me an idea of where to look, then spend 10x the amount of time it would take in plain ol code to find where to make the change.

ugh.

give me ASCII code any day

brudgers2• 13 years ago

[Terms of Service: Let me start by saying that I admire the author's passion and effort to create something wonderful, and that this is not a crap on their project, but rather a reaction to their essay. And the fact that their essay raised a reaction to the point where I wrote something about it is, in this case, a testament to its quality.]

Lisp programs can be represented as trees because they are trees. They can also be represented as lists, and this is because they are also lists.

The important difference between these representations is that trees the primary representation by which computers unravel the intent of a Lisp program and lists are the primary representation by which programmers express intent in their programs.

Sure some people sometimes picture their lisp program as a tree, but usually when a programmer pictures something as a tree, it is when viewing it as a data structure. Picturing a Lisp program as a tree is easier than picturing the tree represented by code in most other languages because of Lisps simple parsing process, but the way in which a programming language is parsed is an arbitrary feature of the language [though one worth design consideration].

Although it is easy to draw a tree. Often in Lisp and other languages, the verbal diagram "a list of lists of ..." is adequate to represent a tree along with a description of its interesting properties. Again, while the interesting properties can be represented graphically, what makes them interesting are the maths underpinning their structure not the aesthetics of their spatial representation.

"The purpose of this project is to assist the coder in developing more complex code faster and easier"

This is orthogonal to Rich Hickey's goal of making it easier and faster to produce simple code.

http://www.infoq.com/presentations/Simple-Made-Easy

We already have a great tool which allows graphic methods to be used easily to create complex code. It is called Excel.

pcmonk9• 13 years ago

Hm, interesting. I've always pictured my Lisp programs as trees. I think it's a more useful representation, but I see what you mean.

You're right that this is orthogonal to Rich Hickey's goal. That's because I think his goal is great for writing simple programs. His goal is to make more programs simple, which is great. My goal is to make it easier to write complex code. As Fred Brooks talks about in No Silver Bullet, there are some problems that are accidental and some that are essential. Hickey is trying to solve the accidental problemms, and I'm trying to make it easier to solve the essential problems. Some logic is just inherently complex -- I want to create tools to make it easier to reason about.

And Excel is not nearly programmable enough to be better than a textual interface to solve the same problems. It's benefit is that it can be used (relatively) easily by non-programmers.

brudgers2• 13 years ago

My impression is that Rich Hickey is seeking ways to allow programmers to solve complex problems, without writing complex programs. He's been fairly successful relative to other people who write new languages.

Excel is entirely programmable using the standard .NET stack and languages [from PowerShell to F#].

If the idea is to target the tool as a new IDE for experienced programmers, how does improve productivity over EMACS or Vim etc.? Or without it being an extension of one of those existing ecosystems, what is gained in exchange for all that is lost?

Which suggests to me that incorporating the functionality as an extension to one or more of those ecosystems might be more simple than competing with them for experienced coders.

cookingrobot• 13 years ago

Excel is a brilliant tool for solving problems that can be expressed as arrays. But its not that nice for trees. A first step to this project could be to make a satisfying experience for editing this kind of data. Here's an example: you have a list of categories and products. In excel today, you'll probably put category in column A and product in column B. But now you have lots of duplication of category ids. You can't change the category in one place like you'd be able to it was explicitly a tree, and rearranging branches of the tree is tedious. Creating arbitrarily deep sub-categories is a mess. If someone could make a good experience for this problem, then it might inspire ideas for the more complicated task of programming.

bane• 13 years ago

http://en.wikipedia.org/wiki/DRAKON

Zigurd• 13 years ago

There are some general problems with the way people think about graphical programming. First, they should stop thinking about it as executable flowcharts and take a step back: There must be a better way than typing characters and pseudo-words to create programs. But that way isn't necessarily any of the graphical programming systems anyone has yet come up with.

This article is a good example of thinking that's not even near the walls of the box, much less outside. And what's that crazy digression about wysiwyg editors? And then the author drops this one: "Thus, this isn’t really about graphical programming." Derrrrrr.

We are barely at a point where we admit that evaluating a language syntax in a vacuum is silly, so people are still surprised that "verbose" Java is popular, despite the advantages static types and an intermediate language confer on the editor, compiler, static analyzer, debugger, etc. A less than holistic approach to graphical languages is probably doomed.

taeric2• 13 years ago

Again I look at how our industry completely ignores every other industry out there. Some things do lend themselves to graphical viewing. Some things do not. Many things lend themselves to a series of graphical representations. The vast majority of things benefit from both graphical representations and textual elaborations.

One of the things I wish programming tools offered, was a way to "zoom in" details wise on a solution. More than just offering an outline, I want there to be some sort of representation of how things went from conception to realization. Consider, the story boards for movies/games. Those story boards are lost in the final product. Worse, when you are looking at a piece of code, you don't really see the "why" of what it was made for.

Sure, you get some of this with descriptive names, but that really doesn't help with what I'm wanting to say here. An example, that I really want to explore some more is graphviz. It can go from text to graphics. This process is fairly static, though. If it doesn't turn out how you want it, you have to edit the text and fire it again. I want something that can possibly go from text, to initial graphics where I manipulate it some more, to final representation all in a repeatable way. (I think this is possible, but I am not familiar with the toolchain.)

More, there is no reason I shouldn't be able to start from graphics, go to a textual details portion, at varying levels of detail, make edits, and view what effect they have in the viewer.

Imagine, something similar to developer tools in chrome/firefox, where in addition to being able to look over at the boxmodel for details, you can physically move elements around.

Of course, I'm a bit of a heretic. I would kind of like to be able to export absolute positioned style sheets for forms designed this way. :) Still not sure on all of the reasons that would be terrible.

MetaCosm• 13 years ago

... wanting Visual Basic doesn't make you a heretic, it just makes you aching for a foregone era a bit. Visual coding has been pushed for ... at least 30 years.

The thing you are looking for is all over the place, in various IDEs, in QT MOC, in literally DOZENS of VPL (Visual Programming Languages) and even to a lesser degree in UML (Unified Modeling Language). Since we have QT MOC in the mix, we could even through in ALL the meta compilation tools, which adds in hundreds of others.

What you want isn't so much unique, as already done and discarded. You end up with tightly bound IDEs and code, you end up with hard to manage binary nonsense, you end up with ridiculously hard peer review and change tracking.

As for tracking the "why" -- use any method you want -- my team uses -- story boards! They work perfect for software.

taeric2• 13 years ago

I was not trying to paint my view as a unique novel thing. Probably what I left out is "well done." I want this product in a well done package.

So, is there a well done version of something like this out there? I should stress that I do not spend most of my time looking for this. I'm too busy getting things done in my current flow to reinvent a new one.

kyllo2• 13 years ago

Good sell, I agree with all the points the author makes, but as Linus says, "Talk is cheap, show me the code."

pcmonk9• 13 years ago

I'm working on it...

kyllo2• 13 years ago

Thanks, it sounds like a cool project. Please do post it here when you have an alpha version, I'd be interested to see it.

seanmcdirmid2• 13 years ago

Was the article flagged?

vinceguidry• 13 years ago

I find myself looking for graphical interfaces to things I already know fairly well but don't really care to drop into shell every time I need to use it. I have an SSH tunnel manager and an rsync manager. I use Sequel Pro instead of the mysql command line.

In fact, I really only find myself using the command line for, you know, running commands. If I find myself there a lot, I'll check myself and start to think, "can I start putting all these things I'm typing into a Rake task or shell script? After all, the more I'm there, the more chance for user error.

I think the only major task I still use a terminal for is looking at log files. I intend to eliminate that soon enough when I can 12-factorify the app I'm responsible for and find a decent graphical log viewer.

chipsy• 13 years ago

I don't see graphics as a silver bullet, but I do think they're underused or misused. Graphics provide a wider variety of feedback; they let you see graph structures and summaries of data, and allow connections to be spatially represented.

All of which means that they excel at connecting the big chunks of the system that build on top of some model of service provision and protocol definition, but poor at engaging in the "direct dialogue" of imperative or applicative computation.

The part that makes people say "it never works" is that we're almost always thinking from an inverted model, where we know how things scale up and down with text, and apply that learned experience to graphical systems. Subsequently, our engineering instinct leads us to put a similar degree of granular agency into graphical coding and allow the standard constructs (looping, branching, subroutines) to appear in it, which massively increases the complexity of the system and brings us away from problem domain thinking.

My recommendation now is - figure out a data structure that could be interesting to visually edit. Make a good editor for that. Allow imperative code a runtime model for accessing it. Allow imperative code to be called from it. Then it stands at least some chance of not being a bad wheel reinvention.

pcmonk9• 13 years ago

I understand what you're saying. My hypothesis is that (some version of) Lisp itself, being a tree, could be interesting to visually edit. All my code, by the way, can be directly accessed from Racket (and vice-versa), so it's definitely not a requirement that the whole program be written graphically. I want to make it a smooth transition from textual to graphical programming.

Graphical Programming: I Really Hope This Is the Future

Related Stories