Rich Command Shells

http://waywardmonkeys.org/2014/10/10/rich-command-shells/

by BruceM • 12 years ago

187 98 12 years ago

BruceM5• 12 years ago

I will probably write a "More Rich Command Shells" to cover things I missed here that are important in some way (like Apple's MPW).

The next thing I want to write though is about building something that has a command shell UI today and thinking about how to do so in a flexible way that works with multiple output devices.

agumonkey2• 12 years ago

Thanks for that article.

I'm often thinking about how to present/"unify" these:

  - command (statements validated by some token),
  - repl (expression, less side-effect driven than the previous one), 
  - unary/immediate mode (each input event is interpretable right away, direct mapping between input and action), 
  - n-ary/non-immediate mode (some parsing occurs, gathering a few tokens into a higher level construct)
  - non keyboard based UI/UX (some sofware like Maya have proper separation, most mouse events have direct translation into object methods)

rsync4• 12 years ago

Please do - I liked this first article very much and would look forward to a follow-up.

josephg• 12 years ago

I miss TermKit[1]. Its a real shame that the developer abandoned it after it got a lot of hype. There's a huge opportunity for someone to come along and make either a new terminal encoding which allows rich, interactive output or extend VT somehow to which allows the same. If you could extend the terminal protocol, you might even be able to get it to work over SSH.

HTML+JS seem like the obvious way to do it - even though the web is an awful platform, its standard, cross-platform and fully featured. Its the perfect worse-is-better solution for this.

I think the hardest part would be figuring out how to reconcile browser-like UI events and file streams. Maybe you'd need to make a standardized event serialization format so your process could receive serialized events via stdin or something like that. I think it'll be a really hard sell if we have to abandon our unix pipes entirely.

[1] https://github.com/unconed/TermKit

Previous hackernews discussion around termkit: https://news.ycombinator.com/item?id=2559734

lallysingh• 12 years ago

HTML+JS seem like the obvious way to do it I think the hardest part would be figuring out how to reconcile browser-like UI events and file streams.

How about Chrome Apps? They've got a mix of file I/O capability and HTML/JS.

pjmlp6• 12 years ago

And this is why UNIX shells feel so primitive.

mozmark• 12 years ago

If you found this interesting, you may also like GCLI (as featured in the Firefox dev tools command line - or you can play with it here: http://mozilla.github.io/gcli/ ).

BruceM5• 12 years ago

GCLI is interesting, but I didn't mention it because I feel like it focuses more on the issue of command parsing / completion.

That said, that's a great topic to cover! I think GCLI does a pretty good job of command completion. The Lisp Machine did as well. I have friends who love some of the router CLIs, but not sure which one(s).

lispm3• 12 years ago

Cisco's IOS is an example.

robbles• 12 years ago

CLOS and CLIM always sounded to me like the names of programs you'd expect to see competing at disc wars, or racing in a lightcycle grid.

tubelite2• 12 years ago

If I might plug a rich command shell I'm developing: http://pigshell.com (Source at https://github.com/pigshell/pigshell)

It provides

- A shell for the web. Runs in the browser, pure client-side.

- File-like abstraction for URLs and other entities exposed by web APIs

- Unix-like style of composing commands using pipes.

- Visualization using HTML

For instance, cp -r /gdrive/<username> /home will backup the contents of your Google Drive to your desktop (see http://pigshell.com/v/0.6.2/doc/gdrive.html for details). Replace /home with /dropbox/<username>, and the same command will back up GDrive to Dropbox. And so on.

While it is already useful, there is still some work, especially around hardening the file abstraction/APIs (and reams of documentation) before before it can be horizontally expanded to support a bunch more APIs and cloud stores. I am actively working on these and expect they'll take ~2 months.

qznc• 12 years ago

Interesting. Why do you think that files are good abstraction for a web shell?

tubelite2• 12 years ago

It's the old "if you have a hammer, everything looks like a nail". Since I'm a filesystem engineer, everything looks like a file :)

Seriously though: the file is a powerful and familiar abstraction. The primary motivation for pigshell was to provide a common minimum abstraction across different web APIs which would enable basic data movement and backup.

That said, pigshell commands actually pass objects across the pipeline; files are a kind of object. For instance,

  cat http://pigshell.com/sample/life-expectancy.html | table2js -e "tr" foo country data | head | printf

extracts data from an html table, converts them to plain Javascript objects and prints their json representation.

judk2• 12 years ago

Are you familiar with Plan9? Everything is a file. Everything.

akavel4• 12 years ago

Given all of that, and also Oberon -- '80s too, and even the Engelbart's Demo -- '68!, I totally every day wonder and can't understand why in 2010's all of the "modern" OSes still provide the developer just with text-only consoles??... okay, maybe for end-users there was a jump (in interfaces and features), and maybe it was significant indeed (movies editing, 3d modelling/sculpting, possibility of instant communication with significant fraction of all the people in the world, to bring up some). But it still feels like even for them, some things were there, and now are not (sorry for vagueness, but I don't have much time now to think about examples, unfortunately. When I'm gonna finally start this blog thing, one day...).

Any theories, anyone? I'm really curious. Still believe this can be improved, and work on some ideas in my free time, but I often wonder why I have to, and I can't already use those beautiful ancient features?

pjmlp6• 12 years ago

Because many seem to be willing to go back to the 70's (vi/sh) instead of embracing the progress you mention.

Since I discovered UNIX, with Xenix in 1994 followed by many other variants, I always looked for ways to replicate such workflows in UNIX.

At the end of the day, Mac OS X and Windows communities are more welcoming to such ways of working.

akavel4• 12 years ago

But, again, why? this can still have many reasons: not willing because of technological obstacles? complexity? because they shown in fact to be somehow worse? (e.g. pure text easier to read, mold, and more universal?) because of some social obstacles? (if yes, of what kind?) what else could that be? (those are just a few of my random quick theories, have I described all possibilities?)

edit: the comment by zorbo seems an interesting answer to my question, but I'm still not 100% convinced; also, I must still ask how can we be sure we're not victims of a text-mode "Stockholm Syndrome" in this regard?

astrobe_• 12 years ago

Because many seem to be willing to go back to the 70's (vi/sh) instead of embracing the progress you mention.

Just because it's old, doesn't mean it's only about nostalgia.

Besides, I for one can't have this kind of nostalgia because in the '70, my fingers were a bit too short.

There is something extremely convenient to be able to use the same environment everywhere, with minimum prerequisites, because terminal emulation software is very common (probably as common as Web browsers).

I also find it more natural to deal with the system with a CLI when you spend your time editing source code or doing sysadmin stuff; it doesn't break the flow. Of course it's totally different if you're a graphic artist or a Web developer.

Sometimes CLI makes more sense, sometimes GUI makes more sense. But accusing people of being stuck in the past, that makes no sense.

richardjdare• 12 years ago

That's a very interesting question.

I'd guess that the technical decisions that differentiated early unix from the lisp machines etc. were based on hardware and cost limitations.(the development of unix is something I'd be interested in reading more about) Then as unix took off, these technologies became entrenched, with a kind of apologetics developing amongst users who liked those systems, or never used anything else.

There's an interview with Bill Joy where he said that Vi's famous modal interface was merely a result of his poor quality terminal and network connection, and now it's used by millions of programmers. I'm not saying that these users are wrong about the benefits of modal editing or the unix text interface etc, but that these are discovered benefits that are then used to retroactively construct a containing narrative of justification that often walls people off from alternatives.

I'd also hazard a guess that many of these technologies were simply inaccessible and/or unknown to the mass of post-pc-revolution developers who create the bulk of our present systems. The number of devs who had access to Lisp machines was small compared to the number who came in with Unix with its success in the workstation market, and tiny compared to the number of devs who came in through home computers/PCs.

I myself grew up with 80s home computers, moving to the PC and Linux in the late 90's. Lisp machines, Oberon etc. were unknown to me until a few years ago, and information about them is still hard to come by. Nearly all the lisp machine sites are web 1.0 with broken links etc. Very few of them try to market the benefits of these systems to people in the wider community.

After I got interested in lisp and lisp machines (thanks pg!) I spent a weekend getting that notorious leaked Symbolics Genera distribution working on Linux, inside virtual box on my mac. After a few hours in the Listener, with its rich output and superior incremental help system I wanted to show it to everybody I knew. It was one of the most incredible things I had ever seen on a computer. For the next few weeks at work I was occupied with the constant thought, "This is not as good as Genera". It's had a huge impact on how I think about software, even my indie game development.

a3n5• 12 years ago

the development of unix is something I'd be interested in reading more about

https://en.wikipedia.org/wiki/Unix

http://cm.bell-labs.com/cm/cs/who/dmr/cacm.html

http://www3.alcatel-lucent.com/bstj/vol57-1978/articles/bstj...

http://cm.bell-labs.com/cm/cs/who/dmr/chist.html

http://www.unix.org/what_is_unix/history_timeline.html

http://www.faqs.org/docs/artu/historychapter.html

vram22• 12 years ago

the development of unix is something I'd be interested in reading more about

http://www.levenez.com/unix/ has a good amount of info and links on the history of Unix and the people who worked on it. I came across the site a while ago and browsed it some. Interesting stuff, some of it at least. Recursively traversing the links should give even more.

akavel4• 12 years ago

Hm, that's a really interesting theory, and the stories and related arguments too, thanks a lot! I didn't think of it before.

In fact, I started hearing of all of those OSes fairly recently too, but I still see myself as a "youngster" in programming world, and thus I thought those technologies were more well known at that time. Also, that they were more well known to Important Figures and Big Companies then, so they could try to build on those ideas.

But now I start to recall, that most of those OSes are usually reported to be used and deployed quite rarely (Lisp Machines e.g. often dismissed because of high price and hardware requirements), so that could have limited their exposure. And also, I'm ashamed to admit I forgot about this (oh the times!), it must be taken into account that there was no Internet at the time, so this also made sharing of ideas harder than now...

a3n5• 12 years ago

For me, because I can't compose strings of gui clicks into an ad hoc list of transformations on text. And I do that a lot in the shell.

There is also no command history for my gui clicks; once you've clicked on something, there's no way to reuse that click in some slightly different way.

Dewie• 12 years ago

It's perfectly possible for a GUI to have a command history, if the GUI is just a front end to a CLI. Then you can click around when you want to explore, and can go to over to the CLI when you know what you want to do and want to establish a more streamlined workflow. Or even click around and do stuff, and then extract a script from that recent history of clicks and interactions.

Some R (programming language) GUI frontend does it in this way.

vinodkd3• 12 years ago

well, not at that minute level, but every application worth its name has an undo log - and that's a pretty close functional equivalent, IMO.

And: applications that have Macro recorder features (Office and many text editors) allow you to actually record your actions - clicks or otherwise - into code that can be rerun.

toddkaufmann• 12 years ago

In CLIM, a graphical presentation is only a representation of an underlying type. Types will have default representation (e.g., basic text like you would get from printf), but you can also override this using with-output-as-presentation (see p. IV-1.30 of [1]).

So, when outputting a number, you could display it in the output stream as a circle of radius N. Then, any input which wanted a number as argument (using "(accept 'integer ..)") would let you type a number or click on that circle.

This could be extended to shell command parser for richer input choices (rmdir can only accept directories, cat won't). Existing utilities could support a "richly typed stream of objects" (graphical ls) or a wrapper or output processor could add it. The latter has the advantage that not all commands would need to be aware, and it could be multi-functional (could work on "ls|grep", "find", "tar tf" output--but you might need to specify type hints).

I think it would be handy to have all my recent files, directories, outputs handy in a list on the side (with keyboard shortcuts too; I'm not a clicker..).

So in no way should your command history have to be compromised just because you have alternative input methods available.

[1] A Guided Tour of the Common Lisp Interface Manager, http://3e8.org/pub/scheme/doc/lisp-pointers/v4i1/p17-rao.pdf

ivanca• 12 years ago

Strange he didn't mention Xiki: http://xsh.org/ http://xiki.org/

zorbo• 12 years ago

Text is the universal interface. You can do things with it. You can strip it, cut it, transform it, send it to other places. Humans can read it, programs can read it, your printer can output it. It can be sent to web APIs, it can be stored anywhere. It's compressible, can be colored and can be copy-pasted and is infinitely extendable. Thousands of protocols run over it.

The command line works with text. The command line remains the best interface I've ever used. It's user friendly, composable and available everywhere. It's easy to automate and easy to extend.

I wish the "command line with pictures" idea would just go away already. It adds nothing for the general public. I can already view pictures on remote machines with X forwarding.

Command line with pictures never made it, because there are ten competing standards. With text, everybody just agreed on ASCII and now Unicode/UTF8. Text has hundreds of ugly clutches on top of it (Extended ASCII, ANSI, Escape codes, etc, etc). It still works. It's still simple. It has its problems, but nowhere near as many problems as GUIs.

Those who don't understand Unix are doomed to reimplement it... poorly.

aduitsis• 12 years ago

Exactly as you indicate, even in "classic" terminals we have features like unicode support, color, geometry reporting, mouse reporting, window title manipulation, arbitrary cursor movement, etc. All these are in widespread use today. These features could not have been implemented in a real tty, but nowadays I think it makes sense to have them. Maybe some are a little kludge-ish, but are we really arguing about their usefulness?

So why shouldn't we have a couple of extra escape sequences where the terminal could, e.g. draw an arbitrary bitmap? You are right, that would certainly require some sort of standard becoming prevalent, but why would a little extra capability (that doesn't break compatibility) be a bad thing?

a3n5• 12 years ago

I wouldn't mind it.

lmm• 12 years ago

Like extending email, there's a a chicken-and-egg problem. What terminal emulator is going to add a command that no program uses? What program is going to use a command that no terminal implements?

By all means implement this thing - I wouldn't be surprised if there were already terminals that did that. But personally I'd be surprised if it caught on. Compatibility matters.

lorenzfx• 12 years ago

For a start I think it would be really awesome, if the terminal could display html (+css).

rsync4• 12 years ago

"Exactly as you indicate, even in "classic" terminals we have features like unicode support, color, geometry reporting, mouse reporting, window title manipulation, arbitrary cursor movement, etc."

Is that true ? For instance, I am not aware that (for instance) the OSX terminal app does mouse reporting or geometry reporting.

Does default xterm ?

pjmlp6• 12 years ago

I rather use objects and function composition in a REPL.

Kudos to Microsoft for bringing to Windows a little bit of Lisp Machine experience with Powershell.

MrBuddyCasino2• 12 years ago

I have used PowerShell, and currently learning Clojure. They never occurred to me as being similar - or did I miss the point?

userbinator• 12 years ago

It's also far easier to help someone over the phone/other "blind" medium of communication with a CLI than a GUI, as you can tell them to press keys on the keyboard and read back the output that shows up, instead of asking them to find and click on things - followed by their thousand-word response (a picture is worth...) describing in exquisite detail everything on the screen except for the exact information you wanted...

That's not to say CLIs are perfect for everything, because there are definitely use cases where a GUI makes sense.

BruceM5• 12 years ago

Okay.

But sometimes, the logical output format for the response from a command is going to be an image. And sometimes, you'd like to view that inline with your commands in the terminal.

That isn't that radical a suggestion. And it doesn't turn anything into a big old GUI suddenly. :)

tbrownaw• 12 years ago

Clearly, the proper solution for this is pervasive hi-def videoconferencing. :)

anon13854• 12 years ago

Text isn't the universal interface in Unix, byte streams are. You can quite happily send non-textual control characters and such around in Unix, or pipe data containing NULLs from one process to another. 'Text' is a very seductive abstraction, but it's one of the most brutal to work with once you start interacting with the real world and have to give up on ascii and deal with encodings and unicode and so on.

Putting commands and data inline is a recipe for disaster and a million command injection exploits. The Unix philosophy has broken the minds of generations of programmers. It leads them to doing things like concatenating strings to build SQL queries or doing IPC with ad-hoc regex-parsed protocols or using a couple of magical characters to indicate that the contents of a variable should be parsed and executed instead of just stored. Take a read of some of the earlier threads on HN about Shellshock, and you will find numerous people blaming Apache for not "escaping" the data it was putting in a shell variable. As if it even could.

Even Unix nerds have at least partially internalised the dangerousness of the paradigm -- "don't parse the output of ls" and so on. The fact that the Unix paradigm (passing everything as strings with magical characters and escape sequences) is broken for the most fundamental computing tasks like working with file names ought to be a damning inditement of the paradigm. Sadly people merely parrot the rote learned lesson "don't parse ls because file names can't be trusted", without thinking about all the other untrusted data they expose to unix shells all the time.

Just this week Yahoo got exploited. At first people thought it was Shellshock, but no, it was just a routine command injection vulnerability in their log processing shell scripts. A problem blighting just about every non-trivial shell script ever written.

The usual reply is "don't use shells with untrusted data". But auditing where any particular bit of data came from can be just about impossible once it has been across several systems through programmes written in a variety of languages, stored on a file system, read back and so on. The only sane solution is to never use shell scripts.

Like the C memory and integer model makes writing secure C code borderline impossible, the Unix "single pipe of bytes that defaults to being commands" paradigm makes writing secure shell scripts borderline impossible.

Unix needs to be taken out back and shot.

rkrzr3• 12 years ago

Agree that text is being abused in Unix all the time. The problem with passing everything around as text is that you cannot reason about anything, because everything is of the same type. One big advantage of object-based systems is that they can catch type errors and notify you of the problem. Text pipelines will simply break because one of your implicit assumptions didn't hold.

jschwartzi• 12 years ago

I would not be half as capable of troubleshooting in my work as I am if I couldn't parse random byte strings on the command line. That alone is why I use Unix for embedded development. Powerful things are often dangerous. For instance, consider nuclear power, motor vehicles, rocket engines, medicine, plastics, et. al..

So how do you provide the same level of capability as C and UNIX but without mixing data and commands? Is there some alternative paradigm that is just as powerful but safe?

mwcampbell4• 12 years ago

Putting commands and data inline is a recipe for disaster and a million command injection exploits. The Unix philosophy has broken the minds of generations of programmers. It leads them to doing things like concatenating strings to build SQL queries or doing IPC with ad-hoc regex-parsed protocols or using a couple of magical characters to indicate that the contents of a variable should be parsed and executed instead of just stored.

Yes! And then to compensate, they have to "sanitize" untrusted input to their systems. I had a meeting yesterday with a developer and a project manager at an organization that wants to work with my company to integrate one of our products with one of theirs. I mentioned the possibility of submitting some data in JSON format to a web API on their end, and the project manager asked about the risk of code injection attacks, by which he apparently meant SQL injection. I had to assure him, based on my knowledge of their tech stack (Node.js, CouchDB, and naturally, JSON) that code injection wouldn't be an issue. My point is that the common abuses of strings by Unix and web developers have led to well-known and widely feared security vulnerabilities which just don't exist in software that's built on a foundation of properly structured data.

See also this classic by Glyph Lefkowitz:

https://glyph.twistedmatrix.com/2008/06/data-in-garbage-out....

pyre2• 12 years ago

Take a read of some of the earlier threads on HN about Shellshock, and you will find numerous people blaming Apache for not "escaping" the data it was putting in a shell variable.

It seems to me that this is the result of Cargo Cult Programming. People know that SQL strings, and user input need to be 'escaped,' so they just think "Obviously this needs to be escaped too! It's user input!" Yet they don't realize that they are trying to put a square peg in a round hole. They just know that pegs go through holes, so they keep banging away at it.

Also, it's always amazed me that there was never some sort of 'standard' way to shell-escape things, even though the shell has been around for ages. Why can't I generate a shell string in the same way that I generate a SQL string? E.g.:

  sprintf("mv %t %t", src, dest);

Where "%t" is a special token that shell-escapes the input (e.g. "My File Name.txt" => "My\ File\ Name.txt"). Instead it's something where people continue to use ad-hoc, incomplete, of 'implemented everywhere' solutions to this.

Note: I don't generate SQL strings with sprintf(), but it's a close approximation of:

  execute('select * from table where id = ?', id);

felixgallo4• 12 years ago

so in one corner: the ability to compose small, comprehensible functions with data streams, delivering staggering riches to the world.

in the other corner: the complaint that sometimes, people can put carriage returns in their filenames and screw up the output of `ls -1`. With no suggestion of what would replace composed functional streams.

Nice try Lennart.

rsync4• 12 years ago

"Like the C memory and integer model makes writing secure C code borderline impossible, the Unix "single pipe of bytes that defaults to being commands" paradigm makes writing secure shell scripts borderline impossible. Unix needs to be taken out back and shot."

What alternative do you advocate/propose ?

Genuinely curious ...

hderms• 12 years ago

I agree with you to some degree. It's incredibly easy to write insecure software if it ever touches a shell.

deeviant• 12 years ago

I agree with you and disagree at the same time.

Command line interfaces are extremely powerful, simple to implement, extend and leverage, but, GUI's have their own advantages mainly in information density, context clues, and ease of operation. There is a world for them to both live in, but I would disagree with OP's approach of bring graphics to command line and suggest the approach of bringing command lines to GUIs.

I have already embedded a command line terminal in several GUI's I've built for industrial machines. The users can see the corresponding command line command pop up in the terminal when they click a GUI button and are free to type commands into the terminal. They can also define a script or batch of commands and easily assign it to a GUI button.

It was amazingly efficient for engineers to work with prototype machines whose functionality and controls were constantly in flux.

Downsides are that exposing a terminal to a potentially non-expert operator is a potentially dangerous, and I had to spend a good deal of time validating and sanitizing inputs(and risk missing something).

In the end, the terminal was removed in the production GUI, but was extremely helpful in the creation of a functional GUI that worked for the operator, rather than the opposite. In the case where the operator is always going to be an expert and is working with a complicated or constantly changing process, I could see a CLI/GUI interface working well.

josh-wrale2• 12 years ago

How about a declarative and idempotent shell? Well, more declarative and idempotent than bash + the GNU core utils.

_mhr_• 12 years ago

How would this be useful?

josh-wrale2• 12 years ago

This comment thread discusses the methodology. https://news.ycombinator.com/item?id=7378764#up_7380392

lispm3• 12 years ago

If you come to Freiheit.com next week (16. Oct 2014) to the Clojure User Group Meeting in Hamburg/Germany, I'll demo the Dynamic Windows user interface of a real Symbolics Lisp Machine. Including its command shell called 'Dynamic Lisp Listener'.

http://www.meetup.com/ClojureUserGroupHH/events/207314372/

cpach• 12 years ago

Cool! Do you know if the demo will be recorded?

lispm3• 12 years ago

It won't.

Scuds• 12 years ago

This demo appears to be running on an emulator instead of an actual antique piece of hardware but you can get the idea.

https://www.youtube.com/watch?v=o4-YnLpLgtk

the AI winter is something you don't hear too much about. http://en.wikipedia.org/wiki/AI_winter

vinodkd3• 12 years ago

Slightly off-topic, but the recent surge in responsive UI had me thinking thus: If we are now driven to merge UI logic across different graphical devices, can we think of apps that span both textual and graphical devices?

After all, the core functionality of the application remains the same. To use a simple example: when you search for a product online, get a search result list and then select one from the list, couldnt this flow be modeled just the same in both gui and text interfaces?

I'm thinking back to the turbo-pascal style applications of the past that produced full-blown IDEs in text, or the wordstars/wordperfects of yore: the UI model that sits behind those apps cannot be much different in principle from the modern day equivalents.

Even farther back, there was a time (and probably still is for college assignments) when cli applications had a prompt-read user input-respond cycle, replete with text-based choices to select from and so forth.

What if we were to merge the two worlds instead of trying to get one to confirm to the other?

fiatjaf• 12 years ago

What about the Aurora/Eve thing? https://www.youtube.com/watch?v=L6iUm_Cqx2s

It is a shell, isn't it?

Rich Command Shells

Related Stories