Turn any application that uses stdin/stdout into a WebSocket server

by adito • 13 years ago

311 88 13 years ago

xxchan• 13 years ago

Amazing how the old is new again. Welcome back, CGI! I hope everyone's aware that this is just a toy and should never be used to do any real work, because most command line tools were never written to be exposed to the internet at large.

baq• 13 years ago

caveat: real work != internet. cgi is still very useful for those super small internal one-shot scripts.

microtonal• 13 years ago

Well, for some folks it works fine on the internet as well:

http://www.mail-archive.com/fossil-users@lists.fossil-scm.or...

tl;dr: sqlite.org and fossil-scm.org run using a simple HTTP server via inetd and some parts of the website are CGI scripts. In 2010 they served over a quarter million requests per day on a tiny VPS with only 3% CPU load.

_ak• 13 years ago

CGI (when used with suexec) is still the tool of choice for your usual low-coat shared webhosting provider, and it's still good enough if all your customers want to do is deploy Wordpress, phpBB and maybe some custom PHP and Perl CGI scripts.

lhaussknecht• 13 years ago

Take a look at how the Leap Motion works. You connect the Leap Motion to your usb port, a demon/service interacts with the Leap Motion and opens a websocket server on localhost so that websites can interact with the hardware device without browser plugins.

X44• 13 years ago

Thanks lhaussknecht That's also what I thought! I don't understand the negativity here!

"Oh, this is a toy. Oh, this doesn't meet my Enterprise needs. Oh, this isn't EAL7¹ Certified, I can't run my Atombomb defense system with it."

Dear Ladies and Gentlemen, you have to admit that this is ueber useful for so many scenarios, that are out of the box, you're thinking in.

I can now hack a firebase clone, just for fun and my prototypes don't require the setup of dozens or a hundred npm packages. Besides the coolness of NodeJS, there are still some use-cases for hackers with this.

1. http://en.wikipedia.org/wiki/Evaluation_Assurance_Level#EAL7...

fit2rule3• 13 years ago

Whats so toy-like about this? Its no toy but rather a useful method of tying differing software systems together .. I can think of many productive uses for this.

nostrademons2• 13 years ago

Problem is that it requires that the app handle all the security aspects of talking to the Internet, because STDIN/STDOUT is an unrestricted protocol. Normally when you write a webapp you have a well-tested parser to validate the HTTP request for you, and you have safe APIs to access parameters, and your templating language auto-escapes dangerous characters for you, and the response goes through middleware to prevent XSRF attacks, and the whole thing is written in a safe language where there's no possibility of buffer overruns.

With this, you have to handle all of that in your application binary. You can certainly do this. However, if you are asking why you need to do this, you are almost certainly not ready to do this.

I agree that this is a pretty useful method of tying software systems together. However, one of the the systems you're tying together is the public Internet, where all sorts of danger lurks. Writing secure parsers is hard. I've written an HTML parser [1], one of the most well-tested ones out there (I spent the better part of a year doing nothing but testing & debugging, and ran literally billions of documents through it), and Google's security team still found 2 bugs in it with about a day's work.

[1] https://github.com/google/gumbo-parser

asb• 13 years ago

The first thing that came to my mind was djb's tcpserver which as far as I can see does much the same thing http://cr.yp.to/ucspi-tcp/tcpserver.html

gwu78• 13 years ago

tcpserver does not pipe input. It sets certain environmental variables, closes fd's 0 and 1 and runs another command line program. It has no quivalent to QUERY_STRING. Very different from CGI or this Websocket contraption. Much safer.

But the parent says that "most command line tools were never written to be exposed to the internet at large", and tcpserver is a command line tool that runs other command line tools, so is tcpserver OK to use for "real work" by his standards? Maybe we should use a GUI tool "written to be exposed to the internet at large"?

pmelendez2• 13 years ago

CGI won't maintain an open connection though

sciurus• 13 years ago

27 years since inetd came into being, and now we've made it web-scale. ;-)

http://en.wikipedia.org/wiki/Inetd

I actually don't mean to be snarky, this program acknowledges its heritage and is a cool hack!

phaed2• 13 years ago

This is beautiful. I can think of a dozen use cases for this right now for one of my pet projects.

schappim• 13 years ago

Call me unimaginative. What uses are you thinking?

phaed2• 13 years ago

I run a popular Minecraft server, we have tons of scripts between the mc server, forum, admin interfaces, and all the other supporting stuff we run. This makes it so easy to create new functionality that interfaces with our various scripts. I'm hacking away as we speak.

schrodinger• 13 years ago

anyone getting an HTTPS warning? It's telling me that github.com cert was signed by an untrusted issuer...

tlrobinson5• 13 years ago

Not on Chrome.

andyjohnson0• 13 years ago

Ok for me. Latest Chrome 31.0.1650.63 on Windows.

mappu• 13 years ago

Chrome tells me it's valid (signed by DigiCert) with a sha1 thumbprint of ‎d7 12 e9 69 65 dc f2 36 c8 74 c7 03 7d c0 b2 24 a9 3b d2 33.

Check your system time?

dutchbrit• 13 years ago

Check your date/time

tlrobinson5• 13 years ago

Basically the same thing in Node.coffee, just because:

    { Server } = require 'ws'
    { spawn } = require 'child_process'

    command = process.argv[2]
    args = process.argv[3..]

    wss = new Server port: 8080
    wss.on 'connection', (ws) ->
      ps = spawn command, args
      ps.stdout.on 'data', (data) -> ws.send data.toString()
      ws.on 'message', (data) -> ps.stdin.write data.toString()
      ws.on 'close', -> ps.kill()
      ps.on 'close', -> ws.close()

(Needs a bit more error handling)

thomasahle• 13 years ago

A nice example on Node. Only I don't quote get the ',' syntax. Is it a scoping thing?

jbrooksuk• 13 years ago

It's a Coffee syntax thing.

  ws.on 'close', -> ps.kill()

That compiles to:

  ws.on('close', function() {
    return ps.kill();
  });

kaoD• 13 years ago

Nice! I love Node for these kind of asynchronous handling of data, it fits perfectly. Keep in mind you can go down the stream route. Instead of:

    ps.stdout.on 'data', (data) -> ws.send data.toString()
    ws.on 'message', (data) -> ps.stdin.write data.toString()

Just do:

    ps.stdout.pipe ws
    ws.pipe ps.stdin

I started using Node before Streams and I often forget piping too!

I still have a question for Node.js experts... would the 'close' events had to be handled then? Streams handle and propagate the 'end' event just fine, but should the underlying resource be closed? I guess killing `ps` is mandatory since ending `stdout` won't kill the process (or will it?), but is closing `ws` still mandatory too?

tlrobinson5• 13 years ago

I thought about using pipe but the WebSocket library doesn't implement the Stream API, does it?

wpears• 13 years ago

If the socket is terminated mid-command (thus emitting 'close'), one would want the process to be killed. Also, the socket doesn't know that no more transmission is required just because one particular stream is over. That is why close is called manually on the socket after the stream emits a 'close'.

This all provided you use a streaming web socket module like https://github.com/maxogden/websocket-stream

pmelendez2• 13 years ago

Sorry if the question is too basic (I don't code in Node.js) but where is the websocket handshaking and the envelop management on this?

alexkus• 13 years ago

{ Server } = require 'ws'

CGamesPlay• 13 years ago

Creating a new server will start listening for connections, and the connection event will only be emitted after the handshake has finished. http://socket.io/ has more info on how the node api works.

    wss = new Server port: 8080
    wss.on 'connection', (ws) ->

joewalnes7• 13 years ago

Author here. Indeed! The very first version of websocketd was actually written in Node. It's great for this kind of stuff.

The reason I switched to Go was to make distribution/installation for end users easier.

AsymetricCom• 13 years ago

Of course, Perl has been doing this for over a decade. Nobody thought it was remarkable enough to write an article about it though.

bsg753• 13 years ago

Curious as to the approach of pulling a project local Go distribution instead of using the default. Is this to ensure Go version compatibility?

minikomi• 13 years ago

Awesome.. I use the same kind of thing to monitor adb output sometimes:

https://github.com/minikomi/pipesock

also in go.

Edit: Doesn't do any receiving, only pipes to a socket what it gets.

babby3• 13 years ago

That image made me chuckle. Had to confirm that it was just sourced from google images; but still. :)

hepek• 13 years ago

isn't this reimplementing netcat -e command?

http://linux.die.net/man/1/ncat

Sanddancer• 13 years ago

Seems more like an inetd to me. Give a command and run it again and again.

joewalnes7• 13 years ago

It's similar to netcat -e and inetd, except it also handles the WebSocket protocol (negotiating the handshake and message framing), so your apps don't need to.

bct2• 13 years ago

Basically, but it also handles some details specific to the websockets protocol.

gren• 13 years ago

    websocketd --port=8080 bash

Then in the browser console:

    ws.send("ls")

I just ported bash to the web :)

mrspeaker• 13 years ago

Ha ha! That's so cool. Hey, off topic... what's your IP? ;)

huhtenberg2• 13 years ago

Don't know his, but I've got the best one 127.123.234.345!

awestroke• 13 years ago

ws.send('cat $HOME/.ssh/{id_rsa,id_rsa.pub,known_hosts}');

ws.send('history');

ws.send('echo "' + pubkey + '" >> $HOME/.ssh/authorized_hosts');

jamescun• 13 years ago

Combine that with Term.js[1] and you could have a fully functioning terminal in the browser.

[1] https://github.com/chjj/term.js

glomph• 13 years ago

There is already https://code.google.com/p/shellinabox/

X44• 13 years ago

The possibilities are endless, just add a websocket proxy as a front-server and you can allow dynamic and even secure control of an entire infrastructure (given that you add auth layer). Here's something you could use for that: https://news.ycombinator.com/item?id=6880487 Or just setup haproxy with it.

tlrobinson5• 13 years ago

So basically replicate SSH in a half assed way?

babby3• 13 years ago

For some stupid reason I hadn't considered that logging stdin/out/err to a web interface for my node.js web apps via websockets. To think, I made a browser-side web IRC interface, and didn't consider this. It would be so useful for my clients, who don't know how, or find it too archaic to ssh in.

To just open up the admin area and see what's going on, provided they're indeed full-permission admins. Then, to actually send input from said interface, that could make ssh'ing into the server something one need not do often beyond initially setting up the app.

Thanks for the idea. Assuming we're not running as root, and the admin side of things is secure, am I not considering any critical pitfalls of this approach? Also, any frameworky cmsy thingers that already do this? Is this new, am I a unique snowflake?

X44• 13 years ago

yup new, but use a restricted jailed and chrooted shell. Ask your #sysadmin of choice =) You can even build a REST like API wrapper for this. That's what I was going to do.

babby3• 13 years ago

Yeah, I thought the same thing concerning the REST api, after mulling it over a bit. Being tied directly to an app could be a bit awkward in practice (Bit of a problem when either can take either's process down).

guard-of-terra• 13 years ago

Is this just xinetd for the ignorant?

X44• 13 years ago

no. you can still chain xinetd.

joewalnes7• 13 years ago

Author here.

If you use this on top of programs like bash, well ermm, you get what you deserve ;).

Here's an example of how I used websocketd to create a little dashboard for monitoring Linux CPU/memory/IO stats. It basically uses websocketd to stream the output of vmstat to a web-page that plots the numbers: https://github.com/joewalnes/web-vmstats

Other useful examples: tailing log files, executing long running job and monitoring output, or interactive querying of datasets that require a long running 'cursor'.

This is not for everyone or everything. Remember that like CGI, a process is forked for each connection so it's not the kind of thing if you want to handle a million concurrent connections on a single server.

However for dashboards, admin tools, quick mashups, visualizations, etc - it's a pretty handy tool.

tlrobinson5• 13 years ago

You should probably include a warning anyway. When I was starting out programming I would do stuff like exec-ing shell scripts from PHP, oblivious to the consequences.

CSDude• 13 years ago

I have a project where small time running tasks requires user input, this would be very nice to have. What would be nice to have is attaching the stdouts of procesess via file descriptors. But I think it could be done easily.

eddywebs• 13 years ago

If the shells script takes input parameters can we pass that and eventually turn it into some kind of web service ?

nitrogen• 13 years ago

By integrating anything shell-based with anything web-based you encounter a plethora of potential exploits and security holes. That said, if you need to run a script that requires command line parameters, and all you have is standard in, then write a wrapper script that reads N lines of text from stdin first, then parses those lines into the parameters you require.

You have to be very conscious of input escaping and validation, though; otherwise your box will be owned in no time.

joewalnes7• 13 years ago

I don't recommend any kind of input parsing in a shell script. You're better of having websocketd fork something like Python, Ruby, Perl, PHP, etc.

Like CGI, websocketd will pass additional URL path and query parameters as environment variables.

So you could access something like ws://somehost/?a=b and you'd access the QUERY_STRING environment variable in your script (which contains "a=b"). Virtually every scripting language already has a library for parsing CGI environment variables so you can do that.

Basically, websocketd tries to emulate CGI as much as possible, but provide WebSocket streams instead of HTTP responses.

ykumar6• 13 years ago

Here is a live version http://runnable.com/UqkIJXqriJwGAATm/basic-count-example-for...

Turn any application that uses stdin/stdout into a WebSocket server

Related Stories