Pigshell – Unix the Web

http://pigshell.com/v/0.6.2/

by nmcfarl • 12 years ago

333 78 12 years ago

pigshell.com

nmcfarl• 12 years ago

More in this comment by the author: https://news.ycombinator.com/item?id=8436716

bilalhusain• 12 years ago

The psty links in README file are broken. A working link is http://pigshell.com/v/0.6.2/psty.py

As mentioned in the README, you need to run psty server locally and establish websocket connection (mount in pigshell). This will enable you to harness local filesystem from pigshell.

kolev• 12 years ago

This is beautiful!

tomw1808• 12 years ago

I just had a glance about the functionality, but for my understanding, this is the best idea right behind ifttt. Pipelining all the web's things with well designed interfaces on a shell. Even if the product itself would be unstable, alone the idea is great enough for a gold-medal. Keep it up!

tubelite17• 12 years ago

Author here.

- Please use a desktop browser (Chrome preferred). While mobile browsers kinda-sorta work if you're just clicking through the examples, keyboard input doesn't work on mobiles (yet).

- Pigshell has few users (in fact, using the plural is a slight marketing exaggeration) so please bear with me for any teething issues, broken links, inadequate documentation...

- One of the practical use cases is copying files across Google accounts, and backing it up to your desktop. (Details here: http://pigshell.com/v/0.6.2/doc/gdrive.html) The nice thing about a unix approach is that the exact same cp command can be used to copy GDrive to Dropbox, or any other supported filesystem.

- Some more D3 visualization examples here, using Mike Bostock's gist viewer: http://bl.ocks.org/ganeshv (Do use the "Open in a new window" links, like this one: http://bl.ocks.org/ganeshv/raw/46cc526d41d16995ccdc/).

sprayk2• 12 years ago

I looked through the four documentation links on the right and was unable to find out how to cat a Google spreadsheet as anything consumable by the shell. I took a guess and tried cat -o gdrive.fmt=txt and csv, but those were not fruitful. Is this possible currently?

edit: found mimeMap.js and saw tsv was an option. That didn't work either.

tubelite17• 12 years ago

IIRC the Drive API does not support those conversion options. The way forward would be for an xlsx parsing JS library to convert it into CSV or a stream of Javascript objects. I didn't find any GPL-compatible library last time I checked.

See https://developers.google.com/drive/web/manage-downloads for a list of export format possibilities. For spreadsheets, the only options are xlsx, odt and pdf.

hbbio• 12 years ago

Good work, congrats!

I played with the same idea a few years ago, but never went that far :)

cf. https://github.com/hbbio/webshell

gmaster1440• 12 years ago

Are write operations possible with pigshell? If I connected my Facebook account and want to post a new status update or add a picture to an album, it doesn't seem like I can do something like this:

pig> echo "Testing status update" >> /facebook/me/posts

tubelite17• 12 years ago

That should be

  echo "Testing status update" > /facebook/me/posts/

I tried it and it WorksForMe (TM)

That said, two caveats:

- The FB filesystem hasn't been worked on in 1+ year.

- AFAIK v2 of the graph API will become mandatory sometime next year and does not allow apps to get lists of friends. Friend list returns only friends who also use the same app... which is pretty disappointing.

fsiefken• 12 years ago

Hi tubelite, on mobile safari and chrome IOS7.04 with an ipad the keyboard comes up after I clicked on this example command: pig:/$ cat http://pigshell.com/sample/photos/bchips.jpg I can type in stuff and after enter the keyboard disappears again. A external bluetooth keyboard might work.

tubelite17• 12 years ago

Yes, that happens, but it is awkward and unreliable. The CLI is a CodeMirror instance (version 2.something), which has known issues with iOS. I need to check if later versions work better on iOS.

linvin• 12 years ago

Ah! Nice feeling that whole of internet as a single (unix) machine.

serf• 12 years ago

reminds me of Plan9 sorta.

crater• 12 years ago

I like the idea. If I'm playing with some facebook data, right now I use whatever language I'm confortable with and use its HTTP request functionality, and than manipulate the data how I want in that language. To achieve the same thing in pigshell I have to write a shell script that manipulates the data. What can pigshell do that I can't just do in another language?

tubelite17• 12 years ago

In theory, the availability of APIs should be enough for the technical user to write their own data extraction routines, connect them to other APIs and so on, using their preferred language and toolchain. In practice, reading up the documentation, writing and debugging code sets up enough of a barrier to entry that it discourages casual usage without a powerful incentive.

The availability of a file abstraction covering many popular cloud services, web resources etc. enables casual, low cost experimentation across these domains, without having to read up each one of the APIs. If something clicks, then it may make sense to write a better "native" implementation.

fiatjaf5• 12 years ago

I'm curious about how are people using this. Is it just a visualization tool for data or what else? Is anyone using it as their primary navigation tool?

tubelite17• 12 years ago

Like I said in an earlier comment, there are very few users right now (cough OK just me cough).

Mostly I have fun with it, e.g. trying out visualizations of interesting data (e.g. http://bl.ocks.org/ganeshv/raw/fada13e56d2c5b7b1a78/)

My Serious Use Case is full and incremental backups of the contents of my Google Drive, both personal and company, to my desktop. Running the Drive desktop app does not backup Google Docs, and you can only sign in as a single user at a time.

basantrajan2• 12 years ago

perhaps stating the obvious here, but with the abstraction pigshell provides of resources on the web, it should be reasonably straight forward to build arbitrary "internet applications" too.

with deferred pipes, split and joins (that pigshell already provides/supports) it should be possible to implement arbitrary work/data flow definitions one desires.

coupled with psty, one could build/harness special purpose "transformers" on the web

pushing it further, one could conceive of setting up pipes between transformers on the web.

of course getting there will require significant work on a security model as well.

[and then someone will figure out a way to interject ads]

fiatjaf5• 12 years ago

CouchDB is already compatible.

progman7• 12 years ago

of course getting there will require significant work on a security model as well.

It could probably be easier than you imagine. The security model could be implemented in psty completely by granting/denying requested resources whose permissions are stored in a database on the psty server, and managed by using long security IDs in the URLs. This way the security model could even be fine-grained. In other words, pigshell itself could be embedded into a secure environment, covered by the permissions of psty.

progman7• 12 years ago

There are so many possible uses cases. For instance,

- Web development toolchain.

- (Multiple) Server administration.

- Control embedded systems. No need for special GUIs there. For instance, simply psty on a Raspberry Pi would be sufficient to present collected data in a nice web graphics on your PC. However for embedded systems I would recommend a reimplementation of psty in a compiled language like C or Nimrod.

mgrassotti• 12 years ago

holy crap this is awesome

jayseeg• 12 years ago

With some practice, or even better, some support libraries, this looks like a great way to quickly explore viz prototypes. Great work!

roywiggins• 12 years ago

Attempting to copy photos out of Facebook albums into GDrive gives me "Exception: SecurityError: The operation is insecure." I'm not running psty locally.

This is super cool, though. There's an undocumented template engine built in it seems? (see /bin/mostliked) which is neat.

Erwin• 12 years ago

Interesting. I'm trying to really go out all the API for my project and was thinking about something like a Mathemathica-like notebook for the local API. E.g. say you want to execute an API call which normally does GET /api/v1/x/y/z; in the "API shell" you type x/y/z and the JSON collection is summarized as a small datatable (not showing all of the 500 rows / 40 columns that might have, but an interesting summary) that you can later interact with.

I'm thinking that such an approach can give the user their own way to explore/modify the data ,especially if you combine this with some kind of pipe lines that let you analyze/mutate/visualize the data without having to write some Python code to access the API (even though with a REST API plus e.g. requests it's simple).

I like the psty idea to integrate with the local system. Sounds like it could also be used to start e.g. emacs on a remote page. Say you have a site with an API that lets you read/write files, you mount it and then can fire up emacs on it via pigshell.

privong• 12 years ago

Is this approach similar to something like flickrfs[0], to make one's flickr account locally accessible? It always seemed like a cool idea (though I never had much luck with flickrfs itself), and it is neat to see the broader utility of psty.

[0] https://launchpad.net/flickrfs

talles• 12 years ago

Error loading root filesystem at http://pigshell.com/v/0.6.2/root.tar: Forbidden

dllthomas• 12 years ago

Have you given thought to exposing this to the regular shell with FUSE?

tubelite17• 12 years ago

The concept of lazy-stream-of-JS-objects and HTML as a display engine underpin pigshell.

I guess it would be possible to adapt the filesystem code to fuse4js. It won't be a drop-in; unix FS ops differ from pigshell's, but it shouldn't be too hard. Personally, I find psty (http://pigshell.com/v/0.6.2/doc/psty.html) a better way to incorporate (a part of) the unix filesystem into the web rather than fuse - which is the other way round.

clebio• 12 years ago

Isn't this -- at least in spirit -- what IPython notebook tries to do? In IPython, you can use shell commands with a bang, e.g. `!ls -l`. And `ipython notebook` uses ZMQ to connect the server and client.

At any rate, awesome work. I've been thinking for a while, offhandedly, about the idea of piping web URLs around. But I don't know how in fact it would work under the hood. I write a lot of API interfaces in my work, and it's all in the details -- every API is different in small but important ways.

tubelite17• 12 years ago

So if you are running psty, here is a fun sequence of commands to try out:

  cd http://dilbert.com/strips/comic/1995-06-24/
  ls
  cat 21021.strip.zoom.gif

Never really liked the colorization of Dilbert. It was so much nicer as a line drawing. Best we can do is remove the colors.

  cat 21021.strip.zoom.gif | to canvas | pixastic -d

And to save your result,

  cat 21021.strip.zoom.gif | to canvas | pixastic -d >/home/dilbert-unix.png

Or to GDrive or Dropbox or even the local RamFS by changing the target directory.

(Scott Adams is fine with personal usage. I checked. And you need psty because it proxies HTTP requests for sites which don't have CORS enabled)

bespoke_engnr2• 12 years ago

Wow, this is a really interesting way to think about web resources. After about two minutes of playing around with this, it gave me a more pleasant experience of the web than I've had in a while.

anigbrowl• 12 years ago

This is what it was like when I first got on the Internet (around 1992); there was no web then, instead you FTPed everything and used resources like Gopher, Archie, and Veronica to find documents.

I find PigShell interesting from a historic point of view but think it also points to a flaw in how the web currently works, with far too much glue and eye candy, which is a direct result of the ad business model.

bespoke_engnr2• 12 years ago

I completely agree. I actually deleted a bunch of my comment before posting it, because I started getting carried away pondering about what the right level of abstraction is for dealing with web resources.

At the risk of sounding like one of the semantic-web xml-standards people, it really does make me wonder if all this horrible, incompatible, site-specific frontend crap is just a fad that will give way to tools (like psty) that ignore all that and just deal with the raw semantic content (and HTTP verbs) at a higher level of abstraction.

Woah, man. We're like...surfing the Matrix, man.

TimWolla• 12 years ago

Does not work in Internet Explorer 11: Exception: The property "slice" of an undefined or null referance cannot be accessed (Exception: Die Eigenschaft "slice" eines undefinierten oder Nullverweises kann nicht abgerufen werden.)

tubelite17• 12 years ago

Sorry, Internet Explorer is not supported currently. Please try Chrome or Firefox, in that order.

dullcrisp2• 12 years ago

I do love this idea, but it bothers me from a security standpoint. Is there a way to do something like this without giving the author explicit control of all of my online data?

tubelite17• 12 years ago

The author has NO control whatsoever of your data.

- Pigshell is a 100% client-side Javascript app. There is no server side. pigshell.com is a dumb static html/js/css server.

- All data (username/password as well as user data like photos and files) travels directly between the user's browser and the provider (e.g. google.com, facebook.com). pigshell.com is not involved at all.

- Pigshell is free software - you are free to download the source code, examine, modify and run it locally yourself. (https://github.com/pigshell/pigshell)

- Our privacy policy is minimal because we know NOTHING about you - not your name, email address, facebook id, nothing. Only your IP address is recorded in apache logs, which is the case with every website you visit.

What is more, we don't want to know. Pigshell has been deliberately designed so as to afford maximum privacy and freedom to the end user.

dullcrisp2• 12 years ago

I don't really doubt you or your intentions, but I guess I might just not understand how authentication works for something like this.

When I try to sign in with Google Drive, it tells me that PigShell (developer email: xxxx@gmail.com) would like access to my Google Drive, as well as my photos and videos. Does that give your API key access to my account, or does only code that I run that's hosted on pigshell.com have access? What do the permissions get tied to? And how would it work if I were to host it locally?

fiatjaf5• 12 years ago

How is the procedure for "installing" new programs? Adding a local directory with javascript commands to the PATH will do it?

What about saving the state of things. I want, for example, to `mount` my local directory with psty everytime I open pigshell, and I wanna have my PATH set automatically. Will a `/local/.config/.pigrc` do this? (This raises another question: file editing. Where is the file editing tool? Maybe a web-based editor somewhere). Well, maybe a gist will do it.

tubelite17• 12 years ago

There are two kinds of commands: Javascript commands (like ls, cat) and scripts (ps, kill, man, csv2js, table2js).

Currently, Javascript commands have to be built-in at "compilation" time. (I need to add a load command which will evaluate a new JS command inside the pigshell closure)

Scripts can be anywhere in PATH.

There is an rc mechanism for doing exactly what you want. At boot time, we run /etc/profile, followed by /local/rc.sh - this is a file saved in the browser's localStorage. Mine is very simple:

  HOME=/home
  mount http://localhost:50937/ $HOME

You can create it simply by

  echo "HOME=/home\nmount http://localhost:50937/ $HOME" >/local/rc.sh

or running "edit /local/rc.sh". Edit provides a simple inline CodeMirror-based editor (actually even the CLI is a CodeMirror editor instance). I use vim on the desktop to do any serious editing. If you run psty, /home is a desktop directory visible inside pigshell, so edit on desktop/run in shell is a low-friction process.

If you manage to mess up /local/rc.sh to the point where pigshell is not "booting", then you can boot it in "safe mode" by appending #norc to the URL, i.e. http://pigshell.com/#norc

progman7• 12 years ago

I downloaded http://pigshell.com/v/0.6.2/psty.py, started it with "python psty.py -a" and did "mount http://localhost:50937/ /dir" successfully.

Unfortunately only "ls /dir/file" works but neither "cat /dir/file" nor "cp /dir/file ." although "file" was marked with "chmod a+r". I get the message "Cross-origin request denied (check if psty is running)". What do I wrong?

aidos• 12 years ago

Kudos, tubelite. This is really cool. I'm not sure I'd ever really use it but it's been fun to play with. Definitely made me think that I'd like to use the web in a more command based fashion.

vezzy-fnord• 12 years ago

So in many ways, this is like having a FUSE/Hurd translator implementation directly from the browser, though it also provides a mini-Unix environment due to needing to implement corollaries to some Unix utilities.

The scope of this project is definitely what makes it nifty.

drivingmenuts• 12 years ago

I have no idea what I would use this for, but I really wish I did.

Very cool.

omouse• 12 years ago

I like the sentiment, however I think it'd be cooler if you could get the Facebook, Google, and other APIs to behave in the "everything is a file" tradition of UNIX.

joseph8th• 12 years ago

A new toy! Thanks!

codehero• 12 years ago

I typed in `oink` but the command was not found...

altrego99• 12 years ago

How do I know what are all the fields available on a file?

tubelite17• 12 years ago

Here is one way:

  ls -d /path/to/file_or_dir | printf

This command prints the stack of file objects. The last one printed is at the "top" of the stack and tells you the fields available. You can then print a field

  ls -d /gdrive/user@gmail.com/somedoc | printf "%(raw.ownerNames)s"

Sorry, a lot more documentation is needed. Working on it.

shykes• 12 years ago

Looks a lot like webshell (http://webshell.io). I wonder if there are other similar tools out there?

itazula• 12 years ago

Wow, this was like the nutshell cracked and the universe spilled out ...

ConSeannery• 12 years ago

This is awesome. It reminds me of a talk I saw recently, linked from hackernews at some point in the last year or so. Something about re-thinking internet infrastructure into a "directory-tree" like model. Does anyone remember it?

oftenwrong• 12 years ago

I don't, but I am interested if you find it. We already have a hierarchy in a way.

The domain tree goes from top to bottom this way <---

The resource tree goes from top to bottom this way --->

So, using numbering for order, you could look at a URI like:

0002.0001.0000/0003/0004/0005

Meanwhile, in an alternate reality...

    $ ls /net | shuf -n 16
    coffee
    net
    рф
    한국
    org
    dating
    pw
    中國
    institute
    இலங்கை
    ninja
    com
    みんな
    su
    政务

progman7• 12 years ago

Congratulations!

The way how you implemented "man" ("man ls" for instance) shows how a convenient shell can look like. Pigshell could be a foundation for a completely new kind of toolchain for web development. I am thinking of seemless integration of JS editors, D3 graphics for pipe output, etc. ... Wait - it's already working :-)

http://bl.ocks.org/ganeshv/raw/b971989c337e958c0531/

If it just were possible to access the bare bone hardware so that we wouldn't be restricted to the JS world (in a closed environment, of course).

pmontra• 12 years ago

The way the man page is formatted is great but I don't like the scrollbar inside a scrollbar effect. It's been traditionally a worst practice. It's better to have the man page take all the vertical space it needs in the main frame and add a Close button to the end of the man.

By the way, from http://pigshell.com/v/0.6.2/doc/psty.html "Websocket server: Every unix app on the desktop which uses stdin/stdout can participate in a pigshell pipeline" so maybe your request has already been implemented.

progman7• 12 years ago

By the way, ...

Thanks, meanwhile I have realized that also :-)

damian2000• 12 years ago

My connection is shaped at 128 kbps at the moment - am grateful that this loads really quickly. Awesome stuff. HN itself is really low bandwidth too.

kybernetyk• 12 years ago

    rm -r facebook

ah, that feels good.

diltonm• 12 years ago

It's very cool, gets back to using the power of the simplicity of UNIX. I found though, that X middle click paste doesn't work.

I made a directory at /mnt/mccomsoft then ran Firefox and didn't see it which is good. :) Then I tried mounting that using your HttpFS with:

  mount http://www.mccomsoft.com/index.html /mnt/mccomsoft

It gave me:

  Cross-origin request denied (check if psty is running)

Which is also good. :) If I load Pigshell from my domain though then it should mount fine.

One user commented below it's like the Internet was a shell and you've cracked it open. I agree.

Pigshell – Unix the Web

Related Stories