Shake: Every Program Can Be a Clojure Function

http://sunng.info/blog/2012/09/shake-every-program-can-be-a-clojure-function/

by sunng • 14 years ago

79 25 14 years ago

Tuna-Fish4• 14 years ago

That's both wonderful and scary. I think the UI of it needs work -- if you want to treat all shell commands as functions, they imho should just return their outputs by default. (Potentially as a lazy list of lines?)

Also, I think that path should not be passed in as an environment variable, but in Clojure somehow. There's no reason for a program to pollute a global namespace with it's internal details.

jmmcd• 14 years ago

Agreed, namespace clashes are an obvious problem. "My program that worked last week now crashes? Oh, it's because I apt-get installed something completely unrelated."

Tuna-Fish4• 14 years ago

Well, (use :as) fixes that one.

Namespaces are a honking great idea, let's have more of those!

Also, I think I just realized an another problem -- if I understood the lib correctly, it scans through the PATH once at require time. This is bad, because Clojure is often deployed in places where processes run for very long times. Even things as immutable as tools in PATH can change when given long enough time.

sunng2• 14 years ago

I'm also looking for better ways to index path. If I can pass some options in (use) and initialize with options, it will be better.

Tuna-Fish4• 14 years ago

I really think "not indexing path" is the better choice. Instead of (uname -a), why not just export a single symbol and do (shake uname -a)? (or even just (sh uname -a)?) Then find the correct program to run at invocation time. It's more clear about what happens, and allows much simpler implementations with much less weird failure modes.

Kototama• 14 years ago

That's great. Next step is implementing piping :-)

pyritschard• 14 years ago

actually, pallet's library stevedore offers the same kind of functionality and has had piping for a good while (as well as other typical constructs):

See: http://hugoduncan.org/post/2010/shell_scripting_in_clojure_w...

The project lives at: https://github.com/pallet/stevedore and is heavily used in pallet, clojure's configuration management and cloud control library

mattdeboard2• 14 years ago

Not to be confused with http://stevedore.readthedocs.org/en/latest/index.html which is part of the virtualenv (or virtualenvwrapper?) installation for Python.

mattdeboard2• 14 years ago

This is actually pretty neat, specifically the part about indexing the executables available in your path and turning them into macros. I was underwhelmed at first because I thought you manually implemented each program.

I can't really imagine a use case for this for me (bash! But with parens!) but still very neat idea and cool execution.

btilly• 14 years ago

Shake draws inspiration from Python's sh. Which looks to me like a port of http://perldoc.perl.org/Shell.html. However after experience, that original version now comes with important warnings about things like metacharacter quoting not being able to be done in a portable and safe way.

It is easy to get things 95% right, and I am sure that Shake does that. But think twice before using it for production programs, particularly if there is any possibility of it encountering untrusted input.

vsipuli• 14 years ago

This was already pioneered by Olin Shivers' SCSH (Scheme Shell). See the process notation chapter in the SCSH Reference Manual: http://www.scsh.net/docu/html/man-Z-H-3.html#node_chap_2. Wish the SCSH project would come alive again: https://github.com/scheme/scsh.

dschiptsov• 14 years ago

A function? Really? Without side effects?)

sunng2• 14 years ago

You can set environment variable SHAKE_PATH to specify path that you want to initialize.

jcromartie3• 14 years ago

This is a really bad idea. It's not even close to useful right now. For instance, you can't use variables in your shell invocation, so this doesn't work:

    (let [x "/usr/local"] (ls -l x))

And it clobbers so many things in the core namespace when 'use'd.

Good luck.

draegtun2• 14 years ago

> And it clobbers so many things in the core namespace when 'use'd.

Yes the ...it indexes all the executables in your path... is different to the approach used by Perl's Shell.pm and I think Python sh.

These simply catch any undefined function calls and attempts to execute them in the shell (found in your $PATH).

  use 5.016;
  use warnings;
  use Shell;
  say cat("~/.vimrc");              # runs /bin/cat
  say incorrect_spelled_command();  # ERROR can't find in $PATH

However nothing is clobbered here:

  sub cat { "This is my cat()!" }

  use Shell;
  say cat('~/.vimrc');  # => "This is my cat()!"

The best approach is to not use the lookup dispatch and instead tell it what executable you want to wrap...

  use Shell qw(cat);

Now incorrect_spelled_execute is never passed to Shell.pm and in fact will give a compilation error (Undefined subroutine). And any cat() subroutine will be overwritten but it does give redefined warning.

cs702• 14 years ago

All these efforts to provide pretty & convenient interfaces to arbitrary executables strike me as being useful only for people who want a powerful REPL as their command-line prompt and/or would rather use Clojure or Python (or whatever other language) for mundane tasks that would otherwise normally be done with throwaway shell scripts. Outside of these limited use cases, I'm not sure this sort of thing is a good idea.

draegtun2• 14 years ago

> We will have a beautiful DSL so you don’t have to quote arguments as string...

If i'm using Perl's Shell.pm (as mentioned elsewhere here by btilly) and wanted to avoid quoting arguments then I could do:

  use 5.016;
  use warnings;
  use Shell ();

  use PerlX::QuoteOperator ls => {
    -emulate => 'qq',
    -with    => sub ($) { Shell::ls($_[0]) },
  }; 

  use PerlX::QuoteOperator uname => {
    -emulate => 'qq',
    -with    => sub ($) { Shell::uname($_[0]) },
  }; 

  use PerlX::QuoteOperator ip => {
    -emulate => 'qq',
    -with    => sub ($) { Shell::ip($_[0]) },
  }; 

  # and then...

  ls();
  uname(-a);
  ip(-4 addr);

But this seems to be an overkill because I could just use Perl's backticks (http://perldoc.perl.org/perlop.html#Quote-Like-Operators)...

  `ls`;
  `uname -a`;
  `ip -4 addr`;

And it works with variables...

  my $x = "/usr/local/";
  my $files = `ls -l $x`;

and piping...

  my $perl_files = `ls | grep .pl`;

Shell.pm also works with variables & piping but I think that backticks are probably Perl's best DSL for dealing (easily) with shell stuff :)

duelin_markers• 14 years ago

Creating this was probably fun and a good learning experience, but I wouldn't recommend trying to push it beyond that point. The problem Shake solves doesn't call for the sort of DSL that uses macros to make Clojure code act like something other than Clojure code.

I'd rather have something like https://github.com/clojure/tools.cli but in reverse: a library for building, manipulating, and running command lines and interacting with the results ... maybe something that supports setting up mappings so that external programs can be exposed as Clojure fns with "Clojuresque" arguments "--rather" "than" "--arrays-of" "strings".

reddit_clone• 14 years ago

A Scsh port to clojure would be nice. (Scheme -> Clojure is not such a stretch).

hugoduncan• 14 years ago

Sounds very much like pallet's stevedore.

http://palletops.com/doc/reference/script/ https://github.com/pallet/stevedore

Shake: Every Program Can Be a Clojure Function

Related Stories