Show HN: Pythonpy – the swiss army knife of the command line

https://github.com/Russell91/pythonpy

by Russell91 • 12 years ago

220 42 12 years ago

github.com

d0m• 12 years ago

Looks so useful, can't believe I never thought of hacking something to do that. Will experiment with it!

ctrijueque• 12 years ago

Nice. Similar to PyP from Sony Imageworks:

http://opensource.imageworks.com/?p=pyp

The Pyed Piper Tutorial: http://www.youtube.com/watch?v=eWtVWF0JSJA

vram222• 12 years ago

Some time ago, I had blogged a series of posts about various ways of doing pipe-like operations in Python, including one experimental project of mine (which is not like real Unix pipes, which are IPC, but is intra-process), and some others, including PyP. Below are links to some of those posts, which may be of interest. Note: some of the posts link to each other, so I've posted them below in reverse chronological order.

Swapping pipe components at runtime with pipe_controller:

http://jugad2.blogspot.in/2012/10/swapping-pipe-components-a...

Using PipeController to run a pipe incrementally:

http://jugad2.blogspot.in/2012/09/using-pipecontroller-to-ru...

PipeController v.01 released - simulating UNIX-style pipes in Python:

http://jugad2.blogspot.in/2012/08/pipecontroller-v01-release...

Some ways of doing UNIX-style pipes in Python:

http://jugad2.blogspot.in/2011/09/some-ways-of-doing-unix-st...

gfxmonk2• 12 years ago

I wrote a similar tool a while back: http://gfxmonk.net/dist/doc/piep/

Mostly out of frustration for PyP not being lazy (on large inputs it reads the entire file up-front, or at least used to).

But it was quite interesting to implement all the standard python idioms (like slicing) in a lazy way, and it's not that complicated a tool. I still use it a lot whenever I have a nontrivial pipeline to write.

Russell914• 12 years ago

Yea, fortunately pythonpy supports lazy iteration over sys.stdin when you really need it. Just like in python, the syntax won't be as nice as using a list. But it works:

  py 'itertools.count(1)' | py 'itertools.islice(stdin, 0, 10, 2)'

However, the number of times that you need this are surprisingly rare. Most lazy operations don't require that each row be aware of the surrounding row context, and using the much simpler:

  py -x 'new_row_from_old_row(x)'

will get the job done in a lazy fashion. Usually, when you need rows to be context aware, as in:

  py -l 'sorted(l)'

  py -l 'set(l)'

it's just not possible to accomplish your task without reading in all of stdin.

Qantourisc• 12 years ago

ls | py -x '"mv %s %s.txt" % (x,x)' | sh has me worried, I can't find the escape code directly. Also you might wish to offer the following api ls | py -X '["mv","%s","%s.txt"] and run those using processes or sub-process. This would ensure the right argument in the right place. Just an idea.

zokier2• 12 years ago

Of course for this case, using shutil (or something similar) would be preferable. Eg ls | py -x "shutil.move(x, "%s.txt" % x)"

Russell914• 12 years ago

Hmm, I just added this example yesterday, and it seems to be highly contested, so I'll exchange it for something less controversial. That being said, if you're concerned about spaces, the best way to do this is either using shutil.move, or by generating output like mv 'file name' 'file name.txt' to pipe into sh. Of course, since you need to wrap your commands with single quotes to avoid bash quirks, it may seem difficult to use single quotes in a pythonpy expression. Fear not though - just use sharp quotes (`), which pythonpy will swap out for single quotes:

  ls | py -x '"mv `%s` `%s`" % (x,x)' | sh

zokier2• 12 years ago

These sorts of things have been one of the strongholds of Perl, glad to see pythonic alternative.

kseistrup• 12 years ago

A simpler version of py is e ⌘ https://pypi.python.org/pypi/e/1.4.5

k_os• 12 years ago

You should put up a donation link. If more people do that and it becomes common practice to donate to useful open source tools / projects then imagine what more awesome stuff could be created. Many people desire to build tools but don't have the financial support to do so.

thejosh• 12 years ago

This is what gittip is for.

And Kickstarter/Indiegogo/Crowdfunding has shown genuine interest to help fund useful tools and libraries lately which is great.

canistr• 12 years ago

I think if Git Tip had a button on a GitHub repo (a la Heroku button), this would be amazing.

easytiger• 12 years ago

Can't disagree more. That completely misunderstands the motivating philosophy behind effort expenditure given to FLOSS.

It isn't about motivating people to do things; that creates abandonware. It is about solving a problem that needs solving.

afarrell• 12 years ago

I'm confused by this. How does contributing money to people developing software for the commons lead to abandonware? And is abandonware a problem if it is free/open-source? Should I avoid publishing my hobby projects that I'm not planning to support?

fsiefken• 12 years ago

is there a ruby equivalent?

judofyr• 12 years ago

    $ ruby -e 'puts (0...3).to_a' | ruby -ne 'p $_.to_i*7'
    $ ruby -e 'puts (0...8).to_a' | ruby -ne 'puts $_ if $_.to_i % 2 == 0'

__debug__2• 12 years ago

Check out https://github.com/grosser/pru

arno_v• 12 years ago

For those too lazy to learn the right shell commands ;)

n0body2• 12 years ago

Or perl.

Demiurge• 12 years ago

Or learned Perl and are too afraid...

S4M2• 12 years ago

I like it! Now I can count the number of files I have in a directory:

    ls |py -l 'sum(1 for x in l)'

And also I can count the number of files whose names matche a certain extension:

    ls |grep .py |py -l 'sum(1 for x in l)'

I am not saying it was not possible before, but I don't know how to do it in bash.

Looking forward to use that for some basic stats on file. The following should sum up the first column of a csv.

    cat data.csv| py -x 'x[0]' |py -l 'sum(x for x in l)'

chmielewski• 12 years ago

Add alias numfiles="echo $(ls -1 | wc -l)" to your ~/.bashrc

foob• 12 years ago

For reference, you can do this using the wc core utility. The -l option makes it count the number of lines so 'wc -l' will do the same thing as your sum expression.

alco• 12 years ago

'wc -l' counts the number of lines. It also supports '-c' and '-b'.

To work with input split into columns, use awk. By default, it assumes the columns are separated with spaces. This can be customized with the '-F' option.

So your last example can be rewritten as

    awk -F, '{s+=$1} END{print s}' data.csv

Denvercoder9• 12 years ago

The first two can be done with wc:

  ls | wc -l
  ls | grep .py | wc -l

mitchty2• 12 years ago

You'll probably want to learn the regular unix way as well. Tools like this are nice, but you're already 99% there as it is for most things.

    ls | wc -l

    ls | grep -c .py

(note the cat and pipe are not needed)

    awk -F',' '{sum+=$1}END{print sum}' data.csv

S4M2• 12 years ago

Ah! I haven't thought of using wc, although I frequently use it to count the number of lines in my code. For awk, yes, I have to learn it. Thanks to you and the others who gave the example in bash.

vram222• 12 years ago

ls | grep -c .py

That won't work if there are files with names like foo.py.txt.

These will:

ls | grep -c '.py$'

or better, because less names to filter out:

ls -l *.py | wc -l

Russell914• 12 years ago

Yea, IMO the best way to do this with pythonpy is:

  py -l 'len(l)'

That being said, I still use wc -l every time. In fact, I always prefer regular unix commands (e.g. grep, xargs, head) to pythonpy when possible. But there are some things, such as:

  py -l 'l[::2]'

which while possible with tools like sed, are just much better expressed with pythonpy.

Lambdanaut• 12 years ago

I usually use Python(or another lang's interpreter) as my calculator, so this will save me a few keystrokes in the future. I think I'll still use short scripts rather than the command line to do more involved things (like find long palindromes in a txt file), but this will save me a bit of time when doing quick calculations. Thanks!

leichtgewicht• 12 years ago

Reminds me of 'js': https://www.npmjs.org/package/js

__debug__2• 12 years ago

I just started something similar for javascript a couple days ago: https://github.com/danielstjules/pjs

cohomologo• 12 years ago

I attempted to install on Windows - it doesn't work due to the lack of os.geteuid(), which is called in the setup.py file to check for root. With a little bit of finagling I could get around that and make it install, but not work due to undefined SIGPIPE in the signal module. It makes sense that this could be UNIX only, but that wasn't stated anywhere on the webpage. Perhaps that should be mentioned somewhere.

Russell914• 12 years ago

Yea, sorry about that. I had an initiative to make a windows version, wpy, a little after launch, but couldn't really get any windows developers interested in making it happen. From my time working on windows, I know this tool would be even more valuable there than on unix. I'll try to find a place to indicate this doesn't work. Perhaps there's a good way to check if the system is windows in the setup.py and warn users.

mgr86• 12 years ago

Honest question. Why would I want to use this? I am already quite happy with awk,sed,grep, etc.

Grue3• 12 years ago

Would've been more useful with a language that doesn't limit one-liners (e.g. you can't have "for x in [1,2,3]: dothing1(x); dothing2(x)").

Show HN: Pythonpy – the swiss army knife of the command line

Related Stories