Set Operations in the Unix Shell

http://www.catonmat.net/blog/set-operations-in-unix-shell/

by ajbatac • 18 years ago

35 4 18 years ago

catonmat.net

rkowalick• 18 years ago

Hey look! If we call a "set" a text file with one number per line, Unix can do "set" operations!

Never would have guessed Unix commands worked well with text files...

sfamiliar• 18 years ago

as a math nerd: i approve.

as a pragmatist?

tuukkah• 18 years ago

I found this very interesting. Specifically, I've never worked through implementing all the set operations in shell myself. You can see these as functional programming, and you often represent sets and multi-sets as lists in fp. In shell, your elements just can't contain '\n'.

Regarding the value of this, I see two main points:

1. When you're shell scripting (see next point ;-), you'll need some of these, and the situation might prefer not calling other programming languages.

2. The shell commands as used here are a fairly compact and convenient base for expressing these and many other computations. Many other environments would have a hard time representing computations processing more data than can fit RAM. Something comparable might be RDBMS-backed LINQ.

alecco• 18 years ago

Very cool, though not something a decent unixadmin can't do :)

Note before you think this is "old crap" this approach scales to multi-core and multi-processor and can handle quite large sets. For really large sets you might need to play a bit with sort flags and perhaps do those steps it in two phases.

Hardcore unixadmins often freak out developers when their month long projects on handling some files in production could be replaced with some scripting and in fact run it orders of magnitude faster. Multiprocessor means multiple L1 and L2 caches, less I/O blocking, etc. :)

Set Operations in the Unix Shell

Related Stories