Gonix – Unix tools written in Go

https://github.com/polegone/gonix

by polegone • 11 years ago

148 80 11 years ago

github.com

tux• 11 years ago

Very nice, please add at least some of this; grep, rm, secure, file, top, trace, ping, whois

bjenk• 11 years ago

What happens when you tail a 32 gig file?

misframer3• 11 years ago

From a glance at the code [0], it looks like it will read it all into memory first.

A better way to do this would be to utilize the io.Reader interface.

[0] https://github.com/polegone/gonix/blob/0b65cd4fb9c6c44357d0a...

polegone7• 11 years ago

Thank you. I will look into this.

microcolonel• 11 years ago

https://github.com/uiri/coreutils << I wrote some with a friend a couple years ago, maybe they suck, or maybe they have usage strings. >.>

mpdehaan2• 11 years ago

While I'm not sure if he posts here, I used to work in the same group with Jim Meyering, who maintained/maintains coreutils. Great guy.

Anyway, he told some great stories of the complexities of POSIX, and what happens in Solaris when you have directories 20,000 lines deep (and how to do it efficently, and the fun of teaching various coreutils commands about SELinux). Lots of it gets surprisingly low level quick.

Coreutils is complex for many good reasons, so while these tools look all nice and clever, they are not dealing with alot of the same kinds of issues.

(I also recall some of the heck Ansible had to go through to deal with atomic moves, groking when something was on a NFS partition, and so on. Not the same thing, but small seemingly easy things get tricky quickly.)

Bottom line is appreciate those little tiny Unix utilities, a lot went into them even when you think they aren't doing a whole lot :)

johnny22• 11 years ago

agreed! so many folks see all that stuff as "bloat". In many cases, that code is there for a reason.

hzhou321• 11 years ago

People are lazy and we took legacy complexity as granted. Time evolves and rarely any problem looks the same. If for no other values, having a second (and third and beyond) look at legacy complexity will worth all the attention.

malkia• 11 years ago

You should totally change the name to Goonix (it just reminds me of one of my favourite movies - Goonies that is)

vortico• 11 years ago

Why do Go programmers always want to redo everything? It's rare to see something actually new written in Go, proving that Go can do everything C can (except make shared libraries and produce small binary sizes) but not that it's actually better.

jonhohle• 11 years ago

The same thing happens in Java… and Ruby… well, maybe every language.

pjmlp• 11 years ago

Because that is the whole point.

If C is still present in the stack, the typical C exploits are possible, which were how many Oracle JVM exploits came to be, for example.

Reducing C presence to the same as Assembly, will just make everything safer in our systems.

Not that it will ever happen in UNIX systems, given how C came to life.

zobzu• 11 years ago

https://github.com/polegone/gonix/blob/master/cp.go

I think it needs some work for actual parity =)

maxmcd• 11 years ago

The author specifically singles out cp as being incomplete in the readme.

electic• 11 years ago

There is no reason to ever want to do this.

taternuts• 11 years ago

except to you know, learn stuff

polegone7• 11 years ago

I'm just doing this for fun and to learn about Go and Unix at the same time.

I am a beginner, and a lot of my code is inefficient and/or incomplete, but by putting it on the Internet I can get criticism and find out where I went wrong.

For instance, some of you have told me that the way I've been reading files is very inefficient, so now I'll try and do it the correct way.

tantalic• 11 years ago

That is a great reason to do this and the best way to learn.

nekopa• 11 years ago

That is the best reason. I am glad HN has people like you who are not afraid to put yourself out there and reminds us all that this is what being a hacker is: doing stuff for fun and learning.

forkandwait• 11 years ago

That's lame. I logged special just to downvote you.

quink• 11 years ago

https://github.com/polegone/gonix/search?utf8=%E2%9C%93&q=re...

Goodbye, memory.

vvpan• 11 years ago

I don't know much about go, but taking a quick look at implementations - they seem to be written by a programming novice, and the are quite primitive. Don't mean to be negative, just my opinion.

polegone7• 11 years ago

I am a novice, and that is one of the reasons why I started this project (to learn).

barakm• 11 years ago

Forget the haters, this is awesome.

GNU started as GNU's Not Unix, reimplementing Unix userland for free. The people who are adamant about calling it GNU/Linux are, on some level, remembering that the userspace is historically a reimplementation.

Give me a busybox that I can 'go build' and that becomes really quite interesting.

mixologic• 11 years ago

Except for the fact that it'll be years before all of the subtle bugs are worked out and you can rely on those apps to be as stable as the ones we've got:

http://www.joelonsoftware.com/articles/fog0000000069.html

eloff• 11 years ago

We have years. They're passing whether we like it or not. In the meantime you can choose whichever implementation of the Unix tools that you want. In the future there may be more options that fit the bill, developed in a safe language without the kinds of buffer overflows and things that exist in C tools today.

microtonal3• 11 years ago

What's wrong with doing this for fun's sake or educational value (both for the authors or people who want to study Go)?

If it turns out to be more widely used, at least it's safer than C code.

mpweiher• 11 years ago

Yeah, just because Joel said so doesn't mean it's true.

I've been responsible for a bunch of rewrites that were resounding successes and typically reduced the bug load (subtle or not) by orders of magnitude.

Animats• 11 years ago

We really need a busybox in Rust or Go. The existing one has licensing problems and security problems, and is built into too many embedded devices.

ploxiln2• 11 years ago

Busybox is GPLv2. That's the same license as the linux kernel; it's not a problem. You just have to release your modified version of the busybox source if you distribute it in a product. The only difference is that busybox developers enforce the license against the many companies who can't be bothered and respond to requests with lawyers.

That said, a bsd-licensed re-implementation of busybox (still in c) that's pretty far along and has gotten support from some of those companies (including Sony) is "toybox" http://www.landley.net/toybox/

fapjacks• 11 years ago

What licensing problems specifically do you mean here?

sagichmal• 11 years ago

There is a busybox in Go. https://github.com/surma/gobox

beoh• 11 years ago

https://github.com/surma/gobox

tezka• 11 years ago

the point being?

iagooar3• 11 years ago

1. Learning a programming language

2. Learning some less-known flags and use cases for the tools we use everyday

3. Reasoning about OS features and how they work

4. Having fun

Please, if you want to be this rude go back to your troll cave.

caipre• 11 years ago

Not meaning to bring up a Rust vs Go debate, but since there are a few comments claiming that this is a waste of time, I figured it's worth mentioning the Rust based re-implementation of coreutils:

https://github.com/uutils/coreutils/

And another by suckless in plain C:

http://git.suckless.org/sbase/tree/README

zarkone• 11 years ago

At least, Rust version has CLI options implemented

barsonme4• 11 years ago

Or my rewrite of GNU's coreutils in Go: https://github.com/EricLagerg/go-coreutils

It's not complete yet, but I couldn't pass up this thread :-)

polegone7• 11 years ago

I must say that yours is much more complete than mine, though.

aikah• 11 years ago

nice. This is the first time I actually read Rust code, and aside from "something macro" that scares me, it's actually mostly readable. Will definitely check out.

cloverich• 11 years ago

That re-write lists its license as MIT, not GPLv2. Is that legitimate? e.g. in general, can you re-implement a library with one license and change it to a non-compatible license?

lorddoig• 11 years ago

From everything I've read I'm pretty sure that if it's a complete rewrite in another language then it's legally "divorced" from the original project. You can own (and ergo license) an implementation of an idea, but you can't own or license the idea itself. Same with patents - you own the how, not the why.

luckydude2• 11 years ago

Heh, this brings me back. I was a young guy at Sun, perl 4 was a thing, I actually argued that we should redo /usr/bin in perl. In the days of a 20mhz SPARC.

Silly me. Maybe it makes sense now.

thwarted• 11 years ago

There's "Perl Power Tools: Unix Reconstruction Project" [0], which doesn't seem to have activity since 2004. I remember something older than 2004, I think, back from when perl was first available on Windows, to bring UNIX command line utilities to Windows through perl.

It's great that they all have inline POD documentation too.

[0] http://search.cpan.org/dist/ppt/

radicalcut• 11 years ago

The "Perl Power Tools" project has been revived by brian d foy: https://metacpan.org/pod/PerlPowerTools

ezequiel-garzon3• 11 years ago

I'm curious: what was your main motivation? I can understand it as a worthy challenge, but it would probably have led to a worse performance than the C-based utilities, no?

deckiedan2• 11 years ago

It would be pretty cool if all applications on your system were written as scripts, especially if they are very simple scripts, as it means you can open any of them in a text editor and see what they do.

It means you can modify any of them just as easily.

I remember one time when I needed information about how much CPU usage a program was using, I could see it in the output from `top`. So to figure out how `top` was getting that info, I had to download the source code, and grep through several .c files. If I could just `vim top`, and read the code, it would be very cool.

As an educational resource, being able to tell all your students "all binaries in /bin and /usr/bin are editable! Read them! Find out how they do what they do!" would be incredible.

luckydude2• 11 years ago

Simplicity mainly, memory management.

while (<>) print;

makes for a pretty simple core of cat.

At the time I had written a source management system (nselite, morphed into teamware) almost entirely in perl except for one C program that was performance sensitive.

And I just really liked perl4 at the time.

aceperry• 11 years ago

Nice, I was thinking of doing something like that, but I don't know as many unix tools as the author does. :-P

giancarlostoro2• 11 years ago

I rewrote "pause" from Windows into my Ubuntu box out of habit of using it in some cases. First it was written in perl, then in Python, I also symlink clear as cls out of habit from Windows. There's small little "hacks" that you can do that are kind of fun.

gizmo686• 11 years ago

Windows' cls actually erases the buffer from previous commands (so you can't scroll up past it), clear does not. I alias "cls=printf '\033c'" to get the Windows behavior.

giancarlostoro2• 11 years ago

I'm surprised nobody has shown someone doing this in JavaScript.

zserge2• 11 years ago

It would be cool to have a "busybox" alternative targeted at Plan9 commands. I personally find Plan9 utils much more logical (and easier to implement!). Something like 9base from suckless, but as a single binary and hopefully in a more modern language (most of the code like sam or rc is not so easy to understand).

SSLy• 11 years ago

I personally find Plan9 utils much more logical

Could you elaborate on that?

zserge2• 11 years ago

Plan9 utils usually have less options and smaller functionality, but you can easily compose them.

"rc": I like it much more than sh for scripting. It's easier to learn and has less pitfalls.

"cat" has no options at all, it just concatenates.

"du": I like plan's "du" utility, which is easier than "find" ("du" simply lists file names recursively, while "find"... Can anyone list all the options of the find utility from memory?).

"rm" is a combination of rm and rmdir (directory is removed only if empty, unless you do "rm -r" to delete it recursively. Only two options: "-r" and "-f"

"tar" is a simple way to do recursive copying instead of cp, scp etc

"sleep" takes seconds only as an argument, but it can be a floating point (no suffixes like in coreutils, e.g. "sleep 1h" or "sleep 3d")

"who" takes no options at all

I'm not really good at plan9 utils, and I still use coreutils much more, but for embedded systems I would prefer to have a tiny number of simple building bricks like "du", "cp", "rm", "tar" etc, and a few smarter commands like awk/sed/grep/rc. I like toybox a lot, but if only that had an rc shell..

iagooar3• 11 years ago

Very neat. I started rewriting GNU's coreutils in Rust and find it to be a nice way to learn the language.

Also, it is interesting how many obscure and less-known features some of the tools provide. In this case I clearly see the 80/20 rule, you can implement 80% of the main functionality in 20% of the time, but if you want to make exact clones you're going to need invest a lot more time.

pixelbeat2• 11 years ago

A few data points on GNU coreutils.

1. It has a fairly good test suite, that rewrites should leverage. That can be easily done by setting $PATH to prepend the dir of the new tools, and running `make check`

2. To give an indication of the size of coreutils:

$ for r in gnulib coreutils; do (cd $r && git ls-files | tr '\n' '\0' | wc -l --files0-from=- | tail -n1); done

985050 total 243154 total

pixelbeat2• 11 years ago

It's also worth pointing out, since busybox was mentioned a few times, that the latest release of coreutils has the ./configure --enable-single-binary option to build as a multi-call binary linke busybox etc.

lttlrck• 11 years ago

The line mentioning using gccgo to make the binaries small intrigued me... it worked! I've only written a couple of small tools in go but the size of the binary always bugged me. It's just a shame that setting up cross-compilation with gccgo looks a lot more involved vs. gc.

Gonix – Unix tools written in Go

Related Stories