Implementing a Tor relay from scratch in golang (2015)

https://tvdw.eu/blog/2015/01/24/implementing-a-tor-relay-from-scratch/

by lox • 10 years ago

109 22 10 years ago

tvdw.eu

dimitar9• 10 years ago

go is a meteor for sure. C,live long and prosper.

HowardStark• 10 years ago

If you don't mind me asking, why do you think so? It seems like a pretty solid language.

lox3• 10 years ago

Don't feed the trolls :) From the article, the author concludes that it's not good for anything where you need performance-critical parts of the code running via cgo. That has certainly not been my experience, but I'd love to hear other people's experiences.

keyle• 10 years ago

Nim might have been a good language for this. As fast to write as Go, direct C access.

nimmer• 10 years ago

And it can generate system libraries. An embeddable "Tor as a library" could be useful for large client applications (e.g. browsers)

lox3• 10 years ago

I posted this because I was interested to see if anyone else had experienced gaps in the golang `crypto/tls` library. The original author is pretty scathing about Go's implementation, but my impression thus far has been that it's an amazing quality part of the Go standard lib.

I'd take it any day over openssl, that is for sure. I wish the author had published methodology on the benchmark comparison, would be interesting to dissect that.

gravypod• 10 years ago

From what I've seen in everyone's work is that Go's STD libs are amazing... for the the exact use case that Google has needed.

When you have to deviate from doing anything that Google/Pike would consider "acceptable" for what they would see as the "plebeian programmer" the STD libs aren't built for it. I'd not say it's a side effect of the language (not even the fact that it's lacking generics). I'd just say it's age. It's too young to be refined.

lox3• 10 years ago

I'd say the std library is built exactly for the "plebian programmer" but such that it scales up well to larger use cases, pretty much exactly how the language itself is designed. I think how they've responded to vendoring package management is a great example of how they do take external feedback, but don't make knee jerk changes, and I really appreciate that.

mSparks• 10 years ago

What tools does go provide to track down memory leaks like he experienced?

tshannon• 10 years ago

Yeah, and it sounds like a lot of his issues ended up being with cgo and openssl. If he'd used the standard lib, he might not have run into many of those issues.

billhathaway• 10 years ago

There were some very significant[0] crypto speed improvements in the Go 1.6 releases. It would be interesting to see how well a pure go implementation would perform.

[0] Russ Cox mentions a ~ 10x performance in a TLS benchmark between 1.5 and 1.6.2 at https://github.com/golang/go/issues/15713

danesparza• 10 years ago

The author doesn't indicate what version of Go they were attempting to use, but in the last two years, Go has seen many improvements by their very active dev team.

Notably, there were several changes to crypto in 1.5: https://golang.org/doc/go1.5#minor_library_changes

1.6: https://golang.org/doc/go1.6#minor_library_changes

and 1.7: https://golang.org/doc/go1.7#minor_library_changes

jasonlai2• 10 years ago

Have you tried Rust? It interacts with C libraries fairly nicely.

tvdw4• 10 years ago

Hi. I wrote the post, happy to answer questions.

* yes, more modern versions of Go would likely mitigate some of the memory pain * yes, crypto/tls is fast now * no, crypto/tls still has insufficient functionality for implementing this. crypto/tls implicitly assumes you want to authenticate the channel through certificates, which Tor doesn't do * I was using go 1.4 * yes, I tried Rust

wbl• 10 years ago

Did you consider a different concurrency strategy to avoid the deadlocks? With separate reader-writer threads you don't have the deadlock you mentioned.

Crypto/tls doesn't support renegotation, which Tor needs, but they are getting rid off.

tvdw4• 10 years ago

There are separate reader/writer goroutines, I don't think splitting them up further would've helped much. The problem is that all connections may end up needing something from all other connections, and as soon as one of them slows down (slow network, etc) its channels start filling up, taking other connections with it :-)

This could've been mitigated by applying backpressure in a bunch of places, and is ultimately a problem of Tor and not Go, but the nature of Go makes it hard to build code to do that.

As for renegotiation: my work on the Go version of Tor had some nice side-effects, and indeed, renegotiation was finally removed :-) https://gitweb.torproject.org/tor.git/tree/ChangeLog?id=55c4...

hittaruki• 10 years ago

So you tried Rust. Then what happened? I am interested since I have been trying out rust myself.

tvdw4• 10 years ago

Rust is awesome. It's likely a better fit than Go for applications like this, as it has more predictable performance[1], and more control over the scheduler (as you have to roll one yourself).

I attempted an implementation of Tor in Rust, but because I implemented it in Go a few weeks before that I got bored quickly. That said, some ideas I had for the Rust version have made it to Tor itself (or soon will), such as my ideas on transparently load-balancing Tor hidden services: https://gitweb.torproject.org/torspec.git/tree/proposals/255...

[1] note that in the land of Tor, unpredictable performance (for example because of GC pauses) could lead to user deanonymization.

daenney• 10 years ago

The article seems to contradict itself, between "the Go implementation broke the speed record" and "Go makes the product too slow."

Granted the cgo stuff and the memory usage would be something that one needs to deal with. Did you talk to go-nuts at all? They might've been able to offer some more insight into all of this, a better way to deal with the cgo related issue and perhaps even make some changes to handle these kinds of cases better.

tvdw4• 10 years ago

Ha, I see how it can sound like that :-)

Yes, it broke the speed record: a multithreaded application outperformed the singlethreaded version. But I wasn't happy with the result. It consumed an order of magnitude more memory, and gc times were potentially harming users (not a widely researched subject, but gc times in low-latency mixnets can likely harm user anonymity). Oh, and it would occasionally crash with OOM errors.

IshKebab• 10 years ago

Go has its own TLS implementation called “crypto/tls“, apparently because agl__, one of the people working on the language, decided so.

I thought it was rewritten because of the terrible quality of the OpenSSL code, which turned out to be a very good decision.

Implementing a Tor relay from scratch in golang (2015)

Related Stories