Golang custom transports and timeouts

http://biasedbit.com/blog/golang-custom-transports/

by ovokinder • 11 years ago

77 6 11 years ago

tptacek3• 11 years ago

I had a similar (though unrelated) problem early on with Golang writing a port scanner, which forced me to hoist my own event loop (build around an explicit select call) into my program: timeout or not, I needed lots and lots of OS file descriptors, and by relying on Golang's own event loop to close stale ones on timeouts, I'd invariably run out.

FiloSottile• 11 years ago

Similar unrelated problem here, too, on the sole machine now running the Heartbleed test.

After recompiling with Go 1.4 (from Go 1.2 - apparently the network poller was rewritten?) I'd start running out of file descriptors really fast, all leaking deep in net/http.ReadRequest.

Turns out http.DefaultServer doesn't have any default ReadTimeout, so when a client goes away before finishing a request I'd lose the goroutine and the file descriptor with it.

So yeah, you probably also want to explicitly set http.Server.ReadTimeout if you are not doing weird streaming/"comet" stuff.

tptacek3• 11 years ago

That's related to my problem, but not the same; what I needed was control over when an fd/goroutine would be destroyed. Explicit timeouts (at least in 1.0, but I think still) couldn't give me that. I needed an obscene number of fds, though.

sebcat• 11 years ago

I needed lots and lots of OS file descriptors

Why?

I ask because I wrote a port scanner professionally once, and I got away with using one fd for network I/O. I can only think of one case where you end up using a lot of fds and that is if you're using TCP sockets and connect(2) where you'd end up with one fd/tested port, and that's not good.

tptacek3• 11 years ago

It had to work unprivileged in userland.

jbox• 11 years ago

If you're going across the network, a slow timeout is often trickier than an explicit failure.

I wrote about handling this in Python: http://www.mobify.com/blog/http-requests-are-hard/

Golang custom transports and timeouts

Related Stories