HN user
Should also note that the "arrays are values" thing doesn't actually matter in practice. I've written a wide variety of go code in the last > 4 years and I can only think of two places I've used an array.
"<-c", regardless of what's on the left side of it will block in every case until there's a messages available in the channel (note that close is a message).
I just notice this section has been removed. I'll be less angry now. :)
This is very much false:
While this will return immediately:
v, ok := <- cYour modified Go server doesn't return "Pong" for "Ping".
The program doesn't read the result, so it doesn't matter. Returning Pong isn't harder, but why write all that code if it's going to be ignored anyway?
It's fundamentally different. - you're firing off all your requests before waiting for any replies, and so hiding the latency in the more common RPC style request-response chain, which is a real problem.
As I said, the program isn't correlating the responses with the requests in the first place -- or even validating it got one. I don't know scala, but I've done enough benchmarking to watch even less sophisticated compilers do weird things with ignored values.
I made a small change that produced semantically the same program (same validation, etc...). It had similar performance to the scala one. If you don't think that's helpful, then add further constraints.
It's not obvious to me what that does. Does that only service one of the 100 clients or one of their 10,000 pings each? Or both?
Did you consider running the go client against the scala server and vice versa?
Also, that's kind of a lot of code. Here's my rewrite of the server: http://play.golang.org/p/hKztKKQf7v
It doesn't return the exact same result, but since you're not verifying the results, it is effectively the same (4 bytes in, 4 bytes back out). I did slightly better with a hand-crafted one.
A little cleanup on the client here: http://play.golang.org/p/vRNMzBFOs5
I'm guessing scala's hiding some magic, though.
I made a small change to the way the client is working, buffering reads and writes independently (can be observed later) and I get similar numbers (dropped my local runs from ~12 to .038). This is that version: http://play.golang.org/p/8fR6-y6EBy
Now, I don't know scala, but based on the constraints of the program, these actually all do the same thing. They time how long it takes to write 4 bytes * N and read 4 bytes * N. (my version adds error checking). The go version is reporting a bit more latency going in and out of the stack for individual syscalls.
I suspect the scala version isn't even making those, as it likely doesn't need to observe the answers.
You just get more options in a lower level language.
We've done a few of them here. Notably:
* http://dustin.github.io/2012/09/09/seriesly.html
* https://github.com/couchbaselabs/sync_gateway
cbgb is an API compatible Couchbase implementation in go. We use it in place of couchbase when we need something tiny to play around with.
seriesly is a time series database for storing and aggregating sample data and doing things like this: http://bleu.west.spy.net/~dustin/seriesly/
sync_gateway is how our mobile team synchronizes data across all your phones and tablets and your central DB.
I have no idea what my response was about. As I said, I was pretty sick that day. Sorry you had to type all this stuff to explain to me that I'm a moron. :)
Then why isn't that the default?
> but I think "<- true" looks nicer than "<- struct{} {}"
And what looks even better:
defer close(doneChannel)
Or turning that around, do you think it requires all 8 of my cores to copy data over the network? Do you think the two lines of code + justification text provides sufficient value for this application to distract from the point of it in order to show why someone should override the default behavior?
Do you believe users shouldn't have any control over the number of cores any particular application consumes?
Have you measured the CPU contention of the application and determined that using more cores is worth the overhead of increased overhead of multi-thread exclusions (vs. more simple things happening directly in the scheduler)?
Overall, it has nothing to do with this article and now even more people are going to copy it in more unnecessary places as a cargo-cult "turbo button" for their programs.
If you are going to use an idiom like that, the least you could do is check for the GOMAXPROCS environment variable and only do this as a default when the user hasn't specified otherwise.
Here's the version I threw together in (what it says here), 12 minutes: https://gist.github.com/dustin/5478818
There's so much code here.
If I weren't sick, I'd submit a new version that:
1. Didn't reimplement io.Copy
2. Didn't avoid io.TeeReader
3. Didn't do weird things to avoid regular channel ranges.
4. Didn't do non-standard date formatting.
5. Didn't reinvent the log package.
6. Didn't try to convince anyone runtime.GOMAXPROCS(runtime.NumCPU()) was a good idea (it's not)
In fact, maybe I will anyway. brb
goroutines are not threads because thread means something and goroutine means something else.
What else would I use? I've been writing tons of go code for nearly three years now. I get fast, concurrent, parallel code in very little time.
I was pretty much production-ready with seriesly in two weeks by myself (though I'm starting to get contributions from other users). Last night, I closed my last open issue "bulk interface" by making an optional memcached binary protocol interface with custom packets for database selection and streaming data in.
Today, I started a new project with a new guy, and got some pretty impressive internal demos working after a couple hours of work.
I get lots of things done fast and reliably. This doesn't happen as much for me in other languages. I went into more details in the follow-up post where I described how I built seriesly (and keep in mind, I wrote this after it had only been alive for two weeks): http://dustin.github.com/2012/09/13/inside-seriesly.html
This is great feedback. You seem to get what I'm going for.
Thoughts on your specific items:
1. I could probably prioritize the query/doc processing and get most of this out of the way, or something like what I've been thinking about for #4.
2. I've thought about this one for sure. It's actually possible to do externally already, just not very magically. I'll learn more when I get more internal people pushing it.
3. I've been tempted to add replication -- not because I need it, but because it's just really easy. master-slave is completely trivial. master-master isn't hard, but requires a tiny bit of state to be tracked I don't have an easy way to do yet. It'd be worth it just for fun.
4. I have a lot of infrastructure for this. To be efficient, I need something like _all_docs that doesn't include the values and/or something like get that evaluates jsonpointer. Then you could pretty well round-robin your writes and have a front-end that does this last-step work. Harvest the range from all nodes concurrently while collating the keys. Once you find a boundary from every node, you have a fully defined chunk and can start doing reductions on it. A slightly harder, but more efficient integration is to have two-phase reduction and let the leaves do a bunch of the work while the central thing just does collation. You wouldn't be able to stream results in that scenario, though.
5. Is this as simple as disabling DELETE and PUT (where a document doesn't exist)?
What kinds of advanced things are you wanting to accomplish?
There are two hard things in computer science. The next blog post I'm planning is related to cache management.
Whisper (backend for graphite), cube, ganglia... possibly more. I used to build things directly on top of rrdtool, but the schema definition can be a pain when you've got a lot of dynamic data.
I think there are some good ones. I wrote this because I could get it running faster than I could get the data I've got adapted to existing ones. That doesn't mean they're bad as much as it means I don't understand the data I've got. :)
The way I like to think about document-oriented databases is that you store what you have when you have it, and worry about what it was later when you need to get things back out of it.
e.g. the big bag of stuff I mentioned in the blog post contains a few things I know I don't need, a few things I think I probably need, and a lot of stuff I just don't want to think about (I might need it later, maybe after some manipulation, etc...). Lob it all in.
The downside of a system like seriesly vs. a system like rrd (or any modern equivalent) is the same as the downside of any nosql database vs. a sql database. By planning up front, I can keep the size down and get more performance by incrementally computing stuff from the beginning. In the meantime, I'll just buy more disk. :)
I'm reasonably happy with the performance, though. There's a good number of visitors on the page right now and this is what they're seeing:
2012/09/11 12:28:50 Completed query processing in 82.54ms, 6,266 keys, 1,280 chunks
BTW, my demo site is running in my bedroom on a slow ARM5-based debian box over my terribly slow DSL. If things get slow, that's why.
The web server itself is a homemade http server I wrote in go and jokingly called "nging". Doing a lot of SSI and transfer compression can be a bit much for that machine, but it got easier than maintaining my nginx config on upgrades.
I happened to have DB logs up and some some queries that weren't me and traced it back here. Good morning, HN.
The two ways (C90 and C99) you can create a variable length item at the end of a structure in C are invalid C++. You can't express them at all.
I work around it by making a one element array and using "sizeof(MyThing) - 1" everywhere when I want to reference the size then use placement new and the ::new operator to do the allocation and initialization of my object.
It works, but it's not very straightforward.
In general, I rather like C++, but it does make a few things harder than C.
go1 has been my goto language for a while now. I don't even understand why, but everything makes a lot of sense. It compiles about as fast as python and I have an easy time making quite efficient code.
Concurrency is obviously very good and that extends to parallelism quite easily. For example, I wrote go-heatmap ( https://github.com/dustin/go-heatmap ) and found I could shave a few seconds off of the colorization phase of a fairly large image I generate hourly by running every column (or row, I forget which) in parallel on my 4 core atom box.
The build/packaging system is everything maven wishes it could be. It's practical to create, publish and use small third-party things (such as my humanize package: https://github.com/dustin/go-humanize ) because the build system just deals with the dependency for you (even transitively) just based on import statements. "go list -json" in a source directory (or pointed to a namespace) will tell you all kinds of things about what your code uses, including transitive dependencies.
I deploy go code I use every day on OS X and Linux, 386, 64 and ARM. I can build all of these programs from any one of these systems so quickly and easily you wouldn't think they're native binaries. In fact, it just took me 20s to build a Windows cross compiler and all of the standard library (with which I built my kmz_example program from go-heatmap on OS X in .49s and now I've got a kmz_example.exe I don't know what to do with).
So you have a good trainer and a bad yoga teacher?
I agree with much of what you've said there. You clearly understand enough that I don't need to go into detail describing why I agree. :) But this isn't quite right:
> There is a reason antirez choose Lua and not JS for a scripting language.
To be fair, I embedded lua in ep-engine well over a year ago, but haven't ever released it. As it turns out, v8 is performing very well lately (certainly faster than plain lua) and people get it more easily. I've done some fun things with lua and it's really fun and easy to embed, but whatever gets shipped has to be supported and lua has some pretty dark areas.
That's really annoying. People have a tendency to find a minor feature missing from a tool and rewrite it from scratch in a new language instead of contributing a feature back.
Personally, I've never used fetch-manifest.rb (even on my mac). Can you tell me what you did and how it didn't work?
The networking stuff is the memcached engine branch. It's pure C and loads ep-engine as a module for doing all the backend junk.
memcached (binary) protocol for both doc manipulation and "changes" helps performance considerably. We looked at inventing a new protocol to stream stuff in and out faster, but we decided to go with the one we already had. Though the internal APIs have been shifting a bit, it was built entirely without touching the couchdb core and gave us a huge performance benefit over http: https://github.com/couchbase/mccouch
rewriting (not core, but slow) components in C is always the right way to do this sort of thing. We've been in the process of building a NIF for document updates for a while now (TBH, I don't know where the source is off the top of my head, but we don't keep stuff closed). We've got massive performance gains in pure erlang, but we expect to cut CPU consumption down more with this new code. It's intended to be available to Apache CouchDB if they want it.
I like CouchDB a lot myself, and use it for lots of projects (including critical parts of couchbase, inc.). If we can't solve the same problems with the new product, we'll know it because we use it ourselves.
We have something higher-performance than _changes and have had elasticsearch support as a third-party plugin for a while. We don't have firm plans to bring it in first-class, but we do think it would be beneficial to many of our users.
ep-engine (the core of membase and what handles all of the data moving around between networks, memory and disks) is almost entirely C++.