mathias_10gen

Actually in C++11 you can just say:

    auto id = mongo::OID("1234...");

Actually, as of 2.4 there is an experimental text search feature built-in: http://docs.mongodb.org/manual/core/text-search/. While it is no Lucene (and is still considered experimental) it does provide simple tokenizing, stop-words and stemming.

One case that could break by this change is if you relied on insert's old default of being a no-op if an object with the same _id already exists. This can be used to efficiently insert a default document without race conditions. Example at https://github.com/RedBeard0531/Mongurl/blob/master/mongurl....

That said, it makes sense for this to be opt-in behavior rather than the default.

Out of curiosity, did you have an index on the ts field that you are matching with?

By the way, if you (or anyone else for that matter) come up with useful benchmarks I'd love to get a copy of them at mathias@10gen.com. I have a few of my own, but I'd like to get some real-world workloads from the community to test potential optimizations against.

There is an official tutorial on http://mongodb.org, just click "Try It Out" in the top banner. Karl Seguin also wrote two great tutorials (one on basics, one on geo queries) at http://tutorial.mongly.com/.

Locking in particular has improved substantially since 1.6. For example, 2.0 introduced yielding in some cases where MongoDB would go to disk rather than page-faulting with the lock held[1]. This has been extended and improved for 2.2, along with increasing the granularity of the lock from process-wide to per-db. There are plans to increase the granularity further in future releases.

[1] To see an example of the difference this makes see http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

Thanks for bringing this to our attention. It has just been fixed in master.

Take a look at the $push and $addToSet[1] update operators in mongodb. They are both for atomically adding one or more elements to an array in an object. On the other hand, MongoDB doesn't support server-side set union or intersection so if you are using those features redis would still probably be a good use there.

[1] http://www.mongodb.org/display/DOCS/Updating#Updating-%24add...

I don't know exactly what your GIS needs are, but it might be worth seeing if MongoDB's geospatial indexing meets them if you are interested in querying free-form data.

http://www.mongodb.org/display/DOCS/Geospatial+Indexing http://www.mongodb.org/display/DOCS/Geospatial+Haystack+Inde...

We try to keep documentation on EC2 best practices here: http://www.mongodb.org/display/DOCS/Amazon+EC2

True, but since the current version of SM that we use (1.7) isn't thread-safe all calls to it need to be inside of a mutex to prevent trashing global state. It is possible that with a switch to V8 or upgrade to the lastest SM (1.8.5?) it is possible to execute JS without a GIL, however that is something that will need extensive testing to make sure that it works properly.

PS - It is important to note that the jslock is completely separate from the dblock and it is rare to hold both simultaneously. This means that if you are running multiple Map Reduces, one can be fetching or writing data to the DB while the other is processing the objects in JS.

Writing to the journal is actually done outside of the write lock most of the time. It does hold a readlock, however as of 2.0 most commits will release the lock before doing any disk I/O.

2.0 has a new index format that should reduce your index sizes by ~20-30% to fit more in RAM. If you haven't looked at upgrading yet, it is probably worth testing with 2.0.1 to see how it performs in your use case. You will need to reindex() or restore from a dump to take advantage of the new index format.

FYI, as of 1.8 MongoDB now has a WAL implementation optionally enabled by starting with --journal. We made it the default in 2.0 for 64-bit platforms.

The global write lock, while important, is probably the most over-written-about and misunderstood issue. The writelock is only held while the db is actually in the middle of an write and tends to only be held for a very brief period (about 10s-100s of microseconds or less) at a time. For example, it is never held across multiple user actions and therefore is closer to a latch than a lock in common RDBMS terminology. The main cases where this caused issues for users are when some data we needed wasn't already in memory so we went to disk to fetch it (~10ms rather than 10us). One of the major improvements to 2.0 was a framework to yield the lock to allow other work while going to disk that is used in the places where this was seen most frequently in production. 2.2 will plug this in more places with the goal to never hold the lock when going to disk. This work is of course in parallel to work to move to DB, collection, and possibly extent-level locking replacing the global lock in almost all cases.

As for safe-mode, the waiting that it does is always outside of the lock. It uses a cached datastructure that is protected by a secondary mutex so that it never has to interact with the global dbMutex. If you choose to wait for replication or journalling of your writes, that will block the connection and therefore your client thread so single-threaded tests will show much worse performance with the db mostly idle. If you use more client threads or asynchronous I/O connections you should see roughly identical throughput in aggregate (see mongostat) although much higher latencies compared with not waiting.

If you have any questions about this feel free to shoot me an email. Replace the _ with an @ and add a .com to my user name.

Even worse is that modern x86 (and probably other) CPUs also have instructions for 16-byte vector registers that can be used to copy or compare data much faster than 1 byte per cycle. Recent versions of glibc use some linker magic to pick the optimal code to use for strcmp, memcpy and friends based on the instruction set available to the CPU at runtime. Of course gcc and glibcxx's developers must not trust glibc and will sometimes replace calls to these functions with thier "optimized" builtin versions that use the lowest common denominator ISA. An easy way we got a 5% boost in througput in mongodb was to force them to call into glibc. https://github.com/mongodb/mongo/blob/master/pch.h#L47-56

Surprisingly good although not complete: http://gcc.gnu.org/projects/cxx0x.html

Unfortunately a lot of companies don't use today's compilers, but use many-year old versions. For example, Apple still ships gcc 4.2 which is over 4 years old now and supports none of these features.

I think Effective C++ by Scott Meyers[1] is the gold standard when it comes to books on writing good clean C++. It assumes that you already know the basic syntax though and already know how to program. This allow him to skip the intro stuff that take up the most space in many books and get strait to telling you what you need to know to write better code.

[1] http://www.amazon.com/Effective-Specific-Improve-Programs-De... (Make sure you get the latest edition, a lot has changed for the better)

Actually since most if not all CPUs have an atomic increment instruction it is quite easy to make shared_ptr<> thread-safe. In fact, that is what boost::shared_ptr<> does. This makes it a good tool for resources that may be shared between threads or for cases where it's just not worth the mental overhead of worrying about ownership.

That said, shared_ptr isn't a solution to everything. Sometimes scoped_ptr (called unique_ptr in tr1) or intrusive_ptr is a better tool. Often if you don't want to share or transfer ownership, passing around a raw ptr or reference is even better and the only option if you want to work with embedded structs (but the newed memory should still be owned by some RAII smart pointer even if it gets passed around as a bare pointer)

That line was referring to restarting just the mongod process on a running machine (eg, when upgrading or changing config flags). In that case, the data should still be in the OS file cache and will result in minor rather than major pagefaults when mongod accesses data so it will not go to disk. The case where you do a full reboot is mentioned two sentences later: "Of course on a full server reboot MongoDB must reheat too."

It was probably either http://try.mongodb.org or http://mongly.com/

Its actually possible to be fully consistent if you include the "tension" documents when fetching the main docs.

For example, in the canonical transaction example of Alice giving Bob $5 you could insert a document in the transactions collection that says Alice gave Bob $5. When you go to fetch Alice's pending balance you fetch her current document, then you fetch all pending transactions than she is involved in. Same for Bob and all other users. Then each night, you pull that days transactions and apply them to each user and update an "as of" field to make sure no transaction can ever be applied twice.

Now you could argue that this isn't fully consistent because it is possible for Alice to simultaneously give $5 to both Bob and Charlie even if she only has $7 in her account. However she will then have a pending balance of $-3 so it immediately reflects her current situation. This is similar to the real world example of overdrawing you checking account. If you wanted to, you could void one or both of those transactions if you never want to allow a user to go negative.

I don't want to turn this into a MongoDB vs Riak thread, but you may want to take another look at Mongo as the points you've mentioned are now handled. As of 1.6 we support automatic fail-over[1] and synchronous replication[2]. In 1.8 we added a journal for durability[3] which will be enabled by default in 2.0 (due out this month - rc2 released today). Optional automatic fail-back[4], for when your preferred primary (if any) comes back online, is also coming in 2.0.

[1] http://www.mongodb.org/display/DOCS/Why+Replica+Sets

[2] http://www.mongodb.org/display/DOCS/Verifying+Propagation+of...

[3] http://www.mongodb.org/display/DOCS/Journaling

[4] http://www.mongodb.org/display/DOCS/Replica+Sets+-+Priority

The defualt mapping of Y is arguably broken since it doesn't follow the normal vi conventions. I have these lines in my .vimrc to "fix" it:

    "make shift Y behave like shift-[cd] (copy to end of line)
    nnoremap Y y$

ci( is another nifty trick that will do the same but will work anywhere between the two ('s not just at the beginning of globalConfig. See ":help text-objects" for more details on one of the more powerful feature of vim.

While this is a very nifty bit of code, I'd be very careful about using it for unit testing as there is a high likelihood of minor behavior differences verses a real mongod which can cause surprises in production.

Also, for anyone looking to embed MongoDB and who doesn't mind using C++ it is possible and rather easy. In fact some of our unit tests and all of our tools other than the shell already do this. See https://github.com/mongodb/mongo/blob/master/tools/tool.cpp#... for the details. The DBDirectClient class implements the full C++ driver interface but does all operations in-process. The downside is that unlike when using our drivers (which have no licensing impact on your code) when you embed mongodb within your application it becomes bound by AGPL.

PS- That is really impressive for a weekend project! You should get in touch with us at 10gen if you're looking for a new job.

MongoDB doesn't require everything in ram either.

FYI - Your _id trick is similar to the ObjectID type mongodb uses by default.

"A BSON ObjectID is a 12-byte value consisting of a 4-byte timestamp (seconds since epoch), a 3-byte machine id, a 2-byte process id, and a 3-byte counter. Note that the timestamp and counter fields must be stored big endian unlike the rest of BSON. This is because they are compared byte-by-byte and we want to ensure a mostly increasing order."

http://www.mongodb.org/display/DOCS/Object+IDs#ObjectIDs-BSO...

One thing you may want to consider adding to your client is a "mongotunnel" command which will set up an ssh tunnel with compression between the user's computer and your mongodb servers. I've noticed some significant performance benefit to this vs. just accessing mongod normally over the internet, mostly due to the abysmal upload rate on my cable modem. Also this allows you to avoid sending data in plain-text over the open internet.

I think those two issues are orthogonal to consistency. In ACID, consistency and durability are two different letters and CAP doesn't even mention durability. Are you referring to another definition of consistency?