Manta: Unix Meets Map Reduce

http://dtrace.org/blogs/brendan/2013/06/25/manta-unix-meets-map-reduce/

by dmpk2k • 13 years ago

119 39 13 years ago

dtrace.org

willcodeforfoo• 13 years ago

Interesting, is Manta open source?

zeckalpha• 13 years ago

The client is: https://npmjs.org/package/manta

susi222• 13 years ago

Those are the client libraries. It seems like the actual manta compute environment is primarily for SmartOS.

http://apidocs.joyent.com/manta/compute-instance-software.ht...

https://github.com/joyent/manta-compute-bin

Also probably a better article:

http://www.joyent.com/blog/hello-manta-bringing-unix-to-big-...

trevoro2• 13 years ago

Official Announcement: http://www.joyent.com/blog/hello-manta-bringing-unix-to-big-...

buster2• 13 years ago

As much as i love Solaris (and ZFS and dtrace and Zones and all the nice things Sun and Joyent brought) i wish this would be available for Linux as well.

Edit: Ok, so it's called "Joyent Manta Storage Service" and the tutorial talks about creating accounts on the joyent cloud. So this is a Service, not a software? Sorry about the confusion then... I thought it would be a new software for SmartOS or Solaris...

dap4• 13 years ago

It's not "for" any particular system. It's a Unix environment.

buster2• 13 years ago

How is it a unix environment? On the first glance i don't really get how it is implemented. That's why i supposed that this only runs on SmartOS. Is the object store software downloadable somewhere? The tutorial mentions creating an account on the Joyent cloud...

trevoro2• 13 years ago

Manta is an Object Store with Compute in-situ. The client is OS agnostic AFAIK - you can use a linux instance as your client if you want.

http://apidocs.joyent.com/manta/commands-reference.html

dsl• 13 years ago

If you like the general concept, you should check out BashReduce. As silly as it sounds, it is quite powerful.

http://rcrowley.org/2009/06/27/bashreduce.html

sturadnidge• 13 years ago

Not just MR, it's a full blown object store too.

http://apidocs.joyent.com/manta/storage-reference.html

oxtopus• 13 years ago

Reading the Manta website, I can't help but be reminded of this webcomic: http://rbxbx.info/images/fault-tolerance.png

epistasis• 13 years ago

I don't see how that comic is relevant to Manta, and it seems to be a bit of an unfair comparison. The Manta web site emphasizes the ability to use common and existing tools to work with the data. In addition to standard Unix pipes, they have bindings for pretty much every language I could think I want to use.

I.e. Manta seems to emphasize the ability to use existing tools in a distributed fashion, rather than forcing you to learn a new language with unfamiliar syntax.

IsaacSchlueter• 13 years ago

Hacker News at its finest.

OMG A DATABSAE THINGIE! IMMA POST THE COMIC WHAT SAYS THE CURSE WORD!

wicknicks2• 13 years ago

The next big data framework must be a programming language. Most languages make the assumption that the data it is processing, is available locally. This was true until the last few years for almost all situations. But not anymore.

tbrownaw• 13 years ago

Why? And where does the language end and the library begin?

justincormack• 13 years ago

Well there are issues of where you do the computation and what can be done in parallel. A language could help by having well defined semantics that makes this easier. Embedding one language in another (as with SQL) is a pain as a programmer. Although no solutions that caught on emerged here despite some promising research.

webjprgm• 13 years ago

I think I misread this the first time. You mean a "big data" framework not a big "data framework", right?

The idea of using a programming language as a database language: MUMPS (aka M), JavaScript as query language to some new NoSQL databases, ORM like Ruby ActiveRecrod which makes it look like you work on data in just Ruby.

As for the "big data" interpretation, I don't have much to say.

CountHackulus• 13 years ago

So like Fortress, X10, Chapel, UPC, or (arguably) IBM InfoSphere Streams.

pepijndevos• 13 years ago

So what is this SmartOS thing?

From what I understand: KVM is hardware virtualization, SmartOS is(a distro for) OS virtualization and Docker is process virtualization?

shykes3• 13 years ago

SmartOS is a distant relative of Solaris. It's an alternative to Linux for the server with all the Solaris bells and whistles (zfs, zones, dtrace) and great tooling for virtualization. Unfortunately it's not very widely adopted and to my knowledge is only supported by one start-up (Joyent) which limits its chances of dislodging Linux. So, a very powerful, highly integrated and ultimately niche stack.

It also inherited a subset of the Solaris community which is jaded by the popularity of Linux and its "primitive" toolset. You can often see them rolling their eyes and making condescending comments as the rest of the open-source world moves on.

The huge potential for all this is to contribute these wonderful tools to Linux, but there seems to be reluctance on that front.

insaneirish• 13 years ago

The huge potential for all this is to contribute these wonderful tools to Linux, but there seems to be reluctance on that front.

The code is there for the taking. FreeBSD has ported ZFS and DTrace. Guys at LLNL are using illumos ZFS code as upstream for ZFS on Linux. Others are working on DTrace.

One can wait for efforts like that to keep progressing, help in whatever way they can, or just use illumos based distros (e.g. SmartOS) now and not wait around. Choice is a wonderful thing.

susi222• 13 years ago

http://www.joyent.com/blog/why-smartos-kvm-dtrace-zones-and-...

SmartOS is something many people overlook. It's an awesome OS. ZFS is IMO enough reason to try it out (no, zfs for linux is not ready). But then zones make it even more awesome (there was a discussion about zones here on HN a few days ago).

This is usually how it is around sysadmins:

Those who know Solaris very well swear by it and implement some awesome very stable running high performance and secure machine setups. Those who don't, hate it because it's not Linux.

It takes a while to become familar with Solaris from a Linux background. But the more you read about it, the more you realize how well designed it actually is.

rsync• 13 years ago

"I’m looking forward to new discoveries now that I can store and process large amounts of performance data."

There it is. The central dogma of "big data".

But there is a dissenting view...

Jaron Lanier said, and I paraphrase: A computer will pass the Turing Test when the human running it becomes dumb enough to be fooled.

I have been very skeptical that the "discoveries" that big data presents to us will only be useful inasmuch as we dumb down our criteria for what discoveries are.

I suspect that anyone that has tried to model data and find real insight with it for any amount of time shares this skepticism.

brendangregg3• 13 years ago

The criteria for discoveries is that they solve real customer issues and solve them fast. This may be direct or indirect, elegant or inelegant, simple or complex.

Last night I helped analyze an issue on Manta regarding job run time, and used DTrace to measure all process run time from all servers. It was a lot of data, but easily awk'd (in this case, not enough data to warrant map reduce), at which point I found run times from a certain set of processes that were equivalent to the job latency - taking us towards the fix.

That's just last night. A large number of real fixes have come from being able to quickly gather and process performance data, and from that spot latency outliers, unexpected workloads, and patterns over time.

I have spent time modeling systems. I wouldn't say I was left feeling skeptical of data-driven analysis, rather, I was left appreciating how complex real computer systems are, which defy accurate modeling. That's not to say that modeling is inferior or not worth doing -- it's another tool. I'll use whatever I can to fix my customer issues the fastest.

helper2• 13 years ago

This looks awesome. I looked through the docs and didn't see much information on what happens in failure cases. If one node that is storing a replica of my data goes down how long until the data is replicated to a new node?

dap4• 13 years ago

It depends on what "goes down" means. The vast, vast majority of such issues are transient: the node panics or loses power and comes back up. There's no impact to data availability or durability unless all nodes storing a copy of the same object are down at the same time.

It's a good question. Sounds like we should document or write up a blog post about the gory details.

helper2• 13 years ago

That would be really useful. Along the same lines, one of the blog posts said that you chose to use strong consistency (CP in CAP). In the event of a failure, how many replicas (out of N) need to be down before the API returns 500s for reads or write (if they are different).

spullara• 13 years ago

Near as I can tell this and Hadoop Streaming are very similar. In fact, you could probably take their open source CLI and wrap it around streaming without too much trouble.

http://hadoop.apache.org/docs/stable/streaming.html

That would at least avoid lock-in to Joyent's infrastructure.

phaet0n• 13 years ago

Wow! I've spent a hour or so looking at the Manta docs and I think Joyent has found the sweet spot as far as abstraction is concerned. Kudos to the Joyent team!

The api [1] is so simple it's absurd. Looking at the design I just keep on asking myself, "but how will it perform?" And the answer is, I don't know. But what I do know is that because of the simplicity of the design, if this compute model catches on, it'll be the start of a new computing paradigm—probably the first genuine attempt at something we can agree is cloud.

The essence of the model is that "caching", in whatever form, is abstracted away. It is up to the architects of the system to ensure that the system performs well on a wide variety of compute scenarios. Let me explain. Usually, in a model like App Engine, you worry about how you'll represent your data in the Datastore, how you'll shard, how you'll use Memcache, and what type of batch jobs you'll run to reduce the amount of dynamic computation that is done on a request. On Manta, you just store your data to their object store and process it as needed. You let the system figure out data locality, what to cache, on which node to perform the compute, etc.

As I said, I have no idea how, or if, this will perform, but as far as abstraction level goes, it's perfect. This is where pricing becomes an issue. At 40µ$/GB/s it works out to 14.4¢/GB/hr with no mention or guarantee of the performance of the system. If the compute is slow, or the node is overloaded for tasks, what can a customer do? What if the compute is not limited by cpu but access to the underlying object store. Sure they say the compute is done at the edge, but what if they stick 4TB of data on a node that's the compute equivalent of an Intel Atom? So many questions about performance!

All that said, congratulations to the Joyent team. The HN response so far has been muted, but I think time will reveal Manta to be an important step towards true cloud computing.

Hopefully, Manta will mature enough to get a chance to kiss a girl.

[1] http://apidocs.joyent.com/manta/api.html

dap4• 13 years ago

Thanks for the praise!

As far as pricing goes, you're charged for the wall clock time spent running inside the zone, so if your task queues up because the system is jammed, you're not paying for that. Task runtime can be affected on busy systems, but this has been in the noise for the jobs we've looked at. We're also looking to see how the system gets used in order to understand whether it would be useful to have other pricing options (e.g., tiers of service with guaranteed resources).

As for performance: we're using the system internally every day, and we've found the performance very good. Of course, the best way to find out is to evaluate it on your own workload. :)

Finally, you'll probably be interested to read Keith's summary of our hardware choices here: http://dtrace.org/blogs/wesolows/2013/06/25/manta-what-lies-...

Manta: Unix Meets Map Reduce

Related Stories