Introduction to Apache Mesos

http://www.antonlindstrom.com/2015/03/29/introduction-to-apache-mesos.html

by aant • 11 years ago

145 54 11 years ago

antonlindstrom.com

preillyme• 11 years ago

I'm curious which of the Marathon bugs caused your downtime?

aant2• 11 years ago

I'm afraid I don't remember exactly which one it was but after an upgrade of the framework it worked perfectly again.

florianleibert• 11 years ago

The 0.7X releases contained a number of bugs. 0.8X has been significantly better!

phildougherty• 11 years ago

Don't poll the apps API if you have a decent number of tasks or it can bring the whole system to its knees.

wmf3• 11 years ago

Maybe this is a good place to ask a question that I've been pondering for a while: Why is Mesos based on the concept of resource offers? AFAIK this is backwards compared to other schedulers where you ask for resources and the schedule gives them to you (or not).

aant2• 11 years ago

I can't quite tell you why it was implemented like this. The Omega paper by Google describes different kinds of scheduling algorithms and their pros and cons.

I don't think that it's backwards compared to other schedulers (many of them are monolithic) but I think it can be improved by using transactions and optimistic concurrency control. This is discussed in the Omega paper and I think there's work to integrate the shared state scheduler into Mesos.

sabraham2• 11 years ago

To load the framework (application) with the logic with the concern of deciding what resources it wants. The scheduler shouldn't care about what you get, just fairness. Two-tiered scheduling achieves this:

Mesos decides how many resources to offer each framework, while frameworks decide which resources to accept and which computations to run on them.

http://mesos.berkeley.edu/mesos_tech_report.pdf

wmf3• 11 years ago

I guess that makes sense if frameworks care about placement (which I generally don't) because otherwise they'd have to express placement constraints to Mesos.

I still can't find documentation about what fairness policies are actually implemented, though.

otterley• 11 years ago

The most puzzling thing for me is why schedulers and executors are so tightly coupled in Mesos. In my mind, those are separate concerns: schedulers should concern themselves solely with locating an available resource for you (and should be part of the Mesos core), and the executor should take care of launching and managing the program to be run.

With the current design, if I want to launch a Docker container, I have to use its scheduler, even though its scheduler might not be as good as some other resource's scheduler. And I might have to accept a regression in the scheduler in order to benefit from a bugfix in the executor.

domenp• 11 years ago

Allow me take the opportunity and ask if anybody here is running Elasticsearch with Mesos (and Marathon) using Docker container?

I'm running Elasticsearch nodes on a dedicated Mesos slaves and I’m still not sure how much memory should I allocate to Elasticsearch task e.g. all available memory as reported by Mesos or some smaller amount to leave something for the system? Please note that I'm asking about memory allocated to the Marathon task not about JVM heap size.

nemothekid2• 11 years ago

I've taken the view of treating our database slaves as special and allocating all the memory on the slave to ES. That way ES (or any other db) always runs on the right slave, and no other potentially long running/resource hungry process runs alongside it (but just enough that's jenkins tasks and chronos tasks can still rune if need be).

AKAIK, there isn't a right way to run databases because the persistent storage layer into mesos is still being baked.

dingdingdang• 11 years ago

the persistent storage layer into mesos is still being baked

That's pretty important point, would have thought persistent storage would have been first point to get working for a project like Mesos. Also, their homepage ( http://mesos.apache.org/ ) outright states: "Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines [...]". What storage are they talking about if not persistent storage?

twoy• 11 years ago

If you don't want to manage ZooKeeper manually to bootstrap Mesos cluster, you need another Mesos cluster haha.

saryant5• 11 years ago

You could run it on top of something like CoreOS, but then you're just swapping Zookeeper for etcd.

eropple4• 11 years ago

Or just use Exhibitor. I don't remember the last time I manually anything'd with Zookeeper.

Between Exhibitor and Curator, I honestly find Zookeeper so straightforward and easy to work with that I don't quite understand the popularity of etcd.

toomuchtodo2• 11 years ago

I'm running a small Exhibitor/Zookeeper cluster dockerized (5 nodes, several hundred clients), and its extremely straightforward. Etcd isn't what we'd consider production-grade yet.

curun1r2• 11 years ago

Between Exhibitor and Curator, I honestly find Zookeeper so straightforward and easy to work with that I don't quite understand the popularity of etcd.

Tools like etcd and consul fit into the Unix philosophy of small, composable tools. Zookeeper is more a part of the Enterprise Java philosophy which many people have written off for various reasons, both rational and irrational.

Having run Zookeeper in the past and now having run Consul in production for the past ~6 months, I can't imagine ever running Zookeeper again, unless I'm using a tool that's built on top of it. Consul is just easier to use/maintain and we've yet to run into any problem with it. Zero problems in six months. In all the time I ran Zookeeper, I could never say that.

rckrd• 11 years ago

An interesting talk about Docker and Mesos by a former coworker, he also contributed to 0.22:

https://www.youtube.com/watch?v=ZZXtXLvTXAE

andyidsinga• 11 years ago

question / thought : seems like mesos could evolve to be an alternative to openstack ...especially if a tenant layer is developed ..yes/no ???

wmf3• 11 years ago

Probably. And OpenShift 3.0 (also on the front page today) is a PaaS (Cloud Foundry competitor) built on Mesos.

josephjacks• 11 years ago

OpenShift is most definitely not built on Mesos. RedHat's plans over time are to integrate with Mesos at a lower level for fine grained resource allocation, but for now Kubernetes is used as the cluster scheduling and orchestrations runtime.

StavrosK2• 11 years ago

Okay, maybe I'm getting old, but I can't really see the point of Mesos. It feels like it's adding too much magic to the mix, and I can't see myself using it to deploy web app servers/databases. Is there a different intended purpose that I'm missing? I suspect it's great for deploying task workers, for example.

bonobo3000• 11 years ago

Yeah i have seen Mesos used for managing big data frameworks like Spark. Any large distributed computing framework has to deal with some of the same problems - managing resources on worker nodes, sending tasks, getting results, scheduling etc. and Mesos/YARN provide a common base here, better than each framework duplicating the same code and solving the same problems.

StavrosK2• 11 years ago

That makes sense, thanks. The workers are pretty independent from each other, too, so this setup would be more fitting for that.

justrudd• 11 years ago

You can use it for web apps/services (https://mesosphere.com/docs/getting-started/service-discover...). But I'm with you on databases. I guess I just like knowing that those hosts are being managed and cared for (pets vs. cattle).

With that said, I often wonder how many people are using Mesos/Marathon before they have any need for it? Using it on for 4 hosts vs. 40 or 400?

tekacs• 11 years ago

There's nothing wrong with Mesos on 4 hosts, though...

Anything more than a single host requires additional co-ordination - picking Mesos is mostly no different than rolling alternate methods for doing so.

It's not choice with the least overhead, but it's a possibility, for those for whom it has the right conveniences...

lobster_johnson3• 11 years ago

I've looked at Mesos briefly, but it seems completely JVM-based, and that you're not prepared to build your whole stack around the current Java stack (Hadoop, Spark, Storm, Kafka, Zookeeper etc.), Mesos has zero utility. Is this accurate?

As someone working on scaling microservices, I keep being disappointed by potentially useful services that turn out to require a JVM-based language such as Java or Scala. For example, Kafka looks very decent, but the high-level client is written in Java; if you're not on the JVM, you're stuck implementing a lot of the client yourself. As far as I know (from the last time I looked at this stuff), the Zookeeper client is similar, whereas Spark and Storm both require that you write processing code on the JVM, and libhdfs is apparently still JNI-based, not native.

For someone using Docker, is there anything competing with Mesos that isn't wedded to Javaland?

senex• 11 years ago

There are frameworks that run on Mesos capable of running Docker containers. Check out Singularity, for example.

CoffeeDregs5• 11 years ago

AFAIK, the Docker capability is largely due to Mesos. Frameworks need the ability to specify a container type so that they can specify that a Docker container can run. But running a Docker container is the responsibility of Mesos. So Marathon and Chronos, both frameworks, are able to specify a job which uses a Docker container, while Mesos is able to run the job which uses a Docker container.

CoffeeDregs5• 11 years ago

[Not using Java -> ] Mesos has zero utility. Is this accurate?

Not at all. My company has a cluster with a few hundred Slaves and we're mostly a Python shop, with some C++ for machine vision.

Mesos certainly has its issues (as does everything), but its awfully nice for micro-services: if you can package your service into a Docker container, then you can launch it into the cluster and Chronos/Marathon/Mesos will take care of making sure that it's run/running.

I missed the deadline for the Mesos conference (I was unaware of the deadline), but I'm trying to squeak in a talk about "Using Mesos at [small] Scale" because we're a small company and Mesos has allowed us to do a bunch of big company stuff.

is there anything competing with Mesos that isn't wedded to Javaland?

Yes, I can look at the source code and see that it uses te JVM (Chronos uses Scala), but, AFAICT, Mesos isn't "wedded" to anything. All of the components are API-driven. I apt-get install it, I run it, I send jobs to it, it works and it behaves well. Better I can poke at the APIs of any of the services to find out what is happening. So we use Marathon for service-discovery and run Chronos, a framework, under Marathon. Makes finding Chronos, which could be on one of 200 machines, quick and easy.

lobster_johnson3• 11 years ago

Thanks for the detailed reply!

I have to say I'm wary of big frameworks like this that insert themselves as a kind of monolithic control structure for everything.

My ideal setup is always one where I pick and mix the best modules for the job, and where I can write some interface glue to let my apps slot into the system, as opposed to writing my apps for a specific API (as tends to be the case with, say, Hadoop, and which of course would tie the whole platform to that API, making it hard to migrate to something else).

Sounds like Mesos is pretty modular and open in that respect?

jacques_chester• 11 years ago

For someone using Docker, is there anything competing with Mesos that isn't wedded to Javaland?

Lattice[0].

It's built from a few components extracted out of Cloud Foundry, all written in Go. In particular it includes Diego, an orchestrator. It can be run locally, on AWS, Google and DigitalOcean.

In terms of the key architectural difference, Onsi Fakhouri explained it to me this way: Mesos is supply-driven, Diego is demand-driven. Mesos keeps a list of pending workloads and waits for a report of available resources that fits. Diego instead receives a request for resources and then stages an auction amongst available cells.

Disclaimer: I have worked on Cloud Foundry (on the Buildpacks team here in NYC) and I am employed by Pivotal, we do the lion's share of the work on Cloud Foundry.

[0] http://lattice.cf/

andrewjf• 11 years ago

Mesos itself is written in C++. Building Mesos will generate Java and Python bindings.

SEJeff2• 11 years ago

Mesos is written in C++ FYI.

lobster_johnson3• 11 years ago

Mesos itself is, but it seems every framework out there is written in Java or Scala: Chronos, Marathon, Aurora, Spark, Singularity etc. Also, it relies on Zookeeper, which is of course Java.

CoffeeDregs5• 11 years ago

As noted in another comment, we, a growing starting with 5 people around software out of 15 people total, run a Mesos cluster with a couple of hundred machines. By FAR our largest challenge has been to adapt our thinking to break the "this machine is 'production', that machine is 'staging'" mindset. We have a production compute infrastructure and you're welcome to launch 'production', 'staging' or whatever jobs into it. The friction around "running Mesos" has mostly been the friction of the air from exhalatoins of joy buffeting our esophagi...

We have a separate, much smaller cluster to test new Mesos (and Chronos and Marathon) versions. But the distinctions "production", "staging" and "dev" have become much more nuanced, so we've settled on discussing the "application" environment versus the "infrastructure" environment. Much as, as a startup on AWS, you wouldn't distinguish the RDS instance of your database (e.g. 9.3.1), you would distinguish the version of your database on an RDS instance, we distinguish the versions of our apps on the production cluster and not the version of the production cluster. One of the team was an ex-Googler and he said that Google did much the same.

The one thing Marathon and Chronos currently lack is a prioritization mechanism so we're building that as a Chronos task that monitors and scales down/up Marathon tasks by their priority (as represented in their id or tag).

KMag• 11 years ago

The friction around "running Mesos" has mostly been the friction of the air from exhalatoins of joy buffeting our esophagi...

Is there some common phrase relating belching and being happy that I haven't heard?

CoffeeDregs5• 11 years ago

Not that I know of, but it'd be awesome to have a word for joyous belching...

Introduction to Apache Mesos

Related Stories