Docker Betas for AWS, Azure, Mac, and Windows

https://blog.docker.com/2016/06/azure-aws-beta/

by petemill • 10 years ago

314 105 10 years ago

blog.docker.com

andor436• 10 years ago

As usual with AWS (or anything I guess) there's more than one way to accomplish a particular goal. How much of Amazon's Elastic Container Service is replicated by Docker for AWS? I'm currently using ECS + Docker but this looks potentially simpler.

petemill2• 10 years ago

Seems like there's not much in the way, at the moment, about attaching EBS volumes, or using your own custom AMI. For example, I want to mount an NFS share on the host to connect some container data volumes to directories inside. I've asked about this on the new forum https://forums.docker.com/c/docker-for-aws

tracker12• 10 years ago

What if your app was designed to work against AWS's data offerings, from RDS, S3, etc? Unless you're self-hosting your DB, concerns about persistence are less of an issue. Also worth considering if you are self-hosting is the replication/redundancy in the platform, you should be able to preserve-restart instances.

Me, I'd rather use what's available than self-host more often than not, but depends on the use-case.

KenCochrane6• 10 years ago

Hello, I'm a developer on this project.

Our goal for this project was to keep it as simple as possible. We tried to only use standard AWS features (CloudFormation, EC2, Autoscaling groups, VPC, ELB, etc) and we are using Docker 1.12 out of the box, with no changes.

All of the scheduling is handled in the docker engine itself, so we didn't need to add anything outside of that.

This version is pretty much an MVP, and we hope that the beta testers will help us test it out, and guide the future direction of the project.

jowiar• 10 years ago

Long-term, where does using something like this make sense vs. Mesos?

zjaffee• 10 years ago

I don't think it does make a difference, these are all just resource/cluster managers. Swarm and Kubernetes are more focused on containers, where Mesos is more robusta and can support running many other things on top of it.

theptip• 10 years ago

Mesos is proven at massive scale (Apple use it for Siri, Twitter use it for their data warehouse). You can run a cluster of 10k nodes if you want. I believe they have run simulations at higher scale than that.

Docker Swarm is not yet proven at such scale. They have run perf tests at 1k nodes, which is high enough for most use cases, but not for truly warehouse-scale computing. 1k nodes is also where Kubernetes tops out at (though expect that to be higher in Kubernetes 1.3, which should be released in the coming weeks).

On the other hand, if you're trying to run Docker container workloads, it's going to be easier in a Docker-first cluster orchestrator like Docker Swarm or Kubernetes; Mesos has Docker support but it's not as tightly integrated, and doesn't implement the full Docker management API.

tbrock2• 10 years ago

The more I use Docker the more I realize that GCE is going to smoke AWS and Azure in the future with regards to deploying a containerized infrastructure.

This is mostly because turn-key kubernetes on GCE is light years ahead of what ECS provides. It almost makes amazon's offering seem like a joke.

This might level the playing field a bit by having Docker itself provide the infrastructure management software instead of it being tied to a particular service.

AlexCoventry3• 10 years ago

A better comparison is probably Amazon's Elastic Beanstalk / Docker integration.

http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create...

tbrock2• 10 years ago

No, elastic beanstalk is terrible.

sheeshkebab• 10 years ago

second on beanstalk/docker - beanstalk does simplify quite a bit about running a fleet of docker images (and non-docker stuff too), uses standard aws tools (cf, elb, rds, ecs, vpc), has zero downtime deployment built in, automated server upgrades, app monitoring, environment cloning etc...

one nick against beanstalk is that it could be faster doing all those things, but it's tolerable considering alternatives.

vonkow• 10 years ago

Terraform + any old CI + Elastic Beanstalk works wonderfully for deploying docker apps on AWS.

moosingin3space2• 10 years ago

The most annoying thing about ECS is how it looks like docker-compose (v1) but is just different enough to be incompatible. At my last job, we were evaluating containerization, and we nearly gave up after seeing how awful ECS was to make work.

thesandlord• 10 years ago

GCE = Google Compute Engine GKE = Google Container Engine

The K is for Kubernetes :)

jdoliner• 10 years ago

Pretty sure the parent actually meant GCE in every spot it said GCE.

thinkersilver2• 10 years ago

ECS wasn't too bad. The approach they took made sense when it first came out but Amazon haven't evolved the idea since then. Personally I think that GCE's edge won't come from their container support since AWS and Azure both support k8s but will come from their value-add analytics and data services like BigQuery, Dataflow and so on. If Azure and AWS can't compete on that front then GCE will start to cannabilise the customers who are only still using the respective cloud providers only for infrastructure. GCE still has a long way to go before it becomes a serious contender.

NegatioN3• 10 years ago

What do you mean it has a long way to go? What exactly is missing in GCE compared to AWS? You just stated something you would be an edge in the future, but no cons it seemed like.

Interested to know because we're considering cloud providers.

dcosson2• 10 years ago

I'm overall pretty optimistic about AWS, but if they do end up losing to another platform I would expect it to be due to this general issue of not iterating on their offerings enough. I can't really recall them ever deprecating a service, e.g. forking something to a v2 and killing off v1after a certain amount of time. So with everything they release they only give themselves one chance to get the core of it right.

skrowl• 10 years ago

This is a rather naive view. AWS and Azure provide A LOT more than just VMs/containers (EC2) as a service.

Just having Docker won't give you the same that you can get from AWS RDS, S3, Glacier, Redshift, Route 53, etc.

jsolson• 10 years ago

Sure, but you can get similar things from Google Cloud SQL, Google Cloud Storage, Google Cloud Storage (Nearline), BigQuery, and Google Cloud DNS.

blaines2• 10 years ago

I can't get GPU instances on GCE and high CPU instances max out at 32 vCPU which is a blocker for me.

After further review, they don't seem to have a product similar to SNS, looks like I might be able to do most of the same things with Pub/Sub though.

moosingin3space2• 10 years ago

Firebase Cloud Messaging (FCM) seems like an alternative to SNS, although it isn't part of GCE.

_asummers2• 10 years ago

As you said, SNS is mostly covered with PubSub but SQS and any sort of RDS for something other than MySQL are absent (notably Postgres). I like GCE a lot and we use it at $work but there's no excuse at this point to not have feature parity.

KB1JWQ• 10 years ago

Very interesting that GCE wasn't mentioned on the slide of supported infrastructures at the DockerCon keynote today.

shykes3• 10 years ago

It wasn't mentioned because we only launched Docker for AWS and Docker for Azure. We only had the bandwidth to ship 2 versions at the level of quality we wanted, so we picked the most frequently requested providers. I think GCE is a great candidate for the next batch of releases. It all depends on demand. Would you use it?

ejholmes2• 10 years ago

Having been running a large scale production system on ECS for well over a year now, I'd like to say that ECS is hardly a bad choice; it's incredibly stable, integrates with proven AWS technologies (CloudFormation, ELB, etc) and it just works without having to manage clustering. All it needs is a layer of usability on top, which you can get from things like Empire and Convox.

While native Docker clustering is cool in theory, I think the vast amount of complexity around networking makes it unsuitable for production deployments at this time. If stability is more important than bells and whistles, ECS is great choice. Docker's production story, outside of simply running containers, has a lot to be desired.

AlexCoventry3• 10 years ago

Can you point to some of the instabilities?

parent54462• 10 years ago

What happened to the good ol' Unix philosophy? The docker command used to be about containers, not service and network scaling in the cloud.

clvx• 10 years ago

It doesn't apply to startup/companies.

Artemis2• 10 years ago

rkt does this: https://coreos.com/blog/rocket

geofft• 10 years ago

You can either have a bunch of different tools that do their own little thing and do them well and need a bunch of perl scripts and sysadmins on call to actually do anything useful, or you can have one large tool that does one meaningful thing and, by virtue of being one tool, does it well.

The UNIX philosophy is fine, it's just slowly escaping from the people who misunderstood it. (Oddly enough UNIX kernels have always been fine, partly because of the bikeshed principle, and partly because experimentally, doing a bunch of tiny things well doesn't actually get you a working OS.)

parent54462• 10 years ago

That I can agree with, but I still feel like there is a reasonable line to be drawn between application containerization and coordination and load-balancing of networked services.

shykes3• 10 years ago

Docker does follow the Unix philosophy: it's just that it's a unix-like collection of tools rather than a single tool.

Under the hood Docker is made of many small components which you can use separately if you want: containerd, runC, libnetwork, SwarmKit, HyperKit, VPNKit, DataKit, Compose, Machine, Notary, etc.

The 'docker' cli gives you a nicely integrated interface to all these components but you can use them separately too.

metalurgist• 10 years ago

I don't think splitting a project up into libraries is what the philosophy describes. If I filter out the 1600+ issue in the Docker project right now, I can see many related to Swarm. Your users now inherit all of the bugs and necessary features of swarm even if they're only interested in using Docker on a single host. The Unix philosophy would also not commend the amount of duplicate effort that the Docker team is pouring into swarm when the problem of clustering have already been tackled by other companies and teams. One unix tool does not develop half of another unix tool, it just uses that other tools and relies upon the experience and expertise of the community. No, If Docker grows a sub tool to solve every problem, it no longer qualifies as meeting the requirements of the philosophy, and it certainly doesn't reap the benefits of compose-ability at the user level. I vote for separate tools and freedom of choice any day over one giant, precomposed tool. I think it's also discouraging that we will now see swarm slowly grow all the features of its competitors, when that engineering effort could have gone elsewhere. To summarize: Don't reinvent, prefer choice to lock in, composition is at the user level, not the tool level (libraries don't count). Plugins were a great step forward, built in swarm was a step backward.

amasad3• 10 years ago

It's the idea popularized by git. Use one namespace and you have related commands that lives under one roof. Compare that with postgressql which installs a bunch of generic names on the top-level (e.g. createddb) which is annoying.

The thing I would be concerned about is bloat. I enjoy and leverage dockers' fast boot up and lightweight containers. I hope they stay focused on this.

thinksccs• 10 years ago

It's the idea popularized by git

It's also one that "source code control systems" have followed on Unix for 40+ years [1],[2].

1- https://en.wikipedia.org/wiki/Source_Code_Control_System

2- http://vasc.ri.cmu.edu/old_help/Archiving/Sccs/sccs.html

Myrmornis• 10 years ago

Seriously. When is postgres going to stop doing that? createdb and dropdb are absurd names to attempt to introduce into a user's path.

jalfresi• 10 years ago

My understanding is it follows the 'plumbing and porcelain' pattern; behind the scenes there are lots of small commands that do one thing and one thing well (the plumbing) and one command that acts as a wrapper over all these commands to keep things simple (the porcelain).

sadcontainer• 10 years ago

Docker's trying to lock in on enterprise customers. This has nothing to do with programming, and everything to do with encouraging lock in. I've lost a lot of faith in Docker. I thought they really cared about bringing containers to developers and fostering an ecosystem, but here we see them trying to oust out sysops competitors. Why compete in the container space when Docker wants to be the king of everything? Why put your trust in Docker as a developer when it's the enterprise they're after? There's still so many improvements to be made to the developer experience, still so many problems to be solved, but Docker is only interested in getting paid by the corporations that could care less. So no, the docker engine shouldn't be defining services and clustering. They touted plugins last year as the answer to extensibility, but of course now we're locked in to their idea of clustering. What their idea of a service is. If I were the Kubernetes team, I would be warning others about Dockers voraciousness. No one is safe as long as there is money to be had.

zymhan• 10 years ago

Am I the only one getting SSL errors trying to connect to their blog?

KenCochrane6• 10 years ago

It happened to me for a brief second, and it is fixed now, are you still seeing an issue?

julienchastang2• 10 years ago

Slightly off topic. What are people doing about user data persistence on the cloud/Docker? Specifically, we are porting a desktop application to the cloud via application streaming technology with Docker, but we would like the user’s data and preferences to not go "poof" when the cloud instance disappears. Ideally, we would like some automagic way to attach, say, the user's dropbox account or the equivalent to the cloud instance. Is anyone working on that problem?

pat2man• 10 years ago

Kubernetes has persistent volumes: http://kubernetes.io/docs/user-guide/persistent-volumes/

You can use something like Glusterfs and distribute your files or it can hook in to your clod provider and create persistent storage automatically on something like EBS.

ShakataGaNai• 10 years ago

When I tried it way back when, he trouble I had with Gluster is that it didn't readily handle nodes randomly leaving and new ones joining. It was more hardware-centric in that if a node left, you were expected to bring that specific node back online. Is that still the case?

dominotw2• 10 years ago

NFS volumes. There are many storage solutions out there for containers.

e12e3• 10 years ago

I think this is where the idea of co-locating compute and storage really shines (eg: Joyent/Manta, scaleableinformatics.com) - moving all state to nfs sounds like a great way to widen the performance gap between RAM and permanent storage... Not to mention that if you really want to spread your data and your application, you would require nfs over vpn (unless encrypted nfsv4 actually works now?).

bkanber2• 10 years ago

A few possibilities:

- If you don't actually need files, design around database instead

- If you need to store files redundantly, you can use a distributed filesystem like HDFS or GridFs

- If you need to store files redundantly, you can use an IaaS like Amazon S3

- If you need to store actual files (like for hard-linking), (redundancy optional), you can use network mounted storage

- If periodic backups of the data is ok, you can run backups that ship to S3 or glacier

Just a few, offhand, I'm sure there are a number of other techniques.

e12e3• 10 years ago

If you don't actually need files, design around database instead

That is fine if your target is eg AWS - but if you want all your infrastructure in docker, the db needs persistent storage...

joshpadnick• 10 years ago

I don't have direct experience with any of these, but the Docker approach here is that Docker exposes a Volumes Plugin API[1] which allows third-parties to implement portable volume plugins that achieve the ability for a volume to persist across hosts and across containers.

A bunch of plugins have been implemented[2], but I haven't personally heard any real success stories of people using them, which doesn't necessarily mean such stories don't exist.

On a separate note, many other teams run stateful services that can handle the complete loss of a node's data. For example, it seems popular to run Elasticsearch in Docker, though again I'm still learning about this, pattern, too.

[1] https://docs.docker.com/engine/extend/plugins_volume/

[2] https://github.com/docker/docker/blob/master/docs/extend/plu...

amasad3• 10 years ago

I'm interested in the streaming technology you mention. Is this something open source or supported natively by docker?

julienchastang2• 10 years ago

@amasad, my colleague created this which may interest you: https://github.com/Unidata/cloudstream

dang• 10 years ago

We merged the threads on these announcements. The Mac beta is at https://www.docker.com/products/docker#/mac and the Windows one at https://www.docker.com/products/docker#/windows.

pgz2• 10 years ago

I don't understand why Docker for Mac is a GUI application. I'd rather get the same features from the CLI.

KenCochrane6• 10 years ago

You can still use the CLI, there is nothing stopping you.

pgz2• 10 years ago

So I assume the new native hypervisor is available from cli as well? Then that's excellent!

moondev2• 10 years ago

You should try it. The toolbar is just there to see the status of the daemon and adjust vm settings. You still use the cli to use docker just like on linux.

tacos2• 10 years ago

They never emailed me from the last "private beta" before announcing this. And now deploying this on AWS or Azure requires me to sign up for yet another "we'll get back to you..." private beta.

Love the tech, hate all this marketing runaround.

EDIT to add: either your beta is ready or it ain't. I understand a gentle initial seed to verify it's not an utter catastrophe but no need to play nanny with my bits across 20+ beta releases. I'm a grown up. Make a disclaimer and let me assess the risk. Your corporate logo looks like a 1970s Carvel ice cream cake--I know what I'm getting myself into.

KenCochrane6• 10 years ago

Check your spam folder, some of the email from the last private beta went in people's spam folder, so they never saw them. Either way it doesn't matter anymore since Docker for Mac and Windows is now public to use without the need of a code.

tacos2• 10 years ago

I'm not sure there's much overlap is between people who can get Docker Machine limping on VirtualBox with storage and networking flowing properly on Windows 10 fast ring and those who don't know how to check their spam folders, but I appreciate the tip.

No messages (and I applied via two accounts for two different platforms). No message telling me "go get the bits!" today. And nothing in spam folders.

But yet another blog post asking me to sign up for yet another private beta... ugh.

EDIT to add: As polite feedback--two separate requests, two separate Docker accounts, one Gmail and one corporate email address, and I did not receive any invite during the private beta nor notification today when it went public.

WestCoastJustin• 10 years ago

Love the tech, hate all this marketing runaround.

Just wanted to offer an alternate point of view or maybe some context on this. There was over 70,000 people that signed up for the Docker for Mac & Windows beta. So, using an invite system was more of a self preservation measure, and not waiting to expose tons of people to bugs, as the beta was refined (through roughly twenty iterations). If a bad update rolls out across all these people, that would not be good, as you can imagine. So, much more self preservation measure, and wanting to catching things quickly in smaller batches, than wanting to market to you.

Here is the direct link to download Docker for Mac without a key https://www.docker.com/products/docker#/mac

Disclaimer: Working for Docker but this is my own thought.

Sanddancer• 10 years ago

They need to put a note on their page that docker for windows is not compatible with docker for windows containers. I've been playing around with docker for windows containers for a few days, then saw this and thought, "cool, an update." I installed, and discovered that while the client is compatible between the two, the daemons they run cannot see the containers created in the other daemon. MS and Docker need to sort this stuff out, because right now, windows containers are nicer for the few images that have been released, but docker for windows allows for the full docker ecosystem.

justincormack• 10 years ago

We will sort this out soon, sorry.

spilk• 10 years ago

Not a fan of the new Windows version as it requires you to enable Hyper-V, which stops any other virtualization (Virtualbox, VMware, etc.) from functioning. The only workaround I've found is rebooting to enable/disable it on demand.

syaz1• 10 years ago

What are you talking about? I had to enable Hyper-V to get Virtualbox or any other VM solution working. But yes you're right that you can't use them at the same time. The new native docker doesn't use third-party VM (boot2docker) and interfaces with the host hypervisor directly. So I had to stop all running virtualbox VMs to get docker to start.

GordonS• 10 years ago

And the Windows version is still Windows 10 only :/

Many large organisations are going to be tied to Windows 7 for a good while yet...

pjmlp• 10 years ago

Because it makes use of Windows containers introduced in Windows 10, instead of bring GNU/Linux with Virtual Box along.

Same thing on Mac OS X, they have migrated to the hypervisor API.

FloNeu• 10 years ago

Now even docker wants me to upgrade to windows 10 ^^ Keep up the great work!

mwambua• 10 years ago

IBM's Bluemix has supported Docker Containers for a while now, but hasn't gotten much limelight... probably as a result of the size of their community and Bluemix's sketchy [but improving] stability. Does anyone have any experience using their container service? And would this be a big improvement?

patwolf• 10 years ago

I've been using the container service for a few months. I have encountered a case where my running containers completely disappeared and was told by support that this was because of a "migration". They claimed to have notified me beforehand on several occasions, but I have no evidence of this. Hopefully this was a one time occurrence and will lead to greater stability.

One other general problem I've had is the time it takes to launch a container. I would expect containers to be an improvement over VMs in terms of how quickly you can launch, but that doesn't seem to be my experience so far.

I would wait a while longer before I would consider their container services for production workloads.

Docker Betas for AWS, Azure, Mac, and Windows

Related Stories