The future of Kubernetes is virtual machines

http://tech.paulcz.net/blog/future-of-kubernetes-is-virtual-machines/

by MordodeMaru • 8 years ago

220 122 8 years ago

tech.paulcz.net

tybit• 8 years ago

As 2018 comes to a close its time to drag out the hubris and make a bold prediction. The future of Kubernetes is Virtual Machines, not Containers.

I’d say that’s less of a prediction than a matter of fact since it’s already happened in 2018 for AWS and GCP.

This post is kind of irrelevant for consumers imo, in that the future of Kubernetes is still the container interface, regardless of whether your vendor decides to run it in a container or a VM.

halbritt10• 8 years ago

How has it already happened? It's true that EKS and GKE nodes are virtual machines, but that's entirely beside the point of this article which seems to make the point that the workloads running inside of Kubernetes will also be virtual machines.

Interesting notion, but I don't see it. The reason for kubesprawl as it is today is a result of the fact that today Kubernetes is hard to tune for disparate workloads which leads most folks to just punt and stand up multiple clusters.

That said, people are starting to figure it out and more tools like the vertical pod autoscaler are coming. Eventually the more efficient choice will be to run disparate workloads across the same set of hardware.

k__• 8 years ago

I had the impression DevOps ppl would cling to containers as the final form of VMs?

But I'm mostly a high level front-end guy doing back-ends with serverless tech only.

stevenacreman2• 8 years ago

It's still containers. On a cloud provider the Kubernetes workers are VM's which orchestrate containers. With Kata Containers you're just spawning containers inside micro-vm's.

int0x802• 8 years ago

That is one big difference, with big implications.

stevenacreman2• 8 years ago

Just wanted to clarify that Kubernetes was still scheduling containers. Even if VM's are being used to isolate them.

It's not all or nothing either. Containerd will support running a mix of containers and kata-containers across workers.

For anyone interested in this topic I wrote about some other container runtimes here: https://kubedex.com/kubernetes-container-runtimes/

jacques_chester11• 8 years ago

It's still containers.

The security profiles of containers and VMs, including kernel-based VMs, are different. VMs still have a significant edge, because the attack surface is smaller and doesn't have many competing missions.

tptacek• 8 years ago

The attack surface argument is debatable, depending on how the system is designers, since virtualization introduces the hypervisor surface.

cyphar• 8 years ago

The attack surface of a container can be massively reduced with seccomp profiles -- there was a paper a few years ago which found that the effective attack surface of a hypervisor was about the same as the attack surface of a locked-down seccomp profile of a container (and LXC/Docker/etc already have a default whitelist profile which has in practice mitigated something like 90% of kernel 0days).

And let's not forget the recent CPU exploits which found that VMs aren't very separated after all.

The fact that Kubernetes disables this (and other) security features by default should be seen as a flaw in Kubernetes. (Just as some of the flaws of Docker should be seen as Docker flaws not containers-in-general flaws.)

raesene93• 8 years ago

it's not really as clear cut as that (IMO).

Shared kernel linux containers can be hardened to the point, where they likely have a smaller attack surface than a general purpose hypervisor (for example look at the approach that Nabla takes)

You then have the hybrid approach of gVisor, still containers, but smaller attack surface than the Linux kernel.

Of course this hardening approach can (and should be) applied to VMs too, which may tip the balance back to them, which is one reason that firecracker looks so interesting.

amscanne3• 8 years ago

An important element of Kubernetes is that it standardizes the infrastructure control plane, and allows different pieces to be plugged in (for networking, storage, etc.).

The "virtual kubelet" essentially throws that all that away and keeps Kubernetes "in API only". For example, with virtual kubelets, scheduling is meaningless and networking and storage are restricted to whatever the virtual kubelet target supports (if useable at all).

Personally, I think the value proposition is tenuous -- you can create VMs today, doing so via the Kubernetes API isn't suddenly revolutionary. Just like throwing something in a "hardware virtualized" instance doesn't suddenly make the whole system secure.

Containers and Kubernetes are compelling for a variety of reasons, improving them to handle multi-tenancy is a broad challenge but I don't think the answer is to reduce the standard to what we have today (a bunch of disparate VMs).

serverascode4• 8 years ago

I agree, I'm not sure the virtual kubelet concept is a great idea overall. It sounds good on the surface, but most of the time these kind of abstractions do more harm then good, as far as I'm concerned. But, I could be wrong. :)

jacques_chester11• 8 years ago

Personally, I think the value proposition is tenuous

Multi-tenancy is a pretty compelling value proposition when you reach any kind of scale. If you're in a regulated sector, it's non-negotiable.

Relying on the cluster as the security boundary is very effective ... and very wasteful.

Containers and Kubernetes are compelling for a variety of reasons, improving them to handle multi-tenancy is a broad challenge but I don't think the answer is to reduce the standard to what we have today (a bunch of disparate VMs).

I think the argument is that rather than the painful (and it will be very painful) and probably incomplete quest to retrofit multi-tenancy into a single-tenancy design, we can introduce multi-tenancy where it basically actually matters: at the worker node.

At first glance it's confusing to go from "one master, many nodes" to "one node pool, many masters". But it actually works better on every front. Workload efficiency goes up. Security surface area between masters becomes close to nil.

Very cheap VMs are the means to that end.

Disclosure: I work for Pivotal and this argument fits our basic doctrine of how Kubernetes ought to be used.

amscanne3• 8 years ago

I'm saying that I think the value proposition for the virtual kubelet is tenuous, not multi-tenancy as a whole.

For a single cluster, "very cheap" VMs solve some of the problems, but leave others unsolved (e.g. they prevent some hardware and kernel exploits, but lots of security issues can still hit you -- like the last two big K8s CVEs). They also sacrifice a lot of the things that make containers compelling on the floor (high efficiency and density), so I don't think they should be spun as a panecea.

You seem to be arguing that one shouldn't bother with multi-tenancy on a single cluster, which is a fine approach, but I do think that the technologies and tools to support the single cluster model are evolving. Calling it a "multi-tenancy retrofit" seems a bit FUD-y to me. Just because there are challenges doesn't mean it's not worth doing.

davidopp__3• 8 years ago

I don't think multi-tenancy has been "retrofitted" onto Kubernetes. Kubernetes was designed with multi-tenancy in mind from the very early releases -- namespaces, authn/authz (initially ABAC, later RBAC), ResourceQuota, PodSecurityPolicy, etc. New features are added over time, such as NetworkPolicy (which has been in Kubernetes for a year and a half, so perhaps not "new" anymore!), EventRateLimit, and others, but always in a principled way. And the integration of container isolation technologies like gVisor and Kata are using a standard Kubernetes extension point (the Container Runtime Interface) so I do not view this work as retrofitting.

Moreover, even today there are real public PaaSes that expose the Kubernetes API served by a multi-tenant Kubernetes cluster to mutually untrusting end-users, e.g. OpenShift Online and one of the Huawei cloud products (I forget which one). Obviously Kubernetes multi-tenancy isn't going to be secure enough today for everyone, especially folks who want an additional layer of isolation on top of cgroups/namespaces/seccomp/AppArmor/etc., but there are a lot of advantages to minimizing the number of clusters. (See my other comment in this thread about the pattern we frequently see of separate clusters for dev/test vs. staging vs. prod, possibly per region, but sharing each of those among multiple users and/or applications.)

Disclosure: I work at Google on Kubernetes and GKE.

davidmr• 8 years ago

I agree, but I think I’ve put less thought into it than you have. Just because I tend to focus on research and low-latency infrastructure, I’ve been really happy with the container-first approach. I really like being able to deal with these processes at the Linux level instead of the VM level and being able to tweak that stuff (cpu placement and isolation, accelerators and rdma network devices, etc.) There’s a reason VMs never really took off in HPC, but I think CRI-O is really poised to change the HPC paradigm, and k8s can be really beneficial in some business applications of HPC.

I definitely understand and agree that multitenancy is super important, but it would be a shame to agree that the bare metal performance is an okay sacrifice.

acje• 8 years ago

What we need is an open technology stack such that we can get full standardization and integration with k8s and other systems. We might get that with RISC-V.

orliesaurus• 8 years ago

This feels like an infinite russian doll problem, putting k8s inside a bunch of vms I mean.. but I am glad that while everyone is still trying to learn how to properly do k8s these folks are thinking of the next thing

dehrmann• 8 years ago

And running a Java services in the JVM. It's not real infrastructure unless you have at least four layers of VMs.

old-gregg• 8 years ago

Kubernetes has a real chance to succeed where OpenStack failed. Good people at AWS have good reasons to be worried and will push us to a proprietary form of "serverless".

serverascode4• 8 years ago

I think they are quite different solutions, and each benefits from the other. Further, they also tend to run into the same concerns at the about the same time in their life time, eg. concerns around complexity.

lykr0n3• 8 years ago

OpenStack's failure is that it's an extremely complex system that looks to be nearly impossible to deploy by yourself. You have to use a Distro that does everything for you- their way.

Kubernetes is simple enough to setup yourself. They have well documented tooling, and a solid do it yourself guide. OpenStack has none of that (that I can find). You select RedHat Openstack (RDO) or Canonical OpenStack (MAAS), and you have to use their all-in-one system to have a deployment- and that requires a narrow set of variables which every environment might not have. Which is insane- and will hinder adoption.

EDIT. Not 100% correct, see below comments.

bamboozled• 8 years ago

The issue with Kubernetes isn't that it's hard to setup, but it's difficult to keep operate long term. Simple things like balancing workloads across availability-zones is still not trivial.

I've heard many stories of people who tried to run Kubernetes themselves in production and didn't have a great experience for it.

serverascode4• 8 years ago

You don't have to use a distro. There are well tested puppet modules, which you could make into your own "distro", as well as openstack-ansible and kolla, and openstack-helm which uses helm to deploy openstack. There is also StarlingX. There is no kubeadm like system however, though I'm not sure how many people will really use kubeadm in prod.

Is it complex? Yes. Is it more complex than k8s? Probably. However, are there multiple open source distros outside of RDO (which is not actually a distro and is instead packaging--see tripleo for a distro like solution based on RDO). MaaS is not an OpenStack distro; it's a way to manage baremetal nodes that OpenStack is then deployed onto using other Canonical related tools. That said, selecting a way to deploy and manage openstack is complex, but the same with k8s.

trjordan2• 8 years ago

This is, like all good HN articles, technically correct and practically incorrect.

It is correct that containers leak, and people know this. Multi-cluster strategies are real, and they shouldn't be. It should be OK to have one big cluster[1]. Until Kubernetes fixes this, there will be some friction to adopt it, based on real use cases like untrusted code and noisy neighbors.

It is incorrect because users (e.g. non-infrastructure engineers) don't know or care about the precise definition of containers and VMs are. The point of "containers" is that I can define something that acts like an operating system from the ground up, and it builds quickly and runs quickly in production.

Kubernetes doesn't win by forcing users to think about VMs. Kubernetes wins by adopting a VM standard that can be built by Dockerfiles. Infra engineers will love it.

But besides them? Nobody will care, because Docker for Mac will look the same.

[1] Maybe 1 cluster per region? There's a whole fascinating topic that starts with the question "when building a PaaS, do you expose region placement to devs?" The answer implies a ton of stuff about what exactly it's reasonable to expect from a PaaS and how much infrastructure your average dev has to know.

georgebarnett4• 8 years ago

It should be OK to have one big cluster

Assuming you're deploying your Kube cluster in the cloud, the costs of having multiple clusters is really reduced. You don't have to allocate physical machines or worry about utilisation as much - you just pick a node size and autoscale.

What that enables is thinking about other concerns when deciding how many clusters and where they are is right for your team.

There are operational reasons why having multiple clusters is a good idea. At the simplest level, making a config change and only risking a portion of the infrastructure is an example.

trjordan2• 8 years ago

Config blast radius, hardware sizing, networking partitions, disaster recovery, dev/prod separation -- lots of good reasons to have multiple clusters!

As a gut check, most of these reasons apply to AWS AZs as well. If your Kubernetes strategy calls for more than one per region per AWS account, it means that you, organizationally, don't trust containers as much as you trust VMs on AWS. And right now, you're probably right to do so.

ec1096854• 8 years ago

You have the blast radius reduction by deploying to multiple AWS regions, similar to what AWS does with their infrastructure.

davidopp__3• 8 years ago

A pattern we're seeing a lot of recently is one cluster per "stage" per region, where a "stage" is something like dev/test, canary, and prod. (In some cases only prod is replicated across multiple regions.) I think this may end up being the "sweet spot" for Kubernetes multi-tenancy architecture. The number of clusters isn't quite at the "Kubesprawl" level (I love that phrase and am absolutely going to steal it) -- you can still treat them as pets. But you get good isolation; you can limit access to the prod clusters to only the small set of folks (and perhaps the CD system) authorized to push code there, you can canary Kubernetes upgrades on the canary cluster(s), etc.

As an aside, something that's useful when thinking about Kubernetes multi-tenancy is to understand the distinction between "control plane" multi-tenancy and "data plane" multi-tenancy. Data plane multi-tenancy is about making it safe to share a node (or network) among multiple untrusting users and/or workloads. Examples of existing features for data plane multi-tenancy are gVisor/Kata, PodSecurityPolicy, and NetworkPolicy. Control plane multi-tenancy is about making it safe to share the cluster control plane among multiple untrusting users and/or workloads. Examples of existing features for control plane multi-tenancy are RBAC, ResourceQuota (particularly quota on number of objects; quota on things like cpu and memory are arguably data plane), and the EventRateLimit admission controller.

There's active work in the Kubernetes community in both of these areas; if you'd like to participate (or lurk), please join the kubernetes-wg-multi-tenancy mailing list: http://groups.google.com/forum/#!forum/kubernetes-wg-multite...

Also, I gave a talk at KubeCon EU earlier this year that gives a rough overview of Kubernetes multi-tenancy, that might be of interest to some folks: https://kccnceu18.sched.com/event/Dqvb?iframe=no (links to the slides and YouTube video are near the bottom of the page)

Disclosure: I work at Google on Kubernetes and GKE.

justicezyx2• 8 years ago

Glad to see people believe there should be one cluster per region, and k8s can be fixed to make that happen.

I am a firm believer of Borg style cluster OS, what Borg lacks is a modern API, but the fundamentals are already there.

Disclaimer: I am with Google's Borg team, with a focus on its client side.

hardwaresofton• 8 years ago

Kubernetes doesn't win by forcing users to think about VMs. Kubernetes wins by adopting a VM standard that can be built by Dockerfiles. Infra engineers will love it.

This is what the CRI does/is, basically. Various projects sprung up to make it possible @ the runtime/kubelet level (kata-containers, frakti, containerd untrusted workloads), but support for runtimeClass[0] is what's going to tie it all together and it's already in Alpha.

Personally I think we should be moving away from docker files -- docker's superior ergonomics pushed the industry forward at the outset, but they lagged in features and compliance with any standards for a long time and were basically usurped. The Dockerfile is a decent format but it lacks a lot of good qualities, like being trivially machine-editable, and the docker client itself has some unfavorable tradeoffs when compared with tools like rkt and podman. I think it's a mistake to standardize on Dockerfiles, but it's definitely a good idea to standardize on the CRI (which Docker now adheres to via a containerd shim by default).

The best thing by far about kubernetes is the CRI[1], CNI[2] (Container Networking Interface), and CSI[3] (Container Storage Interface) standards that are coming out of it. I don't think any one realizes it yet, but they are inadvertently building secure/sandboxed computing for everyone. Containers are just sandboxed processes, and before this, most people were running basically completely unsandboxed processes (both in the cloud and on their personal computers) -- once all this stuff lands, linux is going to have some amazing features for running applications more safely -- production-grade safety for any application you run on your own machine available with standardized tooling.

[0] https://github.com/kubernetes/enhancements/blob/master/keps/...

[1] https://github.com/kubernetes/community/blob/master/contribu...

[2] https://github.com/containernetworking/cni/

[3] https://kubernetes-csi.github.io/

ec1096854• 8 years ago

The nested kubernetes idea is interesting (linked to from the post). However, amazon and google aren’t using nested control planes for their infrastructure. Why is a single control plane good for them, but not for a Kubernetes deployment?

jchw• 8 years ago

"virtual machines" may be the wrong word. Containers is probably still the right word. What we probably want, is actually isolated containers.

And hell, you can nearly get that today. Combining Docker with gVisor is a potential solution to the soft tenancy problem as far as I can tell, and Kubernetes supports using it.

(And gVisor is by no stretch of the imagination a 'VM' - it is, at best, a tiny hypervisor, and maybe less than that.)

mcguire• 8 years ago

As someone who has been a sysadmin and system programmer for 25 years and a crusty rat bastard for almost that long, I have to wonder how long it is going to be until someone realizes that a piece of hardware doing a task is more efficient than 27 layers of virtual machines and a long pair of tongs.

nimbius• 8 years ago

amen. At first i figured containers were a good way to run code with difficult dependencies and weird operating environments, but that still doesnt make sense.

- systemd and uwsgi for example play well when run as a single user per wsgi application under a single nginx/lb.

- php-fpm already handles a ton of overhead from php apps.

- ansible deployments called from gitlab-ci can roll out apps just as well as deploying from the registry.

then i figured maybe they were on to something with the autoscaling thing...but that seems like a meaningless feature. Every good project already has metrics and forecasting...it would be absurd to think a final product like imgur.com or twitter does not know (down to the byte) how much storage theyll need in 4 months and the potential drivers.

auto-scaling infrastructure just betrays the fact that most developers throw resources at load problems instead of waiting for ops to figure out the actual issue.

latchkey• 8 years ago

I helped build an e-commerce marketplace, which is essentially a shopping cart system that was used by third parties. We had no idea how to predict traffic loads because we had no idea when one of our customers would run a successful campaign. Autoscaling on google appengine was a lifesaver for us. Our very small dev team focused on building features because we had zero devops and we didn't have to carry a pager.

rhinoceraptor• 8 years ago

I don't think most developers or admins really have the toolset/skills to debug to root cause. Most people see debugging as the process of making problems go away, not as the science of understanding a problem.

You need both developer and admin skills, as well as a pretty good understanding of the whole stack - the application, the framework/libraries, the application runtime/support libraries, the database, the kernel and the network.

tlrobinson• 8 years ago

Or maybe raw performance isn’t always the most important attribute of a system?

clagio2• 8 years ago

Not directly, but cost usually it is. Being able to serve the same number of customers with less computational power means reducing your cost.

clagio2• 8 years ago

I agree with you, being in the same position. I wonder if the DevOps trend started because of the constant friction between devs and ops. While devs main concern and job is to deliver new features, ops main concern and job is to keep things stable. These different goals caused ops to be seen as an obstacle to dev job. Being the devs close to the business was easy to predict that we would be taken out or at least required to change. I think, we (ops) are mainly the one to blame for the current situation and now we can only adapt or die.

de_watcher• 8 years ago

It has become manageable to do both for one person.

halbritt10• 8 years ago

Never.

shaklee3• 8 years ago

That's kind of the point of k8s. Are you saying containers have too much overhead?

akvadrako• 8 years ago

I don't think he knows what he's saying. At a small scale you have overhead because you need to run the orchestration software.

But everything else (disk, CPU, network) typically amounts to less than 1%.

stubish• 8 years ago

This is why things like MAAS exist, which is a 'cloud provider' that lets you provision bare metal machines. Plug in a new rack of servers, let them netboot, and let the clients spin up a machine matching their requirements in the same way they would spin up an AWS or Azure instance.

We mix bare metal spun up this way with Open Stack VMs, depending on the requirements. And slice up into containers using LXD when that makes sense.

dilyevsky• 8 years ago

Your definition of efficiency is too simplistic. In many cases it’s more “efficient” to have rapid development cycle and low ops overhead than save a few bucks on hardware.

Even if we accept that server cost is key, Google realized long time ago that they can squeeze more out of their fleet if they can overcommit it because many workloads have variable utilization over time. Hence “containers” were conceived.

caymanjim• 8 years ago

Very much this. People time is always the biggest cost. Docker and Kubernetes have been a game changer in time-to-production. Unless you need maximum performance out of the biggest iron available, it's cheaper to buy the next tier of EC2 instance than to waste developer and operations staff time trying to squeeze more out of existing servers. There are many levels of abstraction, but...who cares? It's about productivity per dollar.

barrkel2• 8 years ago

I want to be in a world where allocating hardware is as simple (at the API level) as allocating a block of memory or creating a thread. Kubernetes is a means to that end, and with an API that isn't tied to a specific vendor.

The efficiency of the actual code executing on the hardware is secondary to the efficiency of being able to automate allocation of hardware at the application level.

sigi45• 8 years ago

10 Years ago when i needed a machine for anything, i had to create a ticket, wait a week or so to get what? One machine.

Than i got told, that they have to control it, otherwise backup and restore is not supported and on the other hand, they were only able to install stuff like svn by spending virtual 3 project days! 3! I never asked what it would cost to install some etherpad.

Two years ago in a startup, i logged into aws, clicked around for an hour and had 2 isolated networks, 4 instances, a load balancer, dns server, snapshot and backup configured. A few mouseclicks later i could have had an autoscaler as well.

Ops / own Hardware is a means to an end.

elsonrodriguez• 8 years ago

The future of Kubernetes (for some interesting use cases), is virtual machines. For the rest of us the virtual kubelet project represents a bridge between Kubernetes and Serverless.

I'd much rather pay my cloud provider to run my k8s workloads (billed by pod requests/limits) than pay for a control plane and three nodes just to run my workloads.

state_less• 8 years ago

While the use of namespaces seemed a bit discounted in the article, I think part of the problem is that it's an underutilized and poorly conveyed feature, but maybe that's just me? I'd like to see a better out-of-the-box vanilla K8S user management interface in the dashboard. Maybe the default UX on cluster creation is a user/service account that isn't cluster admin that is limited to 1 or more namespaces. There should be a better dashboard to configure and create users that is front and center when you first create a cluster. You have to work a bit to get something you can use to authenticate as cluster admin like role. This should help direct people towards creating more users and isolated namespaces.

justicezyx2• 8 years ago

Also gVisor.

halbritt10• 8 years ago

My org is using namespaces for tenant isolation. Works quite well.

geerlingguy• 8 years ago

Namespaces++. Unless you’re just running one app in a Kubernetes cluster (where default == that app), namespaces solve a plethora of problems.

ec1096854• 8 years ago

Agree, I'm not sure what the author is getting at here:

"Compounding this is the fact that most Kubernetes components are not Tenant aware. Sure you have Namespaces and Pod Security Policies but the API itself is not. Nor are the internal components like the kubelet or kube-proxy. This leads to Kubernetes having a “Soft Tenancy” model."

That's like saying the Hypervisor isn't "tenant" aware.

raesene93• 8 years ago

+10000

User admin and the reliance on client-cert authentication is one of the biggest weaknesses I see in k8s security at the moment.

There are obviously other options like OIDC available, but it can be tricky to set up and isn't on by default, so instead client certs are used for user auth. and given the lack of certificate revocation, they're really not suited for that.

mattbillenstein2• 8 years ago

Dammit, most of you reading this do not need any of this shit! Build simple things!

TuringTest3• 8 years ago

You don't need them until you're hired by a client with nation-wide deployment requirements, and then you need them.

aprdm• 8 years ago

Not necessarily. Needless to say containers and kubernetes are pretty new technologies and nation wide deployment requirement existed before docker / k8s.

Most of the old big companies aren't using docker or k8s for their core services. They're all using legacy fat apps that are load balanced in baremetal or vms.

013a• 8 years ago

I really worry about cost in this future of leaning on something like ACI/Fargate to actually run the containers.

An m5.large instance (2vcpu/8gb) costs $70/mo on-demand ($44/mo with a 1 year reservation). A similar Fargate runtime costs $146/mo.

A b2ms Azure instance (2vcpu/8gb) costs $60/mo on-demand ($39/mo 1 year reservation). Azure Container Instances at a similar provisioning level costs $176/mo by my calculations.

That's not a small difference. That's, like, 3x.

Point being, I love virtual-kublet from the perspective of a scale-up just trying to get a product out the door. But for established companies, I still think the core idea of a container on a VM you control is going to rule. Fortunately Kubernetes allows amazingly easy flexibility to switch, and that's a reason why it might be the most important technology created in recent history.

hedora• 8 years ago

Also, the instance prices are already some crazy multiple of buying a machine, racking it, and running dozens of VM’s on it for 5 years.

I don’t know the prices for hyperconverged or converged infrastructure off the top of my head, but doubt the amazon rates are even close to competitive unless you’re at a tiny scale (or need a ton of tiny presences in different regions).

markbnj• 8 years ago

I get the multi-tenancy argument, for certain use cases. I'm not sure I understand the point about greater resource utilization. Presumably they mean workloads can be scheduled more densely given a set of hardware resources, vs. containers... but I'd think that containers would score better on that metric than VMs. Can someone expound?

The best part of the piece for me was "kubesprawl," my new favorite word for the week. We've seen it ourselves to some extent, but we are at least aware of it and try to exert some pressure in the other direction. Beyond that I am not particularly bothered by the idea of running lots of clusters for different purposes.

emmelaich• 8 years ago

I'm picking the eyes out the article here but I had to twitch at this ...

Linux containers were not built to be secure isolated sandboxes (like Solaris Zones or FreeBSD Jails). Instead they’re built upon a shared kernel model ...

Solaris Zones and BSD Jails both use a shared kernel.

And I'd bet you they were far from perfect in security isolation.

Now it may be true that security wasn't Linux containers prime reason for being but we have an existence proof that they can be made secure enough -- anyone can get a trial Openshift container for the price of a login.

peterwwillis2• 8 years ago

I'm convinced that Kubernetes is very good for job security, and not much else. Unless you're a managed services host, you should probably not be running it.

Please, for the love of all that is holy, use a cloud services provider if you need K8s-style service features. If you don't, then just cobble together your infrastructure in the simplest way possible that uses DevOps principles, methods and practices.

rantanplan• 8 years ago

Could you elaborate a bit?

peterwwillis2• 8 years ago

K8s is a very complex system. And the more complex the requirements, the more complex one needs to make K8s through additional software that isn't baked in. Complex systems are costly to run, but more importantly, they're costly to maintain due to the typical level of service required, the number of employees needed to maintain it, and the amount of specialized knowledge required. The system is also under constant maintenance due to its short release cycle. Basically, you need to build an entire cloud services team just to keep it running smoothly. (not for "test labs", but for real production services) And on top of all this, if you're running it on your own hardware, you don't even get the benefit of reduced infrastructure costs.

Because this is not only hard to get right, but very costly, it is much cheaper and easier to pay someone to do all this for you. It is almost guaranteed that doing it yourself will not give you any significant advantage, cost savings, or increased development velocity.

On top of this, most people don't even need k8s. K8s is a containerized microservice mesh network. If you don't need containers and you aren't running microservices, you may be trying to fit a square peg in a round hole. Even if you did need k8s, the benefit may be small if you don't have complex requirements.

Most people can get high-quality, reliable end results with simple, general-purpose solutions using DevOps principles and tools. If you're not Google or Facebook, you probably just need immutable infrastructure-as-code, monitoring, logging, continuous integration/deployment, and maybe autoscaling. You don't need an orchestration framework to deliver all that. And by going with less complex implementations, it will be easier and more cost-effective to maintain.

At the end of the day, if you need k8s, use it. But I really worry about most people who hop on the k8s bandwagon because they see a lot of HN posts about it, or because Google touts it.

xaduha• 8 years ago

I'd be fine with that if they were using LKVM (from kvmtool) or that new Firecracker. But as far as I can tell most projects in this area use QEMU still, which is an outstanding piece of software, but doesn't scream micro-vm to me.

Annatar2• 8 years ago

And people still wrangle with this... how can it be easier to struggle like this than learn how to use SmartOS and Triton? Kubernetes is a solution to severe OS virtualization deficiency in Linux, most notably orchestration. You, know, the problem which is non-existant in SmartOS with Triton and large scale configuration management with operating system packages. Every so often this hits "Hacker News" and people will rather struggle than master something new. But one cannot polish a turd.

jskaggz2• 8 years ago

At the risk of tarnishing my reputation amongst the hacker news docker/kubernetes hypecycle elite, have an upvote. The I.T. industry in general is funny. New technologies come and go like pop stars. Docker == ke$ha, Kubernetes == ice cube, triton is fred astaire. They all have their off moments. I personally like my platform stable, performant, secure, and boring. If I spent all of my time keeping up on the latest trends on how to spin up machines, I'd have little time to work on actual product. Something good will come out of this influx of cash, marketing, and cloud sales, eventually. Fits and starts. /me goes back to coding and deploying on triton, while patiently watching the docker/kubernetes show.

Annatar2• 8 years ago

That was a big risk you took. You have guts for standing up to the hype machinery.

CyanLite4• 8 years ago

TLDR: Kubernetes is an anti-pattern.

The future of Kubernetes is virtual machines

Related Stories