The sad state of sysadmin in the age of containers (2015)

https://www.vitavonni.de/blog/201503/2015031201-the-sad-state-of-sysadmin-in-the-age-of-containers.html

by maple3142 • 7 years ago

428 310 7 years ago

_def• 7 years ago

I don't know that much about the state of container systems these days, but to me it seems it's the "comfortable" way in the "security vs comfort" tradeoff. Use where applicable and hope for better days.

ThreeFx2• 7 years ago

But there are some environments where security is (almost) the only relevant factor, e.g. banking IT-infrastructure. I don't want my bank to pull arbitrary images from Dockerhub because that increases their comfort.

Conversely, if I were sysadmin at a bank I would most definitely be concerned what was running in my network.

buzzkillington6• 7 years ago

Conversely, if I were sysadmin at a bank I would most definitely be concerned what was running in my network.

I've worked at a bank. It's a docker file. Running Hadoop. On top of a Linux Container. Running in Rhel 5. On top of a vmware hypervisor.

ownagefool• 7 years ago

In 2015 I was running docker containers for a Gov organisation, where:

- docker containers were built from source

- Dockerfile published with the code

- Built in a new CI environment

- Pushed, Pulled and deployed from the sha

- Collecting network traffic, undertaking protective monitoring, that looks for those backdoors.

Just because you can pull arbitrary bullshit doesn't mean you have to.

Though for the record, the same sysadmins that whine about newer tools are generally the same ones that implicitly trust their older toolsets. Just because you can compile it, doesn't make it secure, so you need to be running monitoring solutions and hedging your bets no matter the tech.

arberavdullahu• 7 years ago

I don't think you should put security at any case in tradeoff, by your argument one can save passwords in plaintext because its easy and comfortable, but its just not acceptable and this would effect not only your app but every app that your users use.

ThreeFx2• 7 years ago

I see the value when running e.g. home instrastructure on some old laptops or other non-critical stuff. Most people have NAT at home and thus whatever services they pull and listen in their local network aren't exposed to the outside world, and they can enjoy mopidy/NextCloud/whatever without going through a lot of hassle setting it up.

Of course once we come to enterprise environments the opaque build/deployment process is atrocious.

contravariant• 7 years ago

If you don't make a tradeoff you end up with passwords to long to remember and security to cumbersome to not circumvent.

Security has a cost, as does the lack of it, hence the trade-off.

x3n0ph3n3• 7 years ago

The trade-off is always between security and accessibility. If you want a completely secure computer, bury it at the bottom of a mineshift and pour a few yards of concrete over it. No one is breaking into that anytime soon!

geocar• 7 years ago

This is not "the tradeoff", but too easily a straw man because bcrypt (et al) cost so very little as to effectively cost nothing at all.

Think about your server consuming resources: It needs a [machine] password stored someplace in order to authenticate itself to those resources. If it stores this encrypted, then an operator must be present to decrypt it when (re)starting the server. If this is stored unencrypted, then it is not and the server can be started unattended (or autoscaled or whatever). That is a security tradeoff because there are both real benefits and risks gained and lost with each approach.

If docker was ever a tradeoff, it was between taking the time to have a thoughtful architecture versus trying to get acquired, and I think we know which way they went.

majewsky2• 7 years ago

Discussed at the time (443 comments): https://news.ycombinator.com/item?id=9419188

And again in 2018 (426 comments): https://news.ycombinator.com/item?id=17083436

mike_hock• 7 years ago

But unfortunately, it's relevant to this day.

majewsky2• 7 years ago

More than ever.

upstandingdude• 7 years ago

ha +1 for the build tool mess. its like everyone goes "duuude this stuff is so complex lets build something on top to abstract it all away" and you end up with another level of complex stuff that builds the stuff that builds the that wraps the stuff that you originally wanted to use...

pram2• 7 years ago

Terrible article IMO. Hadoop is an awful mess, but it has nothing to do with Docker, which is simplistic in comparison.

"Ever tried to security update a container?"

Yes, I have! In fact you can maintain patch compliance in a container pretty much the same way you'd maintain a VM or bare metal Linux installation!

black_puppydog• 7 years ago

Hadoop is an awful mess, but it has nothing to do with Docker, which is simplistic in comparison.

The fact that they can get away with a build system like this is very much due to docker (and curl | sudo bash) allow people to not feel the pain this mess causes, at least not right away.

Yes, I have! In fact you can maintain patch compliance in a container pretty much the same way you'd maintain a VM or bare metal Linux installation!

But then you lose the declarative/immutable nature of docker, no? Except if you mean every time there's a security update, you rebuild your containers?

onion2k2• 7 years ago

Except if you mean every time there's a security update, you rebuild your containers?

That's exactly what you should be doing, at least if you're relatively small scale. Google probably takes a different approach but most of us aren't working at that scale.

pram2• 7 years ago

First: No one installs Hadoop from scratch, and Hadoop isn't built with Docker. Companies use Ambari or other distros like EMR.

Second: Yes, generally you can use a build system like Jenkins, and a registry like Artifactory to automate the process. The docker image is updated, and then you can push it out with your orchestration in whatever method you choose. It's not an obscure or difficult thing to manage..

adrianN2• 7 years ago

It's pretty hard to do truly declarative/immutable things with docker because the typical dockerfile starts with "apt-get update && apt-get install ...". For that use case I think nix is much better.

buzzkillington6• 7 years ago

Unfortunately, the build process for these packages is currently of a disastrous quality, and should only be attempted within disposable virtual machines, as it requires root permissions and will install non-packaged software.

https://wiki.debian.org/Hadoop

If Debian, a distro on the lower side of dramatic, calls it disastrous, it's pretty bad.

regularfry8• 7 years ago

Debian is (mostly) low on the drama, but high on the expected package quality. It's not surprising they'd have problems here.

tomtomtom777• 7 years ago

The article focuses on Hadoop, but the generalization to containers is spot on.

Containers have become the first stop method to hide overly complicated build processes. When you've noticed your wiring has become a complete mess of knots, just put it in a box so no one will trip.

It seems to me that containers are useful for deploying final setups, but should be a no go for packaging tools.

apexalpha• 7 years ago

I would see it more as to dividing your wiring into segments with breakers in between so that one rogue device doesn't bring down the whole network.

Seems a good thing, to me at least.

indigochill• 7 years ago

When you've noticed your wiring has become a complete mess of knots, just put it in a box so no one will trip.

On the other side of the coin, putting stuff in boxes is the essence of architecture. CPU opcodes? Boxes. Compiling readable code into bytecode? Boxes. Objects? Boxes. Functions? Boxes.

Sure, it can be abused. There might be spaghetti in those boxes. The spaghetti should be chopped into digestible chunks (smaller boxes!). But putting it in a box makes that an implementation detail. Putting stuff in boxes is still progress.

hvidgaard3• 7 years ago

I don't think it's fair to blame containers for that. Just because someone programs a shitty product in Visual Studio, does not mean it's a problem with Visual Studio.

What containers does beautifully is dependency containment. I make something and put it inside a container, with the exact dependency it needs and it'll work alongside other containers with a program that might have a different incompatible dependency.

What we're seeing is not so much that developers suddenly decides to throw best practice into the wind. They just do not have to put the same amount of effort into fixing the broken process exactly because we have dependency containment. I believe we would see the same mess without containers, and the process change that would fix one would fix the other.

darkwater• 7 years ago

Containers are great, for example, to internally deploy some internet-facing web component your company wrote but they are not that great as a mean of generic software distribution. But things get complicated when to build the first type of containers you usually need software from the second type. In my case I just mirror good/proven docker images in my internal repo(s) and rebuild images from scratch in the most reproducible manner I can find when the software is badly packaged.

peterwwillis4• 7 years ago

Containers are fantastic for packaging tools. What problems do you see with it?

the_duke• 7 years ago

You missed the bigger point.

Just look at the popular images on Docker hub.

A lot of them involve messy build steps, including downloading binaries or source tarballs without verification.

It's often hard to know which dependencies a Docker image has, and therefore hard to track vulnerabilities and redeploy fixed images.

A lot of docker containers end up either running for a long time, or get rebuilt and redeployed often, but have pinned versions of base images or dependencies which still leaves vulnerabilities in place.

This is compensated a bit by the fact that Docker does provide somewhat decent isolation, and most containers are run in a cloud environment, behind load balancers and with decent security constraints. But there are many pitfalls in the whole process.

Containers provide a lot of advantages, and are certainly where things will continue to go.

But we need to figure out the tooling and ecosystem story to build, verify and update container deployments securely.

Annatar2• 7 years ago

"But we need to figure out the tooling and ecosystem story to build, verify and update container deployments securely."

Already done: Solaris zones. Available in the SmartOS distribution near you. Combine with OS packaging, imgadm and vmadm commands for maximum impact.

oauea2• 7 years ago

Just build your own. It's not that hard.

Before containers we would script the install process using Ansible. With containers we script the install process using a Dockerfile. If you know how to do one, you know how to do the other. Just don't be lazy and use unofficial images created by people you don't know or trust. Just like you wouldn't pull any random role from ansible-galaxy. Or pull a random guy off the street and ask them to provision your infrastructure for you.

Then all you have to do is just rebuild your images every week and you're set. All we had to do for this was have our Jenkins build pipeline run weekly & have it pass `--pull --no-cache` to `docker build`.

GordonS6• 7 years ago

A lot of them involve messy build steps, including downloading binaries or source tarballs without verification

Do they though?

Whenever I write Dockerfiles that depend on external downloads, I always check the hash matches one baked into the Dockerfile itself, and I've always seen others doing the same.

gchamonlive• 7 years ago

I was confused when I read the article. I kept thinking it was actually a problem with hadoop and not docker, since docker is nothing but an abstraction on top of the Linux kernel.

I remember trying to produce a viable alpine based docker image for a flask backend service we had thar produced pdf using latex. Dependencies were really hard to pin down, had to resort to open issues on github, but utimately all packages were on the distribution, so it is not a problem with the tech itself, but the archaic way Java lets people to produce applications

HorstG4• 7 years ago

Agreed, up to a point:

Ever tried to security-update a vendor-provided container which you should never touch because of the vendor's support and warranty conditions? And even if allowed to touch it, the mess that it usually is: app running as root in the container, stuff chmodded a+rwx -R, weird base distro, build script is just a wget, cp -R or "this binary magically appears from somewhere". With a helping of " only works if the directory structure and environment look like the devs machine".

Containers are the ultimate expression of "it compiles, let's ship it" laziness

tetha• 7 years ago

But that's vendors, not containers.

You can say exactly the same about vendors on a VM - applications running as root, bad passwords set, reboots impossible, patching impossible, backups... Or a virtual appliance. Or a third party application you installed and cannot upgrade due to a weird dependency on libc, or java, or a specific windows versions.

Vendors like that are always a mess, no matter the technology. Containers can turn into an equal mess, yes, but that's up to the build chains and ecosystem built around the containers.

fargle2• 7 years ago

I think the article is right on and makes a good point. The title does imply that containers are somehow bad for system administration. But the point is not that a container with a nicely installed, maintained, sensible image and applications pre-loaded on it would be a problem for system administration. Nor would it be anything but trivial to update patches in this situation.

The real point the author is making is that many times containers are misused to paper over problems like lack of reproducibility, dependency hell, etc. They aren't reproducible, certainly not by you. If they were you'd not need the container in the first place, or you could build your own easily. So for this kind of misuse, the analogy to Tucows Windows 95 shareware is pretty close.

Hadoop and Docker are just the examples to make the point. And they are good enough ones, because I could understand exactly what the author meant. There are certainly hundreds of other examples.

Russtopia2• 7 years ago

I suspect Hadoop was just used as an example. For another, I would suggest Concourse (build/CI tool). When I tried it, it was docker images for the default install, vague instructions, and a dev forum which seemed rather hostile to anyone asking for support to build or configure outside of containers.

Might have changed since then, but a big turn off.

Yuval_Halevi• 7 years ago

I saw this comment now from 4 years ago on reddit:

Software packaging and building seems to be becoming more complicated and more disconnected-- particularly as more specific tools continue to be developed. It seems every little corner has their own dependency management and build management solution.

The Docker comments seem a little backhanded. If one looks at a Dockerfile, there's not much to complain about. It downloads a key, it adds a repository, installs a package, and some very simple docker-specific tweaks.

If you distrust Docker's signed image, building your own is as simple as doing a git clone ..; cd docker-nginx; docker build. Docker encourages disposable containers, separating data, and making your own images. There's still a lot of work for Docker to do regarding signed images, but I'd argue running isolated images with documented changesets and simple build files is far different from blindly running 'curl | sudo bash'

https://old.reddit.com/r/programming/comments/33ktc9/the_sad...

jve• 7 years ago

This all boils down to trust.

`curl | sudo bash` is no different than .\install.exe. The question is about trusting the SOURCE and trusting the DISTRIBUTION channel (that HTTP download from scala-lang.org violates this).

Where did you get it? from https://microsoft.com/... or from https://micro.soft.com/...? Whom you trust more? The same with pre-built VM image or whatever... do you trust the party that made this image/container available?

LockAndLol2• 7 years ago

`curl | sudo bash` should never have become a thing in the first place. To me, it's the most "WTF?" mode of installing anything. Especially when the source is HTTP.

In what world is that secure?

Pity Qubes is so heavy, otherwise I'd be using it

eejjjj82• 7 years ago

would you feel better if your Makefile looks like this:

    install:
        [ $(id -u) -ne 0 ] && echo 'please run sudo make install' || curl https//my.thing | /bin/sh

it goes back to what jve said - it's a matter of trust. how often do you blindly run 'sudo make install' w/o reading the entirety of the build script? all the time I bet - it's because you trust the source

eeZah7Ux3• 7 years ago

On the contrary, it's much worse. There's no way for 3rd parties to collect, test and redistribute installers in an organized way.

peterwwillis4• 7 years ago

Trust, but verify.

`curl | sudo bash` is no different than .\install.exe

.\install.exe runs a binary you already have. If it's windows, you can see if it's a signed binary, and it will prompt you for admin access. If it's not signed, you can compare a signature of it to one you got from a secure source.

`curl | sudo bash` downloads the file from a remote source. The URL isn't listed here; was it http? In that case, now a MITM can modify the file before you run it. If it's HTTPS, that's better, but you still haven't compared it to a signature of the original file, meaning it could have been modified on the download site. And hopefully you're one of the seemingly few users that turn on password prompting for sudo access (and hopefully running it in a new terminal, to avoid an earlier sudo's session...)

So they are different. And containers and VMs are the same: if you can compare checksums before running them, or build them from source, you have confidence that they came from a source you trust. Dockerfiles in particular make rebuilding even more trivial than with source Linux packages, so there's not much reason to fret.

cuillevel33• 7 years ago

It also no different from using 'make install' or installing packages from a community repository, like Ubuntu Universe, homebrew, Arch AUR and so on.

There is an absurd amount of trust put in distributions and their package managers. Unless people are paying for it, or use very old software, they probably don't get the level of security they expect.

onion2k2• 7 years ago

Ever tried to security update a container?

The whole point of using a container is that you can destroy it and build a new one easily. The new one should be built using up-to-date packages with security patches applied (and tested, obvs). Using the 'pets versus cattle'[1] analogy, patching a container feels like you're treating it like a pet. You should just kill it and get a new one instead.

[1] https://thenewstack.io/how-to-treat-your-kubernetes-clusters...

dsr_4• 7 years ago

The whole point of having a distro with a good reputation is that you can incrementally outsource your trust to them, one package at a time. OpenSSL needs a fix? One package update per machine, and if you have lots of machines, you update that.

With VMs, you can use the same tools you use with any other machine.

With containers, you are responsible for figuring out which ones need which packages and how to re-build them. Your method is likely to be idiosyncratic, so you not only need to be your own security team, you need to be your own packaging maintainer and infrastructure maintainer.

The two extreme versions of this are the neglect model, in which you just don't care about anything, and the bleeding-edge model, where you automatically build with the latest versions of everything pulled from their origins. Both ways end up with unexpected security disasters.

regularfry8• 7 years ago

The "one package at a time" model assumes you care about per-instance uptime and bandwidth costs. The "reset the world" model assumes rebuilds and reboots are cheap enough not to have to.

I don't think either is necessarily wrong; there's not a huge distance between `apt-get install unattended-upgrades` and a daily (weekly, whatever) container rebuild except that you need to cron the latter. Unless you're using a different OS between the two cases, they can use the same package database, the same tooling, and so on.

hvidgaard3• 7 years ago

If you can build on your distro of choice, you can rely on the same security audit and process. Keep track of the packages you link against and include in your image, and just rebuild your images when a new security update comes along. Then you replace the running one.

lukasLansky• 7 years ago

I don't understand why you think it's hard to tell your CI/CD pipeline to run itself on a schedule. I find this totally consistent with my tooling (I have CI/CD in place for many different reasons, this is one of them) and I don't know how this CI/CD cron makes me "packaging maintainer and infrastructure maintainer".

"With containers, you are responsible for figuring out which ones need which packages and how to re-build them."

Yeah, you need to know how to rebuild your world in ANY CASE. It's not an argument against containers that they require an approach that is the best practice.

pbalau• 7 years ago

The easiest analogy is that containers are like binaries. If you find a problem in nginx, you don't go on and patch /usr/bin/nginx, you uninstall that one and replace it with a binary that has the problem fixed.

m463• 7 years ago

It depends on where you're getting the new one from.

You could be getting mad cows every time.

I think it would be nicer if you could build containers from scratch that didn't tie into a cloud based website or someone's business model.

Sort of like a Dockerfile, but for the prerequisites. Maybe go even deeper to gentoo level.

jwr3• 7 years ago

I recently had a similar discussion with people using npm for building a CSS framework library. I tried to explain the concept of getting a pre-downloaded tarball and using "make" (or similar) to produce target artifacts from source files in a deterministic, repeatable and reliable manner, without relying on any third-party servers being available and without pulling in dependencies that might have changed.

It seems the concept was entirely alien to programmers younger than me.

Gravityloss• 7 years ago

It can be sold as a new invention in a few years.

mateuszf• 7 years ago

It can be done using package managers too, for example using Nix. The problem here is not related to distribution method but rather dependency hell in javascriptland.

weberc23• 7 years ago

I would say the main issue is that Nix is super difficult and requires you to intimately know your entire dependency tree down into Linux particularities. JavaScript makes matters worse, but even packaging a nontrivial Python package pulled directly from Pypi is often difficult.

I want to like Nix, but it is far too pedantic to be practical. And this isn’t even mentioning the usability issues.

jwr3• 7 years ago

To follow up: I ended up building my own Docker images which contained the build tools (also known as the npm burning dumpster tire fire with extras thrown in), source code and all dependencies. This allows me to get reproducible builds, the modern-day analog of "./configure; make".

maple31423• 7 years ago

I sometimes have problem doing `./configure && make` because dependencies varies across different distro, which have some subtle differences that make it not easy as expected.

tjoff• 7 years ago

But it is consistent, which is very important.

regularfry8• 7 years ago

If you can do `./configure && make`, you can also do a `checkinstall` on debianesque systems, which gives you a package so you don't have to do it again.

jwr3• 7 years ago

True. But at least it doesn't auto-download half the internet (with new and improved vulnerabilities of the day) and dump it all in your "project directory" (whatever that might be, I'm still fuzzy on the exact definition).

Tade03• 7 years ago

Truth be told the whole front-end ecosystem relies on those servers being available, so in their absence any development everywhere would grind to a halt - as it almost did during the left-pad scandal.

While I agree that this is not exactly the sanest approach, within the ecosystem there's no incentive to work differently.

Also, like someone else mentioned - dependency hell only got worse over time - setting up a new project you're likely to have several versions of the same library in your node_modules.

tomcooks• 7 years ago

Truth be told the whole front-end ecosystem relies on those servers being available, so in their absence any development everywhere would grind to a halt

There's a world out there of LAMP/LEMP stack web developers that wholeheartedly disagree with this perception.

weberc23• 7 years ago

There’s a pretty straightforward solution to this problem which is to run your own NPM server even if it’s only a caching proxy. For some orgs, the uptime provided by the third party servers is sufficient.

pwagland• 7 years ago

Don't turn it into an ageist thing, unless you are particularly old, there will be plenty of older developers that also don't understand this.

It largely comes from understanding _why_ reproducibility is a good thing, and there are _lots_ of open source maintainers that understand this that are likely "younger than you". The vast majority of developers though focus on other things, that SQL injection attacks are still a thing indicate that we as a community have a _long_ way to go on the security front.

was_boring• 7 years ago

It is way bigger then security, and hits upon quality of life and reliability.

Without a deterministic build I don't know what I check in actually works across environment, or if the deployment artifact works.

radicalbyte• 7 years ago

I like using Artifactory for this, you keep a local copy of all NPM / nuget / etc dependencies hosted next to your build infrastructure.

What shocked me most about NPM is that it used to have absolutely 0 verification built in, yet it was being heavily promoted by very well known, educated and experienced tech celebs. All at a time when it was basically a hobby toy.

etimberg• 7 years ago

Last time I tried to use Artifactory for it's caching it barely worked. Would randomly 404 when a package was not in it's cache yet and you'd end up retrying a CI build for a while until everything was there.

danmckinney• 7 years ago

I don't think it's specific to just younger devs, but there absolutely is an argument for having your own artifact repository that you can control.

JustSomeNobody• 7 years ago

I don’t think it’s age so much as it’s “well, that’s just what everyone does now”.

If you want to work in this industry you have to follow the herd; for better or worse.

I certainly don’t agree with it, but people have bills to pay.

jlgaddis• 7 years ago

Your comment reminds me of the "Lemmings walking off a cliff" animated GIF that was popular once animated GIFs became a thing on the web.

fargle2• 7 years ago

What if "well, that's just what everyone does" is just wrong? I trimmed the word "now" from the quote because lazy sloppy short-term thinking is nothing new. Good configuration management takes a lot of work and pays great returns, only not in the very short-term. It's not surprising that inexperienced devs will fall into this trap.

And just why do you "Have to follow the herd"? Pretty sure just about everything new and creative (good or bad) was a result of not following the herd. I recommend going where you need to go. If a herd starts following you, that's great. At least they'll be slipping and sliding on the shit you leave behind, not the other way round.

kbr2000• 7 years ago

Managers and C-levels surely have appetite for this. In the end it just degrades mostly to trial-and-error, which I'm sure is no problem because the people have to do the real work get sick of it and get replaced (sometimes voluntarily, sometimes with force), or play the game nice out of fear losing their job (thus feeding the beast).

It's not very surprising since 'products' nowadays are more like 'services' instead, and offer some kind of encapsulation: people are not interested much in how something ticks behind the scenes, and so it can drive down the behind-the-scene quality. They also are conditioned to accept lousy excuses (including none) for outages and breakdowns, because they got sold 'magic', and boy that is magical..

You'll might have to work in obscurity to keep up operations quality, which is in turn a driver for your own demise. Game over.

I'd like to suggest to you the following reading material:

- Bullshit Jobs (David Graeber, 2018)

- The Dilbert Principle (Scott Adams, 2000)

- Future Shock (Alvin Toffler, 1970)

y0y• 7 years ago

I know you're taking some flack for making it about age, but I do think there's some merit there. I'm 34. I feel like I'm in the middle in terms of software developer / sysadmin age. Lots of bright minds came before me and there are lots of bright minds out there right now in their early 20s.

A key difference is I can remember a time when network connectivity was flaky. When it was hardly a given. Even when it was available, the download times alone could often constitute a large part of the overall build time.

I think that we have all become complacent with regard to internet connectivity and service availability, but I think the younger you are the more complacent you are likely to be. If github.com goes down entirely, let's be honest - there are a lot of Jenkins builds that are going to be in the red.

IAmEveryone• 7 years ago

People adapt to the situation they experience. Github (and other repositories) tend to be stable, so they are used. Network speeds have increased, so we use the network and expend less efforts on local caching etc.

There's nothing "complacent" about this: previous generations also relied on infrastructure and didn't plan for prolonged power outages or had backup ham radio network links for when AOL was down.

People using nom install instead of custom makefiles aren't ignorant or stupid, they have found better ways to achieve their needs. And if 10 years on the job don't create the need to learn some skill, there is no reason to invest time into it. And I have complete confidence that people would be able to come with some workable solutions rather quickly if the githubcopalypse ever happens.

There is some cultural component at play among the "luddites" here as well, maybe comparable to preppers? It feels like planning for really exciting emergencies when one's skills that have been derided for so long are suddenly needed and safe the day. In this analogy, I guess Makefiles are the equivalent of very masculine hunting and zombie-defending skills.

phaedrus• 7 years ago

I'm 36 and recently was assigned to mentor a new employee in his 20s. We had a moment of miscommunication when I asked him to use git to clone a local repository. He was confused when he couldn't find it on github.com (what I'd sent him was a path to our private network share). I had to explain that, yes you can use the github.com client if you want, but "git" is different from "github". I honestly couldn't tell if he understood the difference.

Now I'm losing sleep worrying if there's any way he can accidentally add a github.com remote to our private repo and push to it. I would be blamed, and I'd have to explain to managers older in turn than me what both git and github are.

saagarjha2• 7 years ago

A key difference is I can remember a time when network connectivity was flaky.

I’m young enough for this to not be something I have experienced, but I don’t have to have to understand that depending on things you don’t have control over can be a bad idea.

pojntfx2• 7 years ago

18yo JS/Ruby/Go dev here. `make` is certainly known to us; it's just that platform-specific build systems (`mage` for Go, `rake` for Ruby, `gulp` for JS etc.) are simply more sane to set up per languages than using `make` for all languages and allows for efficient code reuse.

celticmusic4• 7 years ago

You have make call the platform specific build systems.

AndyMcConachie2• 7 years ago

s/younger/less experiened/

mr__y• 7 years ago

building a CSS framework library

I think that might have been the root cause of the problem.

colinjfw• 7 years ago

Not alien to me, likely younger.

ekidd• 7 years ago

without relying on any third-party servers being available and without pulling in dependencies that might have changed.

There are two different issues here:

1. Not pulling in changed dependencies. This is what "lock files" are for: To limit builds to known version of every dependency. npm was terrible about this for a long time. Most other language package managers are better.

2. Not relying on third party servers to be available. Personally, I've mostly worked with Ruby's bundler and Rust's cargo, and in over 10 years, I've lost maybe two days of work because of package server outages. That's less than I've lost due to S3 outages, less than I've lost due to broken backup systems, and less than I've lost due to complex RAID failures. For the clients and employers in question, this was acceptable downtime.

In the rare cases where one day of noticeable build downtime every 5 years is unacceptable, then it's usually possible to just "vendor" the dependencies, usually by running a one-line command.

For many small to midsize businesses (and many growing startups), a small risk of brief outages is an acceptable engineering trade-off.

hn_throwaway_99• 7 years ago

npm was terrible about this for a long time.

That's putting it lightly. To me "terrible" would mean they just didn't support it, "batshit insane" means they supported lock files but ignored them every time you ran npm install.

My favorite comment from this stackoverflow post: https://stackoverflow.com/questions/45022048/why-does-npm-in...

"Why would you expect something called package lock to lock the packages? Package lock is analogous to how, when you put any key into a door lock, the lock reshapes itself to match whatever key was put in, then opens the door. Now if you'll excuse me, I'm late for a tea party with a rabbit."

dirkf• 7 years ago

At work we also cache all 3rd party dependencies locally. Not because the 3rd party servers might be down but to be sure that one/two/... years from now we can still recreate the same software we delivered at that time to a customer. Our build machines typically didn't even have internet access. If a dependency was not available offline (due to developer error or whatever) we would know quickly.

In certain industry branches this is even mandatory if you want to be seriously considered as a supplier. If a build tool does not support reproducible builds in such a way (both fixing dependency version and getting it from a cache somehow) or makes it difficult then it is considered as a hobby toy that has no place in the workplace.

Even for small businesses I would advocate to take this seriously from the beginning. It's not that hard and will save you headaches later on when suddenly reproducible builds become important.

forgottenpass• 7 years ago

I've lost maybe two days of work because of package server outages [...] For the clients and employers in question, this was acceptable downtime.

Even if the downtime is fine, I don't find it goes over too well if framed as "we rely on a 3rd party service, that we have no contract with nor any guarantees of reliability or product longevity."

mixmastamyk2• 7 years ago

There are other advantages such as build speed. Having local copies is simply faster.

hinkley2• 7 years ago

npm has only gotten worse about lockfiles over time, not better. I wonder out loud some times if they know what the word 'lock' means.

If yarn would fix a particular bug that blocks our workflow I'd be burning political capital at work to get us off of npm as fast as humanly possible.

As to proxies, we have something misconfigured with ours, such that occasionally it gets latest of half of the React or Babel ecosystem and latest-1 of the other half, resulting in dependencies that can't be resolved for a few hours when they increment a minor version somewhere.

winrid2• 7 years ago

Easiest thing to do to solve this would be to have something like artifactory or even squid in front of npm and use exact version numbers. Front end devs are going to use NPM. Solve the other problems separately like you would with any other build system.

eckza• 7 years ago

Sure. And then when something unexpected happens, you have two problems.

asdfman123• 7 years ago

You'll get blank stares and resistance whenever you throw a bunch of jargon at people that they don't understand, though. It makes people feel insecure.

You can't expect new programmers who sound like primarily front-end devs to learn all the hot new stuff and all the old stuff, too.

If you want someone with that kind of knowledge, you need to hire a senior dev with 20 years of experience.

1337shadow• 7 years ago

The holy grail of sysadmin has always been deployment of immutable images, instead of host system pollution, source code mutation during deployment ...

It was doable with things like Packer, KVM or OpenStack, now it's become easier with containers. You can still orchestrate your stuff with ansible or bash scripts.

I most certainly remember how I was doing isolated deployments 10 years ago: https://github.com/jpic/bashworks/tree/master/vps HINT: it's much cleaner and much more automated with containers: with GitLab-CI it's trivial to host a private container bulding infrastructure with a registry and other automation ...

buzzkillington6• 7 years ago

Looked at the Debian wiki the OP linked to:

Debian currently does not include Hadoop packages. There are a number of reasons for this; in particular the Hadoop build process will load various dependencies via Maven instead of using distribution-supplied packages. Java projects like this are unfortunately not easy to package because of interdependencies; and unfortunately the Hadoop stack is full of odd dependencies (including Apache Forrest, which for a long time only worked with Java = 1.5).
If you want to build Debian packages, the most complete efforts can be found at the Apache Bigtop project http://bigtop.apache.org/ . Unfortunately, the build process for these packages is currently of a disastrous quality, and should only be attempted within disposable virtual machines, as it requires root permissions and will install non-packaged software.
These will allow building Hadoop packages on a recent Debian system. However, the resulting packages do not live up to Debian quality standards. In particular, they include copies of .jar files from other packages, and will thus not benefit from security updates done to these packages.
If you are interested in getting Hadoop packages into Debian,

    Coordinate with the Java packaging team
    Coordinate with upstream Apache Bigtop

>to avoid duplicate efforts. Thank you.

https://wiki.debian.org/Hadoop

Jesus Christ.

alfozan• 7 years ago

Who still uses Hadoop anyway?

https://spark.apache.org/ https://www.iguazio.com/data-science-post-hadoop/

buzzkillington6• 7 years ago

Spark is worse because you need Scala as well as regular Java.

I've tried building it for my day job, I would rather have a colonoscopy without sedation.

It's more pleasant and dignified.

dig13• 7 years ago

Sell talk and buzzwords. Either author has no idea that Hadoop is ecosystem and Spark depends on it or deliberately mix Hadoop and Kubernetes, which aren't much related.

And good luck running Spark without Hadoop ;)

kylek• 7 years ago

Spark and dependencies are just as much of a tire fire and you’ll often want Hadoop as well.

mixmastamyk2• 7 years ago

Sounds like this is the root issue, and not significantly about package managers.

No one is willing to invest the time to untangle the build process and fix compat issues in the software. The project is slowly dying out.

0xFFFE• 7 years ago

I recently went as far as setting up Bigtop on a VM to build the Deb packages, I gave up after 2 days. I would rather suffer through 2 root canal procedures back to back without anesthesia.

mitchell_h• 7 years ago

On a bet a couple years ago someone gave me a week to compile and stand up most of the core hadoop services distributed by horton. 4 days in I lost my shit and desk flipped.

adamc• 7 years ago

That triggered memories... happier in Python.

_pmf_• 7 years ago

Give me rsync and a sysadmin any day over a circle jerking dev ops team.

Annatar2• 7 years ago

Replace rsync with OS packages and you got yourself a deal and a half.

ex_amazon_sde• 7 years ago

Amazon and some other FAANGs build and deploy OS packages, and by choice.

Sad to see how anything that is not container gets downvoted here.

_wldu• 7 years ago

Standard Makefiles are really under appreciated today. They are simple and can be used with most any language, version controlled and have been around for decades.

neuronic• 7 years ago

But it's not a product you can sell or artificially inflate its monetary value so out the window it goes.

adgasf3• 7 years ago

Makefiles are too low level to be convenient for anything other than simple projects on one platform.

gerbilly• 7 years ago

If the alternative you are hinting at is CMake, I can't really agree with you.

I tried it recently and was shocked at how awful the syntax and usability was.

It's begging for a tool to auto-detect your dependencies from source. I was trying to port an existing project to it and I gave up when I realised that I'd have to manually create about 30 CMakeList.txt files or whatever they call them.

Meanwhile during the same weekend, autoconf allowed me to build gcc from source with three commands. I know I know, I'm not supposed to like autoconf, and the orthodoxy is that it is garbage ...

It seems every generation ignores the previous generation's tools and invents worse ones to replace them.

If this continues we'll soon go back to banging rocks and sticks together.

outadoc• 7 years ago

They don't solve any dependency management problems.

dfox• 7 years ago

Most ports like systems actually use makefiles to solve cross-package dependency management. (Whether that is good approach and whether writing what is essentially package manager as hairy ball of makefiles is good idea are another questions)

GordonS6• 7 years ago

Agreed (although I so wish they weren't utterly dependant on tabs!)

I even sometimes use makefiles in conjunction with Docker, for example to automatically add the correct tags to a build - it's much easier to type `make build` that to remember a massive invocation for `docker build`.

int_19h3• 7 years ago

Makefiles are problematic on Windows, though. There's no Make out of the box, there's a bunch of incompatible makes that are subtly different in different ways, and then most Makefiles tend to assume sh and POSIX at some point.

t0astbread• 7 years ago

I'm looking at this from a relatively neutral newbie POV. I've tipped my toes in both administering machines the "classic" way and with tools like Docker and Docker Compose. To me both have their benefits and drawbacks but Docker looks a bit better overall. Here's why:

1) The idea of immutable systems rocks. It feels like the logical next step from containers.

2) This is maybe just because of me but I have the feeling services that can easily be (automatically) updated (like with Docker) are gonna be more up-to-date on average than when I have to build every new version myself. (Although that could be automated probably.)

dig13• 7 years ago

The same applies today for nodejs and go, but the same was happening with python, ruby and perl libraries in the past.

However, this only shows that standard distro packaging approach is broken or was broken all the time. First, if you want to distribute you application (or library) via distro packages, you'd either have to build them and publish on your site or have to go through long mailing process to satisfy distro specific packaging guidelines and hope that your package will end up in standard (or extended) distribution. Debian is also notorious for "reviewing" and modifying source code, without contacting authors (remember Debian SSH fiasco).

I had, not once, unpleasant experience with this process.

Today, things are much easier - pack your whatever and put on dockerhub, maven or github. No middleman and no headaches just to reach your users. Sure, there are drawbacks (like mentioned in article), but we'd have to leverage things like GPG and proper package signing. Now, that is totally different story, because cryptography, even the basic one, is PITA for most developers and users. You can't have both :D

lmm13• 7 years ago

You've got to look at this in a context where platform package managers like apt are simultaneously 1) platform-specific 2) jealous, insisting that every language has to conform to their way of doing things and 3) fundamentally not very good, having very limited ability to do things like install packages for a single user or install multiple versions of the same package. Platform package managers like maven have been far more successful because they provide a much better user experience, and frankly I suspect this is if anything because they're so thoroughly isolated from the platform package manager.

marmaduke2• 7 years ago

No I don't. I use Debian derivatives because I can apt get all my stuff without thinking hard, because those package maintainers have done the hard work.

lmm13• 7 years ago

Well, clearly not, since you can't install Hadoop. Before you blame that on Hadoop, remember that it only requires some really quite basic things from its package manager, which apt is nevertheless completely unable to do: use libraries in the normal recommended way, play nice with Java, work cross-platform.

Doxin2• 7 years ago

That's quite a negative view of the situation. The debian maintainers definitely don't insist that everything has to do things the apt-way. They only insist that packages for the official repos do so, which seems sensible enough to me. Everyone is still free to build their own deb packages or distribute their applications through install scripts or app images or any one of a hundred thousand different ways to distribute programs.

But if you want a package in the official repos you have to play by the rules they set, the primary one being that official packages ought to only depend on official packages. It'd simply be impossible to have any guarantees about the quality of a package otherwise.

lmm13• 7 years ago

It's not just a question of being in the official repos. Even if you're running a private repository, the deb tooling is not suitable for building many kinds of software: it's unwilling to play nice with "external" dependencies, and still single-platform and limited in its functionality.

roenxi• 7 years ago

Here here. And the only thing that makes Debian an interesting brand is the quality consistency of the official repository.

If someone doesn't like apt & the main repository they shouldn't use Debian because that is pretty much the only big thing separating Debian from the tens to hundreds of other distros out there. Take apt and the main repo out of Debian and it is basically Linux From Scratch with a bigger team of bugtesters. Distributions aren't 'jealous' about packet management, they are the packet management. It is right there in the name "Debian distribution". The Debian brand, and all linux distro brands, are deeply linked to how they execute packet management to distribute software.

eeZah7Ux3• 7 years ago

jealous, insisting that every language has to conform to their way of doing things

Not at all. The large majority of packaging systems have reasonable and similar requirements for upstreams, like not bundling dependencies or not hardcoding paths.

fundamentally not very good, having very limited ability to do things like install packages for a single user or install multiple versions of the same package

That's completely by design, and for good reasons.

spion2• 7 years ago

The design results with flawed user experience and the users will move past the restrictions one way or another. It's up to system package managers to realize why the old way of doing things is simply not good enough; otherwise, people will just move on.

I'm sad that this is not happening since in the process we're also losing the good bits of system package managers as per this article.

oblio3• 7 years ago

While you could argue about:

install packages for a single user

because of the security angle, Not allowing this:

install multiple versions of the same package

is completely indefensible.

Who are you, package manager, to decide that I shouldn't be able to use different versions of the same package? Do you know more about my context than I do?

zajio1am• 7 years ago

2) jealous, insisting that every language has to conform to their way of doing things

Well, programming languages clearly should not be in business of software distribution. That creates unnecessary tight coupling between language, build system and distribution, causes proliferation of language-specific package managers incompatible with a platform way of doing things.

moonbug2• 7 years ago

Not so obviously as you might think. But, if they are going to do it, they should clearly do it well

buzzkillington6• 7 years ago

Something Java and the Java ecosystem is terrible at. I've had project where we needed all of 7,8,9 and 10 [0] for the data stack: Hadoop, Scala, Spark, HDFS, zookeeper, pyspark. Good luck setting that up on a single machine, containers all the way. The java path nightmares I still have. I'm probably one of the few people in the word who know this much about the JVM without ever having written any Java.

[0] We should have also been following 11 but just didn't have the resources to even think about doing it.

rumanator9• 7 years ago

jealous, insisting that every language has to conform to their way of doing things

Let's ignore for a moment all your baseless assertions and focus on the following question: how do you ensure that a build is reproducible if you have no control on which required dependencies are downloaded by a build system which requires root access?

Is jealousy the only explanation that comes up to you?

Zyst2• 7 years ago

Oh, check out Nix https://nixos.org/nix/

I use it for most of my dependencies, and software in OS X/Windows WSL/Linux

It is a lovely environment to work with. Has completely changed how I feel about package management and the like.

I know you were asking figuratively but something does happen to slot in that space.

solarkraft• 7 years ago

Jealousy is the characterisation of exactly that behavior/aspiration to have everything managed through the OS's package manager. It's a nice effort, but for people not firmly within the ecosystem it's often a hard effort to justify to package something specifically for every distribution.

briffle• 7 years ago

you can use a tool like snap or appimage to pull down the newest full release of your software.. at least then, all the dependancies are 'bundled' into the image, and your not having people go randomly install crap from around the internet with no real way to update.

lmm13• 7 years ago

Reproducible builds were a non-goal for apt for most of its existence.

notacoward• 7 years ago

Language-specific package managers are also "jealous" as you put it. They insist that every platform conform to their way of doing things. They're also promiscuous, in the sense that most have really poor dependency management ("let's build the internet today just in case") and practically no security. The computing world would be a better place if the people forcing their language-specific idioms on the rest of the world had instead spent their time improving the platform package managers that already existed at the time.

smarx007• 7 years ago

So, what would 'pip install s3cmd' look on Windows or macOS?

lmm13• 7 years ago

They're also promiscuous, in the sense that most have really poor dependency management ("let's build the internet today just in case") and practically no security.

Not criticisms that apply to maven. Binary dependencies on released versions are the norm, and all packages in the central repository are signed with GPG, just as with apt.

The computing world would be a better place if the people forcing their language-specific idioms on the rest of the world had instead spent their time improving the platform package managers that already existed at the time.

If those platform package managers had been open to being improved, which I don't think they were (and conversely, deb/apt originated because Debian insisted on doing things their own way rather than improving RPM). Apt's closed-world assumptions seem like a policy decision. Apt not being cross-platform is definitely policy. And surely it's occurred to people in Debian that the ability to have per-user installs, and more importantly non-shared/non-overlapping installs of libraries that are depended on, would be useful; the insistence on single system-wide installs of libraries can only be a policy decision, and one that's turned out to result in a less usable system.

r3trohack3r• 7 years ago

This is a very negative view.

Once you get used to this method of packaging it’s actually really empowering for the end user. You can download the source for anything in the system, modify it, recompile, and install it all with a standard set of commands (no one-off custom build steps for every package).

I feel it’s the closest thing to the spirit of “free software” we have today: software designed to empower the end user to read, understand, and modify their system without a huge amount of obfuscation.

It increases the burden on the packagers to reduce the burden on the users. I would be interested to see how often these features actually get used though. I wish distributions would put this front and center.

https://wiki.debian.org/WhyDebian

bjornjaja• 7 years ago

In my opinion there’s simply too many dependencies to begin with. Some of the simplest things put into a GitHub repository to be “shared” with the world is great—as a gist file. Software needs to be a little more self-contained. Software reuse these days is honestly taking “not reinventing the wheel” to lazy extremes.

System package management systems do things just fine. Ask yourself, “when did I last properly package a Debian library? Do I even know how to do it properly?” Most people won’t be able to answer that because they didn’t.

But truth be told, having each programming language a unique package manager sucks. I’d rather have a consistent way of managing packages on a given system so that I can use various programming languages.

sectiondetail• 7 years ago

In my opinion there’s simply too many dependencies to begin with. Some of the simplest things put into a GitHub repository to be “shared” with the world is great—as a gist file. Software needs to be a little more self-contained. Software reuse these days is honestly taking “not reinventing the wheel” to lazy extremes.

All true. Did we learn nothing from the left-pad incident?

lmm13• 7 years ago

Software needs to be a little more self-contained. Software reuse these days is honestly taking “not reinventing the wheel” to lazy extremes.

The three cardinal virtues of a programmer are laziness, impatience and hubris. I've never seen a decent argument against using ever-smaller libraries (as long as our dependency management is good enough to track them); all of the usual arguments for reusing libraries still apply, even if we take them further than their originators imagined.

System package management systems do things just fine. Ask yourself, “when did I last properly package a Debian library? Do I even know how to do it properly?” Most people won’t be able to answer that because they didn’t.

Well, as we're seeing, Debian is struggling to keep up with packaging of everything that users want. I have actually created some (unofficial) debian packages; it's fine, but it's not a particularly inspiring experience either. Creating a maven/pip/... package is generally a nicer experience.

But truth be told, having each programming language a unique package manager sucks. I’d rather have a consistent way of managing packages on a given system so that I can use various programming languages.

In principle I agree. But I'd definitely need to be able to install independent copies of the same library, different packages for different users, and fundamentally just have a nice development experience when working on that package manager. And that's not something I see the sysadmin tradition being able to come up with.

skywhopper• 7 years ago

Maven is just as opinionated, jealous, and terrible, but its impact is restricted to JVM packages, mostly (hopefully) in the development stage, and so the consistency issues don't come up.

I ultimately agree that the system package managers do a poor job of wrapping language-specific package managers. But you lost me when you suggested Maven has a good user experience compared to most OS package managers. If that was a sly joke, then it was a good one.

hodgesrm• 7 years ago

Platform package managers like maven have been far more successful because they provide a much better user experience, and frankly I suspect this is if anything because they're so thoroughly isolated from the platform package manager.

This is just wrong. Package managements support simple models by design so you don't end up with a mess of conflicting and potentially insecure dependencies in deployment environments.

Use maven in your build pipeline, fine, but distribute a minimal package using tarball/RPM/APT/etc. so that only the minimal runtime dependencies install on the deployment target. In fact, I built my first RPM using the maven rpm plugin. It takes just minutes to set up and use.

mcguire2• 7 years ago

But platform managers like maven are 1) language-specific, 2) jealous, insisting that every OS and extra-linguistic dependency has to conform to their way of doing things, and 3) fundamentally, not very good, particularly with regard to multiple versions of the same package (technically, OSGi could do that...), repeatable builds, and the desire for do-what-I-mean behavior.

Platform managers have been successful because a) they have the same interface across operating systems, b) they're the only game in town if you want to work in that language, and c) everybody loves to re-write existing software in their favorite new language.

lmm13• 7 years ago

2) jealous, insisting that every OS and extra-linguistic dependency has to conform to their way of doing things

Not really. Language-specific package managers are usually capable of integrating with "system" dependencies from outside the language (certainly maven has decent support for this) - they kind of have to be after all.

3) fundamentally, not very good, particularly with regard to multiple versions of the same package (technically, OSGi could do that...), repeatable builds, and the desire for do-what-I-mean behavior.

How so?

Havoc• 7 years ago

And it seems to be getting worse lately with everyone talking about their cloud-y build "pipeline". Which invariably seems to involve stringing together a bunch of different service offerings from startups that might not be here tomorrow.

It's hard to imagine something more fragile

de_watcher• 7 years ago

http://kmkeen.com/maintainers-matter/

AndyMcConachie2• 7 years ago

Things will break and I will need to be able to debug them.

New versions will be released and I will need to be able to upgrade to them.

Bugs will be found and I may need to know if I or my users are experiencing them.

Security holes will be found and I will need to either upgrade, or mitigate them in some other fashion.

All of the above realities mean I want well defined and understandable software. The less well defined and understandable a deployment is, the less I am interested in deploying it. Complexity is a cost, and this cost cannot be waived by simply chucking everying into a container.

Simplicity means cheap. It means time not wasted and it means fewer headaches.

rbc• 7 years ago

The only part of this article that I really disagree with is the title. The main issue here is release engineering, or rather, the lack of it. Software development and system administration have well developed traditions, but release engineering doesn't seem to get much attention in comparison. Given that supply chain attacks are now all the rage in the black hat community, maybe release engineering will start to get the attention that it deserves.

jillesvangurp• 7 years ago

The maven/gradle ecosystem actually does use signatures. That's one of the things they got right. Maven central has no unsigned artifacts. Also, the signatures are actually checked on download. Other maven servers may be a bit more sloppy of course. In any case, if you are running that on a production system, you are doing it wrong. You run that as part of CI. The output of CI is containers which you start on a production machine.

The point about containers is that they should be immutable. Running chef/puppet in kubernetes is not a thing (I hope, probably somebody went there). Updating a container means replacing it with a new one, that hopefully you or somebody else built in a responsible way.

That last part is indeed a problem that we have moved instead of solved. 20 year ago, people were just sticking whatever they downloaded on hardware they bought at the store and banged on it until it worked (very literally sometimes). So, I guess this is progress but indeed hardly ideal. I remember using puppet. Can't say that that is with any level of fondness. I vastly prefer Dockerfile and having CI systems produce containers from those.

Installing things on a filesystem is no longer that common at deploy time unless that filesystem is that of the container you are building using a Dockerfile or you are self hosting kubernetes (aka. reinventing a lot of wheels at great cost). Puppet/chef etc. are still of use if you are doing that but otherwise it has a limited role in IAAS type architectures. The closest thing is perhaps pre-baking AMI images using packer and some tool like ansible, which is nice if you want to avoid having a lot of startup overhead.

Hadoop is complicated, which is why companies exist that host that for you or will help you hosting it on premise in a responsible way. If you want to DYI, you indeed have to do a lot of things and do them properly. Kubernetes has moved that space forward in recent years; so it's easier but this is not for everyone. If on the other hand you are messing with puppet to get this stuff done, maybe reflect on the wisdom of not standing on the shoulders of giants rather than blaming the internet for your self inflicted pain.

axegon• 7 years ago

Talking from a developer's point of view, I've thought about this for a while and I think it has something to do with what I refer to as "anti-imposter syndrome" - the notion that anyone can make software. You see, over the past 10 years or so, self-proclaimed "educational" services/websites/institutions have been shoving down everyone's throat the idea that anyone will be able to create the next FAANG from scratch after just 3 months of training. Which is the same as claiming that I can become an F1 driver in 3 months. Where do I sign up?!?!?! The thing is, F1 results are a lot more visible since you have a point of reference - the top dogs. Chances of scoring a fastest lap or coming even close to them - little to none. Software - not so much - you have a shiny interface and what it does underneath is always a mystery to the end user. And people who have gone through those magical 3 months of training are often lead to believe that the way they are doing things is what everyone is doing and that is how it should be done. Often people who have signed up for those courses are people who have just a smudge above average technical knowledge - they have no idea how OS'es work, what should be considered safe or even why. Don't hate me for saying it, but essentially Windows users. And this is the software they end up building and distributing. 10 years down the line, thousands have picked up random scraps of knowledge from here and there and tried to mash something together. Don't get me wrong, I think technologies like Docker are astonishing and an incredible tools in the hands of people with knowledge and experience. However way too many people have picked up some scraps from there and created the unholy mess the author is talking about.

cuillevel33• 7 years ago

I’m not complaining about old-school sysadmins. They know how to keep systems running, manage update and upgrade paths.

Sadly these ways of olde didn't scale. They could only do it for a handful of servers, change request took months and the systems were not to be 'touched'.

And since nobody is still able to compile things from scratch, everybody just downloads precompiled binaries from random websites. Often without any authentication or signature.

Right, that would have never happened back then, when packages were not signed and freshmeat.net was still a thing. For some reason compiling C code from some website (`wget;./configure;make;make install`) seems to be more secure than `go install`, maybe because nobody understands automake anymore.

zeveb• 7 years ago

They could only do it for a handful of servers, change request took months and the systems were not to be 'touched'.

Those old ways of doing things were indeed slower than they needed to be in many cases, and I'm glad that we sped them up. I don't believe that we needed to throw the baby away with the bathwater though.

The problem now is that we build systems which are unmanageable and unmanaged. 'Throw it in a container and let it run' is not scalable either: it scales neither security not maintainability. Instead of scaling, it just punts.

Enabling one to do something one shouldn't do (i.e., deploy insecure software into production) is not a virtue.

CathedralBorrow2• 7 years ago

The problem now is that we build systems which are unmanageable and unmanaged

Are all the systems out there in production actually unmanageable and unmanaged?

cuillevel33• 7 years ago

After a few decades it's no longer a baby ;)

Software is becoming more complex and moving faster than ever. 'Controlling your dependencies' was always an illusion and a trade-off at best. Often a trade-off against security, inheriting the folder of JAR files from your predecessor and such.

I think of containers and all these new tools as designs which are supposed to help us manage existing complexity. They don't create it, it's already there in the requirements and real-world deployments. The friction we feel, is that some of those tools are not very good (yet), but maybe better than the previous generation (sometimes). On the negative side, there is a lot of nostalgia ("ah, do you remember bare metal...") and unwillingness to change and learn.

z3t4• 7 years ago

So you dont want to run (insert software here, like your IDE) in the cloud, and you dont want to run it yourself because you dont trust it (its open source). Then how do you want it delivered? Snaps? Docker? KVM image?

disintegore• 7 years ago

In an InstallShield package, with a 7-step license activation process.

Let's face it, security is an asymptote. I just want someone I can call and scream at when we eventually get pwned and I'm ready to pay for that privilege.

znpy• 7 years ago

https://xkcd.com/1988/

turk73• 7 years ago

Before containers, I had sysadmins who ran our AIX cluster. Three nodes, not a single one configured like the other. There was zero consistency where it counted and these jokers would go on a terminal and manually configure Websphere and other components and screw it up constantly. We even tried to give them a terminal tool where you could type in one window and it would mirror on other boxes, but they refused to use it.

Puppet, Chef, Ansible, all led the way to something better. Docker and Kubernetes feel so far advanced from what we had in 2012.

PaulHoule2• 7 years ago

Uh... I've built Hadoop from source, it is not that bad.

icedchai• 7 years ago

I've also built it from source, admittedly back in 2012. It wasn't that bad, either! My concerns were building an AMI image, not "packaging" it in a more traditional sense.

PaulHoule2• 7 years ago

I think most of the conventional image building tools are stupid. (e.g. they take something that could be done in 10 minutes and make it take an hour, are prone to failure because of too many round trips, ...)

I made rather complex images that included a large database (e.g. 30 GB) and Java apps, and found the easy way to do it was write a program that analyzes the problem, writes a shell script that builds a bastion and builds the software, then put that in the cloud-init... The machine "phones home" via SQS to tell the controlling server to shut it down and image it.

duelingjello• 7 years ago

Yes, it's worse than that on a couple of fronts. Cloudera, Amazon and others benefit financially from situations of over-complicated messiness, while average SWEs, SAs and others are left with wasted time, money and lack of trust. And worse, lax engineering and operations practices introduce attack surfaces to fraudsters, thieves, organized crime and state-level actors to backdoor, destroy, steal, extort and monetize infrastructure of organizations large and small.

kube-system• 7 years ago

Using stuff blindly from Docker Hub is a bit sketch, but it’s not a hard problem to fix. Read the dockerfile before using it, adjust if you don’t like it, and build and push to your own repo. Then run your vulnerability scans of choice on them nightly.

rb808• 7 years ago

I agree the last decade has been really tough for sysadmins, but the payoff is developers have had a great time and I dont have to rely on those guys any more.

peterwwillis4• 7 years ago

This is a weird mentality that I think comes from being disconnected from ops. It's like you're a mechanical engineer building a truck, and now that you have a 3D Printer that can make any part, you think you don't have to rely on or communicate with the people driving the trucks on the road.

dividedbyzero• 7 years ago

As long as what you're doing isn't too involved, I guess running without ops is completely realistic nowadays. If all you need is a database and a relatively simple monolithic app plus stuff like logging, metrics, and if a little downtime here and there isn't the end of the world, there's lots and lots of managed/cloud offerings that'll cut down ops to a point where it's very much manageable for capable devs. A lot of small shops run like this and do so successfully.

opsthrowaway0• 7 years ago

Flash bulletin from the engine room: you're still relying on "these guys" (and gals), you're just cost-sharing our salaries across whatever pile-o-garbage SaaS you convinced your architect to engage, including my own large cloud provider. Hint: it'd be more effective (and cheaper) to employ us directly, but you do you.

That said, at the end of the day, I'm still sitting in my Aeron and your "great time" just means concentrating an entire discipline of professionals into cloud companies and ISPs. In this dream scenario (for who?), you get to sit around waiting for ticket updates while your revenue craters instead of picking up the phone or using that hole on your face to communicate with a peer. I get to focus on systems at scale and on monitoring and metrics instead of an annoying developer who escalates to their SVP and makes an inter-org ruckus when their TLS certificate expires and they can't find the error in their voluminous Javascript "logs". To be clear: I'm having a great time. There's a suspicious TAM in between me and you now, and you don't have my extension. I prayed for this day across many developer escalations and still can't believe it's here.

Seriously, people who are likely replaceable with Microsoft Access and a few Thoughtworks clones shouldn't throw stones at the professions completely supporting their Blue Bottle habit.

Oh yeah, the Internet keeps on quietly working, too. Tip jar is to your left.

finnthehuman• 7 years ago

All the claims that cloud computing is some boon to developers makes me suspect they might just work at a terribly organized company.

If you make developers pull double duty as sysadmins, then outsourcing surely makes things better for development. But it's still more work than handing the infrastructure and administration over to people that specialize in that. A well managed service is a ticket away, not getting someone who "knows docker pretty good" to have another background responsibility to neglect until it sets on fire.

Throughout the cloud boom I've worked at a company with a great IT department, not just a glorified helpdesk. The division makes sure that we provide a sane deliverable that doesn't need constant handholding and 10 hotfixes a day, and they give us the peace of mind that it's not going to go sideways. They sweat the details so much that the rare issues that have escalated to the head of IT and myself at the same computer investigating something were still relatively minor in the grand scheme of things.

ascotan• 7 years ago

This post reminded me of an old article: https://blog.codinghorror.com/vampires-programmers-versus-we...

The big win for 'containers' is that they separate the concerns of sysadmins from developers. The days of the sysadmin administering the tomcat and apache installs are shortly coming to an end. For my part, i think this is a good thing because devs and sysadmins have different concerns that don't overlap all the time.

disordinary• 7 years ago

I don't agree.

Systems are more complicated to build with a lot of dependencies that's true, but this is mitigated by shipping pre-built binaries and containers which have the dependencies bundled. Yes, those binaries can be from untrusted sources but if you use those you're taking a risk - the same as if you use code from an untrusted source (just because it's not compiled doesn't make it safe, when was the last time your organisation audited anything), ultimately a business should be using banaries from businesses they trust.

Containers actually increase security in the following ways: 1. They run isolated from the host system and if you set things up properly they should be non root. 2. They're immutable and cannot be tampered with. 3. Because they work on layers you can change them without affecting the underlying container, to upgrade you can just replace the layer that you inherit from, simplifying the upgrade path (and being immutable, stateless, and versioned there's no risk to upgrading, you can very easily rollback). 4. In the example listed above like Haddoop you can install the outdated version of Java that it depends on without compromising the host or other workloads on that server.

Of course, this all breaks down if you don't use containers from trusted sources and if you don't have processes in place.

It works in a modern environment with pipelines and automation, it doesn't work as well in a more traditional environment with sysadmins hand wrangling deployments.

Of course the most secure way to do things is to have deterministic builds of a codebase which creates a hash which you can compare with a canonical source, but this is tricky and not all build toolchains support this properly so you'll end up with mixed results. For a streamlined business the best compromise is to get applications from trusted vendors and rely on static analysis and the ability to quickly upgrade if a vulnerability is discovered.

Ultimately, the one of the best ways to secure a system is to establish an unbroken, audited, and automated trust chain and remove the ability for human intervention.

Russtopia2• 7 years ago

Amen. I avoid any container-based or blindly 'curl |sudo' install (the latter only after manual inspection of such a script).

Apps which only offer containers are untrustworthy black boxes. And it indicates devs who are too ignorant and/or lazy to make even basic efforts at distribution-neutral or portable code.

Gentoo and Funtoo keep devs honest to some extent, as packages must be built, not just slurped down in whatever alien form the project decided to use.

james-mcelwain• 7 years ago

I'm not sure about other container solutions, but Docker isn't a black box at all. It's fully introspectable -- I often dump a container's file system to see what's actually included.

IOT_Apprentice• 7 years ago

So nearly 5 years later, is any of this any better? Also, why haven't Linux vendors standardized on a single package manager that is the default for ALL Linux distros? This seems like the equivalent hell that npm gives to NodeJS.

mappu• 7 years ago

why haven't Linux vendors standardized on a single package manager that is the default for ALL Linux distros?

The two distro package managers that "matter" (apt and rpm) are similar enough that this isn't a problem in practice. You can have fpm spit out a deb or rpm or arch pkg with just one flag change, and host it on your own website.

The real differences aren't just mechanical packaging differences, but more a question of philosophies, there are a lot of differences in upstream distro packaging policies themselves. Debian isn't going to let you upload it directly unless it is a good citizen on their distro, stays up-to-date with their versions of all dependencies (so that you only end up with one python, libjpeg etc on your system) and is certifiably free software down to the bones. Whereas Arch will take literally anything in the AUR, and the fast way into CentOS is to be employed by RH (paraphrasing).

(I tried to package a certain app in RPM, but it turns out that OpenSuSE and CentOS have different names for the tzdata package, and until very recently rpm didn't even have a system for specifying alternate dependencies.)

cosmodisk• 7 years ago

"Consider for example Hadoop. Nobody seems to know how to build Hadoop from scratch. It’s an incredible mess of dependencies, version requirements and build tools." And that's exactly what's wrong with a half of the software industry. Some people even take pride in thinking that having dependency hell is a good thing. A lot of build tools look like they are created by aliens to aliens with little thought of how the end user would use it. And if someone dares to say how crappy the whole thing is, they get dismissed for not being smart enough or simply lack dedication to spend unlimited hours trying to build some 500 line code.

The sad state of sysadmin in the age of containers (2015)

Related Stories