Ask HN: Do you run apps bare metal?

https://news.ycombinator.com/item?id=27394058

by sdevonoes • 5 years ago

41 73 5 years ago

news.ycombinator.com

So, I am working on a side project and the way I deploy my (golang) application is basically:

- build binary

- copy binary, config files and static assets to the production server

- do blue green deployment (with nginx) to get zero-downtime deployment

- profit

(This is automated of course! I use Ansible, and I can easily rollback if needed. I can also deploy the same app to multiple machines if needed).

On my local machine I use Docker to test the Go code, but I don't really see the benefit of deploying my Go app in a container. My colleage told me "it's easier to deploy Docker containers. You just pull the image and voila!". I don't see how my approach could be "more complicated". Also, isn't my approach better in terms of performance? If my golang app runs "bare metal" instead of via a container, then sure the performance should be better, right?

PaulHoule3• 5 years ago

Yes.

Try to deploy code without Docker and you have one problem.

Try to deploy code with Docker and you have two problems.

denysvitali2• 5 years ago

It's a nice joke, but IMHO it's not true... unless you don't know how Docker works

edu• 5 years ago

But then, by definition, the joke is true. Even if learning Docker makes sense and has a positive ROI it’a still an extra problem.

PaulHoule3• 5 years ago

It is.

You might have a 10 second build process that does 1 MB of disk IO and it becomes a 100 second build process that does 10 GB of disk IO because you ran it in Docker.

I see people who "know how Docker works" put no care at all in the order of their image builds and end up doing 10-100x more IO than they need to.

Docker is fast compared to the time scale most people deploy servers at but is slow compared to the scale of a healthy software development process particularly if you are not running on a cloud server or are working on a gigabit connection.

If you took the time you spent trying to eliminate entropy from Docker, Kubernetes and such and actually tracked down the entropy that comes in from your environment (like the vandal who set your platform charset to not be "utf-8") you might find you don't have time to check your phone or send your co-workers stupid GIFs on slack waiting for builds.

rovr1384• 5 years ago

You know need to know how your server works, how docker works and how your application works.

You are introducing another layer of complexity.

codegeek• 5 years ago

I think you have it right. Don't over engineer it especially early on. If you need to scale/automate with containers later, do it then. For now, just get that damn thing up and running.

I built a Go web app side project which runs smoothly on a VPS with nginx as reverse proxy, letsencrypt SSL and dead simple supervisor config to run. Boom. Forget containers. Forget docker.

jeffreyrogers• 5 years ago

Yes, I do this too (although you're probably running on a hypervisor unless its your own hardware, so not completely bare metal). This is one of Go's biggest advantages. Deployment is way simpler since you don't have to configure a bunch of dev ops technologies.

0xdba• 5 years ago

I think most containers also run "bare metal", they just are more isolated as far as process/memory/fs, etc.

Having worked with and without Docker for various web apps, it removes some dependency management and server setup at the cost of another layer (or two...) of complexity. It's not always worth it to go to containers.

Docker seems to make the most sense in cloud environments that scale horizontally.

What benefits would you get exactly?

cridenour• 5 years ago

You have it right. Now of course if your project expands and has a front-end build step or some other process, etc. then maybe it's helpful to add a container to not have to get that working in two environments.

But with just a Go binary, I think you're doing it right.

KingOfCoders• 5 years ago

Isn't bare metal without an OS? At least this is what I've assumed when I've read the question.

[Edit] Thanks for the comments, seems I'm just plain old.

andrewstuart23• 5 years ago

Bare metal usually refers to not running in a VM or other actual virtualization tech that boots a nested guest kernel inside the host kernel.

benley• 5 years ago

These days people most frequently seem to mean "without a hypervisor" when they say bare-metal. In this case I guess it's "without containers", which isn't far off. Since it's not a formally defined term I'm happy as long as it's clear what the question means, and this one is explained clearly enough imo.

h2odragon• 5 years ago

It used to mean that to me; too. Language changes I guess.

I once had a small kernel patch for my ultrasparc that kept 14 of the system's 16 CPUs from being scheduled for anything but my application, just before they came up with cpu pinning and other, better interfaces for that... I didn't consider that "bare metal" at the time, but it was definitely thinly wrapped.

leecarraher• 5 years ago

I've always interpreted it as running code directly on the processor with no OS. I made a "baremetal" bitcoin miner for the original raspberrypi, until it burned up the chip(and i learned why OSes are important). I guess it's been updated to mean an application running on the native OS.

einpoklum2• 5 years ago

Isn't bare metal without an OS?

No. When you run with an OS, the OS doesn't intercept every single hardware instruction. It only interjects occasionally (or upon requests) - and the rest of the time, your code (or that of other processes) runs directly on the hardware.

... to be honest, that is also partially true in Virtual Machines, but there, the I/O devices and other facilities you see are mostly virtualized, so that when you interact with them you're actually triggering the VM host to run its code rather than triggering physical hardware to do something.

jandrewrogers4• 5 years ago

In modern virtualized environments, there are usually methods for doing I/O that approximately bypasses the host OS so that your VM is doing I/O against physical hardware. This wasn’t always the case several years ago but virtualized I/O performance can now be very close to bare metal if you write the code for it.

FPGAhacker• 5 years ago

“ In computer science, bare machine (or bare metal) refers to a computer executing instructions directly on logic hardware without an intervening operating system. “

The definition may be changing, but at least Wikipedia still has it this way.

its_nikita• 5 years ago

Not sure if this is applicable to your case, but if the application is/can be used by an end user, then having it available as a Docker image can be really useful to the user, and reduces friction to try out your program.

Many times while looking through a project I find on GitHub, if they have a docker image available to try it out, I can try it by running a single command. But if they have more than 3 installation steps, I usually pass.

rovr1384• 5 years ago

This also depends on what kind of software or if it's a hobby vs work related.

I usually do the opposite, even if there's a docker image available, to try it out I follow the build and manual setup. I need to know _how_ it works even if at the end I use the image. If something breaks, I own this and need to be able to debug it. Depending on the program and its architecture, I also need to know how to scale it.

Brian_K_White2• 5 years ago

I personally can not stand when an app comes with it's own entire surrounding operating system. I already have an operating system whose very job is to provide a standardized environment for processes to run in. I don't want 50 OS's where I used to have 50 processes. I can't stand AppImage, snaps, flatpack, docker images, etc for mere local desktop apps.

It's fine to serve as a reference, and for reproducible builds, and for appliances like a mail server. But I never opt to run an app that way even just to try it out, and I especially can not stand when an app developer only offers that as the only way they distribute an app (there are some, and not open source so you really have no choice). This prioritizes the developers convenience over the users. If it's their right to do so, then it must be exactly equally my right to say I don't like that and try to avoid using their apps.

trog• 5 years ago

Full agreement here. Drives me nuts when I see a Docker image for something that should be a simple zip file and some basic deploy or build instructions, but I can see why people prefer it.

SPascareli13• 5 years ago

By "bare metal" you mean as a regular process in the OS instead of as a docker container?

Then it should really make no difference in terms of performance. A process in a docker container is also a process in the host OS, but with some network and user permission stuff on top.

When I read the title I thought you meant bare metal as with no OS layer, just a go app running in the processor, which I think would be pretty cool to see.

jandrewrogers4• 5 years ago

For any software that is I/O intensive, there is a large difference in performance between a container and an ordinary process, as well as a number of other sharp edges. Containers are only approximately equivalent for CPU intensive workloads.

zozbot2342• 5 years ago

How so? Are you running actual containers (i.e. on Linux bare-metal host) as opposed to docker VM's on a foreign OS?

pests• 5 years ago

How is this true?

A container _is_ an ordinary process.

https://iximiuz.com/en/posts/not-every-container-has-an-oper...

(we just discussed another article by the same author a few days ago on slimmer containers)

"The container process is isolated (namespaces) from the rest of the system and restricted from both the resource consumption (cgroups) and security (capabilities, AppArmor, Seccomp) standpoints. But in the end, this is still a regular process, same as any other process on the host system."

sliken• 5 years ago

I think you are either confusing virtual machines with containers, or using containers where any OS differences are transparently hidden by starting a VPS with a container in it.

Containers have no overhead, you use the system kernel to do I/O for you, just like any other process.

blcknight• 5 years ago

I don't think bare metal means what you think it means.

IceWreck• 5 years ago

For Go applications, docker is unnecessary. Its just a single binary and systemd is enough. You don't need containers for it.

rovr1384• 5 years ago

While it's not necessary, simply saying use 'systemd' is not enough.

How do you handle multiple applications not interacting or affecting the data of another if there's an exploit?

silpol• 5 years ago

for some folks systemd is also 5th wheel in the cart

collinvandyck76• 5 years ago

There are times when a go application needs a lot of additional files for configuration and anything else. Docker is a great way to bundle those in instead of needing to ensure those files are already on all of your compute nodes.

twic• 5 years ago

Yes.

Our contract is that a unit of deployment is a .tar.gz file, and inside that there is an executable file bin/run which starts the app. We build apps in a few languages, but they all have a start script like that.

You can use static or dynamic linking, compiled or interpreted languages, anything you like, as long as the start script sets it up correctly. For Go apps the script can be trivial.

Tarballs must be self-contained - binaries, libraries, config files, static assets. We make an exception for interpreters, because those are huge and change slowly. Those are manually installed in /opt/something. I'd like it if we did something more disciplined and reproducible, but it's good enough for now.

We don't have many machines. Maybe 20? You can get a lot done with 20 full-size physical machines.

I don't see how something like Docker would help us here. It would just be the same, but less hackable.

fimbulvetr• 5 years ago

This sounds like neophobia or NIH syndrome. What you're doing is almost exactly the same as a docker deployment, except you're missing the ability to have some declarative leverage and tagging (unless your place invented that for themselves as well). There very few unhackable things about a Docker/container image, and typically that's for the better. The other advantage is that once you start using docker images, you now have more or less portable infrastructure. EKS, ECS, Google Cloud Run, Heroku, and on and on and on... But then again, it takes some time to realize that serverless is an advantage because if you're still young or your company isn't strapped for cash, they're fine hiring people to stay on call through the night and manage servers and upgrades and so on. Me? My fargate tasks run on servers that never need reboots or upgrades.

raverbashing• 5 years ago

If it works for you don't sweat it.

Docker is fine but if you don't need it then fine (and managing it yourself is usually a pain). But do make sure your systems are up to date (basically your linux distro)

andrewl-hn• 5 years ago

We do it right now, though we run Node instead of Go. For us it's not 'copy a binary', but 'copy and untar', but it's still very simple.

One downside that we find surprizing is that way too many Ops people are more familiar or comfortable in cloud or Kubernetes environments these days. We find it hard to find local talent who are willing to deal with bare metal hardware.

Because of that we plan to migrate to k8s at some point later this year. I'm neither support nor oppose it, to be honest. Introducing an extra Docker build step is annoying, but the idea of adding a few hundreds of Yaml lines to get monitoring, log aggregation, tracing, etc. sounds really nice, too.

arnvald• 5 years ago

How do you do that with Node: do you ship Node binary and all packages together with each app in one tar file? Or do you manage Node versions on the server that runs the apps?

xrd• 5 years ago

pkg "compiles" your app into a binary with a virtual file system inside the app. If you do a "require", the runtime rewrites that call and attempts to load the files from inside the sandbox virtual file system.

You can even provide assets (like say a JS app and associated HTML, CSS) that are stored inside the app. You can then build a full app as a single binary with no extra dependencies or files. There are a few gotchas like when you need native library support in your app (if you are using a sqlite driver) and then you need to ship a .node file with your app. But, generally, it is a great solution once you get the hang of it.

You compile against a specific version of node, and that runtime is included in your binary. You don't need the client to install node or choose a version, you lock that down when you build your app.

I've experimented with a lot of different nodejs packaging tools, but pkg seems to work the best with the fewest surprises.

Xunjin• 5 years ago

You know there are some projects like vercel/pkg which allows you to "bundle" everything inside a single binary, you don't even need the Nose run-time, of course there is some downsides, tho it's pretty solid.

siscia2• 5 years ago

Just to expand.

Container technology run your software just as it was a standard process.

There are no performance implications in running code inside or outside the container.

Here be careful, that on MacOS docker is implemented as a virtual machine. In such case there could be somehow serious performance implications. Especially around IO.

siscia2• 5 years ago

To finish up the point on performance I link this very interesting answer on so:

https://stackoverflow.com/questions/21889053/what-is-the-run...

_wldu• 5 years ago

Yes. Go static binaries are super useful and simple to deploy. IMPO, this is one of the reasons Go has become so popular.

Docker became popular because it fixed a lot of deployment issues for dynamic scripting languages. And really, Go static binaries and/or Java jars solve that same problem. So there is no need to complicate these deployments with Docker.

I do dockerize some Go binaries to run in ECS/Fargate, but otherwise I see no real need or reason to do so.

darksaints• 5 years ago

While Java deployment has definitely become a lot easier in the past decade, I'd contend that java jars dont really solve the same problem. Uberjars get close, but you still have to either deploy the JVM yourself or ensure that your target environment is appropriately provisioned.

andrewstuart23• 5 years ago

Your performance should not noticeably differ (on linux) between docker and just running as an uncontainerized process on the host. There's almost no difference as far as the kernel is concerned; it just tags your process slightly differently (different kernel namespaces and cgroups, but these apply to every process).

Bare metal usually refers to running without hardware or software virtualization (kernels within kernels), and if that were the case (docker on linux does not virtualize) it would be meaningfully faster, but that's not the case here.

The primary benefits you get from Docker may not be immediately apparent, so go for whatever works. Eventually, you'll probably find that it's substantially easier to scale out appropriately and automatically with some sort of orchestration tech, which is nearly all built on containers these days. Don't sweat it til you need it but at the same time, it's easier to learn sooner than when you're knee deep in tech debt for a dozen services.

wisemanwillhear• 5 years ago

Depending on how you setup networking for a docker container, you may see a notable network performance hit. Also, I've seen frustrating issues with the default docker bridge network that didn't play well with other services on our network, forcing us to switch to the "host" networking. The extra complexity & problems added by using a container solution may not give a positive ROI, even if the container abstraction provides more flexibility.

whalesalad2• 5 years ago

It would help to clarify some words:

"Bare metal" means that your application is running on the same OS that is also running the raw hardware (metal), aka no virtualization. Containers are not (generally) the same as virtualization.

The analog to bare metal is virtualized, where the hardware your program is seeing is not necessarily the hardware that is running on the actual host machine.

A docker container could ostensibly be considered running on bare metal. A container is really just isolation but the parent OS/kernel is in command. Here is a graphic that illustrates the differences: https://www.sdxcentral.com/wp-content/uploads/2019/05/Contai...

What you are really asking is do you need an abstraction layer or orchestration tool to manage doing this.

The short answer is that no you do not need it at all. If you can DIY this and are happy with it, that is sufficient. For example, a current deployment process for one of my clients (EC2 environment) involves stopping a custom systemd service, pulling the new binary/deps and then starting the systemd service. Really simple with a small instant of downtime, but within this environment that is not a problem.

einpoklum2• 5 years ago

It's also worth mentioning that even if you do not have a virtualized PC, you might still not be running on bare metal. Case in point: Java applications. They run in a language-specific virtual machine. (Also Scala, Clojure etc.) One could make an argument that this is also true for interpreted languages, but that gets into some splitting of hairs regarding the definition.

whalesalad2• 5 years ago

I don't consider the VM of Java to be "not bare metal" because you can still interface with the unadulterated operating system

sackerhews3• 5 years ago

"Bare metal" is running a program on the bare metal, without an operating system in between.

andrewstuart23• 5 years ago

I've never heard bare metal used that way. Any time I've heard it used, it's referring to a lack of virtualization. What you're talking about is usually referred to as a unikernel.

orev2• 5 years ago

While I have seen this definition used in some very specific circumstances (like high frequency trading where they can afford to hire people to actually write a fully customized app for the hardware), this definition is so specific as to be essentially useless as it would apply to virtually nobody.

Typical IT usage means something is running on an OS that’s installed on the hardware, as opposed to in a virtual machine.

user-the-name• 5 years ago

"Bare metal" traditionally means running with no OS. Your app takes the place of the entire OS.

And you can definitely run a bare metal application virtualised, just like you can run an OS virtualised.

gnfargbl• 5 years ago

There's nothing really wrong with what you're doing. Apps were deployed like that for decades before docker came along.

To my mind the main benefit of containerization would be that the

binary, config files and static assets

would be bundled together inside the container, so there is less risk of you running a half-baked version where (e.g.) the binary gets updated but the other files don't. However since you're using Ansible to deploy, that risk already feels small. You could also consider using the embed feature in go1.16 to bundle all the assets inside the binary.

Arguably there might also some security benefit to running in a container, but I wouldn't want to try and make that argument without knowing a lot more about the specific details of your binary's behaviour.

zozbot2342• 5 years ago

There's a significant security benefit because container will run with separate namespacing of all system resources, and make it easier to drop unneeded privileges. So the attack surface of any security issue within your app will be correspondingly reduced.

tkiolp4• 5 years ago

This is interesting. What if OP is running their golang application under a “golang” user and group? Wouldn’t that be enough to limit the surface area of attack? I’m assuming here that the “golang” user has limited permissions in the OS.

verbury• 5 years ago

The argument for using containers is primarily in your first two steps, far less so for deployment and profit(!). Having all of your apps code and dependencies in a self-contained, er...container, that was portable between different systems and perhaps more importantly, different developers, is arguably the main reason for using Docker.

If for now it's just you and you're not planning on spinning up in AWS ECS, CloudRun, any flavour of Kubernetes etc. then go with what works for you. All good.

analognoise• 5 years ago

Yes, I frequently don't use any operating system and write bare metal C code, usually for Zynq (normally RFSoC or MPSoC class) FPGAs, although I also frequently use a Microblaze soft processor with it's entire memory in BRAM when doing initial system bring up (say for a new board).

Lots of people in here don't know what "Bare metal" means. There's no operating system - at all. Anything else is not bare metal.

To go further: Sometimes (if you're bringing up a new processor design) you don't even get cstdlib - you have to write it (or the processor's architecture won't even support C!). "Bare metal" means exactly that - it isn't the same as "no virtualization"... at all.

sys_64738• 5 years ago

Bare metal generally means running on the host OS without any virtualization layer. Or used to mean that.

Brian_K_White2• 5 years ago

It means running on the hardware. bootloaders, kernels, hypervisors, dos apps, most embedded, grub invaders, efi/bios, are things which run on bare metal.

A process that runs inside an OS, is running inside an OS.

This is like when they redefined "literally" to literally mean figuratively. OK fine "language changes" but then now what word does the job that "literally" used to do? We had a word to express a concept, and now that word means the exact opposite of that concept, or worse, may still meam either one, making it meaningless and non-functional.

yesenadam• 5 years ago

I once complained on HN how appearances of literally in the new sense were driving me nuts. It was explained to me that it's nowadays used as an intensifier—and has been for nearly 100 years, and my impression the change was recent was probably just the Baader–Meinhof phenomenon.

Although unconvinced, I gave up that fight and retreated to my usual protesting that a method and a methodology are two very different things, although it seems everything is a methodology these days. Longer is better?

jasonpeacock• 5 years ago

Ignoring the proper use of "bare metal", you should review the pros/cons of deploying with Docker vs SCP.

Docker is helpful when you have a lot dependencies, or otherwise need to create a reproducible image (configuration, special directory structure, etc.) or when you want to control the process's resources (sandboxing).

Otherwise, because Go compiles into single binaries, if there's no other external dependencies/configuration to manage and you don't need process isolation/sandboxing, then Docker is just another moving part that adds complexity and could go wrong.

Ask HN: Do you run apps bare metal?

Related Stories