HN user
Mix is absolutely awesome. One of the most carefully organized open source projects that I've seen.
Some years ago I made a Mixxx demo video with a DYI "integrated controller". It demos Linux boot to Mixxx, touch screen, beatmatching and some modest effects:
https://www.youtube.com/watch?v=DjHvW4OsQ2Y
Mixxx devs: if you are reading this... cheers :-)
True story ^^^^^
Add to this, that you can have outages due to operational complexity, which tends to happen in complex cloud native setups.
People want simple answers. "It depends" annoys everyone :-\
I wonder if there is some major propaganda push on HN to discourage actual >software engineers from ever running their servers
How to scare a computer engineer in 2024? Invite him to configure a server.
Yeah, the Emperor's New Clouds. Sometime ago I thought I was crazy and everybody else was right :-) but now we are seeing more and more people doing the numbers.
https://logical.li/blog/emperors-new-clouds/
I wonder, though, why the topic is so contentious. This is supposed to be an engineering discipline, not a fight between religions.
Agreed, in general.
But don't forget about the definition of micro :-) This is an optimization problem rather than a conceptual one.
I love Martin Fowler's one pager called Snowflake server - it aged very well and I still use it as a reference in the cloud era. And this text is also good advice IMO.
What I feel that is missing on what he calls the "Microservices Premium" is a clear statement that this premium is paid in "ops" hours. That changes the game because of the "ops" resource scarcity.
In fact, the microservices dilemma is an optimization problem related to the ops/dev ratio that is being wrongly treated as a conceptual problem.
This is the simplest analysis I could come up with:
Thanks for sharing the story. Despite the whole TCO being higher, I wonder how the 8K to 6K reduction happened.
On AWS, fargate containers way are more expensive than VMs and non fargate containers are kind of pointless as you have to pay for the VMs where they run anyway. Also auto scaling the containers - without making a mess - is not trivial. Thus, I'm curious. Perhaps it's Lambda? That's a different can of worms.
I'm honestly curious.
There's also an even bigger problem that can arise, the distribution can just end, such as the termination of CentOS
If you are doing something serious you probably want to chose suppliers in such a way that you can demonstrate you have security and business continuity under control. That means you probably want to use RHEL, Suse or Ubuntu, distributions for which commercial support exists.
(Ubuntu is particularly interesting because you can start with an LTS release for free and activate commercial support if business goes well, without changing your processes.)
You can think about this beforehand or wait until customers require some kind of certification and the auditors ask you for your suppliers list + the business continuity plan, among other things. You will face this if you deliver to a regulated market or if your customers are large enough to self regulate this kind of thing.
LTS not good enough? Well, cloud native does not have LTS comittement and Pipy does not provide security fixes separated from logical changes.
Try to keep your Terraform code stable for two years in AWS, or try to understand the lifecycle of AWS Glue versions from the docs. Or trust that Google will not discontinue their offers :-)
I mean, maintaining software is never easy or effortless but I respect the effort done by LTS Linux providers - they sell stability and security for a fraction of what you pay for cloud native.
reproducibility isn't just on your deployments, it's for development too
Absolutely. Adhoc configurations should be forbidden! It is easy to ensure dev env reproducibility when you run Linux. If you have config management your devs can have VMs that subscribe to the same exact configuration that the staging prod and dev environments have. They can literally have a deplpyment server in their machine, as a VM. Since the configuration is stored on a server and applied continuously, it is hard to screw it.
You can achieve this with Docker as well, if the arrangement is not too complex.
The problem, at least in my experience, comes when you start depending on several cloud native components where local emulations are always different from the real cloud env in tiny details that are going to screw the deploys over and over.
My thoughts and prayers :-\ Wish you a quick recovery!
Thank you as well, for sharing these notes about your setup. Indeed concentrating everything in the same monitoring system is very helpful as it reduces the cognitive load. You can likely do the same with Icinga.
Feel free to reach out on Linkedin if you need some more details. More than happy to share.
Honestly I just think you should be more generous
I am generous in the context for generosity. Turns out that engineering is not about being generous but rather about choosing the most efficient solution for problems that in the end need to be business driven. This requires evaluating requirements, context and tradeoffs. That takes a cold, rational mind more than generosity.
K8s really does address real problems around deployment and it's very well thought out
It's great where it makes sense. It's less than great elsewhere.
Not everything is SaaS, not everything needs scaling, not everything needs 99.99% of uptime, not everything needs a CDN, not every company is VC backed operating at high risk / high reward, etc, etc. Context is better than ideology. If you read the article I posted you will see that stated clearly.
Good luck finding someone to understand what that 'custom Naemon' plugin is doing.
You Kubernetes people get triggered very easily. I was already lucky to have found several juniors that worked in this kind of thing with minimal training. The 'custom Naemon plugin' is 30 lines of bash and you can adapt it to any monitoring system.
Of course this is scary and complicated. I might consider switching to 'Kubernetes operators', which sounds simpler :-)
The longer you wait the harder the pain. The less you wait the more frequent the pain. So it depends on the function that converts intensity and frequency to suffering :p But, most importantly, the fact that LTS gives you a choice is what I was highlighting.
For the scope I operate, which is pretty standard Linux packages (PostgreSQL, MariaDB, Nginx, Docker, OpenVPN, OpenSSH) the changes between 16.04 and 22.04 have been quite OK to deal with.
Like 12 years of life cycle is not enough for you to plan a transition?
You can use the entire life cycle but not one is forcing you to. You can update from one LTS to another every 2 years, or 4 years, or 5 years... you decide.
Also "scalability" is multi dimensional. I've seen, in the same company, infinite scalability in one downstream system whereas the upstream system it depended on was manually feed by fragile human-driven processes because there was not time to fix it. And at the same type the daily ops were "brain frying" because the processes were not automated and not streamlined and the documentation was ambiguous.
So, you had technical scalability in one system but if the customer base grew quickly every other bottleneck would be revealed.
There is more to business operations than technology, it seems.
I am curious about this too but haven't had the time to give it a try. Looking forward to hear about experiences.
The questions are short but the answers would be long. Puppet manages all fine grained OS resources (files, dirs, repos, cronjobs, sudo declarations, firewall rules, etc) and you aggregate those resources into classes which are then pushed to different machines. The classes are parametrizable for the differences between systems.
If I was to write an idempotent script for each native resource I would finish in some years :-)
You chose whatever monitoring system you like the most.
For offline nodes you use whatever the level of criticity of your node justifies. This is something people struggle to understand: not every business needs 99.99% uptime. That said, I never had a downtime in Hetzner. On Digital ocean I had one short forced reboot in 4 years. YMMV so protect yourself as much as necessary.
Deploying on a different provider than Hetzner is the same as deploying on Hetzner except the part of launching the machine which is trivial to script - the added value is making the machine work and Ubuntu/Debian/RHEL are the same everywhere. You don't have vendor lock in with this.
If K8s works for you, enjoy it. Nobody is telling you to stop :-)
I don't mind elaborating - the fact that people are asking me questions reminds me that I need to invest a bit more effort on some articles.
This case is actually pretty simple.
Puppet applies the configuration you declare impotently when you run the Puppet agent: whatever is not configured gets configured, whatever is already configured remains the same.
If there is an error the return code of the Puppet agent is different from that of the situations above.
Knowing this you can choose triggering the Puppet agent runs remotely from a monitoring system, (instead of periodical local runs), collecting the exit code and monitoring the status of that exit code inside the monitoring system.
Therefore, instead of having an agent that runs silently leaving you logs to parse, you have a green light / red light system in regards to the compliance of a machine with its manifesto. If somebody broke the machine leaving it in an unconfigurable state or if someone broke its manifesto during configuration maintenance you will soon get a red light and the corresponding notifications.
This is active configuration management rather than what people usually call provisioning.
Of course you need an SSH connection for this execution and with that you need hardened SSH config, whitelisting, dedicated unpriviledged user for monitoring, exceptional finegrained sudo cases, etc. Not rocket science.
Once you do it for long enough it might be worth it to consider configuration management where you declare native structured resources (users, firewall rules, nginx reverse proxies, etc) rather than writing them in shell.
I use Puppet for distribution of users, firewall rules, SSH hardening + whitelisting, nginx config (rev proxy, static server, etc), Let's Encrypt certs management + renewal + distribution, PostgreSQL config, etc.
The profit from this is huge once you have say 20-30 machines instead of 2-3, user lifecycle in the team that needs to be managed, etc. But the time investment is not trivial - for a couple of machines it is not worth it.
Well that's exactly the point! Creating complex cloud resources with, for instance, Terraform, is less reproducible than a shell script on an LTS system like Ubuntu or RHEL - that's because the cloud provider interfaces drifts and from time to time stops accepting the terraform manifests that previously worked. And to fix it, you have to interrupt your normal work for yet another unplanned intervention in the terraform code - this happened to my teams several times.
This does not happen with Puppet + Linux, because LTS distributions have a long release cycle where compatibility is not broken.
I tried to explain this topic in the article linked above. Not sure how far I succeeded.
I had the exact same business for 18 years :-) and yes, without corporate nonsense it is easy to retain intelligent people. Cheers
My comment about pip is orthogonal to Docker. This is the same with or without Docker - I added a comment on this thread with more detail.
If you really want to open that can of worms, here it goes:
Pipy is an informal source of software that has low security levels and was infested with malware many times over the years. It does not provide security updates: it provides updates that might include security-related changes as well as functional changes. Whenever you update a package from there, there is a chain reaction of dependency updates that insert untested code in your product.
Due to this, I prefer to target an LTS platform (Ubuntu LTS, Debian, RHEL...) and adapt to whatever python environment exists there, enjoying the fact that I can blindly update a package due to security (ex: Django) without worrying that it will be a new version which could break my app. *
Furthermore, with Ubuntu I can get a formal contract with Canonical without changing anything on my setup, and with RHEL it comes built-in with the subscription. Last time I checked Canonical's security team was around 30pax (whereas Pipy recently hired their first security engineer). These things provide supply-chain peace of mind to whoever consumes the software, not only to who maintains it.
I really need to write an article about this.
* exceptions apply, context is king
I'll correct myself:
s/host OS independence/a certain level of host OS independence
And getting containers to run depends on the OS - if you don't control the host, leads to major ping-pongs.
Even within Linux (Ubuntu, Debian, RHEL, etc) when you are distributing multiple related containers there are details to care about, not about the container itself but about the base OS configuration. It's not magic.
I fully agree with you: it is mostly an education problem and you can find people willing to learn right out of univ. Indeed, that is exactly my experience: I successfully onboarded several (carefully selected) junior people into the ops skillset over the years and I have seen them do wonders with customer systems, while enjoying their "ops life", without having fires every day.
The connection of this to the replies above it: I am not sure if this kind of junior people would be easy to retain in a large corporate environment. We certainly can do that in niche consulting.
I mentioned Docker because it interests many developers but on VMs that I control I do not need Docker at all. Deploying with Docker provides host OS independence which is nice if you are distributing but unnecessary if the host is yours, running a fixed OS.
For Python backends I often deploy the code directly with a Puppet resource called VcsRepo which basically places a certain tag of a certain repo on a certain filesystem location. And I also package the systemd scripts for easy start/stop/restart. You can do this with other config management tools, via bash or by hand, depending on how many systems you manage.
What bothers me with your question is Pip :-) But perhaps that is off topic...?
The same thing that happened to devops from 2017-2024 (see: https://logical.li/blog/devops/) is happening with dataops. Hype train and jargon based decisions are taking place.
In the past years I was solving a data pipeline mess on a project which also had a devops AWS mess. First thing I was told was "what we need is a data lake".
Decisions are sticky so take context into account.
Reference:
https://survey.stackoverflow.co/2022/#developer-profile-deve...
Only around 11% of the whole devs identify as devops specialist or cloud infrastructure engineer.
This is why I am saying ops people are a scarce good (unfortunately) from a data driven perspective. Of course my daily life confirms it.