Why are we templating YAML? (2019)

https://leebriggs.co.uk/blog/2019/02/07/why-are-we-templating-yaml

by spiros • 2 years ago

433 654 2 years ago

leebriggs.co.uk

This article made me think it'd be nice to generate k8s JSON using TypeScript. Just a node script that runs console.log(JSON.stringify(config)), and you pipe that to a yaml file in your deploy script. The syntax seems more sane and has more broad appeal than jsonnet, and I'd wager that the dev tooling would be better given good enough typings.

By the way the answer to the question "why are we templating yaml?" is: people are just more familiar with it and don't want to have to translate examples to jsonnet that they copy and paste from the web. Do not underestimate this downside :) Same downside would probably apply to TypeScript-generated configs I bet.

Leo_Verto2• 2 years ago

That's sort of what https://cdks.io does, except the final output is YAML for better readability.

happymellon2• 2 years ago

Fyi https://cdk8s.io/

geethree• 2 years ago

Take a look at cdk8s from Amazon.

https://github.com/cdk8s-team/cdk8s-examples/tree/main/types...

YawningAngel• 2 years ago

You can convert YAML to JSON programmatically, and JSON is valid jsonnet, so you can pretty much copy paste examples from the web into your jsonnet if you find yourself wanting to do that

grncdr• 2 years ago

Others have mentioned CDK, but I want to say that this is almost the exact approach I took on a project recently and it worked out fine. Node script that validates a few arguments and generates k8s manifests as JSON to be fed into `kubectl apply`.

IME, here's no need to involve anything more complicated if your deployment can be described solely as k8s manifests.

sigwinch284• 2 years ago

I would recommend implementing a similar API to Grafana Tanka: https://tanka.dev

When you "synthesise", the returned value should be an array or an object.

1. If it's an object, check if it has an `apiVersion` and `kind` key. If it does, yield that as a kubernetes object and do not recurse. 2. If it's an array or any other object, repeat this algorithm for all array elements and object values.

This gives a lot of flexibility to users and other engineers because they can use any data structures they want inside their own libraries. TypeScript's type system improves the ergonomics, too.

sgarland2• 2 years ago

copy and paste from the web

Hot take, this is a terrible idea, and is why so much cloud infra is monstrously expensive (and bad).

People need to stop making infra easy. It’s not supposed to be easy, because when you make a bad decision, you don’t get to revert a commit and carry on with life. You don’t understand IOPS and now your gp2 disk is causing CPU starvation from IOWAIT? Guess you’re gonna learn some things about operating within constraints while waiting for a faster disk to arrive at the DC! Buckle up, it’ll be good for you.

I’m fully aware that I sound like a grouchy gatekeeper here, and I’m fine with it. People making stupid infra decisions en masse cause me no end of headaches in my day job, and I’m tired of it.

embik• 2 years ago

I am really sad that jsonnet / ksonnet never really took off. It’s a great way to template, but has a bit of a learning curve in my experience. I suspect that is why it’s niche.

If you like what is presented in this article, take a look at Grafana Tanka (https://tanka.dev).

madjam002• 2 years ago

Yeah similarly I'm using Nix to template K8s templates and I've never looked back. Helm is great for deploying 3rd party applications easily but I've never seen the appeal for using it for in house services, templating YAML is gross indeed.

Leo_Verto2• 2 years ago

The fact that it's a purely functional programming language with lazy evaluation is really powerful but steepens the learning curve for devs who haven't worked with functional languages.

The stdlib is also pretty sparse, missing some commonly required functions.

anentropic• 2 years ago

The fact that it's a purely functional programming language with lazy evaluation is really powerful but steepens the learning curve for devs who haven't worked with functional languages.

does it really though? what part do they struggle with?

int_19h7• 2 years ago

The stdlib is also pretty sparse, missing some commonly required functions.

This seems to be the general curse of template languages. For some reason, their authors have this near-religious belief in removing every "unneeded" feature, which in practice results in having to write 10 incomprehensible lines of code to do something that could be easily done in one line of readable code in a proper PL.

twic• 2 years ago

I was reading the description of Jsonnet and wondering why we don't just use JavaScript. Read a file, evaluate it, take the value of the last expression as the output, and blat it out as JSON.

The environment could be enriched with some handy functions for working with structures. They could just be normal JavaScript functions. For example, a version of Object.assign which understands that "key+" syntax in objects. Or a function which removes entries from arrays and objects if they have undefined values, making it easy to make entries conditional.

Those things are simple enough to write on demand that this might not even have to be a packaged tool. Just a thing you do with npm.

ManBeardPc3• 2 years ago

Worse yet, in some places (CI/CD) YAML becomes nearly a programming language. A very verbose, unintuitive, badly specified and vendor-specific one as well.

sspiff2• 2 years ago

In some places working with Kubernetes, people unironically use the term "YAML engineer".

donalhunt• 2 years ago

I've seen memes where SREs complain they have just become YAML engineers. :(

_ea1k4• 2 years ago

yamlops is a real thing :)

zelphirkalt• 2 years ago

In such places one frequently has to remind oneself and others to not start programming in that configuration language, if avoidable, to not create tons of headache and pain.

XorNot• 2 years ago

This criticism doesn't pass the sniff test though: your average Haskeller loves to extoll the virtues of using Haskell to implement a DSL for some system which is ultimately just doing the same thing in practice (because they're still not going to write documentation for it, but hey, how hard can it be to figure out it's just...)

YAML becomes a programming language because vendors need a DSL for their system, and they need to present it in a form which every other language can mostly handle the AST for, which means it's easiest if it just lives atop a data transfer format.

baq4• 2 years ago

maybe yaml should standardise hygienic macros. and a repl.

hiAndrewQuinn2• 2 years ago

Hey now. Your average Haskeller would simply recommend you replace YAML with Dhall.

https://dhall-lang.org/

ManBeardPc3• 2 years ago

I don't know what this has to do with Haskell. I understand that they need a DSL for their system. I just don't agree that it is a good idea to use some general purpose serialization format. In the end they always evolve to a nearly full programming language with conditions and loops. Using a full programming language makes much more sense IMHO, for example like Zig build files or how we use Python to build neural networks. That way I can actually use existing tools to do what I need.

dygd• 2 years ago

Spot on. We use ytt[0], "a slightly modified version of the Starlark programming language which is a dialect of Python". Burying logic somewhere in a yaml template is one thing I dislike with passion.

[0] https://tanzu.vmware.com/developer/guides/ytt-gs/

_ea1k4• 2 years ago

TBH, ytt is the only yaml templating approach that I actually like.

The downside is that it is easy to do dumb things and put a lot of loops in your yaml.

The positive is that it is pretty easy to use it like an actual templating language with business logic in starlark files that look almost just like Python. In practice this works pretty well.

The syntax is still fairly clumsy, but I like it more than helm.

indymike2• 2 years ago

YAML is the Bradford Pear of serialization formats. It looks good at first, but as your project ages, and the YAML grows it collapses under the weight of it's own branches.

Y_Y2• 2 years ago

I had to look up that tree. Invasive, offensive odour, cynaide-rich fruit. That's a a good insult!

MrBuddyCasino• 2 years ago

Even worse, every generation repeats this mistake. I‘m not sure S-Expressions are the answer, but Terraform HCL should never have been invented.

SOLAR_FIELDS2• 2 years ago

I was just telling a colleague today that HCL is great until you need to do a loop. A lot of parallels to this YAML discussion

kevincox3• 2 years ago

I'll take HCL over YAML templating any day. At least it is working with real data structures not bashing strings together.

That being said, yes, it is also an awful language.

marginalia_nu• 2 years ago

It's pretty much repeating the mistake of early 2010s Java, where the entire application frequently was glued together by enormous ball of XML that configured all the dependency injection.

It had the familiar properties of (despite DTDs and XML validation) often blowing up late, and providing error messages that were difficult to interpret.

At the time a lot of the frustration was aimed at XML, but the mid 2020s YAML hell shows us that the problem was never the markup language.

epistasis• 2 years ago

You have a loosely coupled bundle of modules that you need to glue together with some configuration language. So you decide to use X. Now you have two problems.

deathanatos• 2 years ago

Yeah … for CI files (like Github workflows & such), one of the best things I think I've done is just to immediately exec out to a script or program. That is, most of our CI steps look like this:

  run: 'exec ci/some-program'

… and that's it. It really aids being able to run the (failing) CI step offline, too, since it's a single script.

Stuff like Ansible is another matter altogether. That really is programming in YAML, and it hurts.

elicox• 2 years ago

For me, environmental variables are super simple and remove all the complexity of these configuration files.

Kichererbsen• 2 years ago

you can't really check them into source code though, right?

zer00eyz• 2 years ago

I swear this is how we got docker containers... some ruby dev who abused env vars and a SA who was sick of his shit breaking on every roll out and hearing "but it works for me"...

And now installable software is a fucking unicorn!

( This week I keep running into go apps that can be installed from source or as straight down load, with docker as well. Been a breath of fresh air)

throwaway1438292• 2 years ago

The article is mostly talking about things like Helm charts for kubernetes, which aren't possible to define as env vars.

j4yav• 2 years ago

How do you make sure all the right variants of all the environment variables are in the right place(s)?

simiones3• 2 years ago

Except when you need anything more complex than a string or an array of strings, when they become entirely useless.

There is not a single even slightly complex piece of software that uses exclusively env vars for configuration. Even bash or vim have config files, this is not some new idea.

planede2• 2 years ago

Hang on there, array of strings? Environment variables can't handle that either without quirks.

Thev00d00• 2 years ago

In my experience there is a near zero uptake of jsonnet or similar amongst "regular" i.e less ops inclined developers.

gotmpl is a lot easier to grok if you are coming in cold. Yes it sucks for anything mildly complex, but the barrier to entry is significantly lower.

Generation via real programming languages is the future I am hoping for.

Anarch157a• 2 years ago

Jsonnet looks like a case of XKCD-927[0]. I fully agree with you that real programing languages are the way to go for generating anything more complex.

[0] https://xkcd.com/927/

tipiirai5• 2 years ago

I think YAML is a good pick for non-developers / content creators. The front matter section in Markdown files is a good example. Or is there a better, human-friendly alternative?

pvorb2• 2 years ago

YAML is all but human-friendly. It has far too many special features and edge cases for most people. Something simple like Java properties files would solve something like markdown front matter perfectly fine.

Hackbraten3• 2 years ago

Java properties files are a mess. They still require Windows encoding (ISO-8859-1), which is incompatible with UTF-8.

kristiandupont2• 2 years ago

You just pinpointed my biggest peeve with YAML. It looks like it's "human friendly" because there are no scary curly braces. But you still need to get the syntax exactly right, so that benefit is very small. And now you have to keep your finger on the screen while scrolling in order to figure out what a bullet belongs to.

tipiirai5• 2 years ago

Then what alternative do you recommend for content creators? Do you use the alternative in Markdown front matter?

never_inline• 2 years ago

This is least of my worries - just use VScode with plugin which gives red lines on and formats yaml, use yamllint in your CI.

robertlagrant2• 2 years ago

But you still need to get the syntax exactly right

I think it's more that it's declarative that makes it simple. Also you just have to remember simpler rules compared to JSON.

E.g.

  - Apple
  - Orange
  - Strawberry
  - Mango

Is simpler than

  [
    "Apple",
    "Orange",
    "Strawberry",
    "Mango"
  ]

Don't forget to skip that last comma! But not all of the others!

tovej2• 2 years ago

I'd personally go with TOML over YAML for that

rascul• 2 years ago

Why something so complex for front matter? Isn't it typically just a few key/value pairs?

spacebanana73• 2 years ago

I wonder whether doing away with config altogether is a solution.

Just build an application that calls AWS APIs directly when you want to deploy or update an environment.

simiones3• 2 years ago

Preferably written in assembler, to avoid the extra complexity of a compiler, right?

Configuration files have been a common feature of software since OSs exist, basically. They serve a clear and useful purpose, even though they create some problems of their own.

spacebanana73• 2 years ago

For complex environments like those discussed in the article, there’s unavoidably complicated logic.

Code is a good place for logic to live.

Compared to yaml, code is more testable, readable and expressible.

I should’ve restricted my original comment to the kind of situation in the article where different configs are created for various regions and test environments with optional values. Totally agree configs are useful for defining more static values.

roenxi• 2 years ago

I see a problem here. I'm not certain if the sort of person who would choose YAML as their configuration language sees a problem here.

There is a direct conflict between human-centred data representations and computer-centred. Computers love things that look like a bit like a Lisp. Humans like things that look a bit like Python. If you're the sort of person who wants to use a computer to manipulate their Kubernetes config then you'd be secretly annoyed that Kubernetes uses YAML. However, it appears the Kubernetes community are mainly YAML people, so why would they mind that their config files will be horrible to work with once programming logic gets involved? The downside of YAML is exactly this scenario, and I believe the people involved in K8s are generally cluey enough to see that coming.

YAML is a superset of JSON

The spec writers can put whatever they want in their document, but I don't think this is true. If you go in and convert all the YAML config to JSON, the DevOps team is going to get upset. The two data formats have the same semantic representation, but so do all languages compiled to the same CPU arch. JSON and YAML are disjoint in practice. Mixing the two isn't a good idea.

Joker_vD2• 2 years ago

The ironic thing is that, IIRC, k8s manifests were supposed to be machine-generated from the k8s's inception, you weren't supposed to write them by hand... of course, people wrote them by hand anyway, until it became unbearable ― at which point they've started templating them because that's how the things always seem to progress: manually-written text is almost never replaced by machine-generated config-serialized-to-text, it's replaced by templated-but-originally-still-manually-written text.

baq4• 2 years ago

k8s manifests were supposed to be machine-generated from the k8s's inception,

failed spectacularly at not being inconvenient enough for their intended purpose.

one of those cases where unreadable by design would be a most welcome feature.

mortehu• 2 years ago

"YAML is a superset of JSON" only means that any JSON document is a valid YAML document. It does not mean YAML is equal to JSON.

Alifatisk• 2 years ago

If you start templating yaml then you might want to give jsonnet a try

datadeft3• 2 years ago

Indeed why? However the conclusion I have is not to use JSON but to use a type safe configuration language that can express my intent much better making illegal states impossible. One example of such lang is Dhall.

https://dhall-lang.org/

Aeolun• 2 years ago

If I’m going to use a whole language to generate my config already, why would I use anything but the language my application is written in? Everything can export JSON after all.

vlfig2• 2 years ago

Different requirements, different guarantees. Principle of least power. Have a look at https://docs.dhall-lang.org/discussions/Safety-guarantees.ht....

datadeft3• 2 years ago

Because your language might not have a nice type system. For example Python -> JSON is going to produce worse guarantees than DHALL.

happymellon2• 2 years ago

Pulumi over Terraform.

CDK over Cloudformation.

Don't hand craft configuration files, these aren't new lessons. I remember being first introduced to Troposphere, which was pretty awesome.

sdflhasjd• 2 years ago

I need a restraining order against YAML DSLs.

TheFuzzball4• 2 years ago

I just knew this would be about Kubernetes when I saw the title.

The Kubernetes API is fairly straightforward, and has a well-defined (JSON) schema, people should be spending a bulk of their time learning k8s understanding how to use the API, but instead they spend it working out how to use a Helm chart.

I don't think Jsonnet, Ksonnet, Nu, or CUE ever gained that much traction. I'm convinced most people just use Kustomize, because it's fairly straightforward and built in to kubectl.

I'd like a tool that:

- Gives definition writers type checking against the k8s schemas - validation, version deprecations, etc.

- Gives users a single artefact that can be inspected easily and will fail (ACID) if deployed against a cluster that doesn't support any objects/versions.

- Is built into the default toolchain

---

I feel like writing a Bun or Deno TypeScript script that exports a function with arguments and returns a list of definitions would work well, esp. with `deno compile`, etc. but that violates the third point.

baq4• 2 years ago

probably doesn't meet the 2nd requirement, most definitely doesn't meet the third, but:

https://cdk8s.io/docs/latest/

TheFuzzball4• 2 years ago

The second requirement is actually probably the most important - if someone that just set up ArgoCD, Flux, or has their own GitOps pipeline, how much of a headache does using a new compile step present?

Lots of things are simple in isolation: want to use Cue? Just get your definitions and install the compiler and call it and boom, there are your k8s defs! Ok, but how do I integrate all of that into my existing toolchain? How do I pass config? Etc, etc.

The best, fastest tool won't win. The tool that has the most frictionless user story will.

Havoc2• 2 years ago

How would one use the json api without ending up writing a bunch of custom code?

TheFuzzball4• 2 years ago

I think custom code is to be expected, and making it maintainable is what's important.

everything should be made as simple as possible, but no simpler.

Helm et al made it simpler than it was, IMO.

ithkuil4• 2 years ago

I love the idea of keeping it simple and I do try to use kustomize or even plain yaml as installation method as much as possible.

But in practice when managing large systems you inevitably end up benefiting from templating

worldsayshi4• 2 years ago

I've begun thinking that if you start thinking about templating you might be better off building an operator. Operators aren't as well understood and documented. But in my mind an operator is just a pod or deployment that creates on demand resources using the k8s api.

liveoneggs2• 2 years ago

k8s make me miss xml

fsniper2• 2 years ago

For Helm the value is it is not a configuration managemet solution but a package manager. The rest are just methods of writing json/yaml.

I understand the "hate" against yaml, But I don't think it's deserving it that much.

Perhaps timoni will take over with it's usage of cue. At least it's a package management solution.

ryandv5• 2 years ago

The Kubernetes API is fairly straightforward, and has a well-defined (JSON) schema, people should be spending a bulk of their time learning k8s understanding how to use the API, but instead they spend it working out how to use a Helm chart.

This is a general pattern in software. Instead of learning the primitives and fundamentals that your system is built on, which would be too hard, instead learn a bunch of abstractions over top of it. Sure, now you are insulated from the lower-level details of the system, but now you have to deal with a massive stack of abstractions that makes diagnosis and debugging difficult once something goes wrong. Now it's much harder to ascertain what exactly is happening in your system, since the details of what is actually going on have been abstracted away from you by design. Further, you are now dependent on that abstraction layer and must support and accommodate whatever updates may be released by the vendor, in addition to whatever else is lurking in your dependency graph.

DrBazza• 2 years ago

We're using jsonnet for our systems and they have absolutely nothing to do with k8s. I'm not sure it's true to say it has ever gained much traction. It's just a niche case for complex configuration, and isn't the most publicised tool.

It does precisely what we need with zero fuss, cross platform and cross _language_ (we've embedded it in C++, .NET, and JVM executables).

We can use the resulting json config with a vast array of tools that simply don't exist for the alternatives such toml/yaml/hocon/ini whatever. In fact we tried to get HOCON working for non-JVM languages but there was always some edge case.

sanderjd• 2 years ago

Yep, I find kustomize and (especially) helm so confusing, while finding kubernetes yaml files very easy to use and understand.

Joker_vD2• 2 years ago

The title of TFA was actually my reaction when I learned what Helm was actually doing. Initially I thought Helm would take an input file of YAML-with-template-bits, parse that YAML as an object, then use the provided template bits to fill in the parts of that object, then serialize the object back to YAML and write it out. Sounds reasonable, right? Nope, it's literal text substitution, so if you want to have a valid YAML as the output you better count your indentation on your fingers, and track where the newlines go or don't go.

taspeotis• 2 years ago

Yelling At My Laptop

difosfor• 2 years ago

Wouldn't it be a better idea to use an existing programming language instead of cooking up numerous half baked templating languages?

berkes2• 2 years ago

Yes.

Except you then have to sensor that programming language severely. Maybe you can accept some endless loop, but you probably don't want the CI orchestrator to start mining Monero, instead of bootstrapping and configging servers and services.

A solution to that sensorship might be a very limited WASM runtime: one that offers a very few API's, has severely limited resources and timeouts and such. So people can write their orchestration in Python, Javascript or Rust or even Brainfuck if they want, but what that orchestration can do, and for how long it can do that, and how much memory, space and so on it gets, all is very limiting.

While that may work, it's far harder to think of than "lets make another {{templating|language}}" inside this YAML that we already have and everyone else uses.

sigwinch284• 2 years ago

I don't see any practical difference w.r.t. cybersecurity between "I blindly applied this pile of YAML to my production kubernetes clusters without looking at it" and "I blindly downloaded and ran this computer program on my CI runner without looking at it".

A supply chain attack on the former means that your environment is compromised. So does the latter.

armchairhacker• 2 years ago

To me YAML seems like the CoffeeScript of JSON, and unlike CoffeeScript I don’t understand why people are still using it.

I guess XML and JSON are too verbose. But YAML is so far in the opposite direction, we get the same surprise conversions we’ve had in Excel (https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-fr...). Why is “on” a boolean literal (of course so are “true”, “false”, as well as “yes”, “no”, “y”, “n”, “off”, and all capitalized and uppercase variants)? And people are actually using this in production software?

Then when you add templating it’s no longer readable and concise anyways. So, why? In JSON, you can add templating super easily by turning it into regular JavaScript: use global variables, functions and the like. I don’t understand how anyone could prefer YAML with an ugly templating DSL over that.

And if you really care about conciseness, there’s TOML. Are there any advantages of YAML over TOML?

chx2• 2 years ago

Are there any serious advantages of YAML over TOML?

Probably not but you forget YAML came out in 2001 where TOML came out in 2013. Neither are spring chickens but inertia is a hell of a thing. For example, Symfony supports YAML, XML and PHP definitions -- but not TOML. Symfony v2 simply predates TOML and they never got around to ditch YAML for TOML because it's not worth the bothering.

bayindirh13• 2 years ago

TOML is just an .ini file plus some syntactic and computing sugar. I can argue that TOML is actually way older than it is.

lucideer3• 2 years ago

I don’t understand why people are still using it

It's a good comaparator, there are indeed a lot of similarities, but I never understood why anyone ever used Coffeescript whereas I do think I have a solid understanding of why people use YAML.

It's more like Python than Coffeescript really: it's not just about simplicity & brevity, it's about terminators.

Whitespace-dependent languages are often a pain to format / parse / read in many ways - Python has survived this by the skin of its teeth by being extremely strict about indentation, both in terms of the parser & also community convention. YAML hasn't had this - it remains a mess.

However, both have that very attractive property of not requiring terminators, which can't really be understated.

if you really care about conciseness, there’s TOML. Are there any serious advantages of YAML over TOML?

TOML's got some good properties but its handling of structures with a depth > 1 is far from concise, and pretty terrible if I'm honest.

mjevans• 2 years ago

TOML's sections remind me of the directory part of a filename and keys files.

For the content that belongs in a typical configuration file this or the INI style roots are probably the most human approachable formats. For anything more complex maybe a database (such as SQLite?) is preferable past application bootstrap?

fsniper2• 2 years ago

Reading yaml has the enjoyment of reading a love letter where else json has the deterimental feeling of a solicitor email. For writing, yaml is like putting out the draft, you only focus on the meaning not care for else or the form, but for json it is like finishing up your thesis with hard defined structure.

anoopelias• 2 years ago

why anyone ever used Coffeescript

CoffeeScript was the front runner for 'Compile to JavaScript' technology. It was the first time we could write some sane frontend code.

Of course things like TypeScript came along and now we cannot unsee what we have already seen.

jeroenhd4• 2 years ago

I never understood why anyone ever used Coffeescript whereas I do think I have a solid understanding of why people use YAML.

When Coffeescript was invented, it was an advancement on top of the awful Javascript standards at the time. It never went anywhere because Javascript caught up, but Coffeescript had a good reason for existing.

Today, Coffeescript is a remnant of old frontends that nobody has bothered transpiling into Javascript yet, but back in the day it was a promising new development.

hitchstory3• 2 years ago

by being extremely strict about indentation, both in terms of the parser & also community convention. YAML hasn't

This is why I created StrictYAML. A lot of the pain of changing YAML goes away if you strictly type it with a schema but you keep the readability.

Counterintuitively that also includes most indentation errors - it's much easier to zero in on the problem if the error was "expecting status code or content on line 334, got response", for instance.

bencelaszlo• 2 years ago

Why is “on” a boolean literal (of course so are “true”, “false”, as well as “yes”, “no”, “y”, “n”, “off”, and all capitalized and uppercase variants)?

Norway is also "False".

foldr2• 2 years ago

Or more precisely, its country code 'NO' is false. I don't think there are any YAML parsers that parse the literal string 'Norway' as false.

Y_Y2• 2 years ago

And yet it doesn't recognize that the UK is false[0].

[0] https://en.wikipedia.org/wiki/Perfidious_Albion

bayindirh13• 2 years ago

People balk at XML, but its verbosity plus DTD allows it to pull tricks which you can't do on other things.

Well everything has its place, but XML is I think very well suited where you need to serialize complex things to a readable file, and verify it while being it's written and read back.

mcculley7• 2 years ago

Indeed. I get a lot of value out of my strongly typed XML documents. I generally have code that validates them during writing and after reading. Those who don’t understand XML end up learning why it is verbose when they eventually add all of the features they need to whatever half-baked format they are using.

CuriouslyC2• 2 years ago

Indeed, XML is a decent document language because of the quality of tools available and its power/flexibility. I hate when people use it for config files and other things that are usually human edited where readability is paramount though.

theshrike792• 2 years ago

XML + DTD + XMLSchema had things we're still figuring out to do with YAML ja JSON

You could easily generate an UI based on just the DTD and Schema that could be used to fill a perfectly valid XML file.

Validating incoming XML was a breeze, just give it to the validator class along with the DTD and Schema and boom, done.

PH95VuimJjqBqy13• 2 years ago

absolutely, 100%.

When I first encountered XSLT I seriously thought it was the most ridiculous thing I had ever seen. A frickin' programming language whose syntax was XML.

But then I learned it and I don't think I've ever seen another language that could do what XSLT could do in such a small amount of code. The trick was to treat it like a functional language (I got this advice from someone else and they were absolutely correct). Where most people got into trouble was thinking of it as an imperative language.

Pattern matching expressions is the kool kid on the block, but XSLT had that to the nth degree 20 years ago.

account-5• 2 years ago

Personally I prefer INI over nearly all configuration formats.

https://github.com/madmurphy/libconfini/wiki/An-INI-critique...

youerbt3• 2 years ago

I have seen this post on HN before and I wasn't received very well AFAIR.

But I can't help agreeing with its main point: so much complexity to support a few basic data types that are not sufficient for anything complex anyway.

LudwigNagasena• 2 years ago

Well, you can replace YAML with JSON and JS templating without changing the parser. So I guess that’s an advantage over TOML?

__jonas• 2 years ago

In general I am always confused that it lets you use strings unquoted, which is what allows for all these issues with ambiguity of the interpreted data type, Norway problem and all that.

It also just looks odd to me, I don't see why it's necessary to allow this.

hombre_fatal• 2 years ago

It’s great for end users who don’t understand what a string is or don’t have to play the game of finding the hanging single quote when they write the file by hand in a textarea.

On the opposite end of UX, there’s hand written JSON which is just too meticulous in some scenarios when people are writing config without editor support.

afavour2• 2 years ago

It’s because YAML is designed first for readability.

torginus2• 2 years ago

Dunno, to me YAML is the python of markup languages.

YAML is decent at handling things like nesting and arrays, while TOML sucks at it.

I don't dislike YAML that much.

That being said, we knew since the dawn of C macros that templating languages which are not aware of syntax, are AWFUL.

Likewise, writing Helm charts (the place I encountered YAML templating) is just horrible, but would be so much nicer is templates respected the YAML syntax tree and expanded at the right subnode, instead of being a text replace botch-jobs.

wkat42428• 2 years ago

The biggest issue I have with Yaml is that they forbid tabs.

Their argument is that tabs are shown differently in every editor which is actually something I like. When you're looking for something deeply nested you can reduce the tab distance a bit, when that's not needed you can increase it to improve visibility of nesting levels.

And forbidding it makes a one-keystroke action a two or four one.

I really don't understand the python/Yaml hate for tabs, and as a result I don't really use either.

mort96• 2 years ago

TOML has the inline table syntax with curlies, like JSON, and inline array syntax with brackets, also like JSON. It could support nesting pretty well.

Sadly, it doesn't support line breaks in the inline table syntax, so using inline tables for nesting is a PITA; inline tables are pretty much unusable for anything which doesn't fit within like 80-100 characters. Inline arrays can contain newlines however, so deeply nested arrays works well.

Newlines in inline tables will be coming in TOML 1.1, which will make TOML much better for deeply nested structures. Unfortunately, there will probably be many years until 1.1 is both actually released and well supported across the ecosystem.

And of course, inline tables can't be at the top level of the document, so TOML might still not be the best way to represent a single deelpy nested structure.

fkyoureadthedoc• 2 years ago

And then the second layer of hell, a DSL inside the YAML for something like an Azure DevOps pipeline. Truly awful.

vidarh2• 2 years ago

The worst thing with Helm charts is not the YAML, or even the text replace botch-jobs, but that they seem to think that a Go stacktrace is reasonable error reporting. I don't think I've ever worked with a tool with such awfully useless error messages.

But I agree, it'd be better if the template expansion was actually structural and not just text. The huge amount of "| indent 8" etc. in Helm charts is such a stench that by about the second time people encountered that they ought to have made a better template expansion mechanism top priority.

_ea1k4• 2 years ago

Yeah, that's why I prefer ytt over helm syntax. It isn't great syntax, but at least it is aware of what it is doing.

Having said that, yaml has some pretty obvious mistakes. It should have been a lot more prescriptive about data types. Not doing that creates a lot of unneeded confusion and weird bugs.

brobdingnagians• 2 years ago

Having on and no both be Boolean literals, but of opposite values sounds like a horrible decision, a typo doesn't result in a syntax error, but instead in a completely wrong semantic misconfiguration.

BenjiWiebe• 2 years ago

1 vs 0 is another typo of Boolean values with opposite meanings, in quite a few languages.

cmiller1• 2 years ago

Are there any advantages of YAML over TOML

YAML is older and more well supported. I'll explain to you why I ended up choosing YAML for the config files for a CLI utility written in Python that I maintain.

I initially chose TOML for many of the reasons mentioned here but before my first release I ended up switching to YAML. Python added support for reading TOML to the standard library in version 3.11, however it still requires you use an external library for writing. Do I use the built in library for reading and an external library for writing? A chunk of my users are on versions of Python older than 3.11 (generally Windows users who installed Python manually at some point), do I import a separate library for THEM to read the files but use the standard library if ver >= 3.11?

Now that I look at the state of things today I probably would add the tomlkit library to my setup file, but that wasn't very mature at the time, so I just used pyyaml. Changing it now would break compatibility with my older versions that use yaml config files, unless I maintained both paths... which I could do but it's just another source of complexity to worry about. These are relatively simple config files the user has to interact with manually so yaml works fine and I don't see any reason to change at this point.

amelius2• 2 years ago

Does anyone know which format Git uses? Is it YAML? Or TOML? Or something in between?

Gormo2• 2 years ago

Git uses its own ini-style conf format which diverges from TOML.

azangru• 2 years ago

unlike CoffeeScript I don’t understand why people are still using it

Gasp! Does this mean you know why people are still using CoffeeScript?

CuriouslyC2• 2 years ago

YAML is an amazing config language for simple to mildly complex configs. It's easier to read and write than JSON, and it only really breaks apart when you're heavily deviating from nested lists/dictionaries with string values. People use it everywhere because by the time it becomes painful you're already so invested it's not really worth the hassle of switching.

herpdyderp• 2 years ago

I, on the other hand, find it much harder to read and write even in very simple configs. I never know what the indent is supposed to be, I just press my spacebar until my editor stops complaining. I find it really hard to tell if a line is a new entry or a subset of the parent entry.

I'm sure if I used it more it'd become easier, but my whole team doesn't understand it either. Luckily we only need it for GitHub configurations.

coffeebeqn3• 2 years ago

It’s aesthetically pleasing for simple configs. I’m so used to writing JSON by hand by now I don’t find it much easier. At least I never have to think about how a value is going to be interpreted from a JSON since it has a decent subset of types and I can visually tell what it is

nobleach• 2 years ago

Until you find a snippet of config you want to copy into your `application.yml` in Spring or Quarkus (Java frameworks). If it doesn't paste in cleanly (and it rarely ever does) you'll need to go research the schema and find out where to put things. Meanwhile, if you're using a normal `application.properties` file, after you've finished pasting, you can go on with your life.

jampekka5• 2 years ago

I'm still using CoffeeScript whenever I can. It has one of the nicest syntaxes out there, a lot of code fits to one screenful, the logic of the code is easier to see without the clutter of unnecessary syntax and it's a joy to write too.

YAML is probably used for similar reasons.

I don't understand why people want redundant verbose syntax that makes reading and writing code harder. And sadly don't anymore expect anyone to really explain it based on anything tangible.

dewbrite3• 2 years ago

I'm glad I'm not the only one. I prototyped an SPA recently with mithril.js and CoffeeScript and I think there's really something magical about that combo.

Oddly enough I can't stand writing python or js. I do almost all of my actual programming in Rust, because I adore the type system.

dorfsmay• 2 years ago

I wish jdon5.org were adopted more widely, it's JSON,safe because it cannot be executed, but with comments and trailing comma!

Note: YAML is a superset of JSON, which means that any YAML reader can read JSON.

jgalt212• 2 years ago

why people are still using it.

If support for JSON with comments was more widely available / in use, we'd use that. But it's not, so we don't.

foofie• 2 years ago

I guess XML and JSON are too verbose. But YAML is so far in the opposite direction, (...)

YAML is a far better format in terms of being human readable and editable, and supports features such as node labels and repeated nodes that turn into killer features when onboarding YAML parsers into applications.

jongjong• 2 years ago

CoffeeScript is the worst thing that ever happened to the software industry.

CoffeeScript fooled developers into thinking that transpilation was free and had absolutely no downsides whatsoever. The advantages of CoffeeScript over JavaScript were so incredibly marginal. I've never heard a single good argument about why it was worth adding a transpilation step and all the complexity that came with it.

I think even TypeScript isn't worth transpilation step and bundling complexity these days, especially not when modern browsers allow you to efficiently preload scripts as modules and bypass bundling entirely.

About YAML. It's also not worth it though it's not quite as infuriating as CoffeeScript. The advantage of JSON is that it's equally as human-friendly as it is software-friendly. YAML leans more towards human-friendliness and sacrifices software friendliness. For instance, you can't cleanly express YAML on a single line to pass to a bash command as you can with JSON. It's just one additional format to learn and think about which doesn't add much value. Its utility does not justify its existence.

jalapenos• 2 years ago

I guess the real mystery is why so many tech types speak like a infant having a tantrum, about some esoteric trivia, and then have hordes of their kind come and vigorously head-knod it, and all involved think virtue is being done.

People started using things like YAML, obviously, because it reads closer to natural language. It's like a nested bullet list, which everyone can easily read. Readability is important to people. It's why we don't all still write C and Perl.

So it's one thing to say "I think people should be careful about prioritizing readability over precision especially for production systems". It's another to do this narcissistic dramatic faux-incomprehension implying the markup language gained the popularity it did because everyone's stupider than you.

runlaszlorun• 2 years ago

I guess the real mystery is why so many tech types speak like a infant having a tantrum, about some esoteric trivia, and then have hordes of their kind come and vigorously head-knod it, and all involved think virtue is being done.

Ha, great line. And you caught me mid-tantrum and mid-head nod. :)

Helmut10001• 2 years ago

I think YAML is for code what Markdown is for Text: It is easy to read and _can_ produce the same or equal output that more strict and extensive languages. Easy readability makes this tradeoff acceptable for most.

wirrbel• 2 years ago

HJson https://hjson.github.io seems a nice 'in-between' between YAML and JSON without the indentation-based syntax, so closer to the JSON side but with comments and less quotes.

What I don't really get is why the cloud providers / tooling implementors have never drafted up a "YAML-light" that just throws out the rarely-used headache-inducing syntax elements.

AndyKluger4• 2 years ago

Hjson is pretty nice.

Two YAML-light style projects are StrictYAML (a Python library), and NestedText (an alternative spec with only string, list, and dict).

loloquwowndueo• 2 years ago

Oh toml is atrocious and it’s a nightmare trying to understand nesting with all those repeated keys and double brackets.

Cacti• 2 years ago

For everyone who hates YAML, we extend YAML to another use.

It’s like without rules none of you show any common sense. Who cares what the spec says? Obviously you shouldn’t use “oN” as boolean true.

eviks7• 2 years ago

But YAML is so far in the opposite direction, we get the same surprise conversions we’ve had in Excel

This is optional. Besides using a better parser that uses the spec that's long fixed a lot of these listed in the article, another way to avoid the issue is adding more verbosity (that would still not match XML nor JSON).

You don't have this option in XML/JSON, you can't remove all that useless markup (and leave it only when it's useful)

Why is “on” a boolean literal

Because that's what humans use to denote booleans

dragonwriter3• 2 years ago

Why is “on” a boolean literal (of course so are “true”, “false”, as well as “yes”, “no”, “y”, “n”, “off”, and all capitalized and uppercase variants)?

”on”, ”off”, ”yes”, ”no”, “y”, and ”n”, and case variants thereof, are not boolean literals in YAML since YAML 1.2 (2009).

tristan9572• 2 years ago

As far as I know, not even libyaml supports 1.2. What YAML parsing libraries support 1.2?

maximilianburke• 2 years ago

I wonder if we would even be using YAML or TOML to the degree we are now if JSON had support for trailing commas and comments.

Rumudiez• 2 years ago

JSON5 has both of these

slowmovintarget• 2 years ago

Obligatory: https://github.com/edn-format/edn

wim• 2 years ago

Right, I also don't understand why it's considered a feature of many of these languages to introduce so many ways of doing the same thing. Like the boolean example, but also having three different ways to express a list or dictionary? It's the classic Robustness principle which makes it less robust, making reading and parsing more complicated. How about just allowing one syntax and error if it's not according to spec.

dathinab• 2 years ago

XML also has some other issues (no typing, to many ways to have maps but non seems to be the correct way etc.)

JSON just isn't mean to be written by humans (no comments).

But YAML is just horrible, like the whole accidental mistyping issues (NO => false) are just horrible and not acceptable IMHO. That it's a pretty complex thing doesn't help either.

I honestly don't understand why we (e.g. github actions) still use YAML for new thinks even knowing all the issues especially if we, there are many other well suited decent but less wide spread alternatives.

ravenstine• 2 years ago

I've had few to no issues when using YAML for docker-compose.yml files. This isn't to say that use of YAML can't be problematic, but I don't believe it's necessarily bad at all for configuration.

So, why? In JSON, you can add templating super easily by turning it into regular JavaScript: use global variables, functions and the like. I don’t understand how anyone could prefer YAML with an ugly templating DSL over that.

That's a valid use case when the target user is the software developer themself, but access to the language runtime is not something that should be accessible to a technical but non-maintainer user. Granted, it's plausible that a "template" JSON can be defined, which would be spread over a JSON-formatted configuration, but what YAML allows the user to do is define "templates" within the configuration itself and control over where those template structures are extended.

When the user is a developer maintaining a software project, they should probably just use JavaScript for configuration, and not JSON files, except when there's a possibility that the configuration can be intercepted.

zamalek6• 2 years ago

The problem goes deeper. I can't remember who coined the term, but all "implerative" (imperative declarative) languages share the same issue. I don't care if it's JSON, XML, TOML, or YAML, we shouldn't be interpreting markup/data languages. GitHub actions are a good example of everything wrong with implerative languages.

Use a real programming language, you can always read in JSON/YAML/whatever as configuration. Google zx is a good example of this done right, as is Pulumi.

Kris Nóva said it best: "All config drifts towards Turing completion."

hosh3• 2 years ago

I agree with all of this.

If we take it one step further though and think about portability of configuration, I think that is one of the reasons we end up with operators.

Osiris• 2 years ago

For tools that allow configuration in either JSON or Javascript (like eslint), I prefer the JS version. The syntax is similar but has much more flexibility, like being able to use environment variables or add comments.

Pulumi was also a good tool when I was doing kubernetes deployments.

sixdimensional3• 2 years ago

"Implerative" - thank you for this, this is the term I've been searching for to describe the weird blending of the two things.. I immediately Googled it and saw that it has previous uses as well, I would love to know who originated the concept. I see so many times, confusion and arguing about what is imperative and declarative, to the point where I question the value of the terms any longer.

FWIW, I have flirted with my own DSL implementations in a few cases. Certainly, language design is much more complex, but I also felt that once you understand enough of EBNF/parser generators (and some of the simpler alternatives), this is a very powerful option as well.

unshavedyak5• 2 years ago

Oh man, i have a similar issue with NixLang. Though i know it's not "implarative". Many days i just want to write Nix in my preferred language. I wish Nix had made a simple JSON based IO for configuration, because then i could see what the output of something is - and generate the input state from some other language.

Really frustrating. Nix works.. but i just don't see the value, personally. And this is after living on NixOS for ~3 years now, with 4 active Nix deploys in my house.. i just don't like the language.

shortrounddev22• 2 years ago

I've always wondered why we seem to have implemented a whole programming language in yaml or json for so many CI/CD systems rather than just writing quick python scripts to describe the logic of a particular build step, then MAYBE using a JSON or XML file to enumerate the build steps and their order, like:

    build: src/build.py
    test: src/test.py

hosh3• 2 years ago

YAML, or TOML or Json, I think the more problematic issue is using templating at all instead of generating them.

xena• 2 years ago

We're still using the CoffeeScript of JSON because YAML's UX improvments haven't been brought into the upstream JSON spec like CoffeeScript's UX improvements were brought into JavaScript.

PH95VuimJjqBqy13• 2 years ago

stronly agree, I came to that conclusion before k8's even existed because I myself thought to use it as a configuration file format and the second I started realizing some of the ambiguity in it's syntax I walked away from it.

The only thing I disagree with is that Coffeescript is still useful. I had the same reaction to Coffeescript that I had with yaml, Coffeescript _never_ had any real point outside of a segment of people preferring to write javascript in Ruby syntax. The biggest issue Coffeescript had is that debugging meant reading through the javascript anyway so you never really got away from javascript.

I'm a fan of either using a full-blown programming language or ini files, and yes I realize that seems insane to many people but at the end of the day ini files are stupidly easy to edit and if you can get away with not needing a full-blown turing complete language then convention based ini files are vastly easier on the human than yaml or json.

I'm either a greybeard that never got with the times or I'm a rebel, probably depends on who you talk to.

smaudet7• 2 years ago

I'm a fan of either using a full-blown programming language or ini files

How do you persist complex multi-object state? Think nested lists of objects with references to one another.

If your answer is still "ini files", I'm sure it can be done, but only with a lot of custom-rolled code...xml/json(even yaml) for all their issues provided a code-free way of persisting this all - either through use of marshalling (xml) or json/yaml.load().

quotemstr• 2 years ago

This is all because people refuse to use JSON parsers that allow comments

khimaros• 2 years ago

StrictYAML solves most if not all of these problems https://hitchdev.com/strictyaml/features-removed/

AndyKluger4• 2 years ago

StrictYAML is great (and the author is in these comments!), but ultimately it's one specific library, not a format spec, so to depend on it for a project you need every person/tool doing the writing/parsing to commit to use that library (and the programming language it was written for).

Again, it's a great project, but I wanted something similar that is a language-agnostic format specification, so moved on to using NestedText wherever I can.

throwawaymaths• 2 years ago

YAML is fine if you don't do weird stuff with it. (And some stupidity like the Norway problem) A good example is OpenAPI schemas, which are quite legible in the YAML.

TOML has some nasty edge cases like top level arrays, arrays of objects under a key, etc.

mplewis• 2 years ago

The answer is simple: JSON doesn’t have comments, XML isn’t human writable, and TOML isn’t well-supported by common tooling.

ForHackernews2• 2 years ago

YAML has comments and is easier to read and write for humans than JSON.

gregwebs2• 2 years ago

We are switching to cuelang [1]. IMHO it is better designed than Jsonette. Since Kubeenetes already has state reconciliation, the only thing missing in this setup is deletion. But that can now be accomplished with the prune feature. [2]

[1] https://cuelang.org/docs/integrations/k8s/

[2] https://kubernetes.io/blog/2023/05/09/introducing-kubectl-ap...

lantry• 2 years ago

I can second cuelang. We started using it at work and it's so nice. Some of the error messages are a little hard to decipher, but that's acceptable because it catches so many errors up front. The few times I have to write yaml directly, it now feels so tedious in comparison.

Hendrikto• 2 years ago

Why do we need a special purpose language for this? Just for the slightly nicer syntax? Is that worth learning a completely new language?

BiteCode_dev6• 2 years ago

I'm a big fan of Python as configuration for my own projects, but:

- It requires discipline for devs to keep the conf declarative. Discipline is not automatically enforceable, so it's prone to failure.

- No guarantee of reproducibility.

- You need a Python VM (or a starlark interpreter if that's what you like). It's a big constrain.

- If you are a Saas provider, accepting Python as input is really hard to secure.

vlfig2• 2 years ago

a Dhall configuration file will never:

- throw an exception

- crash or segfault

- accept malformed input

- produce malformed output

- hang or time out

in https://docs.dhall-lang.org/discussions/Safety-guarantees.ht...

Still a fan of Python for configuration?

BiteCode_dev6• 2 years ago

This is where I usually pitch in with "Have your heard of CUELang, our lord and savior?": https://cuelang.org/

- Not turing complete yet sufficiently expressive to DRY

- Define schema and data with the same language, in a separate or same file. With union types.

- Generate YAML or JSON. Can validate itself, or a YAML or JSON file.

The biggest drawback being the only implementation is currently in go, meaning you may have to subprocess of ffi.

1oooqooq• 2 years ago

we have a pipeline that ingest very concise cuelang files.

then it generates json files for each application for a tool that will create xml definitions which then are applied to a xls which the architects own, to spit out a yaml that we use to apply our helm charts. the charts deploy a k8s client which then interact with the main cluster via json using the api.

took a while, but we are using the best tool for each job.

ljm• 2 years ago

just throw in a kafka cluster so you can pipe each step through an event bus and you'll have an enterprise-grade deployment setup

BiteCode_dev6• 2 years ago

You used JSON twice, how casual.

Your API should clearly be using protobuf.

planede2• 2 years ago

How does it compare to dhall?

arianvanp• 2 years ago

Dhall's lack of any form of type inference makes it very verbose and difficult to refactor in my opinion. (I'm the author of dhall-kubernetes and never ended up using it in production; funnily enough). Dhall is also extremely slow. We had kubernetes manifests that took _minutes_ to type-check. Cue is basically instant. This matters a lot to me.

I find cue very ergonomic. Also it treating both types and values as values is very neat. You write your types and your values in the same syntax and everything unifies neatly. but I sometimes miss its lack of functions.

Cue also being to ingest protobuf definitions and openapi schemas makes it very quick and easy to integrate with your project. Have a new Kubernetes CRD you want to have type-checked in cue? No problem just run `cue get go k8s.io/api/myapi/v1alpha1` and off you go you have all your type definitions imported from Go to Cue!

Especially for k8s this makes for very fast development and iteration cycle.

I've wanted to take a look at https://nickel-lang.org/ which is a "what if cue had functions" language. but to be honest Cue kind of serves my needs.

gregwebs2• 2 years ago

Cue was designed very much with k8s in mind and developed tutorials and integrations for it early on. Dhall was designed pre-k8s. Dhall had to introduce a defaults feature: before that it was completely unusable for k8s. Dhall has functions, which are natural to programmers- particularly from an FP background, Dhall would be trivial to start using. Whereas it takes some getting used to cue's unifications- but there is enough documentation and integration for getting going with k8s to make up for it. Dhall has unique features for stably importing configurations from remote locations.

lasermike026• 2 years ago

YAML is for human. JSON is for computers. I love YAML.

revskill• 2 years ago

But we are no longer human, we are bots.

tovej2• 2 years ago

We're talking about templating and generating files, but it seems like everyone has just collectively forgotten about M4?

Yes, it can be unsafe if you're not careful, but if you need to bang out a quick prototype it's the best tool there is. It's part of POSIX, and so it will always be available, the language is dead simple, and you can generate any text you want with it.

I wouldn't use it with YAML, but I would probably never template YAML in the first case: just generate JSON and feed it through `yq -y` if you need a quick YAML generator.

somewhereoutth• 2 years ago

Meanwhile in JavaScript land, config is simply another js file, with all the Object and Array literal goodness that that gets us, and the full language environment backing it up.

hiAndrewQuinn2• 2 years ago

In my case, I'm templating YAML because the Obsidian Templater plugin can read YAML frontmatter it asks you for and then fill in a Markdown file with the Mad Libs you choose to populate it with.

I understand this is a niche use case.

skywal_l• 2 years ago

For those who do not know it yet, the now classic noyaml site: https://noyaml.com/

Draiken6• 2 years ago

I agree that YAML templating is kind of insane, but I will never understand why we don't stop using fake languages and simply use a real language.

If you need complex logic, use a programming language and generate the YAML/JSON/whatever with it. There you go. Fixed it for you.

Ruby, Python, or any other language really (I only favor scripting ones because they're generally easier to run), will give you all of that without some weird pseudo-language like Jsonnet or Go templates.

Write the freaking code already and you'll get bitten way less by obscure weird issues that these template engines have.

Seriously, use any real programing language and it'll be WAY better.

krapp• 2 years ago

You shouldn't need the full complexity and power of a Turing complete programming language to do config. The point of config is to describe a state, it's just data. You don't need an application within an application to describe state.

Inevitably, the path of just using a programming language for config leads to your config becoming more and more complex until it inevitably needs its own config, etc. You wind up with a sprawling, Byzantine mess.

sigwinch284• 2 years ago

your config becoming more and more complex until it inevitably needs its own config, etc. You wind up with a sprawling, Byzantine mess.

We're already there with Helm.

People write YAML because it's "just data". Then they want to package it up so they put it in a helm chart. Then they add variable substitution so that the name of resources can be configured by the chart user. Then they want to do some control flow or repetitiveness, so they use ifs and loops in templates. Then it needs configuring, so they add a values.yaml configuration file to configure the YAML templating engine's behaviour. Then it gets complicated so they define helper functions in the templating language, which are saved in another template file.

So we have a YAML program being configured by a YAML configuration file, with functions written in a limited templating language.

But that's sometimes not enough, so sometimes variables are also defined in the values.yaml and referenced elsewhere in the values.yaml with templating. This then gets passed to the templating system, which then evaluates that template-within-a-template, to produce YAML.

SOLAR_FIELDS2• 2 years ago

Yeah, YAML is good at declarative things. It’s when you start using it imperatively eg CI/CD is when it really starts to get ugly.

Draiken6• 2 years ago

The complexity is already there. If you only need static state like you say, then YAML/JSON/whatever is fine. But that's not what happens as software grows.

You need data that is different depending on environments, clouds, teams, etc. This complexity will still exist if you use YAML, it'll just be a ridiculous mess where you can break your scripts because you have an extra space in the YAML or added an incorrect `True` somewhere.

Complexity growth is inevitable. What is definitely avoidable is shoving concepts that in fact describe a "business" rule (maybe operational rule is a better name?) in unreadable templates.

Rules like: a deployment needs add these things when in production, or change those when in staging, etc exist whether they are hidden behind shitty Go templates or they are structured inside of a class/struct, a method with a descriptive name, etc.

The only downside is that you need to understand some basics of programming. But for me that's not a downside at all, since it's a much more useful skill than only knowing how to stitch Go templates together.

gtirloni3• 2 years ago

You don't need an application within an application to describe state.

As shown in the article, you apparently do.

kevincox3• 2 years ago

A really good solution here is to use a full programming language but run the config generator on every CI run and show the diff in review. This way you have a real language to make conditions as necessary but also can see the concrete results easily.

Unfortunately few review tools handle this well. Checked-in snapshot tests are the closest approximation that I have seen.

matharmin• 2 years ago

I have a recent example of rolling out IPv6 in AWS:

1. Create a new VPC, get an auto-assigned /56 prefix from AWS.

2. Create subnets within the VPC. Each subnet needs an explicitly-specified /64 prefix. (Maybe it can be auto-assigned by AWS, but you may still want to follow a specific pattern for your subnets).

3. Add those subnet prefixis to security / Firewall rules.

You can do this with a sufficiently-advanced config language - perhaps it has a built-in function to generate subnets from a given prefix. But in my experience, using a general-purpose programming language makes it really easy to do this kind of automation. For reference, I did this using Pulumi with TypeScript, which works really well for this.

IggleSniggle• 2 years ago

It happens because config is dual purpose: its state, but it's also the text-UI for your program. It spirals out of control because people want the best of it being "just text" and being a nice clean UI.

worldsayshi4• 2 years ago

We need turing completeness in the strangest of places. We can often limit these places to a smaller part of the code. But it's really hard to know beforehand where those places will occur. Whenever we think we have found a clear separation we invent a config language.

And then we realize that we need scripting so we invent a templating language. Then everybody looses their minds and invents 5 more config languages that surely will make us not need the templating language.

Let's just call it code and use clever types to separate turing and non-turing completeness?

pid-1• 2 years ago

That's not my experience after using AWS CDK since 2020 in the same company.

Most of our code is plain boring declarative stuff.

However, tooling is lightyears ahead of YAML (we have types, methods, etc...), we can encapsulate best practices and distribute as libs and, finally, escape hatches are possible when declarative code won't cut.

avianlyric• 2 years ago

That kind of ignores the entire pipeline involved in computing the correct config. Nobody wants to be manually writing config for dozens of services in multiple environments.

The number of configurations you need to create is multiplicative, take the number of applications, multiply by number of environments, multiply by number of complete deploys (i.e. multiple customers running multiple envs) and very quickly end up with an unmanageable number of unique configurations.

At that point you need a something at least approaching Turing completeness to correctly compute all the unique configs. Whether you decide to achieve that by embedding that computation into your application, or into a separate system that produces pure static config, is kind of academic. The complexity exists either way, and tools are needed to make it manageable.

myaccountonhn• 2 years ago

I agree, I think a language like dhall (https://dhall-lang.org/) strikes a good balance.

speleding• 2 years ago

Completely agree, my wish is that anything that risks getting complex uses a Ruby-based DSL.

For example, I like using Capistrano, which is wrapper around rake, which is a Ruby based DSL. That means that if things get tricky I can just drop down to using a programming language. Split stuff into logical parts that I load where needed and, for example, I can do something like YAML.load(..file..).dig('attribute name') or JSON.load from somewhere else.

Yes, you risk someone building spaghetti that way, but the flip side is that a good devops can build something much easier to maintain than dozens of YAML and JSON files, and you get all the power from your IDE and linters that are already available for the programming language, so silly syntax errors are caught without needing to run anything.

mark_and_sweep2• 2 years ago

Agreed, and I almost feel silly for pointing this out, but for writing JSON (JavaScript Object Notation), I'd recommend using JavaScript...

jeroenhd4• 2 years ago

For JSON I'd stick with Typescript to be honest. You end up executing Javascript and producing Javascript-native objects, but the typing in Typescript to ensure the objects you produce are actually valid will save a lot of debugging.

Draiken6• 2 years ago

JS is actually not that great for this IMO. You probably need an NPM package to even deal with YAML because JS has a shitty standard library.

Sticking to a scripting language with a strong standard library is way better.

Any unix system can get Ruby/Python and read/write YAML/JSON immediately without caring too much about versions.

Of course in today's upside down world most developers seem to only know JS, so it would at least be "familiar". Still a bad choice in my view.

The way this industry is going, give it a few years and we'll have React-Kubernetes for generating templates. And I wish I was joking.

andix2• 2 years ago

Helm would probably benefit from something like JSX for YAML/JSON. Just being able to script a chart instead of this templating hell.

otabdeveloper4• 2 years ago

I heard you liked configuration languages, so I made this configuration language for your configuration language generation scripts. It supports templates, of course.

c0l0• 2 years ago

Relevant article (2012): http://mikehadlow.blogspot.com/2012/05/configuration-complex...

martsa1• 2 years ago

This. It's why things like Cloud Development Kit and Pulumi are quite interesting to me.

xwowsersx3• 2 years ago

Pulumi is enticing because it allows you to write in your preferred language and abandon HCL, but it is strictly worse in my opinion. IaC should be declarative in my opinion. That allows for greater predictability, reproducibility and maintainability. In general, I think wanting to use Python or Ruby or whatever language you're going to use with Pulumi is not a good basis for choosing the tool.

There are many graveyards filled with places that tried to start writing logic into their IaC back in the Chef/Puppet era and made a huge mess that was impossible to upgrade or maintain (recall that Chef is more imperative/procedural, whereas in Puppet you describe the desired end state). The Chef/Pulumi approach can work, but it requires one person who is draconian about style and maintenance. Otherwise, it turns into a pile of garbage very quick.

Terraform/Puppet's model is a lot more maintainable for longer terms with bigger teams. It's just a better default for discouraging patterns that necessitate an outsized investment to maintain. Yes HCL can be annoying and it feels freeing to use Python/TS/whatever, but pure declarative code prevents a lot of spaghetti.

jen20• 2 years ago

The existence of the YAML language for Pulumi and the CDK for TF both confound this explanation, it’s just not grounded in reality.

flanked-evergl• 2 years ago

Pulumi is declarative. The procedural code (Python, Go, etc) generates the declaration of the desired state, which Pulumi then effects on the providers.

HCL is also not pure declarative code either. It can invoke non-declarative functions and can do loops based on environment variables, so in that sense there is really no difference between Pulumi and Terraform. The only real difference is that HCL is a terrible language compared to say Python.

I'm actually fairly sure HCL is Turing complete, it has loops and variables. But even if it is not all the way turing complete it's pretty close.

blandflakes• 2 years ago

The fact that HCL has poor/nonexistent multi-language parsing support makes building tooling around terraform really annoying. I shouldn't have to install Python or a Go library to read my HCL.

i_play_stax• 2 years ago

The limitations of HCL are actually a good thing!

I have never seen Pulumi or CDKTF stuff work well. At some point are you simply writing a script and abandoning the advantages of a declarative approach

lamontcg5• 2 years ago

recall that Chef is more imperative/procedural, whereas in Puppet you describe the desired end state

Chef's resources and resource collection and notifications scheme is entirely declarative. And after watching users beat their heads against Chef for a decade the thing that users really like is using declarative resources that other people wrote. The thing that they hate doing is trying to think declaratively themselves and write their own declarative resources or use the resource collection properly. People really want the glue code that they need to write to be imperative and simple.

The biggest issue that Chef had was the "two-pass parsing" design (build the entire resource collection, then execute the entire resource collection) along with the way that the resource collection and attributes were two enormous global variables which were mutable across the entire collection of recipe code which was being run, and then the design encouraged you to do that. And recipes were kind of a shit design since they weren't really like procedures or methods in a real programming language, but more like this gigantic concatenated 'main context' script. Local variables didn't bleed through so you got some isolation but attributes and the resource collection flowing through all of them as god-object global variables was horrible. Along with some people getting a bit too clever with Ruby and Chef internals.

I had dreams of freezing the entire node attribute tree after attribute file processing before executing resources to force the whole model into something more like a functional programming style of "here's all your immutable description of your data fed into your functional code of how to configure your system" but that would have been so much worse than Python 2.7-vs-3.0 and blown up the world.

Just looking at imperative-vs-declarative is way too simplistic of an analysis of what went wrong with Chef.

mrloba• 2 years ago

I agree, and I just want to highlight what you said about generating a config file. It's extremely useful to constrain the config itself to something that can go in a json file or whatever. It makes the config simpler, easier to consume, and easier to document. But when it comes to _writing_ the config file, we should all use a programming language, and preferably a statically typed language that can check for errors and give nice auto complete and inline documentation.

I think aws cdk is a good example of this. Writing plain cloudformation is a pain. CDK solves this not by extending cloudformation with programming capabilities, but by generating the cloudformation for you. And the cloudformation is still a fairly simple, stable input for aws to consume.

asimpletune• 2 years ago

This is how config actually works in Scala.

karlicoss• 2 years ago

I argued that point in my article some time ago https://beepb00p.xyz/configs-suck.html also HN discussion at the time news.ycombinator.com/item?id=22787332

duped• 2 years ago

I think language embedding is kind of a lost architecture in modern stacks. It used to be if you had a sufficiently complex application you'd code the guts in C/C++/Java/Whatever and then if you needed to script it, you'd embed something like a LISP/Lua/whatever on top.

But today, you have plenty of off-the-shelf JSON/TOML/YAML parsers you can just import into your app and a function called readConfig in place of where an embedded interpreter might be more appropriate.

It's just easier for developers to add complexity to a config format rather than provide a full language embedding and provide bindings into the application. So people have forgotten how to do it (or even that they can do it - I don't think it occurs to people anymore)

angarg12• 2 years ago

I'm very happy using Typescript to templatize JSON. You can define a template as a class, compose them if needed, and when you are done, just write an object to a file.

kimbernator• 2 years ago

I once took a job that involved managing Ansible playbooks for an absolutely massive number of servers that would run them semi-regularly for things like bootstrapping and patching. I had used Chef before for a similar task, and I loved it because it's just ruby and I could easily define any logic I wanted while using loops and proper variables.

I understand that Ansible was designed for non-programmers, but there is no worse hell for someone who is actually familiar with basic programming than being confined to the hyper-verbose nonsense that is Jinja templating of Ansible playbooks when you need to have a lot of conditional tasks and loops.

linuxftw2• 2 years ago

Ansible has a great module/plugin system. It's trivial to handle complex tasks or computations in a custom module or action.

awesomeMilou2• 2 years ago

I agree. And to make matters worse, the DSL on YAML has grown so large in features, it may as well be a programming language now.

aarmenaa• 2 years ago

Chef vs Ansible was the first example that popped into my mind. I had a very love/hate relationship with Chef when I used it, but writing cookbooks was definitely one of the good parts.

jayd16• 2 years ago

Because the security surface of "any language" is tricky and most (all?) popular languages do not have nice data literal syntax better than JSON and YAML.

lamontcg5• 2 years ago

I agree that YAML templating is kind of insane, but I will never understand why we don't stop using fake languages and simply use a real language.

The problem is language nerds write languages for other language nerds.

They all want it to be whatever the current sexiness is in language design and want it to be self-hosting and be able to write fast multithreaded webservers in it and then it becomes conceptually complicated.

What we need is like a "Logo" for systems engineers / devops which is a simple toy language that can be described entirely in a book the size of the original K&R C book. It probably needs to be dynamically typed, have control structures that you can learn in a weekend, not have any threading or concurrency, not be object oriented or have inheritance and be functional/modular in design. And have a very easy to use FFI model so it can call out to / be called from other languages and frameworks.

The problem is that language nerds can't control themselves and would add stuff that would grow the language to be more complex, and then they'd use that in core libraries and style guides so that newbies would have to learn it all. I myself would tend towards adding "each/map" kinds of functions on arrays/hashmaps instead of just using for loops and having first class functions and closures, which might be mistakes. There's that immutable FP language for configuration which already exists (i can't google this morning yet) which is exactly the kind of language which will never gain any traction because >95% of the people using templated YAML don't want to learn to program that way.

anon2913• 2 years ago

I mean... Nix satisfies every single one of what you mentioned and people say its too complicated. It's literally just the JSON data structure with lambdas, which really is basic knowledge for any computer scientist, and yet people complain about it.

It's fairly straightforward to 'embed' and as a bonus it generates json anyway (you can use the Nix command line to generate JSON). Me personally, I use it as my templating system (independent of nixpkgs) and it works great. It's a real language, but also restrictive enough that you don't do anything stupid (no IO really, and the IO it does have is declarative, functional and pure -- via hashing).

In Nix's favor:

1. Can be described in a one page flier. An in-depth exhaustive explanation of the language's features is a few pages (https://nixos.org/manual/nix/stable/language/)

2. dynamically typed

3. Turing complete and based on the lambda calculus so has access to the full suite of functional control structures. Also has basic if/then/else statements for the most common cases and for intuition.

4. no threading, no concurrency, no real IO

5. definitely not object-oriented and no inheritance

6. It is functional in design and has an extremely thin set of builtins

7. FFI model is either embed libnix directly (this does not require embedding the nix store stuff, which is a completely separate modular system), or use the command line to generate json (nix-instantiate --eval --json).

Note: do not confuse nixpkgs and NixOS with the nix language. The former is a system to build linux packages and entire linux distributions that use the latter as a configuration language. The nix language is completely independent and can be used for whatever.

hnlmorg2• 2 years ago

There’s plenty to choose from that support embedding: Python, Perl, Lua. Heck, even EMCAScript (JavaScript, VBA, etc).

As another commenter rightfully stated, this used to be the norm.

I wouldn’t say LOGO is the right example though. It’s basically a LISP and is tailored for geometry (of course you can do a heck of a lot more with it but its strength is in geometry).

int_19h7• 2 years ago

What we need is like a "Logo" for systems engineers / devops which is a simple toy language that can be described entirely in a book the size of the original K&R C book.

I would argue that Tcl is exactly that. It's hard to make things any simpler than "everything is a string, and then you get a bunch of commands to treat strings as code or data". The entire language definition boils down to 12 simple rules ("dodekalogue"); everything else is just commands from the standard library. Simple Tcl code looks pretty much exactly like a typical (pre-XML, pre-JSON, pre-YAML) config file, and then you have conditionals, loops, variables etc added seamlessly on top of that, all described in very simple terms.

pseudonom-• 2 years ago

Dhall is the FP config language you're thinking of, I think.

trealira2• 2 years ago

I think Scheme would work, as long as you ban all uses of call/cc and user-defined macros. It's simple and dynamically typed, and doesn't have built-in classes or hash maps. Only problem is that it seems like most programmers dislike Lisp syntax, or at least aren't used to it.

There's also Awk, although it's oriented towards text, and doesn't have modules (the whole program has to be in one file).

It probably wouldn't be that hard to make this language yourself. Read the book Crafting Interpreters, which guides you through making a toy language called Lox. It's close to the toy language you describe.

ParetoOptimal3• 2 years ago

What are your thoughts on:

- https://dhall-lang.org/ - https://toml.io/en/

eternityforest3• 2 years ago

I think I'd rather just have logicless templates than use anything dynamically typed...

Jinja2 makes a lot of sense when you're trying to make it hard to add bugs, and you also don't want everyone to have to learn Rust or Elixir or something.

It would be interesting to extend a template language with a minimal FP language that could process data before the templated get it.

ForHackernews2• 2 years ago

Throwing in a plug for https://dhall-lang.org/

Dhall is a programmable configuration language that you can think of as: JSON + functions + types + imports

valty4• 2 years ago

The problem with imperative languages in configs is that they become harder to read. Webpack configs always devolve into this.

We need better tooling to allow tracing a how final configuration values are being generated.

And a _live programming_ environment so we can see the final generated configuration in one view.

m4632• 2 years ago

I wonder if there isn't a place for both:

1. a full-blown language that can generate complex output

2. a declarative static data file

I hope I'm not just pulling my punches with #2

on the other hand, some complexity spirals out of control, especially when people use it without any need. Some great things come out of creating boundaries.

larve2• 2 years ago

Serendipity strikes as I'm implementing an emrichen interpreter in golang after getting too annoyed about templating YAML as a string.

The reasons I like yaml is that I can see the tree structure directly, and to my lisp brain it is extremely easy to read. Furthermore, in our age of LLMs, I find LLMs to be able to generate "correct" YAML more easily than JSON, since the tree depth is encoded in every line, and doesn't require matching larger structures. It also uses an actually significant amount less tokens.

I find it extremely easy to have LLMs generate decent DSLs by asking them to use a YAML output format, and found it very robust to generate code out of these (or generate an interpreter for the newly created DSL).

I didn't know about !tags until sunday, which is quite shameful, but I find that the emrichen solution is actually quite elegant, and really kind of feels like a lisp macro expander.

Overall, YAML is just good enough for me to get shit done, I can read and skim it quickly, LLMs do well with it, and it's easy to work with. It has aliases, multiline strings and some other Quality of life features built in.

https://github.com/con2/emrichen

efunnekol• 2 years ago

I totally agree with you on LLM usage. I have recently switched from JSON to YAML for requests and replies from LLMs (GPT-4 specifically) and I find it much better: fewer tokens used, more readable if you are looking at the http requests and responses and you can parse it on the fly in streaming responses. The last point lets you do visual updates for the user, which is pretty important if you need to wait 1+ minutes for the full response

larve2• 2 years ago

I'd be very curious to know what kind of previews/streaming YAML applications you are building with LLMs. I have building a v0.dev kind of thing with streaming update on my TODO list.

brainwipe• 2 years ago

Having XSLT flashbacks. Send help.

pachico• 2 years ago

I keep coming back to tanka (https://tanka.dev/) and hoping it had more traction in the industry.

sigwinch284• 2 years ago

Relevant: https://noyaml.com/

YAML and its ecosystem is full of footguns and ergonomics problems, especially when the length of the document extends beyond the height of a user's editor or viewport. Loss of context with indentation, non-compliant or unsafe parsers, and strange boolean handling to name a few.

It becomes even worse when people decide that static YAML data files should have variable substitution or control flow via templating. "Stringly-typed programming" if you will. If we all started writing JSON text templates I think a lot of people would rightly argue we should write small stdlib-only programs in Python, Typescript, or Ruby to emit this JSON instead of using templated text files. Then it becomes apparent that the YAML template isn't a static data file at all, but part of a program which emits YAML as output. We're already exposing people to basic programming if we're using YAML templates. People brew a special kind of YAML-templated devops hell using tools like Kustomize and Helm, each of which are "just YAML" but are full of idiosyncracies and tool-specific behaviour which make the use of YAML almost coincidental rather than a necessity.

Yes, sometimes people would prefer to look at YAML instead of JSON, in which case I suggest you use a YAML serialization library, or pipe output into a tool like `yq` so you can view the pretty output. In a pinch you could even output JSON and then feed it through a YAML formatter.

The Kubernetes community seems to have this penetrating "oh, it's just YAML" philosophy which means we get mediocre DSLs in "just YAML" which actually encode a lot of nuanced and unintuitive behaviour which varies from tool to tool.

Look at kyverno, for example: it uses _parentheses_ in YAML key names to change the semantics of security policies! https://kyverno.io/docs/writing-policies/validate/ . This is different to (what I think are the much better ideas of) something like kubewarden, gatekeeper, or jspolicy, which allow engineers to write their policies in anything that compiles to WASM, OPA, and Typescript/Javascript respectively.

We engineers, as a discipline, have decades of know-how building and using general purpose programming languages with type checkers, linters, packaging systems, and other tools, but we throw them all away as soon as YAML comes along. It's time to put the stringified YAML templates away and engage in the ecosystem of mature tools we already know to perform one simple task they are already good at: dumping JSON on stdout.

Let's move the control flow back into the tool and out of the YAML.

carlosrdrz• 2 years ago

Can someone help me understand what is the advantage of using jsonnet, cue, or something else vs a simple python script (or dialect, like starlark), when you have the need of dynamically creating some sort of config?

I've used jsonnet in the past to create k8s files, but I don't work in that space anymore. I don't remember it being better or easier than writing a python script that outputs JSON. Not even taking into account maintainability and such. Maybe I'm missing something?

ants_everywhere• 2 years ago

They're various points along the Turing complete config generator vs declarative config spectrum. Declarative config is ideal in lots of ways for mission critical things, but hard to create lots of because of boiler plate.

A turing-complete general purpose language is entirely unconstrained in its ability to generate config, so it's difficult to understand all the possible configs it can generate. And it's difficult to write policy that forbids certain kinds of config to be generated by something like Python. And when you need to do an emergency-rollback, it can be hard to debug a Python script that generates your config.

Starlark is a little better because it's deliberately constrained not to be as powerful as Python.

Jsonnet is, IIUC, basically an open source version of the borgcfg tool they've had at Google forever. My recollection is that Borgcfg had the reputation of being an unreadable nightmare that nobody understood. In practice, of course, people did understand it but I don't think anyone loved working with it.

Brian Grant, creator of Kubernetes, wrote up his thoughts on various config approaches in this Google doc: https://docs.google.com/document/d/1cLPGweVEYrVqQvBLJg6sxV-T....

IshKebab2• 2 years ago

I definitely wouldn't use Python because it isn't sandboxed, and users will end up doing crazy things like network calls in your config.

Starlark is a good option though.

People will talk about Jsonnet not being Turing complete, but IMO that is completely irrelvant. Turing completeness has zero practical significance for configs.

ekimekim• 2 years ago

To add to the sibling comments, after going from a jsonnet-based setup to a Typescript-based one (via pulumi), the biggest thing I missed from jsonnet was the native object merge operations which are very useful for this kind of work as it lets you say "I want one of these, but with these changes" even when the objects are highly nested, and you can specify whether to merge or override for each individual key.

But ultimately this was a minor issue and I think it's far more important that you use something like this (whether a DSL or a mainstream PL) and that you're not trying to do string templating of YAML.

rockwotj• 2 years ago

Honestly I find all these different config languages either too much to learn or they get too unwieldy quickly.

I have arrived that I think using typescript to generate JSON as being the ultimate solution.

Easy JSON support, optional typing, you already know it, and adding reusable functions and libraries is understandable. Just prevent external node modules, and have a tool that takes a typescript file with a default export of some JSON and renders the JSON to a string on stdout.

Sharlin• 2 years ago

I guess templating your configuration is just an instance of the old wisdom that any problem is solvable by adding a layer of indirection.

break_the_bank• 2 years ago

Obviously biased but we at Kurtosis are trying to solve this problem through Starlark.

We took Starlark added a few more of our instructions that make our Starlark container native. The complex Starlark definition supports

- Composition - you can import a remote definition and just use it - Decomposable - you can break things apart - Parametrizability - want one of a service and 10 of the other, just pass an argument - Portable - It runs pretty much anywhere

Our runtime takes the Starlark and creates environments in both Docker and Kubernetes; from one definition

Our CTO wrote this - https://docs.kurtosis.com/advanced-concepts/why-kurtosis-sta...

We are source available https://github.com/kurtosis-tech/kurtosis

Here is a popular environment definition - https://github.com/kurtosis-tech/ethereum-package that protocol developers use to setup custom Ethereum test environments

neallindsay2• 2 years ago

My personal philosophy is that string interpolation should not be used to generate machine-readable code, and template languages are just fancy string interpolation. We've all seen the consequences of SQL injection and cross-site scripting. That's the kind of thing that will keep happening as long as we keep putting arbitrary text into interpreters.

Yes, this means I don't think we should use template files to make HTML at all.

Alternatives to using template languages for HTML include Haml (for Ruby) and Pug (for JavaScript). These languages have defined ways to specify entire trees of tags, attributes, and text nodes.

If you don't like Python-style significant indentation, JavaScript has JSX. The HTML-looking parts of JSX compile down to a bunch of `createElement` expressions that create a web document tree. That tree can then be output as HTML if necessary.

Haml, Pug, and JSX are not template languages even though they can output HTML. Likewise, `JSON.stringify(myObj)` is not a template language for JSON. Generating machine-readable code should be done with a tool that understands and leverages the known structure of the target language when possible.

kevincox3• 2 years ago

This is the essence of the problem! Yaml and templates are just distractions. It just boils down to the fact that "string" is a very general type and we use it lazily.

My personal rule: Every time a value is inserted into a string it must be properly encoded.

I wrote a full blog post around this a while back https://kevincox.ca/2022/02/08/escape-everything/. But the TL;DR is that every string has a format which needs to be respected wether that be HTML, SQL or human-readable terminal output. Every time you put some value into a string you should be properly encoding it into that format. But we rarely do.

Izkata• 2 years ago

My personal rule: Every time a value is inserted into a string it must be properly encoded.

This is how Django templates have done it for over a decade. You have to go out of your way to tell it not to escape the values if for some reason you need that.

lioeters3• 2 years ago

Haml, Pug, and JSX are not template languages even though they can output HTML.

That's nonsense, unless we go by your idiosyncratic definition of what a template language is ("fancy string interpolation").

Haml (HTML Abstraction Markup Language) is a templating system that is designed to avoid writing inline code in a web document and make the HTML cleaner.
Pug – robust, elegant, feature rich template engine for Node.js
JSX is an XML-like syntax extension to ECMAScript without any defined semantics.

OK, I'd agree that JSX is not strictly a template language.

But in the end, all of these compile down to HTML. Not by string interpolation, but as a language that is parsed into a syntax tree, then rendered into HTML properly with an internal understanding of valid structure.

YAML with templating is fancy string interpolation, it's not a template language (or at least a poorly implemented one).

neallindsay2• 2 years ago

I am aware that Haml and Pug call themselves template languages, but they are not. In a template language, the source is a "template" that has some special syntax to fill in some bits. I don't think that's a very idiosyncratic definition. Pretty much any programming language can output a bunch of text, but most of them are not template languages. Java has XMLBuilder, but that doesn't make it a template language for outputting XML. But PHP is a template language, even though it's not recommended to use it that way anymore.

int_19h7• 2 years ago

Not all template languages are string template languages, though. If you consider PHP a templating language for text, for example, then by the same logic XQuery is a templating language for XML.

PaulHoule• 2 years ago

You’re going to hate absolutely everything involved in configuring Kubernetes because you’re configuring Kubernetes.

gtirloni3• 2 years ago

I love YAML and I curse it every single day that I'm working with Helm charts.

People ask me what I'd use to deploy apps on Kubernetes and I say I hate Helm and would still use it for a single reason: everybody is using it, I don't want to create a snowflake infrastructure that only I understand.

Still, back in the day I thought jsonnet would win this battle but here we are, cursing Helm and templates. That's the power of upstream decisions.

resters• 2 years ago

In my view, the presence of YAML templating is a red flag in any codebase or system.

YAML got its popularity with the advent of Ruby on Rails, largely due to the simplicity of the database.yml file as an aid in database connection string abstraction that felt extremely clean to Java programmers who were used to complicated XML files full of DSN names and connection string peculiarities.

The evolution of the database.yml file into something arguably as complex as the thing it was intended to replace is described in the article below:

https://dev.to/andreimaxim/the-rails-databaseyml-file-4dm9

rco8786• 2 years ago

I guess I'll ask the dumb question - why not use Javascript directly instead of Jsonnette?

IshKebab2• 2 years ago

Integrating a modern JS engine into your program is complicated and probably pretty huge.

anon2913• 2 years ago

A modern JS engine suitable for config generation and much more is one C File: https://duktape.org/

There is no reason you need an optimizing JIT compiler to parse a configuration file.

davedx• 2 years ago

Yeass. I encountered jsonnet thanks to Ory Kratos, it’s great. Yaml is an awful hack that’s only still around in devops because of chance and circumstance. I hate IT.

HeckFeck• 2 years ago

I'd like to quote a snarky sig I once read on Slashdot:

You know what's good about YAML? No, me neither.

Personally, I love using a Perl hash for my config files. But maybe I'm just mad.

liveoneggs2• 2 years ago

I, too, loved hiera and puppet.

tonnydourado• 2 years ago

Helm becoming a de facto tool is up there with null in the top biggest mistakes our industry committed.

Gormo2• 2 years ago

The fundamental mistake being overreliance on huge piles of idiosyncratic middleware to accomplish simple tasks (like config parsing).

kungfufrog• 2 years ago

There's 2 things on the horizon here for Kubernetes that give me hope. KCL, its own configuration language, and Timoni, which builds off CUE and corrects some of the shortcomings of Helm.

Though these days, OLM and the Quarkus operator SDK give you a completely viable alternative approach to Helm that enables you to express much more complex functionality and dependency relationships over the lifecycle of resources. An example would be doing a DB backup before upgrading to a new release etc. Obviously this power comes at a cost.

valty4• 2 years ago

It's funny how little developers think about how to do configuration right.

It's just a bunch of keys and values, stored in some file, or generated by some code.

But its actually the whole ball game. It's what programming is.

Everything is configuration. Every function parameter is a kind of configuration. And all the configuration in external files inevitably ends up as a function parameter in some way.

The problem is the plain-text representation of code.

Declarative configuration files seem nice because you can see everything in one place.

If you do your configuration programmatically, it is hard to find the correct place to change something.

If our code ran in real-time to show us a representation of the final configuration, and we could trace how each final configuration value was generated, then it wouldn't be a problem.

But no systems are designed with this capability, even though it is quite trivial to do. Configuration is always an after-thought.

Now extend this concept to all of programming. Imagine being able to see every piece of code that depends upon a single configuration value, and any transformations of it.

Also, most configuration is probably better placed into a central database because it is relational/graph-like. Different configuration values relate to one another. So we should be looking at configuration in a database/graph editor.

Once you unchain yourself from plain-text, things start to become a lot simpler...of course the language capabilities I mentioned above still need to become a thing.

Syntonicles2• 2 years ago

This is something I'm trying really hard to do with a client. They have a bunch of 1500+ line "config" files for products, which are then used to make technical drawings and production files. The configs attempt to use naming scheme to group related variables together.

I want to migrate to an actual nested data-structure using (maybe) JSON - and these engineers absolutely will not write code, so config-as-code is a no-go, in addition to the disadvantage you mentioned.

My next thought was that there should be a better way to show the configuration, and allow that configuration to be modified. I was thinking maybe some sort of visual UI which where the user can navigate a representation of the final product, select a part and modify a parameter that way.

Is that along the lines of your suggestion? If not will you please expand a little? Configuration is the absolute core of this application.

valty4• 2 years ago

Sounds like you need an SQL database. You could use SQLite.

Then provide a GUI to modify that database. You could add a bunch of constraints in the database too to ensure the config is correct.

Usually when there is plain-text files though, it's because they want it that way. It's easier to edit a text file sometimes than rows in a database. Cut/copy/paste/duplicate files and text. Simple textual version control.

reissbaker2• 2 years ago

I think Steve Yegge got it right when he wrote:

I know, I know — everyone raves about the power of separating your code and your data . . . But it's [not] what you really want, or all the creepy half-languages wouldn't all evolve towards being Turing-complete, would they?

https://sites.google.com/site/steveyegge2/the-emacs-problem

Templating YAML is the same, but... Honestly, Jsonnet is too. If you're going to generate JSON — by god use a normal programming language. You have to teach your team one fewer thing (approximately no one knows Jsonnet); it already integrates with your existing build system; if you wrote a useful util function in your main codebase you can reuse it; if you have a typechecker or a linter you can use it; etc etc.

orwin• 2 years ago

Does it mean that Lisps 'code is data' is the right idea in the end?

reissbaker2• 2 years ago

IMO yes, although sadly Lisp is a pretty unwieldy language to do it in — it truly needs a little more syntax sugar for readability's sake. (Also a decent type checker would help.)

Ensorceled• 2 years ago

I feel like I wrote something on usenet with a similar title about imake in the early 90's ...

nhumrich• 2 years ago

I will tell you exactly why we template yaml. Its the exact same reason every code base has ugly parts. And that's the evolution of complexity.

At first, you have a yaml file. No templates, no variables. Just a good old standard yaml. Then, suddenly you need to introduce a single variable. Templating out the one variable is pretty easy, so you do it, and it's still mostly for humans to edit.

Well, now you have a yaml file and template engine already in place. So when one more thing pops up, you template it out.

8 features later, you wonder what you've done. Only, if we go back in time, each step was actually the most efficient. Introducing anything else at step 1 would be over-engineering. Introducing it anywhere else would lead to a large refactor and possible regressions.

To top it off, this is not business logic. Your devs are not touching this yaml all that much. So is it worth "fixing", probably not.

transfire• 2 years ago

Is Kubernetes using YAML 1.1 still? Because some of the complaints I hear shouldn’t be an issue with 1.2.

Moreover the YAML spec allows for specifying the tags recognized per application.

So on two counts, if “on”, “true”, “false”, as well as “yes”, “no”, “y”, “n”, “off”, and all capitalized and uppercase variants, are all boolean literals, it is not YAML’s fault.

Gare• 2 years ago

Unfortunately, almost everyone is still using YAML 1.1

PyYAML still doesn't support 1.2

andix2• 2 years ago

A lot of people don't know that JSON is valid YAML. You can mix JSON into your YAML wherever you need it.

So if you have issues expressing something complex in YAML (like nested arrays) just write JSON instead.

neilwilson• 2 years ago

Obligatory reference to Rubynetes - using Ruby and rspec to generate the YAML files that k8s loves.

[0] https://www.brightbox.com/blog/2020/02/24/rubynetes-getting-...

[1] https://www.brightbox.com/blog/2020/02/12/rubynetes-kubernet...

[2] https://www.brightbox.com/blog/2020/02/17/using-openapi-to-v...

EdwardDiego• 2 years ago

If you're using Helm to deploy your own apps, I feel that's a code smell. I'll add jsonnet for your own apps to the that list.

Just use dumb YAML, maybe kustomize if you really need, but if that's not sufficient, consider that a sign that you're not carving the wood the way it's telling you to.

Any form of templating for creating your own application manifest is another moving part that allows for new and fun errors, and the further away your source manifest is from the deployed result, the harder it is to debug.

If you really want to append a certain set of annotations to each and every pod in a cluster, instead of using shared templates (and enforcing their usage), there's other approaches in K8s for these kinds of use cases, that you have a lot more control over.

booleandilemma• 2 years ago

There are people out there who, for various reasons, like to make things more complex than they have to be.

mordae• 2 years ago

The real question here is, why the fuck are you not using a restricted LISP dialect, generating an S-expression using macros.

imwillofficial• 2 years ago

You guys are missing the big reason.

Most devops folks are former sysadmins with no developer experience.

The tools needed to be able to be picked up quickly without too much pain.

This is why our devops tooling in popular usage is not as robust as developer tools.

Know your audience.

anon-3988• 2 years ago

I have no idea why people are so against code generation. If I want hundred different kinds of YAML or code, I just fully generate them.

marttilaine• 2 years ago

For anyone struggling with Helm YAML syntax errors in their day job, I shamelessly advertise my browser-based debug tool Helm Playground:

https://helm-playground.com/ - https://github.com/shipmight/helm-playground

ddejohn• 2 years ago

Hey, I used this the other day. Thank you.

asylteltine• 2 years ago

So true. I abhor yaml. It’s impossible to know correct indentation without a plugin that shows it which isn’t available in many cases where you edit yaml. It’s whitespace sensitive. The data types are not obvious. It’s just all a round bad

I love json. It’s explicit and easy to read. We should just be using json for everything that needs to be human readable.

osigurdson• 2 years ago

Perhaps we need something along the lines of an infrastructure description language. Some of these yamls get pretty long. Using a real language (like Python) is probably not constrained enough however.

danfritz• 2 years ago

Give it a couple of months: Why the f*ck are we templating JSON

This article feels very ironic to me hence the sarcasm.

Give me code as configuration and I'll be interested

brodouevencode• 2 years ago

It's fashionable to hate YAML. And sometimes rightly so. But what are the alternatives? JSON, XML, INI, TOML, Dahl, Cue, JSONNET, HCL, your programming language of choice. Also let's agree on the target use case - in that YAML is largely used for configuration and operational tasks. If I were to rank-order the features necessary in a good configuration language they would be 1) readability 2) data/schema validation 3) stackability/composability 4) language support 5) editor support 6) industry adoption. So let's do a comparison:

YAML is fairly easy to read, has schema validation with the right library, and is pretty ubiquitous. It can get unwieldy like JSON though.

XML is big, ugly, unreadable. No one likes XML despite it's robust schema validation capabilities.

Your programming language of choice doesn't work because of the target use case unless you truly are a build-run group.

INI it too simplistic for many environments.

HCL is included because I'm a bit of a Terraform fanboy and it has great features like validation, readability and composability. However you're not going to find it in the wild as a general purpose configuration language - outside of Terraform it just hasn't taken hold.

Does anyone really use Dahll? (Serious question.)

JSON is nice because everyone understands JSON. JSON is not nice because all the brackets, braces, quotes, etc get in the way and make sufficiently large configurations hard to read. With the right library you can get schema validation.

JSONNET suffers from the same problems that JSON does, but adds more operations which makes sufficiently large things very hard to read.

TOML is nice and reminds me of INI in it's simplicity.

Cue looks and smells like JSON, has schema validation, but is much more readable.

If I were to rank-order these options it would be 1) Cue 2) TOML 3) YAML 4) JSON 5) your programming language of choice 6) JSONNET 7) INI 8) HCL 9) XML 10) Dahll (maybe?). My point here is that while YAML has a lot to be desired it's still very useful for most implementations and is better than many of the alternatives.

int_19h7• 2 years ago

I'd take an XML config over a YAML one any day. This isn't to say that XML is great, but its warts are well-known, and they are generally not of the kind that makes it easy to shoot yourself in the foot. Mostly the problem is that it's verbose, and to some extend, redundant.

JSON is also fine, esp. if it is JSON5 (with comments, unquoted keys, and other such minor improvements). I find that braces, brackets, and quotes don't get in the way - if anything, they make the structure clearer.

motbus3• 2 years ago

Yaml template is horrible but jsonnet also hurts my feelings :/

jaxxstorm• 2 years ago

Author of the post here.

It's always been relatively shocking to me that this is still relevant 4 years after I wrote it. Helm is as Ubiquitous as ever, despite attempts to replace it with Jsonnet, Cue and programming languages.

I've personally moved on from Jsonnet and would recommend Pulumi to anyone experiencing this problem.

jmbwell• 2 years ago

If only we had yet another system to solve this problem. Oh look, the author has one.

xkcd 927.

byyoung3• 2 years ago

any1 want to invent jsonc with me? It will be json but with comments

linsomniac• 2 years ago

Ansible convinced me that doing programming tasks in YAML is insanity, so I started an experiment: What would Ansible be like if it's syntax were more like Python than YAML. https://github.com/linsomniac/uplaybook

I spent around 3 months over the holidays exploring that by implementing a "micro Ansible", I have a pretty solid tool that implements it, but haven't had much "seat time" with it: working on it rather than in it. But what I've done has convinced me that there are some benefits.

usrbinbash6• 2 years ago

The problem is very specifically the fact that YAML, as a config language, sucks.

I have no idea why people started using it. "bUt jSOn dOeSn'T HaVe cOmMenTS" ... oh gimme a break! You want a comment in JSON?

    {
      "//": "This is a comment explaining key1.",
      "key1": "value1",
      "//": "This is a comment explaining key2.",
      "key2": "value2"
    }

There. Not so hard. Writing a config parser that just ignores all keys starting with "//" is trivially easy...if it's necessary to ignore them at all that is, because most config parsers I have seen couldn't care less about unknown keys, let alone repeated keys.

So what other "reasons" were there for YAML?

Oh, the human readbility thing. Yeah. Because syntactically relevant whitespace is such a joy in a data serialization format. It's bad enough when a programming language does that (and I am saying this as someone who likes python), but a serialization format? Who thought that would make things easier?

And then of course there are other things filled with joy and happiness...like the multiple ways to write "true" and "false", because that's absolutely necessary for some reason.

"Oh but what about strict-yaml?!" I hear the apologies coming...great, so now I have the ambiguitiy of what parser is used on top of the difficulties introduced by the language itself. Amazing stuff. If that's the solution, then I'd rather not have the problem (aka. the language).

But despite all[1] these[2] problems[3] and more, YAML somehow became the goto language for configuring pretty much everything in DevOps, first in containerization, then in cloud, and everything in between. And as a result, we now have to make sure our config template parsers get whitespace right. Great.

So bottom line: The problem here is maybe 1/3 the complexity of config files and 2/3rd the fact that YAML should have never been used as a configuration format in the first place. It's benefits are too small, and it's quirks make too many problems for that role, outside of really trivial stuff like a throwaway Dockerfile.

Want config? Use JSON. And if you need something more "human friendly", use TOML.

[1] https://github.com/cblp/yaml-sucks

[2] https://changelog.com/posts/xml-better-than-yaml

[3] https://noyaml.com/

eviks7• 2 years ago

That ugly child doesn't cut it, these are the comments people want:

key1=value1 # this is a proper key1 comment, if you move a line, it stays with key1

key2=value2 # it also doesn't break the table

# and you don't need to write a config parser

# nor modify a syntax highlighter

# nor make sure other people use your comment style

usrbinbash6• 2 years ago

If people absolutely want those, they can use TOML, which supports single line and inline comments.

Given a choice, I'd even opt XML over YAML.

rho4• 2 years ago

The cycle seems: Invent a new static information format (XML/JSON/HTML/...), reduce verbosity, add GUI, variables, comments, expressions, control flow, validation, transformation, static typing, compilers, IDE support, dependency management, and maybe a non-backwards-compatible major version etc. And you end up with yet another Java/C# clone, just inferior because it was never meant to support all these things.

wodenokoto• 2 years ago

Like many other things, Azure services can be deployed using JSON. Of course it’s not just json, it’s an entire language of deployment definitions and templating language hidden within json markup.

But next to that, Microsoft came out with bicep, which is a domain specific language for defining resources. It comes with a full language server and is honestly quite nice to use (if only azure services had some sort of reasonable logic to them)

I think k8 and friends need their own biceps.

throwawaaarrgh• 2 years ago

If you can write Python, Perl, Ruby, etc. Hell even yq in the shell. Then you have a full programming language that can output YAML or JSON in any way you want. No weird DSL, no twisting yourself into knots.

Just write normal code, make any data structure, print it as any data format. Call the code, output to temp file, use file, delete file.

Is it clunky? Yes. But it works, and you can't get any simpler.

Keep it simple, silly.

kube-system• 2 years ago

Templated yaml is readable if you do it right.

hellodanylo• 2 years ago

If need anything more complicated than simple $var substitution, it's time to use a general purpose scripting language with appropriate libraries to generate your data structure. A half-baked template DSL will never work.

im3w1l• 2 years ago

From my vantage point it seems to have happened roughly like this

First we had arbitrary code that did something. Then we thought, "hey wouldn't it be nice if we could do this declaratively in a standard way using configuration instead?" Then we could reason about it more easily. But then came the realization "this declarative system isn't quite powerful enough, what if we could sprinkle some logic on top of it". "Hey wouldn't it be nice if we could go back to doing it declaratively? I guess we can just add the missing features to our not-turing-complete config language". "Wow now it can almost do everything I want.... but there is just this tiny little thing I want to do in addition, let's do templating!" etc

globallyunique• 2 years ago

We wrote a backend service at Lyft in Python and at some point needed to do some string interpolation for experimentation. In a rush someone implemented this in YAML (no new deps needed). This ended up being the bane of the teams existence. Almost impossible to test if something was going to break in runtime, could only verify it was valid yaml but many other things were infeasible, super hard to debug - it soured me on YAML for years.

drybjed• 2 years ago

YAML, TOML and JSON can be ingested to represent the same data structures internally, it's just a few lines of code to decide which load() function should we use for a particular file. Why not support all three formats in your applications for configuration and just let users decide, which one they want to use? Put a 'config.json' in '/etc/app/conf.d/' and you get the same data, as with 'config.yml' or 'config.toml'. Then users can use whichever format they prefer for the input data.

cyanydeez• 2 years ago

clarification: YAML and TOML can represent JSON, but the opposite isn't true.

MrDarcy• 2 years ago

I tried this, it significantly complicated documentation and support after release. Lots more logic handling conflicting cases in two otherwise identical files, etc.

hosh3• 2 years ago

This article is not convincing me about yaml or json templating.

The only thing I can think of is that generating these files requires picking a language platform. I chose Ruby to generate the k8s manifests I need.

If you are picking something that is meant to be language-agnostic, or to have very little ramp-up time, then sure, templating. It just comes at a cost where the templating language itself approaches that of a full blown, Turing-complete language as more features gets added, often with shaky foundations (such as HCL).

gtirloni3• 2 years ago

I think we're barking at the wrong tree here (speaking about Kubernetes workloads).

This ugly mess of low-ish level details will never go away. Developers that are trying to focus on developing apps will never enjoy these things and that's fine.

Something like score.dev which abstracts things even further seems to be the way to go as the interface that is exposed to developers.

pyuser583• 2 years ago

Yeah I knew that answer would be Helm Charts.

I love Helm. Helm is magic. Helm makes my life tolerable.

But I’ve templates way too many yaml files.

If it were up to me XML would be default and we would use XSLT for templates.

I’m pretty sure you can use XSLT on JSON, which is logical identical to YAML. So something could be worked out.

Maybe that should be my next open source project.

tristenharr• 2 years ago

Clearly lots of people have tried to replace YAML with something else and that hasn’t worked. What’s your wishlist on making YAML actually work for declarative systems? Or can it work?

Would things like..

A great LSP with semantic autocomplete, Native cross-file imports, and conditionals based on environment make things feel different/better?

What should modern declarative systems be doing in your opinion?

bilalq5• 2 years ago

I'm completely done with configs written in YAML. Easily the worst part of Github Actions, even worse than the reliability. When I see some cool tool require a YAML file for config, I immediately get hit with a wave of apprehension. These same feelings extend to other proprietary config languages like HCL for Terraform, ASL for AWS Step Functions, etc. It's fine that you want a declarative API, but let me generate my declaration programatically.

Config declared in and generated by code has been a superior experience. It's one of the things that AWS CDK got absolutely right. My config and declarative definition of my cloud infra is all written in a typesafe language with great IDE support without the need for random plugins that some rando wrote and never updated since 2 years ago.

weebull2• 2 years ago

GitHub actions would suck whatever you "configured" them in, because you are trying to describe a program in a data structure.

Ansible makes the same mistake, as do countless other tools.

tomjakubowski• 2 years ago

"because you are trying to describe a program in a data structure"

(cries in lisp)

latchkey• 2 years ago

Ansible makes the same mistake, as do countless other tools.

My favorite example of this is chown/chmod taking 4-5 lines, in yaml. Sure you can do it a bunch of different ways, sure it allows for repeatable commands. But, it just sucks.

lolinder5• 2 years ago

you are trying to describe a program in a data structure

This describes 100% of software development, though! Programming is just designing data structures that represent some computation. Each language lends itself better to some computations than to others (and some, like YAML, are terrible for describing any kind of computation at all), but they're all just data structures describing programs.

The problem isn't that GitHub Actions tries to describe a program in a data structure, the problem is that the language that they chose to represent those programs (YAML and the meta language on top) is ill-suited to the task.

hintymad3• 2 years ago

The same reason I don't like AWS' Step Functions. The spec in JSON is horrible. On the other hand, Step Functions is pretty scalable and reliable and can take practically unlimited throughput. It's a good story for how a product can succeed by getting the primitives right and by removing just the key obstacle for users. Now that Step Functions has gained momentum, they can construct higher-level APIs and SDKs to translate user spec to the low-level JSON/YAML payload.

treflop2• 2 years ago

If configs had well-adopted schema support, it wouldn't be so bad.

bilalq5• 2 years ago

Even then, it gets messy. From a tooling standpoint, how will I load your schema? How will my editor respect it? How do I run a validator against it? I know XML kind of solves some of these problems, but it has its own thorns and despite what anyone says, it is not easy to work with. XSD, XSLT, etc. So much complexity that needs to be managed in a different way in every runtime. And then type safety goes out at the boundary where it connects to your code.

jrockway3• 2 years ago

At this point, I even prefer plain JSON to YAML. What pushed me over the edge is that "deno fmt" comes with a JSON formatter, but not a YAML formatter. It's a single binary that runs in milliseconds. For YAML auto-formatting you basically have to use Prettier, and Prettier depends on half of NPM and takes a good 2 seconds to startup and run. So, I literally moved every YAML file in our repository at work that could be JSON to JSON and I think everyone has been much happier. Or, at least I have been, and nobody has complained to me about it.

Various editors also support a $schema tag in the JSON. I added this feature to our product (which has a flow that invokes your editor on a JSON file), and it works great. You can just press tab and make a config file without reading the docs. Truly wonderful.

YAML has this too with the YAML language server, but you need your tab key to indent stuff, so the ergonomics are pretty un-fun. JSON isn't perfect, but at least the text "no" is true.

matsemann• 2 years ago

At work we're currently expanding to another country. Which means that many services now need a country label etc., which is fun when you're adding "no" to all our existing services. Luckily it's quick to catch, but man... why?

bilalq5• 2 years ago

I prefer JSON to YAML as well. The lack of comments is a problem though. But I feel like this is a false dichotomy. Both kind of suck for this need, but I can accept that JSON is at least reasonable to work with if you need language agnostic config.

DecoySalamander• 2 years ago

AFAIK Prettier has 0 dependencies and runs fast enough that triggering formatting on save wasn't ever noticeable (granted I've never tried it with YAML specifically). Curious what kind of setup you had to push it to 2 seconds - maybe bulk formatting in CI for whole repository?

Footkerchief• 2 years ago

In the case of GitHub Actions, it's made more painful by the lack of support for YAML anchors, which would provide a bare minimum of composability.

https://github.com/actions/runner/issues/1182

throwaway8943456• 2 years ago

These same feelings extend to other proprietary config languages like HCL for Terraform, ASL for AWS Step Functions, etc. It's fine that you want a declarative API, but let me generate my declaration programatically.

Yeah, I've had the same sort of opinion since the bad old AWS CloudFormation days. I wrote an experimental CloudFormation generator 4 years ago where all of the resources and Python type hints were generated from a JSON file that AWS published and it worked really well (https://github.com/weberc2/nimbus/blob/master/examples/src/n...).

Config declared in and generated by code has been a superior experience. It's one of the things that AWS CDK got absolutely right.

Is that how CDK works? I've only dabbled with it, but it was pretty far from the "generate cloudformation" experience that I had built; I guess I never "saw the light" for CDK. It felt like trading YAML/templating problems for inheritance/magic problems. I'd really like to hear from more people who have used AWS CDK, Terraform's CDK, and/or Pulumi.

bilalq5• 2 years ago

It's an annoyingly OOP model with mutations and side-effects, but if you look past that, it's pretty nice. The core idea is you create an instance of a CDK "App" object. You create new instances of "Stack" objects that take an "App" instance as a context parameter. From there, resources are grouped into logical chunks called "Constructs" which take either a stack or another construct as their parent context param. The only things you should ever inherit from are the base Constructs for Stack, Stage, and Construct. Don't use inheritance anywhere else and you'll be okay.

The code then looks something like this (writing this straight in the comment box, probably has errors):

    // Entrypoint of CDK project like bin/app.ts or whatever
    import * as cdk from 'aws-cdk-lib'
    import { MyStack } from '../lib/my-stack.ts'
    const app = new cdk.App()
    const stack = new MyStack(app, 'StackNameHere', someProps)
    
    // lib/my-stack.ts
    // Imports go here
    export class MyStack extends cdk.Stack {
      constructor(scope: Construct, id: string, props: MyStackProps) {
        super(scope, id, props)
        const bucket = new s3.Bucket(this, 'MyBucket', {
          bucketName: 'example-bucket',
        })
        const lambda = new NodejsFunction(this, 'MyLambdaFn', {
          functionName: 'My-Lambda-Fn',
          entryFile: 'my-handler.ts',
          memorySize: 1024,
          runtime: Runtime.NodeJS_20X
        })
        bucket.grantRead(lambda),
        tracing: Tracing.Active
      })
    }

The best part is the way CI/CD is managed. CDK supports self-mutating pipelines where the pipeline itself is a stack in your CDK app. After the pipeline is created, it will update itself as part of the pipeline before promoting other changes to the rest of your environments.

The equivalent CloudFormation for the above example would be ridiculously long. And that's putting aside all the complexity it would take for you to add on asset bundling for code deployed to things like Lambda.

TL;DR: Infrastructure-as-code-as-code

hintymad3• 2 years ago

An often-heard benefit for using YAML is that JSON does not have comment. What I don't understand is why we would switch to a whole new language. Just add a filter before loading the configuration, which can't be harder than switching to YAML, right?

Another reason for YAML is that it is easier to read. That I don't understand either. The endless pain of dealing with configuration does seem come from saving a few seconds of parsing off braces and brackets, but from not being about easily figure out what goes wrong, especially when what's wrong is a missing space or tab embedded in hundreds of lines of configurations.

p10_user• 2 years ago

Just make another named list key called "comment". Problem solved.

int_19h7• 2 years ago

https://json5.org/

m4632• 2 years ago

I like json a lot.

That said, I think json would benefit from only two things:

1) comments

2) allow extra commas, like ["a", "b", "c",] or {"a":"b", "c":"d", }

or more properly:

  {
    "a":"b",
    "c":"d",
  },

EDIT: and json5 does both, plus a few more niceties. (hmm. too much?)

mekster• 2 years ago

Not sure why people just don't settle with TOML.

didip• 2 years ago

For real, I want a real language (Lua/JS/Lisp) for configuration but without 3rd party imports so that it's secure and predictable.

SergeAx• 2 years ago

Config declared in and generated by code has been a superior experience.

And here we are at the point in time when people are plainly forgotten about compiled programming languages.

w0m• 2 years ago

living in a yaml-world; i honestly hate it.

gofreddygo• 2 years ago

let me generate my declaration programatically

this sort of thing looks and sounds like the right thing to do. Till you do it, on a largeish project with multiple teams that have experienced attrition.

I quietly added a layer of yaml generating code to make it bearable.

enonimal• 2 years ago

JSON not including comments has caused so much labor for the world.

imajoredinecon• 2 years ago

It’s funny that the post ends up recommending Jsonnet, which is a very close relative of the Google-internal “GCL,” which everyone at Google hates

Pick your poison, I guess.

gsky• 2 years ago

I forgo Kubernetes because of my hate for YAML.

ryandv5• 2 years ago

I'm reminded of DHH's article, "Rails is Omakase" [0] back during the time when "convention over configuration" [1] was a common refrain, meant to avoid the proliferation of (YAML or XML) configuration files by assuming sensible defaults and pre-selecting various parts of the solution stack or architecture, instead of letting it be freely specified by the developer.

You lose a few degrees of freedom and flexibility in your implementation this way, but at the same time you also don't need to wade through pages and pages of configuration documents.

Everything is cyclical. I'm waiting for the next "omakase" offering that provides a sane low-configuration platform for building "cloud native" apps. Right now it looks like we're in an analogue of the XML hell that prompted the design philosophy of Rails and "convention over configuration."

[0] https://dhh.dk/2012/rails-is-omakase.html

[1] https://wiki.c2.com/?ConventionOverConfiguration

alkonaut• 2 years ago

Two things need to die and die quickly:

1) “blind configuration” Such as a CI pipeline config that you commit but can’t validate before you do.

2) Complex markup for config. Anything that can’t fit into a single screen of flat .toml or .ini style config should be code.

solatic• 2 years ago

YAML is fine for human-maintained configuration. Yeah it has its footguns (like Norway) but if you're actually writing human-maintained configuration then you quickly pick these up with practice and they turn into a non-issue.

If your configuration is complicated enough that it needs to be generated, then use a real general-purpose language to generate it. Not crummy pseudo-imperative constructs bolted onto the YAML.

At the very least, systems that take YAML as configuration should also take JSON as configuration. GitOps-style systems should allow you to define, not just system.yaml config, not just system.json config, but also system.js config, that is evaluated in some kind of heavily-restricted sandbox.

schappim• 2 years ago

The sidekick configuration file is a great example of when you’d want to template YAML.

In that instance the queue name is derived dynamically from the hostname.

JohnMakin• 2 years ago

I wrote a monstrosity of a terraform module that takes pre-existing helm charts/templates, feeds some json into them via terraform, translates the results to HCL, and deploys them.

It's kind of a rube-goldberg machine that I made as a bespoke solution to a weird problem but it's been fairly pleasant to work with so far.

m000• 2 years ago

Yes, templating YAML is crazy. But is the answer jsonnet? That's even more batshit.

Why hasn't anyone opted for a "patch-based" approach? I.e. start with a base YAML/JSON file, apply a second file over it, apply this third one, and use the result as the config. How you generate these files is entirely up to you.

vimax• 2 years ago

Yes. The answer is a "config.d" directory, this has been known to linux package managers for a long time. It is the only way for multiple packages to contribute to configuration without fighting over ownership of the one true config file.

shp0ngle• 2 years ago

now I play a bit with nix and its config language.

i like yaml now.

jrockway3• 2 years ago

Yeah, I'm very sad that helm won. We do OSS k8s stuff at work, and 100% of users have asked for us to make a helm chart. So we had to. It is miserable to work on; your editor can't help you because the files are named like "foo.yaml" but they aren't YAML. You have to make sure you pipe all your data through "indent 4" so that things are lined up correctly in the YAML. What depresses me the most is that you have to re-expose every Kubernetes feature in your own way. Someone wants to add deployment.spec.template.spec.fooBars? Now you have to add deploymentFooBars to your values.yaml file and plumb it in. For every. single. feature.

It's truly "worse is better" gone wrong. I have definitely done some terrible things like "sed -e s/$FOO/foo/g" to implement templating... and that's probably how Helm started. The result is a mess.

I personally grew up on Kustomize before it was in kubectl, and was always exceedingly happy with it. (OK, it has a lot of quirks. But at least it saves you time because it actually understands the semantics of the objects you are creating.)

I like Jsonnet a lot better. As part of our k8s app, we ship an Envoy deployment to do all of our crazy traffic routing (basically... maintaining backwards compatibility with old releases). Envoy configs are... verbose..., but Jsonnet makes it really easy to work on. (The code in question: https://github.com/pachyderm/pachyderm/blob/master/etc/gener...)

I'm seriously considering transpiling jsonnet to the Go template language and just implementing everything with Jsonnet. At least that is slightly maintainable, and nobody will ever know because "helm install" will Just Work ;)

But yeah, I think Helm will be the death of Kubernetes. Some competing computer allocator container runner thingie will have some decent language for configuration, and it will just take over overnight. Mark my words!

thinkmassive• 2 years ago

I have definitely done some terrible things like "sed -e s/$FOO/foo/g" to implement templating

Next time you reach for this, check out envsubst for a slightly improved solution that’s somewhat standard (at least common).

On the topic of templating or modifying helm charts using jsonnet, you might find Tanka helpful:

https://tanka.dev/helm

dewbrite3• 2 years ago

But yeah, I think Helm will be the death of Kubernetes. Some competing computer allocator container runner thingie will have some decent language for configuration, and it will just take over overnight. Mark my words!

I want to believe this.

Everywhere I've worked we're still rawdogging tf/hcl and helm though, because change is scary.

At least I get some relief in my personal projects. :')

hintymad3• 2 years ago

I can't remember how many times I heard or saw the argument "but that is in YAML", which implies that the configuration(or god forbid, the code) is simple and well designed. I find it hilarious.

And a worst contender is embedding text template like jinja in a YAML config and forcing everyone to use such abomination to change production config via deployment. Yes, I'm talking about Terraform or the like. Why people think this kind of design is acceptable is beyond my comprehension.

boringuser2• 2 years ago

Config-as-code is this strange nightmare made for non- programmers to trick them into writing extractable formal logic.

seadan83• 2 years ago

I'm confused on two points:

(A) why not use the yaml syntax that is not whitespace sensitive. In the authors example, that could be: {name: Al, address: something}

(B) do env variables not go a long way to avoiding the need for a template? Instead of generating a complete YAML, put env variable placeholders in and set those values in the target environment. At this rate, the same YAML can generally be deployed anywhere. I've seen that style implementated several times, works pretty well.

I do agree that generating config itself, and not just interpolating values - is potentially really gnarly. I do wonder, instead of interpolating variables at deploy time, why not use env variables and do the interpolation at runtime?

killthebuddha• 2 years ago

I would love for someone to eviscerate the following idea:

Every deployed process receives exactly 3 environment variables, NONCE, TAGS and CONFIG_DB_PARAMS. Every process is bootstrapped in the same way:

  1. Initialize config db client.
  2. config = db.fetchConfig(tags)
  3. Use NONCE to signal config changes if needed.

Of course there's some environments where this Just Won't Work. I'm wondering if there are some very serious issues with this approach in a "standard" web application environment. It seems so straightforward but I've literally never seen it done before, so I feel like I'm missing something.

b33j0r• 2 years ago

Separate generated content from maintained content. Works for me. But on the specifics here, from a very python POV.

Strict YAML is easier to maintain than json if you have deeper than one or maybe two levels of nesting, multiline strings, or comments.

So, I build my config systems to _generate_ YAML instead of “templating YAML.”

PyYAML extensions and ruamel.yaml exist—Though kind of out of date, and more new projects are using TOML. (From project description: “ruamel.yaml is a YAML parser/emitter that supports roundtrip comment preservation”)

Confession: but yeah, not when I use ansible. Ansible double-dog-dares you to “jinja2 all the things” without much in the way of structured semantics.

oalders• 2 years ago

YAMLScript now exists: https://yamlscript.org/

vitiral• 2 years ago

I'm designing a simple dev environment from scratch.

My solution for this is a sandboxed lua for programatic configuration:

https://github.com/civboot/civlua/tree/main/lib/luck

I can't stand JSON (for many reasons) so I created a serialization format that combines it and CSV for nested objects

https://github.com/civboot/civlua/tree/main/lib/tso

I wish the industry would standardize on a solution like this. IMO you shouldn't use a "real" language unless you can lock it down to be determinisitic. JSON is supposed to be human readable but fails for lots of real-world data like multi-line strings or lists of records.

CSV is more readable but doesn't supported nested objects.

5Qn8mNbc2FNCiVV• 2 years ago

The worst part about this mess is that it's all fine and dandy with any other tool as long as it's first party but as soon as you want to use another software you're stuck reading and understanding the spaghetti that Helm inevitably becomes and hope that you can configure it the way you need to.

It's come to the point that I don't even think about using most charts and just build them myself. The issue with that is that software like Prometheus, Loki, Grafana or f.e. Postgres operators are so complex that it's almost impossible do "fix" them.

This really is like that dog sitting in a burning house and saying "This is fine" because that's how most go on with their day after hitting their head for a few hours and running dozens of pipelines.

willhbr• 2 years ago

I actually wrote about my thoughts on the tradeoffs between purely-config and purely-code:

https://willhbr.net/2024/01/18/the-code-config-continuum/

20after4• 2 years ago

I'm no fan of YAML but for an example of templating YAML that is tolerable, take a look at esphome[1] (and I suppose also home assistant).

In your main yaml file you have something like this:

  packages:
    left_garage_door: !include
      file: garage-door.yaml
      vars:
        door_name: Left
        door_location: left
        open_switch_gpio: 25
        close_switch_gpio: 26

Then in garage-door.yaml you can reference the vars directly with ${door_name} syntax.

It's the best version of templating YAML that I have experienced.

1. https://esphome.io/guides/configuration-types#packages-as-te...

Why are we templating YAML? (2019)

Related Stories