Ignition: A new CoreOS machine provisioning utility

https://coreos.com/blog/introducing-ignition.html

by polvi • 10 years ago

67 55 10 years ago

HorizonXP4• 10 years ago

Alex this looks great and likely addresses a lot of the problems I had with cloud-config. I was one of those with a Bash script that I used to generate multiple files, so this is great.

My biggest issue so far is CoreOS' naming of Ethernet interfaces on VMWare ESXi. It always uses some eno* name for each interface. I have a unique case where each VM I spin up has up to 10 interfaces.

I've solved this by adding net.ifnames=0 to my grub.cfg. It requires that I reboot the machine at least once to get it to take.

If I could have predictable interface names using Ignition, then I'm set!

jsmthrowaway• 10 years ago

My biggest issue so far is CoreOS' naming of Ethernet interfaces on VMWare ESXi. It always uses some eno* name for each interface. I have a unique case where each VM I spin up has up to 10 interfaces.

That's a systemd decision, not CoreOS, and also impacts the 7.x series of RHEL derivatives and anything that uses systemd > v197, really. Your way is one of three to revert it.

https://www.freedesktop.org/wiki/Software/systemd/Predictabl...

HorizonXP4• 10 years ago

Ah, fair enough. But it looks like that introduces another problem. When you spin up a VM in ESXi, it generates a new MAC address for each interface. So it would be great to create systemd .network files that bind to those MAC addresses. Except you don't what MAC addresses you need to match against. The only flow I see working is:

1) Create a VM with your interfaces, but don't boot it. 2) Take the MAC addresses and enter them into your .network files, created via cloud-config/Ignition 3) Mount the cloud-config/Ignition file to your VM and boot

Which is a bit painful to do manually when you have 10 VMs, with 5-10 interfaces each. I'd love to automate this, and if you happen to have a suggestion here, I'd really appreciate it.

Edit: Oh, and when I said it had an eno* number, it's more like eno16777736, which is not very predictable at all.

justinsaccount• 10 years ago

Are you talking about things like eno16777736 ? it's a vmware issue...

http://serverfault.com/questions/636621/why-is-my-eth0-calle...

HorizonXP4• 10 years ago

Yes, precisely. The comment below identified one of 3 possible routes to turn off the naming, one of which I'm using (net.ifnames=0 in the kernel config).

If I could find a better way to match the interfaces, it would be fantastic.

otterley4• 10 years ago

Why do you need predictable interface names if you're using DHCP?

HorizonXP4• 10 years ago

I am not using DHCP.

brandonbloom4• 10 years ago

Apologies for the somewhat off-topic remark, but ... Can we please stop creating new JSON-based config files? Sane config file formats must support comments.

kkirsche• 10 years ago

100% agreed. This is the reason I personally find YAML to be a great config alternative to JSON due to their ease of converting between the two

brandonbloom4• 10 years ago

YAML isn't a particularly good config format either, since it has a lot of unnecessary complexity. For example, bare words are strings, but some bare words, such as "true" are booleans. Which wouldn't be so bad, but did you know that "yes" and "no" are also booleans? I've gotten YAML files from an i18n-system that screwed this up in a pretty bad way.

jonaf• 10 years ago

In fact, YAML is a superset of JSON, and any valid JSON is also valid YAML. This becomes more obvious when you realize you can put any JSON objects inside of your YAML document and any YAML parser will correctly deserialize it.

Sanddancer2• 10 years ago

I wouldn't apologize for that, you're addressing the elephant in the room here. JSON has turned into the new XML. There seems to be a pervasive undercurrent that if it looks website-ish, it's got to be good. So it seems to be used quite a bit in lieu of looking carefully at the problem and figuring out the best structured format. Configuration files tend to have a lot of ambiguity in them, going to a format that makes it all that much harder to pull out the why part of settings does nothing to improve on that.

geofft2• 10 years ago

There seems to be a pervasive undercurrent that if it looks website-ish, it's got to be good.

I don't get that from the article. There's a stated reason, that it's a stricter format with much less risk of type confusion, which sounds plausible enough to me. There's also the benefit that tool support is good. Why does the web need to be relevant at all?

benley5• 10 years ago

Honestly, I don't think JSON config files are a big problem. What's shitty is when humans are expected to manually create and edit plain JSON, but what's stopping us from using whatever tools we want to generate those JSON configs?

There are some very nice tools out there for this; I have taken to using jsonnet (http://jsonnet.org/), which is excellent as a configuration language. It's pretty comparable to Hashicorp's HCL, but it's a standalone thing.

Summary: Write your config files in a language of your choice. Render them to JSON for your apps to consume. Be thankful that every app doesn't have its own special snowflake config format like they used to.

pm• 10 years ago

Any suggestions for a format that is sane?

mintplant• 10 years ago

TOML.

https://github.com/toml-lang/toml

ddispaltro• 10 years ago

My personal favorite is HOCON from Typesafe, https://github.com/typesafehub/config#examples-of-hocon

cookrn• 10 years ago

Hashicorp has created HCL https://github.com/hashicorp/hcl

pjmlp2• 10 years ago

I am going to be downvoted, but for me XML is sane and I don't bother to use anything else when given the option.

benley5• 10 years ago

Jsonnet is a really excellent one, imo: http://jsonnet.org/

There is also Flabbergast, which is comparable but a little less intuitive for those accustomed to Python-like languages: http://flabbergast.org/

The flabbergast homepage has a comparison table that covers most of the things mentioned in this thread: http://flabbergast.org/#comparison

benley5• 10 years ago

This paper also pretty interesting: http://www.masella.name/~andre/2015-srecon-andre_masella.pdf

otterley4• 10 years ago

At long as an app isn't strictly validating the JSON, you should be able to add keys like { "comment": "this does xxx" } to a JSON configuration object.

e12e• 10 years ago

There's that, or:

"Comments in JSON

I removed comments from JSON because I saw people were using them to hold parsing directives, a practice which would have destroyed interoperability. I know that the lack of comments makes some people sad, but it shouldn't.

Suppose you are using JSON to keep configuration files, which you would like to annotate. Go ahead and insert all the comments you like. Then pipe it through JSMin before handing it to your JSON parser."

https://plus.google.com/+DouglasCrockfordEsq/posts/RK8qyGVaG...

pjmlp2• 10 years ago

I just keep using XML, when the decision falls on my side.

- It supports comments;

- A very flexible format;

- I don't care it is verbose, it should be managed by tools anyway

- Can be automatically validated;

- Plenty of tools to choose from

darren0• 10 years ago

The transition away from cloud-init and YAML is really quite odd to me. Nobody enjoys editing JSON files. Forgetting commas, no multi-line strings, no comments, escaping characters in strings, etc. Just reading the documentation of ignition should be enough to illustrate the pain it is to manage multiline content for unit files in an JSON string.

But why abandon the cloud-init format in general? Again, why would somebody want to learn a new configuration syntax. Using CoreOS already requires you to know and use systemd units (most other distros this really isn't required knowledge), so that adds two steps to users learning/using CoreOS.

nikolay5• 10 years ago

YAML is good for simple configs. As the article says, JSON is much easier to programmatically generate than non-standard YAML. You still can store files in YAML, but compose and convert them into JSON in user data.

seanp2k25• 10 years ago

Again here, now you have 2 problems. Why not just go with a simpler config language and skip converting it and all the complexity that adds? It's possible to make this extremely complex, to the point where the config is in a database with ACLs and requires a DB server system (which itself requires config) to host (e.g. WebLogic) , but what do you gain?

chrissnell• 10 years ago

I'm totally with you. We're big users of CoreOS and while there are aspects of cloud-init that I find janky, it's at least mostly easy to read and understand. The use of JSON is nuts. Just looking at the one-shots consolidated to a single line with \n's gives me a headache. This is moving forwards?

What I would prefer is a compiler. You feed it a directory of unit file drop-ins and app config templates and it builds a single artifact that can be served over HTTP and pulled by the server booting up. This could allow for dynamic configuration and automation but still makes it easy for the admin to piece the config together.

seanp2k25• 10 years ago

A compiler. So now you have another language which compiles into JSON. So now there at least 2 problems, more if we want to be able to edit these configs on the systems directly, since we'll need some kind of tool chain.

Why do we keep re-inventing this wheel over and over again? Very plain text configs have worked for decades.

robszumski• 10 years ago

Check out bootcfg, a server that sends out customized cloud config or ignition files based on mac address, IP, serial number, etc

https://github.com/coreos/coreos-baremetal#bootcfg

rdtsc• 10 years ago

Running before systemd helps ensure that all services established by Ignition are known to and can be managed by systemd when it subsequently starts. This allows systemd to do what it does best: concurrently start services as quickly as possible.

I think this looks like a slight jab at systemd ;-)

Systemd thinks it does a lot more other things best

wmf2• 10 years ago

I think that was poorly explained on their part. Digging deeper into the docs, it looks like Ignition writes unit files that systemd runs; e.g. networkd is still used to configure networking.

thebeardisred• 10 years ago

Affirmative. This is just a mechanism to make sure that interfaces can be properly plumbed, disks formatted with the desired states & filesystems, etc.

songgao• 10 years ago

I'm curious about how it works. I must be missing something, but if it runs before the network and file system are setup, how does it pull configure from a http:// URL (say on bare metal) or oem:// which is on FS? Does it run with a default setup for network/fs and let systemd redo everything later?

_0w8t• 10 years ago

JSON is not language that "makes it very easy to write tools to generate new configs or manipulate existing ones". One cannot realistically consume JSON in a shell script, and even with tools that have native JSON support manipulating the config require the knowledge of document semantics as the language itself does not define how to combine or merge 2 documents into one.

For example, consider an application that comes with a complex JSON (or YAML as that language also share this shortcomings) that essentially describes default settings. I cannot just define an extra file that specifies a couple settings that I alter/delete/add, I have to have my own copy of the original file with my changes resulting in painful merge when the original is updated.

What I have found is that things like .ini files or config files in style of ssh_config work much better in practice. It is easy to generate and process them in any language with a notion of text IO including shell scripts and the merging functionality can be provided independent of semantics so I can keep config fragment outside the main file/files.

dghubble• 10 years ago

Some useful Ignition configs can be found in the https://github.com/coreos/coreos-baremetal examples. For those wondering about the format, they're converted to JSON and served to machines.

Ignition: A new CoreOS machine provisioning utility

Related Stories