NestedText, a nice alternative to JSON, YAML, TOML

by nestedtext • 6 years ago

302 284 6 years ago

Specifically, lines that begin with a word or words followed by a colon are dictionary items; a dash introduces list items, and a leading greater-than symbol signifies a line in a multiline string. Dictionaries and lists are used for nesting, the leaf values are always strings.

No doubts that there are use cases for this, but calling something that casts everything to strings an alternative to the above formats seems like a bit of a stretch.

xorcist• 6 years ago

It's not a bad idea, as text file formats are all strings anyway, to let the application do the conversion. It's the one with the domain knowledge anyway. Perhaps it could do with a companion library to specify the data format and emit parse errors as appropriate. But to keep that out of the syntax makes a lot more sense than the insanity that is yaml.

dagw2• 6 years ago

I've been using toml for my latest project and the one thing that really bitten me is that [1,2,3] is a valid array [1.1, 1.2, 1.3] is a valid array, but [1, 1.5, 2] isn't valid and throws an exception due to it having heterogeneous types.

baq• 6 years ago

I’m not sure the world needs another sexpr/sgml isomorphism, though it does look pleasing to the eye for the given examples. What this doesn’t solve though: yaml+jinja use cases, code as text, schemas outside of the document, everything else that makes a language out of a syntax tree.

alexchamberlain• 6 years ago

For example, in JSON 32 is an integer, 32.0 is the real version of 32, and “32” is the string version. These distinctions are not meaningful and can be confusing to non-programmers.

I'm really struggling with this assertion; IMHO one of the problems with JSON is the lack of more sophisticated scalar types.

That being said, this appears extremely readable, so my concerns could definitely be alleviated by a decent schema.

comex• 6 years ago

data type does not change based on seemingly insignificant details (2, 2.0, 2.0.0, “2”)

…only because NestedText does not support numeric types at all. That seems like throwing out the baby with the bathwater.

clarkevans• 6 years ago

In the design of YAML, Ingy made the case that we shouldn't have types for scalar values, that they should all just be strings. His argument was that each application knows what it needs and should have the ability to direct how those scalars are processed. If the format needs to be standardized across applications, one can use a schema system layered on top. In retrospect, I think he was right and I was wrong. For example, Fast Healthcare Interoperability Resources (FHIR) serialized as JSON treat numeric values as decimals, e.g. `2.3` is not a floating point value. Moreover, since parsing numbers is slow, it should be deferred till you actually need to interpret the numeric value.

crazygringo2• 6 years ago

Indeed.

It's less of a problem if you're using it for configuration files, where a program knows what key's values need to be cast to an integer or float.

But it seems disastrous if you wanted to use it for storing or transmitting data, above all between applications. You're immediately throwing out the possibility of being able to serialize and then deserialize data in basically any programming language.

I shudder at the idea of an API that accepted NestedText, where I'd need to worry about whether my floating-point output was compatible with its floating-point string parser. Yikes. I want the serialization format to handle that. Isn't a major criticism of JSON that it doens't have a built-in datetime representation?

Mindless2112• 6 years ago

You wouldn't use this for arbitrary object serialization. Lists can only contain strings, not other lists or dictionaries.

As for stringification, JSON's data types are mismatched with pretty much everything that isn't JavaScript to some degree. If you need to serialize a 64-bit integer to JSON, you serialize it as a string because the parser on the other end is probably going to try to parse it as a double-precision floating point number. Once you've started serializing numbers as strings anyway, it's not too far to "serialize every scalar as a string".

xg15• 6 years ago

To be fair, this seems to be designed specifically for config files. Just as JSON should be used for data transmission but not for config, I could imagine, this should be used for config but not for data transmission.

dan-robertson3• 6 years ago

I think avoiding numeric types is a good decision. It tends to eventually cause problems when one implementation converts numbers to doubles, another to either doubles or longs depending on whether they have a . or e, and another which converts them to bignums (or passes them as strings to the caller).

One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”.

There are also issues when something looks numeric but doesn’t parse (eg 1.2.3, 3/2, 12in, 4h30m2s, 2:30, 2020-02-29, etc). One way to deal with these is a tokenisation rule like in Common Lisp: if it is a valid number syntax then treat it as a number, otherwise it’s a symbol, but this can lead to issues (eg you would need to know that when your number needs more than float precision or otherwise doesn’t follow the rules, it should be in quotes. It seems crazy to pass that detail on to the poor sod who has to write the config file).

kanox5• 6 years ago

One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”.

The benefit of standard numeric and boolean types is that different tools can exchange data in a well-understood way.

Getting rid of yaml's 30 ways to write "true" and "false" by making everything is a string just means that you now have 30 tool-specific ways to write "true" and "false".

The "everything is a string" approach already exists in shell scripts and TCL and it's not really that great.

zamadatix• 6 years ago

That's not the only way to deal with them, you can specify type like bson.

The nice thing about that is it solves the problem rather than hoping it doesn't matter or assuming each program's validator will think to note all of the possible data types not compatible with the program natively. If a u32 is defined in the file and you've only got doubles to work with it's a given you'll have to deal with it in your tool specific validation. For everyone else it's well defined.

The downside is it's a bit more verbose and if you have all of that info already it's pretty easy to jump to just using a binary format which will be more efficient anyways.

thayne• 6 years ago

My ideal config language would have unambigous syntax for at least 64-bit signed integers and doubles (so, for example the spec for the language requires 56 to be parsed as an integer and 56.0 to be a double. Additional types would be ok too, as long as the syntax is unambiguous and obvious.

Phrodo_00• 6 years ago

I am of the opposite view: As long as there are no edge cases, supporting more types in data languages is good, it leads to failing faster.

Too many types could get overwhelming, but I like where amazon's Ion is [1]. It actually supports multiple number types, with decimal being the default for values with a dot.

you would need to know that when your number needs more than float precision or otherwise doesn’t follow the rules, it should be in quotes

Not really. The configuration value should either be a number or not, which is determined by the application reading the config. As a config writer you only care to make the type match (so, in json, if the application uses number you make sure you use number, and if it expects a string you use that)

(disclaimer: I work for amazon, but have nothing to do with Ion other than having used it. Opinion is my own, not my employer's, yadda yadda)

[1] http://amzn.github.io/ion-docs/

jiggawatts6• 6 years ago

I think avoiding numeric types is a good decision.

Only if this format is intended for use-cases that never need to deal with numbers.

One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”.

That statement couldn't possibly be more wrong.

Number parsing (and encoding!) is a decidedly non-trivial problem. You need to concern yourself with -- at a minimum -- all of the following:

- Unsigned 64-bit numbers.

- A series of digits that would be bigger than a 64 bit whole number. Convert to float? Truncate in some way? Error?

- NaN

- Infinity

- Negative zero

- Denormal numbers.

- Differentiating between decimal/currency types and floating point numbers. Not all decimal values can be exactly represented as floats!

- Efficiently encoding floating point to use the minimum digits without losing precision.

- Parsing those minimal numbers with perfect "round-tripping".

- Doing the above efficiently.

- Securely too! Efficient parsers cut corners on sanity checks. I hoped you fuzzed your parser...

The above can easily amount to many kilobytes of extremely complex code. Look up "ryu" as an example of what Google came up with to make JSON number parsing reasonably efficient.

Meanwhile, reading a fixed-length number from a binary format can be done in a single machine instruction. One. It might not even take an entire CPU clock cycle! Okay, two, if you need to bounds-check your buffer, but there's ways to avoid that.

Afterwards, the bounds check is again literally just two machine instructions in complexity. That's not the difficult bit!

The difficult bit is the parsing.

barumi• 6 years ago

I think avoiding numeric types is a good decision. It tends to eventually cause problems when one implementation (...)

It seems you're mixing up the language definition with implementations that try to follow the language definition.

If different implementations have different results then either they are buggy or the language has some important holes in the specification.

Either way,the solution to this problem is not less validation.

One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”.

That's the whole point, isn't it?

I mean, if you already acknowledge the fact that this parsing and validation is a basic requirement, why handle it as an afterthought and force developers to add their own hand-rollef absurd and unnecessary type checks and type coversions?

Wouldn't it simply easier to let the language and the parser do that already?

I mean, no one ever complained that JSON had string types. In fact, one of JSON's main complaints is that it doesn't support enough types, such as timestamps.

7952• 6 years ago

I wonder if you could just include explicit type information?

  name(str): Dave
  age(int4): 22
  dob(date): 2020-02-01
  photo(base64):TWFuIGl....

thaumaturgy2• 6 years ago

How would you include "\n >" in some multi-line text in this format?

mastax• 6 years ago

    >
    > >

secondcoming• 6 years ago

Do people really think significant indentation is a good idea?

Edit: A lot of people seem to be accidentally clicking the downvote icon.

wnoise• 6 years ago

Yes.

OkGoDoIt• 6 years ago

Apparently. It’s my single biggest pain point with Python, but apparently a lot of people think it’s a good idea.

agumonkey• 6 years ago

I'd rather cringe a bit at it than fight on editor style configuration in reviews

snazz• 6 years ago

It works fine in Python. It's nice because the visual indentation lines up with how the interpreter actually reads the code. You don't end up in a situation like this contrived example:

  if (something == somethingElse)
    doSomething();
    --counter; // added afterwards

when whitespace has actual meaning.

Your editor adds the whitespace to match the current scope, so you don't have to type it manually.

What's your issue with significant whitespace?

sepples2• 6 years ago

I don't mind it that much for programming languages like Python. The one annoyance I have is related to editor tooling: In languages that use braces to denote blocks, I can paste a block of code wherever and have my editor autoformat it to the correct indentation level, while in Python I have to be more deliberate with increasing or decreasing the indentation of a pasted snippet, and sometimes I'll make a mistake in the process that isn't caught until later.

I dislike significant indentation for configuration because the nesting tends to get deeper than for programming languages and it can be harder to see what's going on.

Netch3• 6 years ago

You don't end up in a situation like this contrived example:

Modern languages repair this with mandatory block braces: {C style} (Rust, Go, Swift...) or Modula/Ada style (if x then foo else bar end). C style is easier for editors.

What's your issue with significant whitespace?

1. Many cases when leading whitespace is still spoiled, including editors (which use can't be avoided due to corporate tool limiting), web formatters, etc. Last decade spreading of mobile-aware tools made this worse.

2. With grouping by indentation, it's hard to determine block start if multiple blocks are ended at the same line, like:

and let's imagine the most nested block occupies 2-3 screens (not rare for configs or code, even if it is against good coding rules). You have no means to decide what block start is to be matched when you ask editor to find match.

I code in Python continuously since 2004 and I deem grouping by indentation is bad idea. Not fatally bad, of course, but with explicit grouping it would be a bit better.

tobyhinloopen• 6 years ago

I don’t have strong opinions about it

ratww• 6 years ago

Why not? I'm not a Python developer, but bad indentation is considered a faux pas in most production codebases and is already enforced by linters, so I don't really see the issue with enforcing it at the language level.

sings• 6 years ago

I love Coffescript, Sass and Pug for exactly this reason. Just my opinion, though.

jarym• 6 years ago

With NestedText any decisions about how to interpret the leaf values are passed to the end application, which is the only place where they can be made knowledgeably. The assumption is that the end application knows that Enrolled should be a Boolean and knows how to convert ‘NO’ to False.

That assumption is... not applicable to most scenarios I come across and will likely lead to issues being pushed downstream and introduction of subtle bugs and defects.

liversage• 6 years ago

The special interpretation of NO in YAML has led to bugs and defects:

I once disabled our product for the entire country of Norway for a day because `NO` in YAML evaluates to `false`

https://twitter.com/aarondjents/status/1307692593493553160

pas2• 6 years ago

It'll lead to incompatible NestedText codecs/serdes :(

There should be at least some support for some standardized representations (basically JSON + ISO8601 datetime + some encoding for embedding whatever stringified serialization, eg. just how HTTP uses chunks and unique boundary tokens).

baxter001• 6 years ago

Interesting insights into the thought process of the author, there's very little as to positive statements of why you'd want to use this format, but they do however spend a great deal of time discussing neutral features and edge cases of currently wildly adopted alternatives.

One might call this 'argument by gotcha' interesting to imagine the kind of lifeworld that makes one think this is a worthwhile form of persuasion.

didibus• 6 years ago

I like it. For use cases that are for non technical users to manipulate and maintain it makes sense to me.

In Emacs, org files are often used for configuration similarly.

While I'm a fan of EDN most of the time, which doesn't suffer from the issues they mention JSON having, not having to wrap things in quotes for text, and using indent for nesting definitely would make it nicer for non technical people. So I can see a use case for this as a simple text interface for forms that is targetted at non-technical users.

oblio2• 6 years ago

You know what I want? Schemas. And clear error messages.

I want to know beforehand what I can put in a config file and I want a fast and hard failure if what I put in there is not good.

And this should be implemented at the file format parser level, with hooks for apps to add on top of the default behavior, so that every app that implements this format gets these things almost for free.

already_exists• 6 years ago

Like in Windows where you configure by clicking check boxes that can get disabled if invalid, with tooltips explaining what they do, additional help if you press F1, etc. ?

It would be nice if we had such tools.

pas2• 6 years ago

There's JSONSchema, and there are GUIs for handling/inspecting them.

zmj• 6 years ago

Protobufs have a text representation.

andrewg• 6 years ago

Yes indeed - it's actually pretty nice. You just define a message for your configuration schema:

  message Config {
    repeated Server server = 1;
  }

  message Server {
    string address = 1;
    int32 port = 2;
    bool standby = 3;
  }

And then you use the text representation in a config file:

  # main instance
  server { address: "127.0.0.1" port: 4567 }
  # backup instance
  server { address: "127.0.0.1" port: 9876 standby: true }

And load it into a message instance:

  Config config;
  google::protobuf::TextFormat::ParseFromString(input, &config);

quietbritishjim5• 6 years ago

As of protobuf 3 they also have a canonical JSON representation, which you can access from all the supported languages.

vlovich1233• 6 years ago

Haven’t you described cap’n’proto, protobuf, thrift, flatbuffers etc?

I know cap’n’proto also has fantastic support for using the schema for config files. You can just compile any constant as a stand-alone serialized message that you mmap into your code in a safe way. It can’t do complex math and things (at least yet) but you can express lists, dictionaries, and reference other constants, so as a config file replacement I love it. I’ve also found the format to be far more regular and consistent than you get with things like text protobuf (you’re still using the schema language instead of another format)

quietbritishjim5• 6 years ago

You can just compile any constant as a stand-alone serialized message that you mmap into your code in a safe way.

Are you suggesting using a binary format for your config files? I think most people would find that more trouble than a decent text format.

... than you get with things like text protobuf

You can just use protobuf's canonical JSON representation (thought the lack of ability to use comments is annoying).

michaelmior• 6 years ago

JSON schemas are available for a number of JSON/YML config formats from JSON Schema Store[0]

[0] https://www.schemastore.org/json/

pookeh• 6 years ago

We had that a decade ago. It was called XML and XML Schema. All IDEs support it.

JSON was a huge step backwards in the name of simplicity. And now when we are going to add similar functionality to JSON, something else is going to come out in the name of simplicity (like NestedText).

thaumaturgy2• 6 years ago

Magento is a popular codebase that made XML-based configuration a fundamental part of its architecture. The results were terrible and caused numerous headaches and countless hours lost to trying to troubleshoot inscrutable configuration issues. The Magento 2 codebase began a shift away from XML for configuration, although it still uses some.

There may be room for an argument that Magento did XML badly (it did many things badly), but I don't believe I've ever seen XML done well.

m463• 6 years ago

I think XML sort of failed simplicity.

In a minute I can read and write json from most languages I use.

In the same amount of time, I'm still wondering if I should use a tag or attribute in xml. cdata? expat?

It's not that xml isn't a good technology. It's that it's not appropriate for general use, especially in comparison to simpler alternatives.

kanox5• 6 years ago

We have it today for JSON, it's called JSON Schema and many IDEs support it.

jeffbee2• 6 years ago

XML is fatally flawed because you can't safely put one XML doc inside another one. Because of this rather fundamental problem, it never was any good for anything, and it never will be.

Netch3• 6 years ago

We had that a decade ago. It was called XML and XML Schema.

It would be true if XML was not full of all this SGML debris like "entities" (really, uncontroller macros), if real schema formats was flexible enough (I needed <c> inside <a> and <c> inside <b> when they totally different), etc.

But when a config reader tool has to deal with 40+-year legacy of enterprise guys wanting to embrace the universe, but all this doesn't allow to control contents without external measures like regexp checking... that simply shuts up facing real world.

irrational• 6 years ago

I spent years working with xml, xslt, xml schema. Frankly when I first saw json I thought it was terrific. Nothing has changed my mind since. Why do you feel like it is a huge step backwards?

atombender2• 6 years ago

JSON Schema [1] is actually a mature standard now, with decent tooling support, mostly through OpenAPI (formerly Swagger), which extends it with support for endpoints.

It's much simpler to use than XML Schemas, and arguably results in cleaner data models, since it doesn't have anything analogous to XML namespaces that allow for arbitrary mixing of schemas.

[1] https://json-schema.org/

RobIII• 6 years ago

You know what I want? Schemas.

I can see this work perfectly fine in typed languages like C#: `NestedText.Deserialize<T>("nestedtext")` where the deserialize method handles the actual mapping of nested text objects to `T` by providing the deserializer a class / classes that handles the string -> scalar(s) mapping for the given T. That would, sort of, function as a Schema.

I think the only thing, from glancing over the project, that would need to be supported to make this really useful is nested lists/dictionaries. I don't see how this can be done but maybe I'm missing it.

setr• 6 years ago

You can always do that, defining the schema in the client to produce sensible checks, even with JSON. The problem is that wherever the spec is underspecified is another place where two different clients can deserialize differently, and both be correct.

And the problem with stringly typed systems is that everything is underspecified

rudolph9• 6 years ago

Check out https://cuelang.org/

verdverm• 6 years ago

Came here to say the same, Cuelang is by far the best config system and paradigm I have tried. All else seems so last century, though Cuelang has its foundation in NLP systems from last century :]

vitiral• 6 years ago

Never seen this, it's awesome! Might be an improvement over jsonnet, which was my favorite approach

tootie2• 6 years ago

You want XML from 15 years ago? Yes, me too. Schemas and includes.

oblio2• 6 years ago

I've used XML. I don't want namespaces, I don't want the verbosity, I don't want entities, I don't want the security vulnerabilities.

I should have mentioned that I want something simple and readable.

fn1• 6 years ago

Slightly off-topic, but yes, having fail-fast deserialisation is great.

I wrote a json/kotlin-serialisation library once and purposely restricted some json-features to achieve that:

1. Fields can arrive in any order - this is standard

2. Field names are matched case-insensitively - so keyA and keya are the same, because who would use two variables differing only by case. Serialization keeps the original casing of the name.

3. Missing fields throw an error. if they are nullable, they have to be explicitly set to null - so that you can be sure the serialization side upgraded to the latest version of a protocol if a field was added, and things don't just work by chance.

4. Nullable strings are not coerced to empty strings or anything like it. Kotlin is null-safe, so if it's a string, it has to be "". If it's, for whatever reason, a nullable string, you can set it to null.

5. Enums are also serialized case-insensitively - so you an write "keyA": "eNumVaLuE" if you want - typos should not break the code here, no on would you two enums differing only by case. IIRC booleans could also be TRUE, tRuE, truE etc. (but NOT t or f, or yes or no, or 0 or 1 or empty).

6. Superfluous properties are silently ignored.

These rules were a great tradeoff for quick development, mixing languages and having fail-fast behavior with a stable protocol.

(https://medium.com/@fabianzeindl/generated-json-serialisatio...)

hn_throwaway_99• 6 years ago

For example, in JSON 32 is an integer, 32.0 is the real version of 32, and “32” is the string version. These distinctions are not meaningful and can be confusing to non-programmers.

Au contraire mon frere. What's the point of this data format if it's only intended to be for humans and not computers, because for computers the data type is critically important. For example, I've more than once seen the extremely ill-advised idea to treat zip code as a numeric, which completely screws your data model once you want to support zip+4 or international postal codes that contain letters.

thewakalix2• 6 years ago

Isn’t that a perfect example of a situation where stringly-typed-data-by-default would help?

masklinn• 6 years ago

No, because you’re pushing the choice on the applications giving them more opportunities to screw up. If your data is already typed and something is a string odds are the application will either just follow that lead or look up why the data format authors picked that type in case it’s important or relevant.

Myto• 6 years ago

But this is an interoperability nightmare. Number, date and boolean formats are going to be a pain in the ass to deal with.

rch• 6 years ago

I'm mostly using HOCON these days. Strange that it's not included in the alternatives section.

gkoberger• 6 years ago

This seems... worse?

I can't figure out what's happening here. Is everything just a string? They complain about how YAML handles booleans (fair criticism), however I can't see how NestedText fixes this.

This feels far inferior to TOML (everything NextText does, and much more) or JSON (very unambiguous, works natively in most languages).

I see this a lot in startups and open source projects. They exist solely as a criticism of their competition (which is fair!), but don't work to understand the real problems and fix them in any meaningful way. You can spot them when they talk a lot about what's wrong with the competition ("AWS is too expensive", "YAML is too ambiguous") but never explain their solution.

geerlingguy• 6 years ago

"It's a complex problem so we just cast everything to a sting and require applications to handle the typing" is what I interpret it as.

Honestly after a few years using YAML you remember to always quote anything you know you need to be a string, and the different multi line types.

An argument against this is "why not just design it to be simpler" — well, then it's less useful in as wide a variety of applications, and stands less chance of actually becoming a standard like JSON or YAML has. And you end up with the XKCD problem.

YAML came in and solved a few of JSON's most glaring problems (multi line and comments) with a usable approach.

This new format seems like it's "YAML, but a little different."

If I wanted to usurp YAML, I'd focus on the greatest pain points for most, whitespace and schema support.

robocat• 6 years ago

YAML came in and solved a few of JSON's most glaring problems

AFAIK they were both independently created in 2001 and YAML was not created in response to JSON.

jeffnappi• 6 years ago

Sadly my comment will not add value to the conversation, but I came to the comments looking to see if someone made the XKCD reference. To my delight, here you are. #927 always delivers - https://xkcd.com/927/

timc3• 6 years ago

YAML just introduced other problems.

mongol4• 6 years ago

Interesting. Maybe the time is ripe for a next generation of config formats. It looks decent. But if you just open a file with this you might think it is YAML if there is nothing else to hint you?

xixixao2• 6 years ago

This actually looks great. Many people are complaining about the lack of data types, but most of the time you

a) Do not have values of different types occupy the same fields

b) Have a schema defined (explicitly or implicitly as part of the parsing), especially because you're likely working with a type system

This isn't good if you want `x = parseJSON(blob)` kind of API, but that's definitely not what you want for any kind of human-editable config.

It seems simpler than TOML, I'd give it a try.

spullara• 6 years ago

I don't like any machine readable format that doesn't have some indicator that it is a complete document (like JSON or XML does). I've had a production issue where a format like this one was used where the file was read before it was finished writing it and ended up with a corrupt configuration as half of it was elided without any way to know.

kortex6• 6 years ago

This is the best criticism of this entire thread.

It'd be easy to employ ... as a document separator / end indicator that could be checked for.

kzrdude• 6 years ago

You could add that yourself - so an application specific marker, using this format.

stickfigure3• 6 years ago

Is it missing "list of dicts"? All the examples show only lists of simple strings. If so, that seems like a major problem.

Taking the main example, what do you do if you have more than one vice president?

ivan_gammel• 6 years ago

This is the biggest problem with this format! You probably won’t need it for configs, but they are written by engineers which are fine with other formats. For business data missing list of dictionaries is a deal-breaker. This is yet another shallow project similar to dribble-design portfolios of some people: looks cool at first glance, but has so big problems that it won’t fly.

mkl4• 6 years ago

This supposed "biggest problem" doesn't actually exist. Lists of dictionaries work fine, and I don't see any reason they wouldn't be useful for configs either. Maybe you should actually try it before dismissing it so shallowly?

This:

  list of dicts: 
      - 
          key 1: Hi 
          key 2: there! 
      - 
          key 1: I'm a list 
          key 2: of dicts!

parses as

  {'list of dicts': [{'key 1': 'Hi', 'key 2': 'there!'}, {'key 1': "I'm a list", 'key 2': 'of dicts!'}]}

arnon• 6 years ago

Oh great, another alternative that seems to not really fix anything significant, while creating a slew of other things that will be criticized.

It just shifts problems to other areas, which some may find even worse.

dan-robertson3• 6 years ago

What is the representation for objects in the “name and arguments form” (eg enums in rust, variants in Haskell, other tagged/discriminated unions)? It seems like they are becoming more popular in various programming languages.

What is the way to represent a map/dictionary where the keys are not strings? In json I also don’t have a good way except I guess a list of objects with a key and value field but it feels heavy. In Lisp I would probably use an alist (whereas a plist might be used for the “string” (or symbol, rather) keys). In fact, how does one even write a list of lists with this?

richm31415• 6 years ago

Misspelling "Topeka" does not inspire confidence . . .

jeremy88832• 6 years ago

Interesting idea to remove number and string types. The downside being that you spend more time writing parsers of the data. But if you had some sort of separate schema that generated all of that for you, it would no longer be such a big deal.

They mention that their key/value pairs are ordered. The downside here is that not all languages (eg. javascript), support them.

I also prefer them to be unordered. The downside with ordered dictionaries, is that you need to always be asking "does sequence matter here?". So it adds an additional thing to think about, more tests need to be written, and certain optimisations can't always be made.

xixixao2• 6 years ago

Both Objects and Maps in JS are in stable insertion order.

jeremy88832• 6 years ago

Apologies, it looks like pre es6, they were unordered objects. But now there are rules that guarantee insertion order.

kevincox• 6 years ago

This looks fantastic. I was recently reading a config file from Rust and ended up going with JSON5 but this is simpler to read. In a language like Rust you don't need the types because you specify that in the struct anyways. Sure, this means that there are effectively more types than strings but the user doesn't need to differentiate.

In Rust you do something like this:

  #[derive(Clone,serde::Deserialize)]
  struct {
    an_int: u64,
    a_float: float,
    ordered_map: linked_hash_map::LinkedHashMap<String, chrono::DateTime<chrono::Utc>>,
    unordered_map: std::collections::HashMap<i32, String>,
  }

So there is no need to worry about if maps are ordered or if a value is an integer, real or string in the format itself. Ironically for Python (which the reference implementation is in) it does seem much more annoying to have to manually call `int()` on each element.

I'm just a little sad that tabs are disallowed. I really think the best rule for indentation-sensitive languages is that each line must either have the same indentation in which case it is the same level, same indentation plus any amount in which case it is the next level, or the exact indentation of any previous level in which case it is a dedent. These "solutions" which just forbid tabs are half-assed and ones that try to convert tabs to a set amount of spaces just lead to confusion.

Additionally it would be nice if there was an example of a dict inside a list. I think it would work like the following but can't confirm from reading the site.

  -
    key: value
  -
    key: value
    other-key: other-value

kseistrup• 6 years ago

Correct. You'll get:

    [{'key': 'value'}, {'key': 'value', 'other-key': 'other-value'}]

haxorito• 6 years ago

Is it just me or NestedText is kinda subset of YAML? I personally see very little advantage of using this over json or yaml, but having more flavors and variation is always a good thing.

Edit: forgot to mention. I personally don’t like most general data serialization formats for configuration, the one I can probably tolerate is XML, but even that I only use when it’s part of requirement. The way I usually implement program behavior configuration is through runcommands(runcom, .rc)

tgv• 6 years ago

Very YAML. I checked the example, and the difference is that YAML places the > following the attribute name, and this format places it before the text.

BTW, I happen to like YAML as a configuration format. It's much more readable than JSON. It's not that suitable for serialization, and probably shouldn't be used to create huge config files, but for the rest, it's as good as it gets.

kortex6• 6 years ago

I like this quite a bit. Keep stringly-typed text files as strings; let code sort it out.

Pydantic would work great in conjunction with this to coerce types.

bArray• 6 years ago

Why choose this over the other options? Just some thoughts:

1. I like that comments are part of the standard. I wrote my own C++ JSON parser that allows for comments too.

2. Is it strict about indentation? One thing you can never get programmers to do on significantly sized teams is consistent indentation. Is that tab+space? Or spacex5? Is it going to break if a tab sneaks into Git? (Setting up Git push rules just annoys and confuses people.)

3. "without the syntatic clutter of JSON" - I happen to like it. I can compact it quite far if I need to. I also like the fact I can spit it out over a debug server and JS will just magically start reading it.

4. Something really cool would have been the introduction of typed data. One way we achieve this via JSON is to create a template file which would declare something like (in a file named 'template.json' or something):

    {
      "data" : { "type": "float", "default": 0.0, "min": -1.0, "max": 1.0 }
    }

Obviously this requires checking in the code, but it does build up some kind of format checking and sanity. It can also warn you that it's using a default rather than a config defined value.

It would be nice if there was then the ability to define type syntax... But I fear this might be going too far.

5. Another thing I do with JSON is inheritance. So you define a 'parent' property at the top of a file, the values are loaded from the parent and then the child loads theirs over the top. Why have this? We usually need some per-application configuration but mostly it stays the same. It saves having to write it multiple times. You can even break it down into sections to keep each configuration file smaller.

(NOTE: For inheritance, a top tip is to implement "maximum depth", encase you get into a loop.)

Hurtak• 6 years ago

For second point: most of the languages have linters that can enforce unified indentation, also there is https://editorconfig.org/ standard

_hofstadter• 6 years ago

https://cuelang.org

- clean syntax

- logical & unified

- roots in LINGO

- forked from Golang

https://cuelang.org/play

eyelidlessness• 6 years ago

This seems on its face to be a significant improvement on the goals of YAML, but I think the tradeoffs it makes will likely move YAML’s problems into a different place, creating a whole different set of difficulties understanding what a given data is, means, or does.

The problem with human friendly formats is that the thing that typically makes them human friendly is removing things that make reading and editing difficult, but make disambiguation possible. If the format ever needs to be read by a machine, something has to do that disambiguation.

If it’s not provided by the format, you’ve turned every usage into a potential source of bugs that would otherwise be restricted to interchange/stack implementation incompatibilities. In other words, now your format can have a different set of expectations even on the same system.

The natural response to that problem will be to bolt on validation, types, and documentation that is provided arbitrarily (and with varying quality).

IMO, efforts in human friendly formats should focus less on stripping out funny characters, and more on which minimal set of funny characters provide:

- Good readability

- Good editability

- Clarity of structure

- Clarity of data types

- Reasonable tolerance and flexibility for variance in arbitrary formatting/style preference (particularly in delimiting long form/multiline text and annotations), because no one can agree what good readability or editability means

- A flexible type system that allows machines and humans to know what a given datum is without variation or surprises

- Maybe humans should just use a GUI?

mistercow• 6 years ago

I generally find that the biggest problem with human friendly formats like YAML, which I think this also has, is that they tend to decouple readability from writability, and this encourages all sorts of complexity and polymorphism that seem superficially expressive, but end up just being difficult to work with. I've seen so many cases where YAML schemas turn into a quasi-DSL, because the developer thought that it was more important to have a clean looking configuration than one that is easy to edit. The result is that things like indentation get really weird, because the developer didn't optimize for having a sane underlying model.

A great comparison for this is CircleCI's config syntax and that used by GitHub Actions. The Circle format is extremely error prone; about half the time when I'm modifying a Circle config, I'll end up pushing a broken config, even though the YAML syntax itself is valid. With the GitHub Actions format, I almost never screw it up. I don't think it's a coincidence that if you convert a Circle configuration to JSON, it looks twisted and bizarre, whereas if you do the same with a GHA config, it looks perfectly ordinary and sensible.

If you think of YAML as "a prettier version of JSON", and design as if your users will work primarily with JSON, you can do fine with it. If you think of it as a medium for building your own configuration language, you'll make something awful. The problem is that any human friendly format is going to inherently encourage the latter.

mikepurvis• 6 years ago

See also the travesty that is Ansible's YAML-based DSL, which includes fun stuff like an in-line replacement language with tokens enclosed in braces, which of course you have to quote in some cases so that pyyaml doesn't think they are dicts.

kstenerud• 6 years ago

This is basically my goal with https://concise-encoding.org

Also one more goal is twin binary and text formats that are 1:1 compatible, so that you can write it in text and transmit in binary.

I'm still finishing up the reference implementation, and then will start on the schema.

billconan• 6 years ago

Hjson is the best format in my opinion https://hjson.github.io/

alphachloride• 6 years ago

Looks good. Another very worthy competitor the #2 position of the best config language.

These felicitations were brought to you by the .ini gang

transfire• 6 years ago

Don't quite get the use of > for multiline strings. If you don't use > does it mean the string folds lines? Or is it an error?

vortico• 6 years ago

Looks similar to CSON, at least superficially. https://github.com/bevry/cson

draegtun• 6 years ago

Which in turn has more than a passing resemblance to Rebol...

  ; Comments!!!
 
  ; An Array with no commas!
  greatDocumentaries: [
      'earthlings.com
      'forksoverknives.com
      'cowspiracy.com
  ] 
 
  importantFacts: [

      ; Multi-Line Strings! Without Quote Escaping!

      emissions: {Livestock and their byproducts account for at least 32,000 million tons of carbon
  Goodland, R Anhang, J. “Livestock and Climate Change: What if the key actors in climate change we
  WorldWatch, November/December 2009. Worldwatch Institute, Washington, DC, USA. Pp. 10–19.
  http://www.worldwatch.org/node/6294}
 
      landuse: {Livestock covers 45% of the earth’s total land.
  Thornton, Phillip, Mario Herrero, and Polly Ericksen. “Livestock and Climate Change.” Livestock E
  https://cgspace.cgiar.org/bitstream/handle/10568/10601/IssueBrief3.pdf}
 
      burger: {One hamburger requires 660 gallons of water to produce – the equivalent of 2 months’
  Catanese, Christina. “Virtual Water, Real Impacts.” Greenversations: Official Blog of the U.S. EP
  http://blog.epa.gov/healthywaters/2012/03/virtual-water-real-impacts-world-water-day-2012/
  “50 Ways to Save Your River.” Friends of the River.
  http://www.friendsoftheriver.org/site/PageServer?pagename=50ways}
 
      milk: {1,000 gallons of water are required to produce 1 gallon of milk.
  “Water trivia facts.” United States Environmental Protection Agency.
  http://water.epa.gov/learn/kids/drinkingwater/water_trivia_facts.cfm#_edn11}
 
      more: http://cowspiracy.com/facts
  ]

Some references: https://en.wikipedia.org/wiki/Rebol | http://www.rebol.com/article/0108.html

jayd16• 6 years ago

I'm always disappointed when one of these new languages comes up and there's no railroad diagram.

temporallobe• 6 years ago

Lack of data types and schema support aside, there is one significant flaw with this, and it’s the one thing I always hated about YAML: the lack of delimiters. In a code editor that matches start and end blocks, having delimiters is a huge advantage. Heck, even in minified JSON, I can easily identify start/end blocks simply by placing the cursor on the open or closed parenthesis. Also how would you “minify” a format like this where whitespace is significant? I think this is a nifty as purely an alternative for simple (shallow data) uses cases like configuration files, however using this for data exchange, serialization, portability, and deeply nested, complex data structures could be problematic, just as it always has been for YAML.

dreamcompiler2• 6 years ago

The Python community would like a word.

dreamcompiler2• 6 years ago

I like this idea very much; two comments though:

1. Doesn't org-mode already solve this problem nicely, in a more widely-standardized way? (Maybe not; I'm just learning org-mode ATM.)

2.There is no "i" in Topeka.

mkl4• 6 years ago

1. Does anything use org-mode for data files (other than Emacs and things trying to be compatible)?

2. The addresses are of course fake; the zip codes don't match either. Same with the phone numbers. "KateMcD@aol.com" may be real, but "margaret.hodge@uk.edu" isn't.

karmakaze2• 6 years ago

I see two problems with this for my use.

"The format holds dictionaries (ordered collections of name/value pairs), lists (ordered collections of values) and strings (text) organized hierarchically to any depth."

1. Ordered dictionaries are not conveniently supported in all languages I tend to use.

2. The only element type is string which means that parsing of common types has to be separately and differently by each implementation that uses such a file with potential for unspecified differences.

jiggawatts6• 6 years ago

Ordered dictionaries are not conveniently supported in all languages I tend to use.

Ordered dictionaries are a fundamental, extremely basic data type present in every language I've used in three decades.

a) What modern, widely used language doesn't have them?

b) Why?

c) Okay, so you've picked a bad language that has only hashtables. You can still implement ordered keys using an array of the keys & values in sorted order, and a hashtable of keys to the array indices.

fanf22• 6 years ago

In the C++ and Rust standard libraries you have a choice between unordered maps and maps sorted by key, but not maps ordered by insertion.

karmakaze2• 6 years ago

The value of these representations and libraries is that they work in many places, not "It works for me."

Animats• 6 years ago

Maybe what we need is a widely used UI for trees, and editors for them. The editor reads a schema and tells you what blanks to fill in. Export XML, JSON, S-expressions, whatever - any tree structure.

The trouble is, open source cannot do good GUIs. If a problem is best expressed with a GUI, open source consistently blows it. See Gimp, Blender, Inkscape, FreeCAD, all of which are notably worse than their commercial competitors.

saghm• 6 years ago

The trouble is, open source cannot do good GUIs. If a problem is best expressed with a GUI, open source consistently blows it.

What about browsers? Firefox and Chromium are open source (and fit under any reasonable definition of "GUI")

afarrell• 6 years ago

Proposal for someone with more free time than I:

Improve the format of org-mode. Make it primarily easy for humans to interact with but also easy for cheap scripts to parse and manipulate. Create a super-fast CLI for it which ships with the ability to read keybindings from a file. Ship with emacs keybindings as a default but also a file with the spacemacs keybindings. Add the ability to run the CLI as a daemon that can be started from neovim.

Keep the format open so someone else can write some npm package for including an editor in VSCode or a webapp.

jfoutz2• 6 years ago

this has two of my favorite features!

1. comments aren't part of the parse tree, so I can quickly strip all comments by a quick parse print cycle. This is really helpful if you want to do automate changes to the files.

2. I can freely truncate the file at (almost) any point, and still have a valid config file. It's not obvious up front, but if your file gets snipped at some point, it's won't cause parse errors (usually) consumers of the file.

bowmessage• 6 years ago

Could be missing the start of the file too, in rarer cases.

Wonder what the best start of file and EOF-equivalent sentinel would be to use for YAML-esque files.

Something like a double colon?

  ::
  item1:
    - etc..
  ::

makapuf• 6 years ago

2. Add someone says in another comment, isn't that a mistake? What if my file is accidentally cut? In which cases would I want to snip my file without editing it?

epalm• 6 years ago

Regarding #2, would you really be ok with loading half a configuration file? Or am I misunderstanding your point?

jfoutz2• 6 years ago

The parent comment is irony. I've been wrestling with some declarative configuration, and it's getting to that frustrating scale where things are almost the same but a little different.

2. isn't too bad in yaml, since json is valid yaml, a broken file can be reliably picked up with leading '{' and trailing '}'

With regard to 1 - both comments and nonstandard formatting like the above {} hack, are currently hard.

Anyway, got to spend some time outside, and away from screens. Apologies for the snark.

ChicagoDave• 6 years ago

There is such a thing as JSON Schemas if you really want to nail down types.

I don't see this thing getting any traction. JSON is pretty much the standard at this point and I don't see anything changing that anytime soon.

dragonwriter2• 6 years ago

There was a time you could, with as much justification, say that about XML. Not that long ago, actually.

delhanty3• 6 years ago

This is an interesting idea. I don't know whether it will take off.

One thought that occurred to me several times in the last year or so is that roughly the level of abstraction offered by NestedText might make sense as part of a hierarchy of abstractions that could be built on.

We already have that with text files. Because end-of-line character combinations are special, text files in a given encoding are already more structured than streams of characters.

So, assuming UTF-8 character encoding:

  CharacterStream
    Text (EOL character combinations)
      NestedText
        MyNestedTextFormat (domain specific semantics)

With non-text files, this has already happened more than once. For example, both Zip files and Sqlite files are used as base formats for specifying other formats.

rswail2• 6 years ago

ASCII has always had separator and message delineation characters [1][2] that can be used instead of the CR, LF, SP, TAB, "|", "," etc. They are not "human readable", but are very easy to parse.

They are even UTF-8 transparent and can be easily converted to/from CSV, TSV, PSV, and, with a definition of the equivalent of a "close brace" could allow for a multi-level hierarchy.

[1] https://en.wikipedia.org/wiki/Control_character#Data_structu...

[2] https://en.wikipedia.org/wiki/C0_and_C1_control_codes

delhanty3• 6 years ago

I don't know whether that's an "aside" comment, but it's interesting and informative anyway!

Another variant of the idea that I was mulling over was to base a roughly NestedText level of abstraction on UTF-8 transparent characters, and then combine that with what Animats was talking about as a standardized GUI for trees, dictionaries etc.

A recent trend is that programming languages have more than one bijectively equivalent syntax. For example, ReasonML [0] and OCaml are two bijectively text based syntaxes. That idea could be extended to a NestedText like syntax being bijectively equivalent to a text based syntax. Editors like Visual Studio Code infer that sort of information continually on the fly, but it sort of gets lost in the toolchain. Compilers could operate at a higher-level of abstraction than lexing/scanning. Git merge might also work better if could operate at a NestedText like level of abstraction.

[0] https://en.wikipedia.org/wiki/Reason_(syntax_extension_for_O...

samatman4• 6 years ago

Yep.

We were recently storing tokens in a database, and I chose to use SOH for the metadata and SOX for the text.

One byte width, no collisions with printable text, and that's what they're there for.

I'd love to see a CSV replacement that used SOH ... SOX for headers, RS as "commas", and GS as "newlines". You'd be able to cleanly concatenate multiple files, since the first line is no longer special, and you'd be able to have commas, newlines, and in fact any printable text whatsoever inside the data.

And the semantics are perfectly clean. Again, that's what they're there for. Some small challenges for hand-editing that a competent text editor can easily rise to.

rswail2• 6 years ago

JSON is a serialization protocol, not a configuration syntax. It's designed to be written/read by machines. It's convenience for humans is that it can be relatively easily be read or written by humans as well.

Protobufs is similarly a serialization protocol combined with an RPC layer for server and client stubs. Its canonical serialization format is binary.

Neither are particularly well suited for human written configuration files. A "JSON without the quotes around keys and allowing Javascript comments and commas at the end of objects or lists and a mechanism to escape multi-line strings" would probably cover most of the required cases.

YAML is an attempt at that, but also attempts to solve a bunch of other problems in a complicated and fault-inducing way (eg, relying on indentation for hierarchy).

snarfy• 6 years ago

Indentation is used to indicate the hierarchy of the data

After dealing with 500+ line kubernetes configurations, this is a bad idea.

Chakazul• 6 years ago

Indentation by tab is good for shallow data or code e.g. Python. Space is bad. For deeper levels, I'd go for a visible, countable symbol Level 1 .Level 2 ..Level 3

yunruse• 6 years ago

This format seems to support one-space indents, so you could use an editor which highlights spaces easily and set indent level to 1 for visibility. The alternative is unfriendly config or another method of doing the same thing, and both are antithetical to having a simple data format.

bxparks• 6 years ago

I have had great experience using JSONNET (https://jsonnet.org/) as a configuration language. It supports variables, inheritance, operators, functions, substitutions, types, with just the right amount of power, expressiveness, and simplicity.

In my opinion, JSON is best used as a wire-protocol. It is awkward as a configuration language.

YAML works for short configs, but becomes unmaintainable for longer configs. I think the primary problem is that the indentation is significant. I also think the language spec is far too complex.

INI format works for short configs, but also becomes unmaintainable for longer configs. Ironically I think this is because INI is too primitive, the opposite problem of YAML, but has the same effect.

I am not familiar with TOML or DHALL, mostly because I stopped looking after I implemented the JSONNET system and liked it so much.

Addendum: I have used text-formatted protobufs in limited situations with good results. But I don't think that protobufs is a good general purpose configuration language.

Addendum2: The amazing thing about simplicity of the INI file format is that I was able to write a "single line" sed program to parse it in a bash script. The following finds the value of the $key in the $section in the $config_file INI file (definitely works on GNU sed, I think it works on MacOS sed too, not 100% sure though):

    sed -n -E -e \
        ":label_s;
        /^\[$section\]/ {                                                       
            n;                                                                  
            :label_k;                                                           
            /^ *$key *=/ {                                                      
                s/[^=]*= *//; p; q;                                             
            };                                                                  
            /^\[.*\]/ b label_s;                                                
            n;
            b label_k;
        }" \
        "$config_file"

thangalin• 6 years ago

I think the primary problem is that the indentation is significant.

An editor problem, perhaps? We don't maintain office documents using vim; why edit structured configuration files using a plain text editor, if doing so is arduous?

My text editor[0] abstracts the underlying hierarchical data format behind a tree-based widget[1]. Whether YAML, JSONNET, NestedText, CSON, XML, or TOML backs the widget becomes an implementation detail.

[0] https://github.com/DaveJarvis/keenwrite

[1] https://dave.autonoma.ca/blog/2019/07/06/typesetting-markdow...

woah• 6 years ago

One might say that this ain’t a markup language

summitsummit• 6 years ago

wth this is the exact syntax i use for my notes

eecc3• 6 years ago

I’m getting fed-up with this constant reinvention of the serialization game. Back in the days I used to be skeptical of IT and informatics precisely for this reason: always arguing about slight variations of the same mundanities: XML, XSD, IDL, ASN.1, Avro, JSON... Emacs vs. Vim, Weakly vs. Strongly typed, and so on...

What can an ICT professional claim at the end of its career? “Hey, I’ve argued about shit all my life!”

RedShift1• 6 years ago

I feel the same way. I've settled on JSON everywhere. The only thing I don't have in my toolkit is an easy binary ser/deser format.

IshKebab• 6 years ago

Also comments, schemas, hashmaps with anything other than string keys, and sparse reads.

Wilem823• 6 years ago

Your statement implies that arguing is something bad. Arguing is presenting arguments that attempt to show (prove) why something is true or false. It's the most important tool (convincing each other) in the progress of civilization. If you don't try to convince each other, the only alternative, in the end, is just shoot whoever you disagree with.

And it also diminishes your value as a team member. If you can't convince others, means the reasons you present are weak and nobody would be interested in listening to you and therefore there's not much reason in having you around.

eecc3• 6 years ago

And it also diminishes your value as a team member. If you can't convince others, means the reasons you present are weak and nobody would be interested in listening to you and therefore there's not much reason in having you around

Jeez, you extrapolated a personal observation all the way to a character assassination and firing letter paragraph.

Standups and 1on1 with you must be a blessing... a joy

divbzero• 6 years ago

Presenting arguments isn’t bad if it’s about something important, but I think by “Hey, I’ve argued about shit all my life!” the GP takes issue more with the “shit” than the arguing.

amelius• 6 years ago

Yes, when coding always ask yourself how you would tell your grandchildren about what you accomplished.

pelasaco• 6 years ago

The only thing that I can think when I see it https://xkcd.com/927/

offtop5• 6 years ago

Needs to ship with an editor ( or a vs code extension).

One of the biggest issue is just validating the actual data

Printenhart• 6 years ago

So you can't use ' and " in the same key. Does this mean NestedText can't represent all possible key values?

sriku• 6 years ago

I'm not a fan of rigid formats for human-entered metadata structures like config. I think it violates a kind of Postel's law .. where there ought to be considerable flexibility for the human to specify the structure (and therefore in what the format reader accepts). If you want rigidity, just use an appropriate programming language or, say, s-expressions. The rigid formats ask for their own linters to be created and maintained.

In my indian music notation publication tool Patantara, I use a text preamble format that's limited and far simpler for people to use than any of these "data structure capable" formats - including NestedText/YAML/JSON. Here is a quick description -

1. You specify key value pairs with lines of the form -

    key with allowed spaces = any textual value

2. The first " = " on a line separates the key from the value. The key is normalized to lower case and spaces in the key are normalized to single space, and the key is LR-trimmed. The textual value is mostly kept as is (except for LR-trimming), but can accumulate more content.

3. Lines which don't have a " = " are just strings that get appended to the value of the immediately preceding key. So

    key = value1
          value2
          value3

will result in the key "key" having the single string value "value1\nvalue2\nvalue3".

4. If the same key is given multiple times, the values are concatenated with line breaks. So

    key = value1
    key = value2
    key = value3

gets you, for the key "key", the value "value1\nvalue2\nvalue3".

5. The reader doesn't care about data types. However, the application can decide how to parse the string values. One convenience I use for lists of items is "comma or whitespace separated values", where a list of values is given as "value1, value2, value3" or "value1\nvalue2\nvalue3" or "value1 value2 value3" or any combination including "value1, value2,\nvalue3 value4".

Nested structure is not possible with this format (unless you impose more structure on the value in the application), but it serves very well as a metadata format for text files in my application.

Ref: https://blog.patantara.com/blog/2017/04/16/controlling-who-c...

krick2• 6 years ago

So, this is basically YAML, "but better". I can repeat once more that "easily understood and used by both programmers and non-programmers" is unapologetically stupid concept that can never succeed. So I see how all of this will sound all too familiar to anybody with a little experience, which makes them to automatically dismiss this YAYAML.

But YAML is really quite complicated, and JSON (which shouldn't be used for config files at all) and TOML (which I love and wish it would gain more popularity) aren't exactly alternatives to YAML. So, I would be actually totally ok with "YAML, but better", as a way to deprecate YAML.

Now, it is clear from the start that this cannot deprecate YAML, because it doesn't even have booleans and numbers. But, surprisingly, I can accept this as well: ok, let's just assume that being good at dealing with strings may be enough.

The problem is, it isn't clear at all from the docs, if this is better than YAML at anything. It raises dozens of questions. I'll start with the most basic ones (using [] as a wrapper/delimiter): how do I represent values [ a], [a ], ["a"] and [""] in this file format?

zelphirkalt4• 6 years ago

What kind of values are [<whitespace>a] and [a<whitespace>] supposed to be? They look like typical YAML syntax traps to me.

Why should JSON never be used for configuration? It is sufficient for declaratively expressing anything I have encountered. Do we really need references or other stuff from YAML? For configuration this seems unnessecary, provided that the program, which interprets the result of parsing the JSON is well written.

krick2• 6 years ago

What kind of values are [<whitespace>a] and [a<whitespace>] supposed to be?

I don't understand the qustion. They are supposed to be exactly that, [<whitespace>a] and [a<whitespace>]. I assure you, I've encountered many situations, where whitespace at the beginning or the end of the value is actually meaningful for reasons you (creator of the app) have no control over.

Why should JSON never be used for configuration

Many reasons, actually, but the most important (IMO) being that original JSON specification doesn't support comments, nor most actual parser implementations do. Configuration file that doesn't support comments is trash and causes very real inconvenience for users. Using additional key values for comments (even if such atrocity doesn't bother you conceptually) isn't a solution in many cases (for example, when your intention is to comment out a list item).

user5994461• 6 years ago

JSON lacks comments and will fail for a missing or extra comma, so it's not great for configuration written by humans.

You can use HJSON which is the json with comments. It's fully compatible with json so easy to introduce into anything that does json. https://hjson.github.io/

reificator3• 6 years ago

[<whitespace>a] could be a markdown string that begins with code.

msla• 6 years ago

I have an even stupider question: How do I have a key with a colon at the end?

    this:: right
    this: left

Will my program be able to tell its right from its left?

pydry• 6 years ago

basically YAML, "but better"

That was my intention behind this, too:

https://github.com/crdoconnor/strictyaml/

The general structure of YAML is fine I think but its feature set grew a little bit out of control.

The "cleanliness" of the format leads to one of its inherent weaknesses - syntax can't be used to encode type information so you either need a schema to encode type information (strictyaml approach) or have magic conversions (yaml approach) or to assume strings (strictyaml w/o schema/nestedtext).

The interesting thing I discovered about schemas building this is that it kind of pays to make them extensible and build them in a turing complete language. Schema validation done using a non-turing complete language (e.g. jsonschema) allows for cross language usage but it ends up being a kind of blunt object.

simonh2• 6 years ago

I'll start with the most basic ones (using [] as a wrapper/delimiter)

So far as I can tell, this doesn’t use [ or ] in its syntax at all, so all the values you give would be represented exactly as strings without any problem.

reificator3• 6 years ago

So far as I can tell, this doesn’t use [ or ] in its syntax at all

Yes, that's why the GP chose those characters to delimit their example strings. I'll try again using ` characters to delimit the example strings.

so all the values you give would be represented exactly as strings without any problem.

But what would those string representations hold? If I parse the below strings in a javascript context, what would I get?

    ` a`, `a `, `"a"` and `""`

Do I get this?

    " a", "a ", "\"a\"" and "\"\""

Or do I get this?

    "a", "a", "a" and ""

Or do I get something inbetween?

westurner2• 6 years ago

From "The description of YAML in the README is inaccurate" https://github.com/KenKundert/nestedtext/issues/10 :

I will mention something else. The section about the "Norway problem" is not quite accurate. Some YAML loaders do in fact load no as false. These are usually YAML 1.1 loaders. YAML 1.2's default schema is the same as JSON's (only true, false, 'null and numbers are non-strings).
Any YAML loader is free to use any schema it wants. That is, no loader is required to to load no as false. Good loaders should support multiple schemas and custom schemas. The Norway problem isn't technically a YAML problem but a schema problem.
imho, YAML's biggest failing to date is not making things like this clear enough to the community.
Note: PyYAML has a BaseLoader schema that loads all scalar values as strings.

miki1232113• 6 years ago

One cool advantage this format has that is not mentioned anywhere is its potential for localization.

If everything is a string, and you need to parse values yourself, you can tread "prawda" and "fałsz" as booleans, instead of "true" and "false".

marvinblum2• 6 years ago

How is that useful?

miki1232113• 6 years ago

It is, when you're accepting data from users, instead of data from other programs.

An end user with no programming experience could reasonably understand a file written in this format. This can't be said about json or yaml, not when they don't know what "true" and "false" mean.

tgsovlerkhgsel• 6 years ago

Standardization is much more important than perfection. This is a horrible idea.

hermitcrab• 6 years ago

My Easy Data Transform software already supports yaml, json, xml, csv, tsv and Excel. The last thing I need is yet another format!

kokizzu2• 6 years ago

lemme piggyback a bit '__') i love this format, but i don't know the name: https://stackoverflow.com/questions/57470117/name-for-junipe...

alpaca128• 6 years ago

Just give me a JSON with support for comments(and ideally trailing commas) and I'm happy. I don't think there would be a need to simplify that further, and barely a way to do so without making sacrifices at some other important aspect.

tln• 6 years ago

Check out JSON5

torresjrjr• 6 years ago

It's kinda suckless, and I love it.

I think they made some good decisions. All types are strings and left to the application to parse and document. Excellent.

And, I think the desicion for the angle-bracket-multiline syntax was great because I think one could create a parser with awk/grep. Simple.

epicureanideal2• 6 years ago

I think the idea of leaving interpretation of the values to the end application is a good idea. I forget the details, but there was something about this idea related to the TCP/IP stack, with error correction being done at the application level rather than at one or more intermediate levels.

fanf22• 6 years ago

You are thinking of the end-to-end principle http://web.mit.edu/Saltzer/www/publications/endtoend/endtoen...

nicexe• 6 years ago

I read a bit but I'd like to see a comparison of the same data structure in JSON, YAML and NT.

I would also like to see YAML converted to NT and back to YAML just to see if there is any information loss.

jmull• 6 years ago

This cites as an advantage over JSON:

Unicode characters without encoding them

But JSON doesn’t generally require special encoding for unicode characters.

Technically, JSON is UTF-8 encoded, but that’s true of NestedText too so that can’t be what they mean.

Also, of JSON it says:

in JSON 32 is an integer, 32.0 is the real version of 32...

But JSON doesn’t distinguish integer from real. It only has number.

On the format itself, I wonder if it needs some testing and hardening. The definition seems ambiguous, but maybe just needs a formal grammar. Just from reading it, tab handling seems like an issue. I think you could have documents that look right but have invisible issues due to tabs. E.g., it sounds like tab can be a character in a dictionary key name, which looks like indentation.

textman• 6 years ago

NestedText is a file format for holding data Then why not call it NestedData? NestedText is misleading because text and data are different. That was a distinction many users of XML failed to make when they used it to store data and ended up dissing XML for being obese when it was designed for large text documents (wherein XML tags are an insignificant portion of the document), and not data.

malodyets2• 6 years ago

Every time we have a conversation that touches on any serialization or configuration file format, it doesn’t take long for someone to pull out the flail and start beating on XML again.

XML might not be the best format for everything, and I for one am glad to use other formats for simple structured data. But when it comes to representing complex content, there is no other format that even comes close to being as useful and usable.

* All digital publishing of ebooks uses XML inside ZIP files.

* All contemporary mainstream word processors (Word, LibreOffice) use XML inside of ZIP as the basic file format.

* Automatic customized conversion processes from Word to InDesign or from InDesign to EPUB use XML at the heart.

* Let’s not forget the web itself, which is still mostly SGML in the form of HTML. Not XML per se, but only different in the details.

Not only is XML the only practical serialization format for working with publication content, but the presence of mature schema tooling is intrinsic to making publication automation robust in a given context.

I’m very glad for JSON and JSON schema in the domain of APIs. But in the domain of content data, it’s all XML.

Every serialization format has a domain for which it is most appropriate (whether or not it is the best choice in that domain.)

I’m really liking the shape of nested text for the domains in which I would have used YAML.

prussian• 6 years ago

I actually think this is a great idea. There are a few immediate questions I have though.

1) how does one define a key with a ": "

2) as others said, significant whitespace, specifically trailing, seems to lead problems with keys that have nested key/val, array, multiline string.

3) the github repo associated with the parser has an isue that questions how to verify if the file was truncated.

I think Deco: https://github.com/Enhex/Deco better solves the problem the OP has. It is provably delimiter collision free, unlike this (see my point 1 Unless I'm mistaken).

laserbeam• 6 years ago

There's a pretty good usecase which I highly doubt will ever be implemented: copy/paste between applications.

I don't think this will ever happen, but having a human readable plain text output whenever you select any data from a UI is a very powerful idea. EVE Online did this back in the early 2000s and let you copy virtually anything in UI. This led to awesome third party tools that could work with the game with a minimal learning curve from the user (everything was "paste what you copied from the game"). It did use a custom serialization format... But a plain text one nonetheless.

IO via copy paste between a wide variety of apps in a known, and very simple, structured format would be awesome in my books.

EADGBE• 6 years ago

Lol, “Topika”

NestedText, a nice alternative to JSON, YAML, TOML

Related Stories