Designing a Good DSL

by zonotope • 8 years ago

115 34 8 years ago

NickBusey• 8 years ago

Interesting article, though the title should perhaps be `How not to design a Bad DSL` since the vast majority of the advice is apparently what NOT to do.

nerpderp83• 8 years ago

If the no-go space is larger and easier to fall into, it makes sense to have a more detailed danger map. It might stop a lot of folks from even taking the journey, which unlike real journeys is proper choice. If you are making a DSL for your own enjoyment, don't transmute your joy into someone else's pain. If end users have real issues that your DSL will solve, by all means, make it.

traverseda• 8 years ago

I'd like to see a parser-generator built using these principles, as most of the DSLs for building DSLs are "bad DSLs", by this definition.

wcrichton• 8 years ago

Arguably parser combinators satisfy the author's requirements, e.g. https://github.com/Geal/nom/

nixpulvis• 8 years ago

Does this count? https://github.com/nixpulvis/parser-combinator

bd82• 8 years ago

Have a look at this: https://tomassetti.me/parsing-in-javascript/#chevrotain

This is Not exactly a parser generator but an internal JavaScript DSL for writing parsers, but it does meet the criteria for a more verbose syntax.

Twisol5• 8 years ago

(EDIT: I replaced asterisks with <AST> in the pattern below, since HN doesn't have a convenient way to escape asterisks last I checked.)

First, most of its syntax beyond the very basics like "X+" or "[^X]" is impossible to remember. It’d be nice to know what "(?<!X)" does without having to look it up first.

I realize regex can be hard to remember, but I think this is a little overblown. "(? ...)" is the general form of an extended group -- something for which "special behavior" occurs. The character sequence following that determines what the special behavior is. In this case, "<!" means "negative lookbehind": "<" for "lookbehind" and "!" for "negative". Compact, but mnemonic.

[a-z]{3,10}://([^/?#]<AST>)([^?#]<AST>)(?:\?([^#]<AST>))?(?:#(.<AST>))?

Yes, this is very dense. But it's not really meant to be read at a glance -- URLs do not have an exceptionally simple pattern, and they have multiple parts, many of which are optional. You could write a recognizer for URLs explicitly, but I doubt it would be as immediately recognizable in full. I consider it more important to obtain a high-level understanding before a low-level understanding, and (for me at least) I can see that this regex matches URLs up front, setting my expectations for the details later.

I think a PEG or combinator parser would be more self-documenting, so I'm not saying regex can't be pushed too far. But it isn't nearly as unstructured as it looks.

Many DSLs were designed to reduce amount of non-DSL code to the absolute zero. They try to help too much.

I 100% agree here. A DSL should be laser-focused on doing one thing well. Any language should be composed of orthogonal features, each laser-focused; a DSL is just a language with a very small handful of features geared toward a specific domain. (Of course, ideals are rarely realized in full.) I particularly like GraphQL as of recently; it has a really nice feel to it.

I found an interesting paper [0] on design principles for DSLs while composing this comment. I've only skimmed it, but it looks quite nice.

[0] https://arxiv.org/abs/1409.2378

Twisol5• 8 years ago

Incidentally, here's how I might write that URL regex with comprehensibility outweighing all else. Notice that I enable free-spacing [0], which can usually be enabled by passing a flag rather than embedding it in the regex. In Python, the re.X (or re.VERBOSE) flag does the job.

    (?x) # free-spacing
        (?P<protocol> [a-z]{3,10} ) ://  # http://
        (?P<domain>   [^/?#]* )          # google.com
        (?P<path>     [^?#]* )           # /path/to/resource
    (\? (?P<query>    [^#]* ))?          # ?id=123
    (\# (?P<fragment> .* ))?             # #section

[0] https://www.regular-expressions.info/freespacing.html

jmts5• 8 years ago

Claiming that regular expressions are too terse is a bit much. There are only three (!) fundamental operators in basic regular expressions (four if you include parenthesis), with all other non-language specific operators being derived from that (ignoring precendence rules):

1. concatenation, to append regex A or regex B: AB

2. alternation, to select between A or B: A | B

3. kleene star, to repeat A zero or more times: A*

4. parenthesis allows specification of a sub-expression: (A)

The following are all derived/syntactic sugar:

[ABCD] -> (A | B | C | D)

A+ -> AA*

A{2} -> AA

A{2,4} -> AA(|A|AA) or A(A|AA|AAA)

A? -> (A|)

Just about everything else is implementation specific (if choice of special characters and available operators isn't already). That means you either need to be using the features regularly to remember them, or you have to look them up anyway.

Regular expressions are terse not because they are badly designed, but because by definition the description of regular languages is inherently minimal. It is part of their beauty. Without this minimalism, every tidy little one liner we have to perform some simple match becomes a multi-line specification in Backus-Naur form.

The world needs to get over this fear of regular expressions from ignorance and continued misinformation. They are not magic or impossible to understand. They are an elegant description of a very simple state machine which steps through a string one character at a time, nothing more.

Edit: corrected derivation of A{2,4} a la Twisol and jbnicolai.

Twisol5• 8 years ago

Minor nitpick: A{2,4} becomes AA(|A|AA), not AA(A|AA).

jrochkind1• 8 years ago

`AA(|A|AA)` is valid regexp syntax?

Holy moly it is! I had no idea you could put a | right after parens like that! I guess (|A) is nothing or A? Woah.

jmts5• 8 years ago

Well spotted. Thanks for providing the correct derivation.

jbnicolai• 8 years ago

Agreed. The problem is the risk of small, relatively hard to spot & nearly impossible to properly debug mistakes.

A{2,4} -> AA(A|AA)

Twisol5• 8 years ago

And this is why we have the syntax sugar.

jmts5• 8 years ago

Well spotted.

I would argue that risk of mistakes should not limit the expressivity of a language or have it added to the pile of bad ideas. It is better for users of the language to be aware of potential pitfalls, and use the language appropriately.

See my reply to shafte for some elaboration.

shafte• 8 years ago

To be fair, you are not including more advanced operators, like positive/negative lookahead/behind (which is the specific example the article uses), capturing and non-capturing groups, greedy vs non-greedy kleene stars, etc.

As you say, they are implementation specific, but that's part of the problem: the basic regular expression syntax is insufficient for many tasks, so people take to extending it in complicated and syntactically opaque ways. That's the sign of a bad DSL, not a good one.

jmts5• 8 years ago

Maybe I'm arguing semantics here. To be clear, my point is that I do not agree it is reasonable to declare that regular expressions are a bad DSL simply because it is possible (however common) for people to write difficult to read, or difficult to understand regular expressions. It is the responsibility of the author of the expression to ensure that it is readable and understandable - to the extent that they should exercise restraint when possible use of an available feature would hinder readability and understandability.

There is absolutely no need for the example regex of http-like strings to be written the way that it is - there is only the want of the author, because they have a hammer and they are looking for a nail. If anything, using a regex for such a thing sets a bad precedent because anybody who wishes to come along and add user@password support to it is going to extend it and make it worse.

A more understandable way to process such a string would be to split it into constituent parts and use regex only for validation. Split at the :// for schema, split at the next / for path, etc. Turn these into functions, and keep the regexes simple.

Regular expressions are notorious because they are abused, not because they are evil.

mirimir• 8 years ago

What drove me crazy at first was that any character can be used as a delimiter. That's very useful, I admit. But it complicates understanding examples found through searching ;)

cestith• 8 years ago

Perl6 goes far beyond regular expressions.

https://docs.perl6.org/language/grammar_tutorial

https://docs.perl6.org/language/grammars

Even with Perl5, if you're using too many regular expressions, you might want to check CPAN.

https://metacpan.org/search?p=1&q=parse&size=500 (over 4k results searching for "parse")

In particular, Parse::RecDescent, Parse::Yapp, Parse::Lex, Parse::Flex, Regexp::Common, Net::IP, NetPacket::IP, ... really most of the things you'd want to parse from Apache::ParseLog to Parse::DNS::Zone

The inclusion of (not so) regular expressions in a language doesn't mean one needs to abuse them.

greenyouse• 8 years ago

Would anybody mind sharing links to good DSLs?

From Clojure I think hiccup is a good example.

https://github.com/weavejester/hiccup

9214• 8 years ago

Embedded DSLs for:

* implementing other DSLs - https://www.red-lang.org/2013/11/041-introducing-parse.html

* description of visual interfaces and specification of 2D drawing operations - https://www.red-lang.org/2016/03/060-red-gui-system.html

* low-level programming - https://static.red-lang.org/red-system-specs.html

EdCoffey• 8 years ago

I'd add a few extra points:

1. Don't (unless there's an extremely good reason to do so). Ask yourself: is there a net benefit gained by forcing a developer to learn and use your DSL, ignoring their likely familiarity with general purpose languages that could solve the same problem? For starters, any application where humans won't be reading/writing a large amount of the DSL is probably out.

2. Avoid anything that resembles natural language. If you're writing a true natural-language interpreter, you're not writing a DSL. If you're writing a DSL that looks like natural language, people will be tempted to apply the grammar rules they already know from that language, rather than the strict grammar of your DSL, resulting in frustrating errors. It's a whole lot easier to memorise the rules in a language of keywords and symbols, because you don't have to first banish your existing knowledge of natural language.

3. Don't try to accomodate "non-technical" users performing intrinsically technical tasks. There's no point creating a "friendly" DSL over HTML and CSS if the people authoring in the DSL still require an in-depth knowledge of the box model, responsive web design etc. All you've done is kicked the can down the road and created a false sense of capability.

bocklund• 8 years ago

3. Don't try to accomodate "non-technical" users performing intrinsically technical tasks. There's no point creating a "friendly" DSL over HTML and CSS if the people authoring in the DSL still require an in-depth knowledge of the box model, responsive web design etc. All you've done is kicked the can down the road and created a false sense of capability.

Should be #1 in the OP. So underrated.

carlmr• 8 years ago

These rules are so much better than the article.

ellisv• 8 years ago

Another non-verbose DSL example is Java date and time format string:
"YYYY-MM-DD'T'HH:mm:ss.SSSZ"

Oh that crazy Java formatting date times using ISO 8601!

Although the author didn’t really mention any good DSLs I do appreciate the tips in the second half.

Designing a Good DSL

Related Stories