A Formal Spec for GitHub Flavored Markdown

https://githubengineering.com/a-formal-spec-for-github-markdown/

by samlambert • 9 years ago

475 89 9 years ago

libeclipse• 9 years ago

I'm really happy to see this. It's actually quite frustrating that although markdown is so nice, it barely has a consistent standard. It's almost impossible to use it cross-service.

Hopefully now that Github has standardised their own flavour of it (and quite a nice flavour too), more people will start to use it.

Of course there is the obligatory XKCD: https://xkcd.com/927/

zzleeper• 9 years ago

I would argue that it is consistent now.

At the lowest level, you have commonmark. Then, you have extensions at the top, such as GFM.

If Pandoc/Github/Reddit/SO/kramdown switch, that accounts for almost all front- and back-end cases that I care about.

And given that the first four were actively involved with commonmark, I would take as given that they will support commonmark or a superset of it.

kccqzy• 9 years ago

Pandoc already supports different flavors of markdown (commonmark included) and you can add/remove extensions to base flavors.

legulere3• 9 years ago

It's a specification based on the commonmark specification. Both are not a formal specs. They are more of an informal specification with some edge-cases listed (in contrast to the original markdown specification which has known unspecified edgecases).

UK-AL3• 9 years ago

I really wish their was concise formal spec for markdown, rather than a multi-page essay. It makes it incredibly difficult for anyone trying to create something to parse it. There is no mechanical way for go from spec -> parser.

I think its quite difficult to do though.

adrianN2• 9 years ago

Is it really useful to write a formal spec for Github Markdown? The software they use to parse and render it is open source. If you want to know how exactly something works, you can read the source.

oblio• 9 years ago

Is there any common spec where you can mechanically go from spec to parser? HTTP, SMTP, DNS, HTML, CSS, Javascript, Ruby, Python, C, ... I basically know of nothing in widespread use with a spec that can actually be converted directly into working code.

Ericson23142• 9 years ago

I hate calling things formal which aren't. Also, I hope this puts to rest any idea of Markdown being "simple".

legulere3• 9 years ago

https://github.github.com/gfm/#disallowed-raw-html-extension...

Why this? This is not a working blacklist to prevent XSS (e.g. onload="...")

Xylakant• 9 years ago

It's not meant as an xss prevention but as a safety to prevent rendering errors.

tanoku• 9 years ago

Hi there! As the spec explains, this is a Markdown specific blacklist that prevent the tags that would otherwise "break" the content of the Markdown document.

A document that contains these tags will not be parsed properly by an HTML5 compliant parser; the parser will "swallow" other chunks of Markdown content that come after the tags. Hence, we disable the tags altogether.

This is an UX feature, not a security feature. XSS prevention, and a plethora of other security checks, are performed by our user content stack -- but this functionality is shared for all markup languages in GitHub (MD, RST, ASCIIDOC, ...), so it's not discussed in this spec.

scrollaway2• 9 years ago

Wow, TIL about the <plaintext> tag. Here I thought I knew most of the corner cases of HTML.

dbbk• 9 years ago

What's bonkers to me is that there is no closing tag, everything after it is no longer parsed as HTML.

simplehuman• 9 years ago

This is great! A couple of years back, there was a failed attempt at standardizing this - http://www.vfmd.org/ and http://www.vfmd.org/vfmd-spec/specification/. GitHub given it's popularity will surely succeed more.

zeveb• 9 years ago

There was also Common Mark (http://commonmark.org/), which failed IMHO mostly due to John Gruber taking offense at their first choice of name, Common Markdown. Will formalising this as 'GitHub Flavored Markdown' similarly cause offense?

Ajedi324• 9 years ago

GitHub Flavored Markdown has been a thing for a while. The only difference is that now they have a formal spec for it.

Also, CommonMark failed? News to me. Last I heard it was still under active development, years after the drama with Gruber.

criddell3• 9 years ago

IIRC, their first name choice was Standard Markdown. I don't blame Gruber for being upset at that.

mwfunk• 9 years ago

Who knows, but it'd be pretty irrational if it did. Whether it was fair or not, the "Common Markdown" name caused a kerfluffle because of the implication that it was claiming to be the One True Markdown. The name "GitHub Flavored Markdown" only implies that it's base Markdown plus GitHub extras, plus it's been known by that name for a long time now.

gregmac• 9 years ago

Did it 'fail'? It seems to have a decent amount of use. And it fixed a bunch of problems with the original Markdown spec (or lack thereof).

I suppose since one of their goals was to get Github and StackExchange 'out of the markdown business' but neither use CommonMark, and further, Github now has put work into creating their own spec, they failed in that aspect.

steveklabnik5• 9 years ago

The Rust ecosystem has used Markdown for a long time, but is moving to CommonMark as we speak.

Given this news from GitHub, it's very exciting.

derrickdirge• 9 years ago

TFA is literally all about how GFM is based on the CommonMark spec.

pkamb• 9 years ago

Previous discussion on the name: https://news.ycombinator.com/item?id=8270771

nerdponx3• 9 years ago

It didn't fail. CommonMark is the standard implemented in Pandoc, and the projects share an author.

Ajedi324• 9 years ago

GitHub's spec here is based on [CommonMark][1], which has been around for a while now, and [was originally authored][2] by a group of representatives from GitHub, Reddit, and Stack Exchange.

[1] http://commonmark.org/

[2] https://blog.codinghorror.com/standard-flavored-markdown/

LoSboccacc• 9 years ago

based on, compatible with, or is common mark?

I'd hate to see them pushing a different spec around, that would solve nothing

forsaken2• 9 years ago

This is great -- lack of a standard that was actually used (unlike CommonMark) was one of my main issues with Markdown (http://ericholscher.com/blog/2016/mar/15/dont-use-markdown-f...) -- It's really great to see GitHub leading in this department, and it gives me hope that one day we might actually have Markdown that is portable between implementations.

SEJeff• 9 years ago

Perhaps you missed Jeff Atwood (codinghorror)'s standardized markdown spec, which is about as close as you'll get:

http://commonmark.org/

forsaken2• 9 years ago

Yea, I mention commonmark in the post, and referred to it in the "standard that was used" part -- commonmark is great, but wasn't widely adopted.

Updated my original post to be more clear.

zellyn• 9 years ago

This is actually very closely based on CommonMark, according to the article.

jotux• 9 years ago

The github spec is literally CommonMark with extensions.

https://github.github.com/gfm/

http://spec.commonmark.org/0.27/

jorams2• 9 years ago

It's odd how neither this post, nor the spec, nor GitHub's "Mastering Markdown" help page[1], nor the more complete "Basic writing and formatting syntax" page[2], mentions the fact that GitHub treats every newline as a hard break.

CommonMark contains this little sentence to work around its specified behavior, which is left untouched in the GFM spec:

A renderer may also provide an option to render soft line breaks as hard line breaks.

I'd say whether or not it does this is a rather important thing to mention. When I write Markdown documents for GitHub I have to change my editor settings, only because of this.

[1] https://guides.github.com/features/mastering-markdown/

[2] https://help.github.com/articles/basic-writing-and-formattin...

y0ghur7_xxx• 9 years ago

It's odd how neither this post, nor the spec, nor GitHub's "Mastering Markdown" help page[1], nor the more complete "Basic writing and formatting syntax" page[2], mentions the fact that GitHub treats every newline as a hard break.

Well, actually it does say that "Hard line breaks are for separating inline content within a block."

https://github.github.com/gfm/#hard-line-break

So two spaces at the end of the line for a <br>, and an empty line for <p>.

jorams2• 9 years ago

Right, but that's not what GFM does. GFM turns any single newline into a <br>.

kivikakk4• 9 years ago

GFM itself leaves that line unchanged because we don't actually change that option in our implementation — the reference implementation `cmark` (which we built upon) supplies the "hardbreaks" option, described as follows:

--hardbreaks Treat newlines as hard line breaks

We turn this option on when rendering issues, issue comments and so on, but leave it off when rendering blobs (such as README.md). Both are GFM, one just uses this option to make it more conducive to communication.

yuchi• 9 years ago

This is a very important bit of information.

Do you mind to make sure this is reflected in docs?

tbrock2• 9 years ago

Good to know. Why is the setting different for issues than for blobs?

dreamcompiler3• 9 years ago

This is great news! Does anybody have a recommendation for a Javascript parser for Formal GFM? (I know there are a million JS MD parsers; I'm looking for a good one that will let me serve GFM docs over HTTP and render them on the browser.)

steveklabnik5• 9 years ago

https://www.npmjs.com/package/marky-markdown

dreamcompiler3• 9 years ago

Looks good. Thanks.

harmonyinfotech• 9 years ago

Is it ok if I promote my domain here? I read the guidelines but it doesn't mention anything regarding self promotion. Sorry if it ain't appropriate but anyone looking for a relevant domain (markdown.in) please get in touch or any suggestion if it is better to develop it.

erlend_sh• 9 years ago

GitHub devs & Markdown enthusiasts at large, please consider contributing some brainpower to these last remaining issues that are blocking the v1.0 release of CommonMark:

https://talk.commonmark.org/t/issues-we-must-resolve-before-...

rcarmo• 9 years ago

I have to wonder why this isn't done in the form of a context-free grammar, like Hitman[0] uses. Specs in English are still too vague for my liking.

https://github.com/chameco/Hitman

daigoba66• 9 years ago

Is it even possible to define a context-free grammar for markdown?

jws2• 9 years ago

It is not, and not just because of a few idiosyncrasies like C which require context.

Fundamentally, markdown was specified as some pattern matching and English description. The original specification was not done thinking of productions and grammar rules, and you typically don't get there by accident.

See http://roopc.net/posts/2014/markdown-cfg/ for a detailed exposition of how the '*' character in markdown is sufficient to ruin any chance of a CFG.

I have sometimes pondered how to make a markdown-like language with a simple production based grammar. I have not succeeded and would appreciate any pointers. The criteria being that is has to have something like the minimal intrusion into the prose of markdown.

strikedout• 9 years ago

Why no ~~strike out~~ in spec?

kivikakk4• 9 years ago

Right here: https://github.github.com/gfm/#strikethrough-extension- :)

oneeyedpigeon• 9 years ago

Is their no corresponding equivalent of INS? I would have thought something like

    +new text+

could be used...

flippyhead• 9 years ago

Jeesh, about time!

patrec• 9 years ago

Now if only org-mode could define a sane, parsable format.

alphapapa• 9 years ago

What do you mean?

tomcam• 9 years ago

web2py handled all of these issues and made its markup language extensible with its markmin specification: http://www.web2py.com/init/static/markmin.html

IgorPartola• 9 years ago

At the risk of starting a mini-flame war, is RST a more cohesive format? If one was to pick one of the two formats to start using for personal documentation, which format should one choose?

coddingtonbear• 9 years ago

For personal documentation (assuming you mean notes): follow your heart. Personally, though, I wouldn't use RST for any non-python public documentation at this point, but for my personal notes, it's hard to beat the extensibility of RST.

mdaniel• 9 years ago

I enjoyed this article http://eli.thegreenplace.net/2017/restructuredtext-vs-markdo... and since the article didn't provide supporting evidence for the Linux/OpenCV/LLVM assertion: https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html and http://docs.opencv.org/2.4/doc/tutorials/introduction/how_to... and https://github.com/llvm-mirror/llvm/blob/master/docs/index.r... respectively

I do think rST is waaay more expressive, but I also recognize that in many of the instances one would want to use markup in a chat or PR situation, the expressiveness likely wouldn't be well received if the trade-off is verbosity.

This is something in life that I file away with competing regex standards: my brain just has to switch languages based on the app in which I'm typing (between markdown, pseudo-markdown (ahem, Slack), org-mode, rST, etc).

jostmey• 9 years ago

HTML used to serve as a simple way to format a document. Now HTML is too complex for that purpose. Introduce markdown. In ten years, markdown will be too complex for formatting documents.

I am big of markdown in case I didn't make that clear. I love it

hawkice• 9 years ago

It wasn't an issue of simplicity, it was about HTML being extremely hard to sanitize so it isn't turing complete. Having formatting (that can still line up with your sites styles) while knowing you can't get script injection is pretty useful.

wtetzner• 9 years ago

I don't think it's that HTML is too complex. It's more that Markdown allows documents to be marked up in a way that "natural" and easy to read if you're reading the plain text version of it.

Siecje• 9 years ago

What's wrong with http://www.vfmd.org/ ???

charonn0• 9 years ago

So that's why my project wikis have suddenly stopped rendering markdown properly. I've been trying to figure out WTF was going on since yesterday!

A Formal Spec for GitHub Flavored Markdown

Related Stories