GitHub dropped Pygments

http://www.greghendershott.com/2014/11/github-dropped-pygments.html

by mkonecny • 12 years ago

184 129 12 years ago

jarcane• 12 years ago

It is still difficult for me to express my opinion of this move without simply resorting to strings of profanity. Frankly, I suspect Atom has more to do with it than anything.

walterbell10• 12 years ago

Why is Atom more strategically important to Github than existing developers? Is there an economic belief that Atom will reach some larger base of untapped developers who don't know how to use other editors?

If a vendor cannot be trusted to host your content without regression, why would you trust that vendor to supply your mission-critical editor?

stingraycharles• 12 years ago

No, but the article highlights that maybe they are using Github's traction to get a better syntax highlighter for Atom. As in, instead of waiting for a large community of Atom users contribute Atom syntax highlighting, they simply push the same engine onto Github and let that community take care of the syntax highlighting they actually care bout.

If this were the case, it would be slightly evil, but very clever business-wise. But we don't know, so all this is is a small conspiracy theory.

cespare• 12 years ago

I assume the new syntax highlighter is way, way faster than pygments. It's written in C++ rather than Python. (Atom uses the same grammar format, but a Node implementation.)

walterbell10• 12 years ago

Why not give existing users the choice of migration and timing?

scott_karana• 12 years ago

Giving users the choice of "slow" in a webapp means that both hardware and software deployment/management/purchasing is greatly complicated.

walterbell10• 12 years ago

Dear downvoters, have you heard of the Streisand effect? This thread now has the most downvotes (-14) of the many comments I've posted on HN. There is nothing factually incorrect in my posts. Why does this question deserve five downvotes?

Why not give existing users the choice of migration and timing?

Edit: -20 downvotes. Previous record was -3.

dhaivatpandya• 12 years ago

I don't think it is necessary for Github to provide choice for this sort of thing. It isn't a ground-breaking, massive change to their main product. To me, it seems like an unnecessary choice that would only create confusion and make their product more difficult to navigate and use.

cremno• 12 years ago

Your assumption is correct. The blog post links to an issue in which an official statement (and a graph) can be found:

https://github.com/github/linguist/issues/1717#issuecomment-...

By using TextMate grammars we also get some nice features like highlighting SQL inside Ruby heredocs. But the main motivation was improving performance.

joemaller1• 12 years ago

Most important link in this thread.

wbond• 12 years ago

What is actually being used for syntax highlighting now? Linguist doesn't appear to contain any C++.

lexcorvus• 12 years ago

The new highlighter may be faster, but is syntax highlighting frequently a bottleneck at GitHub? That seems unlikely.

rsanheim2• 12 years ago

In large diffs, it can definitely be a problem.

pjc502• 12 years ago

Given the discussion going on at https://news.ycombinator.com/item?id=8650952 , this sounds like it might also increase the risk of exploit.

isomorphic6• 12 years ago

This sort of thing feeds my paranoia about GitHub being a giant single point of failure in the open-source world.

I know the argument: Someone, somewhere has a copy of each repo checked out, so we (the nebulous "we") could reconstruct everything from the diaspora of ".git" directories.

It just bothers me to think how dependent OSS has become upon GitHub.

Demiurge4• 12 years ago

I don't think it's paranoia. I think it factually is a single point of failure, which is a bad thing(tm). And GMail is a single point of failure for my (and my companies) e-mail, and that bothers me too, despite the fact that I'm not doing anything about it :)

isomorphic6• 12 years ago

Google is a giant single point of failure. Others would rush to fill the vacuum, but it'd be a brutal few months (years) while we all relied upon Bing.

Is there any GitHub-esque outfit waiting in the wings that provides free OSS hosting?

shurcooL3• 12 years ago

This planet is also a single point of failure. If a meteor strikes, all our code will be lost.

Sir_Cmpwn2• 12 years ago

The counter-argument is true, though. I have a server that I'm able to log in via SSH. I just backed up one of my repositories like so:

    sircmpwn@homura ~/s/K/kernel master> ssh irc.sircmpwn.com git init --bare kernel
        Initialized empty Git repository in /home/sircmpwn/kernel/
    sircmpwn@homura ~/s/K/kernel master> git remote add backup irc.sircmpwn.com:kernel
    sircmpwn@homura ~/s/K/kernel master> git push backup master
        Counting objects: 4735, done.
        Delta compression using up to 8 threads.
        Compressing objects: 100% (1765/1765), done.
        Writing objects: 100% (4735/4735), 7.62 MiB | 650.00 KiB/s, done.
        Total 4735 (delta 2954), reused 4699 (delta 2930)
        To irc.sircmpwn.com:kernel
         * [new branch]      master -> master

walterbell10• 12 years ago

There is value in the aggregation of repositories which is separate from individual repos.

rdtsc• 12 years ago

Good points.

There is a silly joke about that even. It usually goes like --

"Gee, I wish someone would invent a decentralized version control system".

hobarrera3• 12 years ago

Well, if GitLab supported cross-site pull requests, we'd have just that.

jacquesm• 12 years ago

Why do you think that that is limited to open source? Plenty of businesses utterly rely on github in their workflow.

isomorphic6• 12 years ago

It's not limited to open source, but commercial enterprises pay for GitHub's services, so at least there's some sort of service agreement. OSS depends upon GitHub's largesse (which is extensive). However, if GitHub changes its mind (deciding, for example to focus on selling code-editing software), or if something goes wrong with the commercial side, then OSS is left high and dry.

esfandia• 12 years ago

Would a P2P version of GitHub make sense? Each outward-facing local repo would provide a GitHub-like interface for browsing code, submitting PRs etc., and would in addition route repo search requests to neighboring repos using something like a Gnutella protocol.

curun1r• 12 years ago

I don't think that would work well...Github has too much momentum. But I think open protocols and folding more of the functionality that has been proven out at Github into the core SCM could have a very positive impact.

To start with, pull requests could be implemented in the main git tool. They're no longer experimental and many, if not most, git users rely in them in some form or another (Github, Gitlab and BitBucket all support them). Folding them into the core would just standardize all the implementations.

It would also be good to define protocols for collaboration. Off the top of my head, that could mean a fork:// browser protocol that would allow a BitBucket user to fork a Github repository and seamlessly submit pull requests back upstream. Some of that is possible today, but there would be some new requirements around federated authentication to enable this (i.e. how to allow a user who is registered with a different service to create a pull request).

If the mechanics of interoperability are standardized, people will develop competitors to Github and things will get more decentralized. But, as mentioned above, Github has a lot of momentum and no incentive to cooperate with other providers. The only way they're really forced to work with others is if these changes and standards are coming from the core Git project.

biomechanica• 12 years ago

Well, there was a project called GitChain*; however, it seems that the project is looking for a new leader. Was a very cool idea.

-> http://gitchain.org

paulannesley• 12 years ago

Lightweight but great for LANs like the Rails Camp that it was built for: https://github.com/toolmantim/bananajour

shurcooL3• 12 years ago

GitHub being a giant single point of failure

What is our definition of a "point"?

If you have an external hard-drive backup of your laptop, that's 2 points of failure, right? If someone else has 10 external hard-drives that they keep in different places, that's 10 points, yes? But what stops you from calling all those hard-drives "a giant single point of failure"? If all of them are destroyed, the data is lost.

I just don't get these arguments... The chances of all GitHub data being lost is probably less likely than BitBucket and SourceForge combined.

isomorphic6• 12 years ago

By "failure," it never occurred to me to mean a technical or operations failure.

I'm thinking: shut down due to the business model not working, or some other business-model variant, such as gradually getting intolerably crappy like SourceForge. Cash flow isn't great; let's introduce "sponsored" code. Etc.

luch• 12 years ago

You're right about the 10 hard drives : if you've bought them identically at the same time, it's likely they all gonna fail in a short time span. I've recently heard about the "3,2,1" rule concerning backup :

  - 3 images of your data (1 original + 2 copies).
  - 2 types of media (e.g. extHD and DVD)
  - 1 offshore location (e.g. bank vault, your parents' house,etc.)

Of course it's not a golden rule, but it can prevent catastrophic failures.

hueving• 12 years ago

It's organizationally a single point of failure. The company could easily decide to shutdown access to all of it's servers and everyone relying on github for issue tracking, etc would be screwed.

Demiurge4• 12 years ago

In this case my definition of a point is a single remote to github.com/<project> in all the OSS projects .git/config

A single issue tracker with hundreds of issues for all these projects. A single database of commiter permissions. Etc.

agumonkey2• 12 years ago

Should there be a limit on the size of any organ, to avoid this bit bus factor ? Like hierarchies starting to be needed after a group reaches 150 people.

isomorphic6• 12 years ago

Well, the current Internet pendulum has swung to nearly the polar opposite of federated services, much to the dismay of my inner software-libre graybeard.

Even a duopoly (!) would be preferable to a single vendor.

Dewie• 12 years ago

How hard is it for people to switch, if they want/have to? They might be dependent on GitHub issues. That might be preventable by using some issue tracker that can actually be put under git (like Ditz), though I guess it makes collaboration with non-owners of the repository more difficult. Other than network effect, what else is it that binds people to GitHub?

hobarrera3• 12 years ago

Not 100% accurate. Issues and pull requests (as well as their respective comments) are not in the git repository. They're not easily cloneable either.

I'd love to see all that metadata in a separate git repo (just like wikies).

abhinavg• 12 years ago

I honestly don't see a problem here. They decided to change a backend library for a non-essential system in their product. Most services don't ask for permission or make announcements when they make changes like this.

The approach seemed to be, if things break, people will report it and we’ll fix it.

While this may not be the best approach, the number of languages supported is too high for a person to check each one manually. Generally, I imagine they wouldn't expect a change like this to break anything significant.

[people] use it as a portfolio. [..] To suddenly doink the appearance of people’s portfolios is unfortunate.

It is very unlikely that syntax highlighting errors in GitHub will affect someone's chances of getting a job.

Sure, this switch could cause some issues but they don't seem to be severe enough to kick up a fuss over.

Demiurge4• 12 years ago

It is very unlikely that syntax highlighting errors in GitHub will affect someone's chances of getting a job.

And if it does, the problem lies not with the syntax highlighting, not even close...

conradev• 12 years ago

One other thing to note is that it's not just syntax highlighting, it's also recognizing the language a given repository is written in.

Some of my repositories that use Logos[1] are now incorrectly classified as a combination of Ruby and Scala[2].

[1] http://iphonedevwiki.net/index.php/Logos

[2] https://github.com/conradev/Tweaks

lelandbatey• 12 years ago

Yep. And now Github thinks my Makefile is JavaScript: https://github.com/lelandbatey/link_grab

muyuu• 12 years ago

Also, not a great idea to rely on some "cloud" 3rd party service to host your portfolio IMO - at least not exclusively. Maybe I'm old-fashioned.

That said, I don't think anyone will bin a candidate because Github didn't highlight his or her code properly.

regularfry• 12 years ago

It's not that people necessarily choose to host portfolios on github, but more that employers will treat whatever's on there under your name as a portfolio regardless.

Argorak3• 12 years ago

I beg to differ. Github pushes its code rendering as far as possible, including providing widgets to embed code snippets into you blog. It is also the _main content they display all the time_. That's not "non-essential".

Just switching the library and breaking things at a whim is problematic.

Also, the number of languages may be 316 (including some oddballs like "Unified Parallel C") but that's still a possible number to check for at least for major, obvious breakages. Still, for people that do use Unified Parallel C, adequate highlighting might just be the reason to choose that platform and use it to write your blog in, instead of writing a custom highlighter for prism.js.

Sorry, if you business is code and you decide to support 316 languages, expect people to hold you on that promise.

That said, also: errors happen. But that isn't a reason to give them a pass, just not to put too much weight on such things. It doesn't break the platform at large, but terribly inconveniences some users, and they are very right in being upset, too.

tomcorrigan• 12 years ago

expect people to hold you on that promise

What promise did github make?

greghendershott• 12 years ago

I didn't expect my blog post to be on the front page of HN. Here's a TL;DR summary:

For many languages this is a significant and distracting degradation in the presentation.

I could understand GitHub removing highlighting completely because they feel speed is the overriding priority. That would be even faster than what they're doing now. Languages would look "plain" instead of "wrong". Not my first choice, but a reasonable choice.

The situation now is that they've replaced a library that had been handling highlighting thoroughly, with a variety of text-editor lexers that mostly are not. People like me who already contributed to Pygments, aren't feeling motivated to do this all over again for no good reason. So it seems likely the lexers will remain poor for quite a long time. Which is unfortunate.

Finally, at the time I wrote my blog post, I was speculating about the motivation because GitHub hadn't explained why, yet. Someone later did explain ("because speed") in the issue thread.

res0nat0r• 12 years ago

Are the majority of languages now broken, or just a small niche subset that don't see much use vs. ruby, python, etc?

If a few minor languages hardly anyone uses as compared to the whole site might need some fixing, this still seems like a win from Github's side of things since per the graph the change did in fact significantly improve render times.

alayne2• 12 years ago

Racket is a fringe language. Github has about as many Prolog repositories as Racket repositories.

If Racket syntax highlighting was causing performance issues that were noticeable to Github, performance must have really sucked. Why should Github let Racket drag down its capacity?

walterbell10• 12 years ago

Is that the message an open-source vendor wants to send to developers: your code is welcome, unless it's a new and unproven "fringe" language?

How would a "fringe" developer feel about that vendor's brand after their language has become successful? Would they like/trust/recommend the vendor to upcoming developers?

alayne2• 12 years ago

Racket isn't new. It's a scheme which has been in development for around 20 years.

Reading their thread, it sure sounds like they want to work with the community to make Racket code viewing a good experience.

hawkice• 12 years ago

GitHub: a product so close to our hearts that even extremely well meaning changes that really do help in many cases can sting sometimes.

I normally wouldn't understand this type of thing (others say they don't see the problem and it's quite clear where they are coming from), but in a way I _do_ see the author's point of view. When you build something people really care about, any change, no matter how minor, has the opportunity to impact someone. That's why we all build things, isn't it?

ivan4th• 12 years ago

Seems like Common Lisp syntax highlighting suffered, too.

slantedview• 12 years ago

It's worth noting that even syntax highlighting on common languages like Java is currently messed up on Github. I hope it's fixed eventually, but kinda lame to take something that wasn't broken and break it.

needusername• 12 years ago

Linguist has trouble telling C and C++ apart.

_pmf_• 12 years ago

GitHub is not your personal vanity site.

latkin• 12 years ago

F# highlighting is also now totally gone with this change. Code is highlighted with some random lexer than doesn't even understand // comments. Rather frustrating, given that a lot of F# development is centered around GH, and GH themselves use F# in a couple of places.

Browsing the issues list, this isn't just "fringe" languages, either. Perl, PHP, Go, and Clojure all appear to have regressed to some degree.

shurcooL3• 12 years ago

Interesting coincidence, as I've just changed the highlighter in my Go library for offline rendering of GitHub Flavored Markdown [1] 23 hours ago.

[1] https://github.com/shurcooL/go/commit/6aad35a0a60fd67927f446...

ttwwmm• 12 years ago

Rendering of reStructuredText disappeared in GitHub Enterprise ~6 months ago... I wonder if the reasoning was the same. More limited resources in the GHE VM environment?

frontsideair• 12 years ago

I understand the author's exasperation, but the post is surely filled with loaded words and wild speculations. It may get the message across, but this is not the way to start a conversation.

Matthias247• 12 years ago

I don't know about pygments but my experience with writing a custom highlighter for Sublime Text (aka Textmate, Atom and what Github seems to use now) was that it is not really a good and reliable system for highlighting.

It is really easy to highlight simple things (keywords, numbers, ...). However when it comes to more complex scenarios (e.g. where the type of a word depends on the previous one) then the singleline regex based mechanism shows it weakness. Due to that many language support plugins will yield wrong results when you start to split things like function declarations over several lines, even though it's perfectly legal in the languages. Some things can be worked around with the start/end regexes, but nesting those multiple levels deep can get quite akward and I don't think that they were thought of for things beyond braces and multiline comments.

Therefore I don't know if Githubs move here is a really good choice. However I think their main motivation might be that this file format already has such a big ecosystem due to Textmate, Sublime and Atom and the parser has a high performance so that they went for it.

jessaustin2• 12 years ago

I really would have expected a highlighter written in Coffeescript to do a better job highlighting Coffeescript.

https://github.com/atom/highlights#using-in-code

(Generic identifier looks the same as class identifier looks the same as a property in an object literal looks the same as a string. I guess maybe orange is just the color used left of an equal or colon, but in that case the string color should be different.)

GFK_of_xmaspast2• 12 years ago

"""pygments.rb had an interesting history of trying to use a Python library in Ruby on a high-traffic web site. They had tried various approaches. The final approach — piping to a long-running Python process — seemed to work well. """

That doesn't fill me with a lot of confidence in this thing.

GitHub dropped Pygments

Related Stories