Github now supports rendering tabular data

A massive amount of time and energy is wasted by news organizations on the question of how to show tabular data. This is an incredible feature, and one I'm assuming can be hackable/extended in various useful ways.

Anyone stress-test it with bulk data yet?

7952• 13 years ago

It would be good if browsers had native support and rendering of SQLite in the same way that Chrome works with PDFs.

nonchalance2• 13 years ago

Hopefully they'll add sorting soon

thedays• 13 years ago

1) Agreed - allowing users to click on a column heading and then sort it by ascending or descending values would be killer.

2) Adding embed codes for these tables would also be incredibly useful for news organisations and non-profits.

jmduke2• 13 years ago

The money line:

Building software is about more than code.

GitHub's ably and subtly positioning themselves as attractive for a much wider audience than programmers.

MartinCron• 13 years ago

Or making their product more attractive for programmers who do more than just write code. Either way, I think it's brilliant.

toomuchtodo• 13 years ago

Programmers are data manipulators, no? Github is about streamlining data manipulation (storing your code, helping you with CSVs/GEOJSON). They're just lowering the bar to make data manipulation more accessible.

One day, everyone is going to be a programmer. Is everyone a professional driver? Not at all. But most people do know how to drive a car.

hcarvalhoalves• 13 years ago

Cool feature. What happens on gigantic files? Does it limit, paginate, lazy-load?

gjtorikian5• 13 years ago

The limit is 600kb; I'll have to publically document that. Only 3% of GitHub's one million CSV files go over that limit.

As you might imagine, shoving in that many rows into the DOM kills any browser. G

jmduke2• 13 years ago

I'm actually kinda surprised that the number is that high.

Part of me wants to make a stumble-upon that grabs a random csv from GitHub. I feel like there's probably some really interesting data waiting to be explored.

acdha2• 13 years ago

I'm tempted to say you leave an enable option and report it to the browser teams as a performance challenge – it's the kind of thing which should work better than it does.

I noticed that more than half the time in Chrome isn't HTML parsing but rather layout recalculation - I suspect you'd be able to avoid much of that if you were able to set some sort of min/max width on the columns and containing table with overflow:scroll-x or hidden. Perhaps set a fixed width when the server sees the size is large and/or by measuring the rows while rendering the template?

mjs7231• 13 years ago

In similarly large tables, I found if you shove about 1000 rows into the DOM at a time inside a setTimeout(0) loop, most browsers behave quite ok.

chrisseaton2• 13 years ago

This is great, but their basic text editor isn't usable for me for what I think are pretty reasonably sized files (170kb of Markdown). This doesn't have to be because the editor they use (Ace) works fine with it, so I don't know what they're doing to it. A tech support person said it was only there for small files, not for serious editing.

I was working on a project and wanted non-techies to edit Markdown for me on the site, being able to preview the results, but couldn't make it work on these files.

So I wish they'd fix that rather than more editors for other file types.

dbond• 13 years ago

Have you tried http://prose.io/ ?

endergen• 13 years ago

Wow, works great on my iPhone. Love it!

gjtorikian5• 13 years ago

Can you link me to such a file?

As a former developer on Ace and now employee at GitHub I've honestly never run into a problem. I can't promise anything but I'd like to see what you mean.

BHSPitMonkey2• 13 years ago

I just tried this with a (very superficial) test case[1] at 189KB and the editor seemed fine, both for editing source and generating previews. I even used the editor to double the file's length, saved, and reopened without problems.

[1] https://github.com/BHSPitMonkey/Large-Markdown-Test/blob/mas...

chrisseaton2• 13 years ago

I've sent you an email with the exact file. Cheers.

eranation• 13 years ago

I was wishing for this for a while, but didn't imagine they'll actually do it. This is awesome (Now all I need is to have it editable...)

pearjuice• 13 years ago

Bracing myself for the Github IDE. If they do that right, any browser can become your development machine.

jackmaney• 13 years ago

"git actually has a simple design, with stable and reasonably well-documented data structures. In fact, I'm a huge proponent of designing your code around the data, rather than the other way around, and I think it's one of the reasons git has been fairly successful […] I will, in fact, claim that the difference between a bad programmer and a good one is whether he considers his code or his data structures more important."

--Linus Torvalds

(source: http://programmers.stackexchange.com/questions/163185/torval...)

malingo• 13 years ago

I'm getting off topic, but this quote brings to mind Dijkstra's remark that "our intellectual powers are rather geared to master static relations and that our powers to visualize processes evolving in time are relatively poorly developed."

Does anyone know whether this has indeed been shown to be true?

hoka• 13 years ago

Along the same lines, a quote from 1975(!):

"Fred Brooks, in Chapter 9 of The Mythical Man-Month, said this:

Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious."

http://www.dreamsongs.com/ObjectsHaveNotFailedNarr.html

picardo• 13 years ago

Unfortunately, Git isn't a good way to collaborate on massive csv files for two reasons a) git stores changes at the file level, rather than row level, b) for massive datasets, you quickly run into file size limits. That's why I am looking forward to dat[1], a collaborative DVCS that's going to blow Git off the water for this kind of tasks, even if it achieves only half of its objectives.

[1] https://github.com/maxogden/dat

dchest• 13 years ago

git stores changes at the file level, rather than row level

It applies delta compression when storing blobs in pack files.

rpedela• 13 years ago

git does not handle large files well. Actually none of the open-source VCS do a good job. I think that was picardo's main point.

zhemao• 13 years ago

This isn't precisely true. Git stores changes at the "blob" level. Large plaintext files (like a CSV) are generally split into several blobs. You are correct that git works badly for binary files, like images and compiled executables, But plaintext files like CSVs should be OK.

pit• 13 years ago

Is GitHub turning into an operating system?

drhodes• 13 years ago

Not sure, but it might be turning into an IDE. They've got a good part of it in place. Something like FP Complete: https://www.fpcomplete.com/

polskibus• 13 years ago

In next iteration of this feature, please provide an option for sticky column and row headers - ie. scroll table down with headers in place. Such solution helps inspect large files with many columns. Surely you can do better than rendering a standard html table?

jloughry• 13 years ago

I second this motion. Pull request, anyone?

devcpp• 13 years ago

Where to? Do we have access to the source of this?

On that topic, I have never found much of Github's source code, which is extremely hypocritical, and makes me fear the day we'll want to move away from them. I'd be glad to be wrong and find out that I had just never looked in the right place.

cdcarter• 13 years ago

I've wanted this for so long in my docs. It's so easy to build a doc ecosystem around my code on github between rendering md, and shoving YARD output into gh-pages!

ape4• 13 years ago

Is the viewer open source?

gjtorikian5• 13 years ago

The viewer is just Ruby's CSV library, iterated within an ERB.

j4mie• 13 years ago

It would be nice if GitHub added better support for column-oriented data in diffs, too. This article shows how it could work, with an example implementation in a fork of GitLab: http://theodi.org/blog/adapting-git-simple-data

coherentpony• 13 years ago

Great. Nested comments, anyone?

roycoding• 13 years ago

A tabular/graphical JSON explorer would be interesting as well.

kyle_martin1• 13 years ago

I totally agree. This the next logical step.

anonymoushn• 13 years ago

It looks like this (ruby's CSV library) defines CSV to be some unknown superset of RFC 4180, so just following the RFC should get this feature to work.

Github now supports rendering tabular data

Related Stories