HN user

vsbuffalo

934 karma

Research scientist and software engineer in computational epidemiology.

x.com/vsbuffalo :: vincebuffalo.org

book: Bioinformatics Data Skills, published by O'Reilly http://shop.oreilly.com/product/0636920030157.do

Posts35
Comments80
View on HN
www.pcg-random.org 1y ago

C++ Seeding Surprises (2015)

vsbuffalo
36pts25
vincebuffalo.org 10y ago

A Genealogical Look at Shared Ancestry on the X Chromosome

vsbuffalo
2pts0
www.sacbee.com 10y ago

UC Davis chancellor received $420,000 on book publisher's board

vsbuffalo
1pts0
medium.com 10y ago

Just Landed Is Shutting Down

vsbuffalo
215pts133
www.newyorker.com 10y ago

The Space Doctor's Big Idea

vsbuffalo
1pts0
circres.ahajournals.org 10y ago

Ben Franklin and Open Heart Surgery (1974) [pdf]

vsbuffalo
24pts4
circres.ahajournals.org 10y ago

Ben Franklin and Open Heart Surgery (1974) [pdf]

vsbuffalo
5pts0
grisha.org 10y ago

Linus on Understanding Pointers (2013)

vsbuffalo
2pts0
www.youtube.com 10y ago

Making things with maths [video]

vsbuffalo
1pts0
github.com 10y ago

SDSL – Succinct Data Structure Library for C++

vsbuffalo
79pts38
vincebuffalo.com 11y ago

The Mighty Named Pipe

vsbuffalo
385pts96
www.archive.org 11y ago

“A Communications Primer” by Charles and Ray Eames (1953) [video]

vsbuffalo
36pts0
www.technologyreview.com 12y ago

For One Baby, Life Begins with Genome Revealed

vsbuffalo
4pts0
julialang.org 12y ago

Shelling Out Sucks (2012)

vsbuffalo
1pts0
www.youtube.com 12y ago

Matrix Singular Value Decomposition Film (1976)

vsbuffalo
2pts0
www.rsi.deas.harvard.edu 12y ago

Harvard RSI Action: Preventing RSI

vsbuffalo
2pts0
kbroman.github.io 12y ago

Tools for Reproducible Research

vsbuffalo
1pts0
en.wikipedia.org 12y ago

An Open Letter to Hobbyists — Bill Gates (1976)

vsbuffalo
2pts0
news.ycombinator.com 12y ago

Ask HN: Will we be using Unix derived OS's for the next 50 years?

vsbuffalo
11pts13
madhadron.com 12y ago

The iniquities of the Unix shell (2011)

vsbuffalo
2pts0
cpp-next.com 12y ago

Want Speed? Pass by Value (2009/C++)

vsbuffalo
1pts0
attractivechaos.wordpress.com 12y ago

Heap-free and lock-free parallelized C "for" loops

vsbuffalo
7pts0
www.newyorker.com 12y ago

What's Wrong with Jeremy Seifert's OMG GMO

vsbuffalo
2pts1
www.slideshare.net 12y ago

GNU Parallel – The command line power tool

vsbuffalo
106pts28
aaroniba.net 13y ago

How I Cured my RSI Pain (2010)

vsbuffalo
106pts79
medium.com 13y ago

A Liberal's Defense of GMOs

vsbuffalo
3pts0
jeremiahwarren.com 13y ago

DIY Matrix bullet time slice rig with fan and GoPro

vsbuffalo
3pts0
irri.org 13y ago

Super salt-tolerant rice, after 34k crosses (and 1 survived)

vsbuffalo
5pts0
nbviewer.ipython.org 13y ago

Write a genome assembler in Python

vsbuffalo
3pts0
www.drbunsen.org 13y ago

Custom notebooks with QR codes as digital tethers

vsbuffalo
2pts0

I really like all plotting systems in R. First, I used base graphics for a few years—and loved it. You learn your way around par(), commit esoteric argument names to memory (oma, mar, mgp, mfrow, etc). It feels powerful — you're just drawing on a screen; its history traces to the original pen plotters. Second, I learned lattice. You can't help but fall in love with lattice after a year or two with creating panel plots in base graphics. The biggest learning curve with lattice is panel functions, but once you learn to throw a browser() in a panel function for stack variable introspection, you can do anything. Somewhere on a dusty bookshelf is a well-worn lattice book I splurged on while taking an R course at UCD.

I like this article, because I think for production graphics, the author has a point. If you're placing lines, points, and labels on a screen — you can create anything. You can draw polygons and arcs. It's like drawing with raw SVG. But I'd have a hard time thinking of an exploratory data analysis situations I wouldn't reach for ggplot2 first. Since it looks at dataframe column types (integers, factors, numerics), it automatically matches these two the appropriate type of color gradient. Coloring a scatter plot by a potential confounder is one additional argument to aes(), e.g. aes(x, y, color=other_col). More than once during EDA I've done this and seen some horrifying pattern in data that shouldn't be there. That's a powerful tool for one extra function argument — the cost of checking for a confounder with color (or shape) is essentially near zero.

I'd make the case that this is a more costly operation in base graphics, and is thus much less likely to be done. You may already have your plots in a for loop to create panels, then you have a few extra lines for adjusting margins and axes (rather than facet_wrap(~col)). It took a lot of code to set that up — there's already a lot of cruft when you just need to do a quick inspection. Then you need to create a vector of appropriate size of colors, and then map this to data. Sure it's easy-ish, but it takes at least double the time as color=some_col. In EDA visualization, I want every single barrier to checking a confounder to be as small as possible—which is what ggplot2 does.

That said, I really liked this article because I do agree that going from EDA visualization to production is a hassle. Just after reading this, I remade some production ggplots with base graphics and love the simple aesthetic — which to mirror in ggplot takes a lot of hassle.

What I really long for is a lower-level data to visualization mapping (like d3) in R. d3 is a pain to learn, but it's really the only data abstraction (even though it is a low-level abstraction) that is seemingly limitless in what it does and can do. I always hope for a general data-join grammar like d3's to be the norm, built on top of base plotting (analogously: svg elements), and then have abstractions like ggplot for tabular data built on top of that.

You're treating this sample-is-the-population issue as if it's resolved in the statistics literature. It is not. Gelman has written on this [1][2], as the issue comes up in political science data frequently. As Gelman points out, 50 states are not a sample of states—it's the entire population. Similarly, the Correlates of War [3] data is every militarized international dispute between 1816-2007 that fits certain criteria—it too is not a sample but the entire population.

Treating his population as a large sample of a process that's uncertain or noisy and then applying frequentist statistics is not inherently wrong in the way you say it is. It may be that there's a better way to model the uncertainty in the process than treating the population as a sample, but that's a different point than the one you make.

[1] http://andrewgelman.com/2009/07/03/how_does_statis/

[2] http://www.stat.columbia.edu/~gelman/research/published/econ... (see finite population section)

This isn't the most useful statistical figure. The heart feature is novel — the increase could be due entirely to folks trying it out. A more meaningful figure would be to look at the folks who have constant favoriting habits (with stars that is) and see how their behavior changed. Personally, I'm more reluctant to heart tweets, as my Twitter account is mostly professional and it feels a bit unprofessional to "heart" a colleague's tweet.

I shared this in light of the recent article by Matt Ridley, "On the Myth of Basic Science"[1] which I find less honest and filled with anecdotal examples that fit his particular narrative.

[1] www.wsj.com/articles/the-myth-of-basic-science-1445613954

Completely agree. I can't understand how professional developers are able to give up their text editors for iPython. Don't get me wrong, I love iPython/jupyter, but the slick interface comes with huge productivity drops due to the lack of a real text editor.

I love fountain pens and highly encourage everyone to try them (with a good notebook helps). First, ballpoint pens are wasteful — 1.6 billion pens a year are thrown away[1]. Fountain pens are reusable, ink is comparatively cheap and lasts forever, and finding your ink is a fun and personal experience (I really like the "bulletproof" Noodler's inks which are waterproof, bleach proof, etc.). Fountain pens last forever — which is why folks still hunt around for 40+ year old used ones.

Second, it really does make writing fun. I hated writing — my handwriting is messy, it's slow, and it's not as easy as typing. As the article argues, a good fountain pen makes it much, and in my experience much more enjoyable.

Third, it doesn't need to be expensive. Get a Lamy Safari (EF), a Lamy converter and a bottle of Noodlers ink. I also love my Faber Castell Loom[2] (it's the smoothest pen I own), and I carry around a Kaweco Al-Sport[3] everywhere (it's the perfect pocket pen).

[1] http://www.epa.gov/superfund/students/clas_act/haz-ed/ff06.p... [2] http://www.gouletpens.com/faber-castell-loom-metallic-orange... [3] http://www.jetpens.com/Kaweco-AL-Sport-Fountain-Pen-Fine-Nib...

Tufte CSS 11 years ago

Well done. Though shouldn't the sidenotes use HTML5's <aside></aside>? How does the vertical alignment of sidenote callout and sidenote work?

Really awesome stuff! I think there's a very minor bug in which missing pixels are treated as black, which is what adds a black border around the output image in the second example.

I hate the best editor debate because I think it's distracting from what's more important — what both editors can learn from each other and what both need to do to improve. I used Emacs for years, switched to Vim because of RSI, then recently switched back to emacs+evil. Frankly, for what I do most (R, R+knitr, C++ with clang autocomplete), no single editor is great. First, there's too little ability to switch between modes within a single buffer in both Vim and Emacs. The feature's entirely lacking in Vim AFAIK, and poly-mode in R uses a high level hack that (1) doesn't play well with other modes (including evil) and (2) has so thoroughly destroyed my documents in the past I refuse to use it now (mostly because it uses many buffers behind the scenes, which destroys undo history).

In general, if you want flawless R support in certain blocks of text (as in a .Rnw file) in between LaTeX blocks that are fully connected to AucTeX, well... you're out of luck. And Vim... Vim-R-Plugin is useful, but it's sort of a painful hack to use tmux just to get R and Vim to talk (and I'm saying this even though I love Tmux).

Vim has YouCompleteMe, which is smooth as silk compared to Emac's options (which are painful and poorly integrated, especially with clang). But some lower-level issue in Vim causes this constant error message in Vim whenever YouCompleteMe uses clang — bloody annoying. So overall, both editors have huge issues that would require serious overhauls or tedious bug fixing in various modes. Sure, Emacs does AucTeX better, but until it does everything better (or Vim does everything better) it's a flawed editor. Both are flawed editors. But sadly everyone thinks the best course of action is to start fresh — which usually creates a feature-poor flawed editor on a new shiny foundation, that fails to attract developers because it's feature poor. (apologies for ranting -- jetlag).

From Vim to Emacs 12 years ago

I thought this was the case too, but YouCompleteMe is really terrific for Vim! I wish emacs had something similar.

If you're using a large genomic dataset, you shouldn't be sorting your results in text format anyway... it would take way too much time and temporary disk space. What you'd probably want is a row filter to extract out the rows of interest.

For repeated queries, this isn't efficient. This is why we have indexed, sorted BAM files compressed with BGZF (and tabix, which uses the same ideas). Many queries in genomics are with respect to position, and this is O(1) with a sorted, indexed file and O(n) with streaming. Streaming also involves reading and uncompressing the entire file from the disk — accessing entries with from an indexed, sorted file involves a seek() to the correct offset and decompressing that particular block – this is far more efficient. I definitely agree that streaming is great, but sorting data is an essential trick in working with large genomics data sets.

Hate to sound like Steve-Jobs here, but: "You're using it wrong."

I don't quite agree — say this individual needs to sort a file by two columns. Should they really load everything into memory to call Python's sorted()? With large genomics datasets this isn't possible. Trying to reimplement sort's on-disk merge sort would be unnecessary and treacherous.

It's easy to forget how much engineer went into these core utilities — which can be useful when working with big files.

Beyond functions there is nothing good. There are three object-oriented programming systems, none of which are simple or straightforward.

I don't think this is quite fair. R was greatly inspired from Scheme and stole a lot of good ideas. S4 is very similar to CLOS, and does a fairly good job. If you come from Python or Java, this system looks crazy, but being unfamiliar with a language style does not make the language bad.

To this day I don't have a good mental picture of genomicRange (a foundational bioinformatics package in Bioconductor for NGS data analysis) data structure and how to manipulate it (one main data structure, I believe).

It's a dataframe attached to ranges. This design extends lower classes like IRanges, and all the accessors, setters, etc are consistent. You can manipulate the range part with integer range operations and manipulate the dataframe part as a dataframe. I would check out the IRanges vignette — this is very helpful.

R, more than any other language I know, is an implementation-defined language.

This is the most severe problem for sure. I'm cross my fingers someone forks R's implementation and does it right (which should break a lot of packages and code).

Best of Vim Tips 12 years ago

I was a long time emacs user, and switched to Vim due to typing pain. Even with Evil, I can't see moving back. Vim + Tmux + YouCompleteMe > Emacs. There's just no comparison. No Emacs shell will ever be as good as zsh, and there's always some latency for some reason when using shells running in emacs. And, there's no youcompleteme.

I use vim (after years of being a happy emacs user — switched due to RSI), but I've never liked its help system. Compared to emacs, it's incredibly hard to navigate and it just doesn't have anything as powerful as C-h k (what does this key do?).

This looks neat, but is the method open? Is there a command line version?

I'd argue that while open source is a desirable feature in most software, it's a necessary feature in scientific software. When our results depend on running through software, that software can't be a proprietary, closed-source black box.

But thanks to fracking and the shale revolution, peak oil and gas have been postponed. They will run out one day, but only in the sense that you will run out of Atlantic Ocean one day if you take a rowboat west out of a harbor in Ireland.

As an economist, how can the author not see that the hidden environmental costs are growing? As markets demand natural gas within a cost range while resources are limited, the solution becomes to pass this cost off on the environment. The environment doesn't have a powerful advocacy and can't fight back.

I am disappointed that economists buy these models so readily without doing real accounting as to all costs. I think this is a problem in modern economics — we can easily measure monetary costs through accounting and prices, but other costs are hard to measure so the model treats them as residuals. Then, they run predictions with their models that completely ignore environmental costs. Don't argue markets don't have a cost on the environment when you've failed to include environmental costs in your models.

GNU Screen v.4.2.1 12 years ago

Neat changes, but I'm still going to stick with tmux. Tmux not only works better, it's changed the way I develop. Everything is done in a Tmux window with Vim and shell. Tmux is how you can create a shell-based IDE with splits and different functionality. I came from Emacs, and liked the IDE-ness of it, but I can capture almost all of that behavior with tmux + vim.

TeX reminds me of GCC — huge, complex codebase, unhelpful error messages, and designed to just be a compiler/typesetter. I wish someone were to rewrite an LLVM-like TeX system. Simpler code, could be used to write libraries to do live syntax checking and formatting, beautiful error messages, and fast typesetting times.

This has given me a new appreciation for (competent) managers. A good manager that's put time in as a programmer can truly be a master metaprogrammer, with people as their tool, not programming languages.