Evaluation of C, Go, and Rust in the HPC environment [pdf]

http://octarineparrot.com/assets/mrfloya-thesis-ba.pdf

by steveklabnik • 11 years ago

145 64 11 years ago

GitHub repo with all of the code: https://github.com/mrfloya/thesis-ba

Most interesting part of the thesis, for me, is page 42:

In this case it was even an advantage that the Rust version was developed last since it revealed a critical error in the other implementations.

Null strikes again!

yiyus3• 11 years ago

He would have found the problem in Go too if he was using the comma-ok form:

    n1_idx, ok1 := g.nodeIdx[n1]
    n2_idx, ok2 := g.nodeIdx[n2]
    if !ok1 || !ok2 {

When you do not use the comma-ok form, you are specifically asking the map to return the zero value for non initialized elements (which is something fine to do in many cases).

heinrich5991• 11 years ago

Well, you're implicitely asking for it. In Rust you have to explicitely specify your intent, i.e. not by omitting something, but rather by specifying that you want to ignore it.

MordodeMaru• 11 years ago

I wonder what would've happened with the results if biicode would've been used as deps/package manager for C.

jryan49• 11 years ago

Yeah it would of been nice to see the dev time split up into setup and actual programming.

semi-extrinsic2• 11 years ago

Would've been nice with some internal links in that pdf, especially in the table-of-contents (links to sections and subsections). This is simple with hyperref.

ThePadawan• 11 years ago

Has the link has changed since your post? The ToC is linked for me.

semi-extrinsic2• 11 years ago

Ah, sorry, it's just Quickoffice's pdf viewer (on Android) that somehow manages to not understand the links.

ericjang• 11 years ago

I'm a bit surprised that the author didn't mention julialang.org in the report - it's much more suited for technical / scientific computation, while Go and Rust seem better for systems-level concurrency/design.

petermonsson• 11 years ago

I'm willing to cut the auther some slack here. A bachelor thesis is limited in scope and a bachelor student can't know all languages.

tines• 11 years ago

I feel that it's a shame to test C rather than C++, because the latter offers so many advantages to developer productivity -- the paper cites memory management and type-related errors as being major causes of developer unproductivity, and C++ has automatic memory management and templates and such.

Also, it doesn't restrict the use of C libraries, so even if you've been writing C for HPC code all these years, there should be minimal technical difficulties in switching over, except for the time it takes to learn C++, which shouldn't be long if you know C well.

bsdetector• 11 years ago

The author has a "lack of equal experience in the three evaluated languages", especially C, but is comparing them for development time and SLOC based on a sample size of one developer and one very small program.

He rewrites a distributed program as a threaded one, because Rust and Go didn't have a distributed computing library, but says Go and Rust have "similar performance" in HPC.

He says he was surprised at the C performance, but there's no investigation of a cause. Maybe just because C is slow...

I don't mean to be overly critical, but these results in the paper mean absolutely nothing. It's a waste of time.

pnathan4• 11 years ago

pretty sure all programs were multithreaded.

I do agree that the advisor should have pushed back lots harder on the methods; it's a weak paper, even as an experience report.

hurin5• 11 years ago

I do agree that the advisor should have pushed back lots harder on the methods; it's a weak paper, even as an experience report.

I agree - this seems to be mostly the advisors fault, by the looks of it it's not that the student didn't put a lot of work into it but methodological problems are what advisors are supposed to be there to address; I saw a lot of this back in college, at least with technical papers it's easy to tell what's wrong, with the humanities it's just a complete disaster.

istvan__• 11 years ago

I agree with your assessment only thing I did not understand is the Erlang integration with other languages. I am not sure if he is saying that NIFs are bad or he is saying there are is not support for other languages. I think non of these are true...

http://www.erlang.org/doc/tutorial/introduction.html http://www.erlang.org/doc/tutorial/nif.html#id64792

Dewie3• 11 years ago

It's a bachelor thesis... probably doesn't aspire to be the new authority on HPC. :)

hurin5• 11 years ago

Yes, because the point is being able to write (on a resume): successfully presented thesis on "Evaluation of C, Go, and Rust in the HPC environment".

The advisors should have encouraged a smaller and more manageable scope with a more thorough methodology where the results would actually be worth something.

panic• 11 years ago

I agree that the paper isn't very convincing. One conclusion you might draw is that learning to write performant code in C is harder than learning to write performant code in Rust. That could just be a side-effect of Rust's much stronger standard library, though.

dgacmu• 11 years ago

The problem with reading this in BS thesis form is that it hasn't gone through the wringer enough. Most of the performance comparison results vs C are bogus because (to its credit, as noted), the author accidentally compiled some important support libraries for the C version without optimization and used those on the cluster. Oopsie.

As briefly mentioned in the previous chapter this performance regression might have been caused by the two unoptimized libraries that were compiled on the development laptop and copied to the cluster.

mrfloya• 11 years ago

Hey author of the thesis here.

You are correct this was definitely an oversight and I would have redone the measurements if time allowed for it. However I just want to state that the libaries were built with optimizations just not on the target platform which might not have been clear from the qoute.

acadien2• 11 years ago

Hey Floya, I'm writing this here hoping you'll see this comment. You're getting a lot of unjust responses in here. The commentors are reading your thesis with the eyes of a trained skeptic expecting peer reviewed, collaborative work that takes years of experience to produce. They're nitpicking flaws and mistakes we all make when we first start researching.

Your thesis is in fact really impressive and very nicely written! Your conclusions and future work section is excellent. Really very well done. Congratulations on finishing it. I hope you don't let the irrationally negative feedback here dissuade you from continuing your work, you certainly have a bright future ahead of you.

angry_octet2• 11 years ago

Hope the HN experience hasn't felt too bruising. The use of go and rust makes it quite interesting for those of us interested in correct code, not just fast code. And most of the criticism is that you didn't write the paper the commenter would have wanted to see, or didn't write a graduate level paper. And good on you for publishing it on github.

pnathan4• 11 years ago

Interesting work.

It has some pretty basic flaws that are pervasive that the reader should be aware of. First off is the "productivity" measure: the last time I looked around, this was nearly impossible to quantify for software development. The author chose SLOC and development time as stand-ins for productivity. SLOC has a connection with code quality and time of development, but development time is well known to vary[1].

In particular, development time in this thesis is linked to results in a pretty classic "Psychologist's fallacy"[2]; the author generalizes their experiences as conclusive. This implies in particular that "development time" in this thesis should be thrown out for meaningful conclusions, as the sample size is 1. It is, however, an interesting experience report.

Another basic flaw is the connecting of 'modern' with 'good'. The author remarks the main disadvantage of C and Fortran are their age; this crops up here and there. Workflow tooling in Go and Rust are major focuses by the developers: C workflow tooling is usually locally brewed. This does not make C-the-language worse.

More on a meta level, my advisor in my Master's work drummed into me that I should NOT insert my opinion into the thesis until the conclusion. So I found the editorializing along the way very annoying.

And finally, and very unfortunately, the experience level of the author appears to be low in all three languages; this is significant when it comes to implementing high performance code.

---

Now, for the interesting / good parts of this experience report.

Standout interesting for me was the Rust speedup on the 48-core machine. I did not expect that, nor did I expect Go to make such a good showing here as well.

In general Go performance/memory made a surprisingly good showing (to me) for this work. I shall have to revise my opinion of it upward in the performance axis.

I am both surprised and vaguely annoyed by the Rust runtime eating so much memory (Would love to hear from a Rust contributor why that is and what's being scheduled to be done about it).

Of course the stronger type system of Rust catching an error that C and Go didn't pick up is both (a) humorous and (b) justifies the type community's work in these areas. I look forward to Rust 1.0!

One note by the author that is worth calling out stronger is the deployment story: Go has a great one with static linking, whereas C gets sketchy and Rust is... ??. I believe Rust has a static linker option, but I havn't perused the manual in that area for some time. For serious cloud-level deployments over time, static linking is very nice, and I'm not surprised Google went that route. It's something that would be very nice to put as a Rust emission option "--crate-type staticbin".

Anyway. I look forward to larger sample sizes and, one day, a better productiv

[1] http://www.amazon.com/Making-Software-Really-Works-Believe/d... The situation is actually far worse than just varying developer time.

[2] https://en.wikipedia.org/wiki/Psychologist%27s_fallacy

steveklabnik4• 11 years ago

Would love to hear from a Rust contributor why that is and what's being scheduled to be done about it.

I haven't spent time with the code yet, so I can't comment. It shouldn't be the runtime, though, as our runtime is about as big as C or C++'s. My first thought would be that Vec's growth factor may be poor in this instance, but without reading the code, who knows.

Rust is... ??

Rust statically links everything but glibc by default. Experimental support for musl was added last week.

pnathan4• 11 years ago

Oh, cool! Thanks for the information!

madez• 11 years ago

I read more than once that rust has no runtime. Could you please clarify whether it has a runime, and if it does, what it consists of?

gaius• 11 years ago

Any HPC language evaluation that doesn't include vs FORTRAN is pointless. It's still king of computation for a reason.

Manishearth• 11 years ago

... the reason is mostly legacy. All your libraries are written in fortran. Everyone learned fortran in the past and they don't want to switch.

As an example, I've seen physicists use fortran for parsing and transforming the output of a program. The cluster at our department has an awesomely updated ifort but no python2.7 (I think it has 2.2 or 2.3 or something). C++ isn't used directly either.

But yeah, there should have been fortran and some other languages there.

runlevel1• 11 years ago

Experiments aren't usually written in FORTRAN.

The programmer is by far the slowest part of HPC. It often makes sense to use the language you're fastest at instead of the language that will run the fastest.

It's very common for experiments to be written in languages not know for their performance (Java, Matlab, Perl, and Python for instance).

For reference, here are some of the packages commonly used in experiments[1]:

    BLAS        Fortran                 http://www.netlib.org/blas/#_software
    NCBI BLAST  C++                     ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/ncbi-blast-2.2.30+-src.zip
    BFAST       C                       http://sourceforge.net/projects/bfast/
    BioPerl     Perl                    https://github.com/bioperl/bioperl-live
    Bowtie      C++, C                  http://sourceforge.net/projects/bowtie-bio/files/bowtie/
    Clustal     C, C++                  http://www.clustal.org/omega/#Download
    cp2k        Fortran                 https://github.com/cp2k/cp2k
    Gromacs     C                       https://github.com/gromacs/gromacs
    HTSeq       Python                  https://pypi.python.org/pypi/HTSeq
    MUSCLE      C++                     http://www.drive5.com/muscle/downloads.htm
    MrBayes     C                       http://sourceforge.net/p/mrbayes/code/HEAD/tree/
    OpenFOAM    C++                     https://github.com/OpenFOAM/OpenFOAM-2.3.x
    SAMtools    C                       https://github.com/samtools/samtools
    SNAP        C                       http://korflab.ucdavis.edu/software.html
    fftw3       C                       https://github.com/FFTW/fftw3

[1]: I went down this list and picked some of the ones I remember using (been out of the HPC world for a couple years): https://portal.tacc.utexas.edu/software It'll be a bit skewed towards genomics.

east2west• 11 years ago

The list is not just skewed toward bioinformatics; a majority of them is bioinformatics software. I would not call BioPerl or HTSeq high-performance-computing software. Some of the software are written by academic programmers with all attendant baggages. I am not aware of any bioinformatics software written in Fortran, which is not popular in the bioinformatics community. But this is not to say Fortran might not be a good fit. The Broad Institute is rewriting some of its tools in C++ with help from Intel to leverage vector instructions. Who is to say if they had started with Fortran they might not have gotten vector instructions for free earlier.

sampo2• 11 years ago

fftw3 C

FFTW is C code, but nobody wrote the C code. The C code is generated by code written in OCaml.

http://en.wikipedia.org/wiki/FFTW

drwells2• 11 years ago

My biggest criticism of this work is that it does not consider C++. Most of the big HPC projects today are written in C++, not C, and to the best of my knowledge this transition happened in the 90s.

buster• 11 years ago

Not Fortran?

physguy1123• 11 years ago

There is still a lot of Fortran being written, but a lot of it is on legacy projects (scientific codes have a habit of lasting a long time). While there are some cool things like coarray Fortran and some groups require that everything be done in Fortran, C++ is a favorite for new projects afaik.

drwells2• 11 years ago

I have not done a census of the various libraries around, but all of the free finite element libraries in common use (deal.II, libmesh, and FEnICs come to mind first) are written in C++. I suspect Fluent and Abaqus are written in C (maybe Fortran?) but they are a bit older.

Trilinos is another C++ example, while PETSc and HYPRE are written in C.

My tentative conclusion from this is that newer projects tend to use C++ and slightly older projects use C.

Yes, there are things like ODEPACK, QUADPACK, and FFTPACK, but those are not under development anymore (as far as I know). The only widely used Fortran library still under development I can think of is LAPACK.

I did not count the occasional 'bespoke' code. There are still some Fortran applications in development for particular purposes (like MOM), but those are harder to survey.

sampo2• 11 years ago

Most of the big HPC projects today are written in C++

My experience is limited to one single field, but all weather prediction models and climate models that I know of, are in Fortran. And I would count them as big HPC projects.

angry_octet2• 11 years ago

An interesting case study, though not really HPC or scientific computing related. It would be good to see some profiling to understand exactly why each implementation was slower in particular ways. From a HPC context, linking with Fortran and C codes, and exchanging data with them, is very important.

ohquu2• 11 years ago

Why are there so many bold terms and phrases throughout the text? It's extremely distracting.

garthk• 11 years ago

I often find bold text helpful for key points, or for the first mention of key terms to help orient me in a document. There was something about the mix in this document that felt inconsistent. I found myself pausing to try to figure out why the author wanted my attention on this particular word or phrase.

dibanez• 11 years ago

Well, Go and Rust can do better threading on this problem than a not-really-optimized C code with OpenMP. Thats a start. Message passing has to be shown next, and then one can decide whether the installation and runtime of Go/Rust/Python/Julia on something like an IBM Blue Gene/Q with its custom compute node OS is easier than a link flag.

deadgrey192• 11 years ago

Keep in mind, this is a bachelors thesis. It's not a peer reviewed research article in a respected systems journal or conference. I mean this in both a positive and a negative sense. Positive in sense that the purpose of a bachelors thesis is to introduce undergraduates to an extended, self directed project with an element of research to it. As someone who has supervised many such projects, I think that this is a pretty good one. Great work! At the same time, since the author is an undergraduate, presumably with a full coarse load, this is not a high quality research publication. The results are probably a bit shaky and the methodology is perhaps a bit off. But that's ok. Every excellent piece of work has to start somewhere.

tomrod• 11 years ago

Given the comments in this thread, I think the author should be thrilled! What great feedback!

deadgrey192• 11 years ago

Agreed!

short_circut3• 11 years ago

"In times where compilers are smart enough to basically rewrite and change code for performance reasons it is completely inexcusable that the order of source arguments to process is still that relevant."

After reading the comments on C and the difficulties the author had in the compilation process.... All I can say is Garbage in garbage out. I have worked in the HPC enviornment for a number of years using both Fortrash and C. The errors the author encountered would have been avoided by someone with any experience developing large softwares. I learned these things after working on my first project and now they barely, if at all, factor into my development time. Make sure you build and keep a dependency graph while you are developing and this becomes a non issue.

chm• 11 years ago

    Fortrash

Why the hate? One could misuse any tool, but that doesn't make the tool bad at what it does.

short_circut3• 11 years ago

I don't disagree. Fortran is brilliant at what it does.... but I find many of its object oriented features to be very obtuse compared to C and C++. The way it uses pointers drives me insane. Building certain datastructures in fortran just takes too much time.

I kind of wish I could find a good way to make Fortran and C play nicely together. (There are some compiler dependent peculiarities that I have encountered that make it difficult.)

Fortran's intrisics and file handling capabilities make it indispensable in the work I do.... So it is really a love hate relationship.

BuckRogers• 11 years ago

Wildly offtopic- reading papers like this bring to mind for some reason, that the next evolution of CPython, the one that deals with the Python2/3 rift should be written in Rust.

Sanddancer• 11 years ago

Looking through the links, and methodology makes me question a lot of things, most of them regarding his choices in how he allocated memory. First and foremost, he was using the glibc malloc, which belongs in a museum, not in a running system. One of rust's big advantages in performance comes from that alone, given all the memory that's being allocated and destroyed. Which leads to the second question of how he did allocate objects. He knew he was going to allocate a number of objects, yet he didn't pre-allocate a large chunk of memory ahead of time to make it so he wouldn't have to call malloc and friends as often. I wouldn't be surprised if spending a bit of time just tuning memory didn't cut significant time from how long execution takes. I really wonder how much time was spent in the system context of all three of those; I'd be willing to bet that the final part of the examination would probably fare much different had he used the same malloc for the C program that rust and go already use.

vans• 11 years ago

80 pages for that, with one user and one implementation of one algorithm ? Sorry but the rate knowledge/size is very low, so don't be impressed by the page number, it harms the paper more than anything else. Even for a bachelor thesis, it's mostly "filling"

Evaluation of C, Go, and Rust in the HPC environment [pdf]

Related Stories