Energy Efficiency Across Programming Languages

https://sites.google.com/view/energy-efficiency-languages

by foob • 9 years ago

188 139 9 years ago

sites.google.com

flavio81• 9 years ago

According to their normalized "global" results, something interesting i see:

1. Pascal, surprisingly, the most memory efficient of all. I should take a look at the implementation they used.

2. Rust a good alternative to C which leads in "energy efficiency" and speed.

3. Common Lisp most energy-efficient and fastest and smallest memory footprint of all the dynamic programming languages in the list -- like Python, Ruby, Lua, Perl, and even Racket (which fares pretty well)

4. PHP, JRuby, Ruby, Typescript, Perl, Python, being massively slow than the fastest languages, for example Ruby being 59 times slower than Rust or C.

I agree that these languages (in item 4) are "acceptably fast" for many applications, but we can't say they are "close to the speed" of the fastest languages, even more if we consider the fastest dynamic+interactive languages like Lisp (Python being 21 times slower than Lisp.)

5. I wonder which implementation of Lua they used. Lua can be pretty fast, one of the fastest dynamic languages out there.

goatlover2• 9 years ago

I have noticed a significant speedup using LuaJit, similar to PyPy compared to normal Python. I'm guessing if LuaJit & PyPy were used, the performance for Lua & Python would be similar to JS in these results.

mamcx3• 9 years ago

1. Pascal

Pascal is a "hidden" gem in the area of languages. Sadly not enough pus for it, but imagine if it have the push that other languages have...

agumonkey3• 9 years ago

Now I need to finish my lisp.pas

open-source-ux• 9 years ago

"Pascal is a "hidden" gem in the area of languages"

I agree. Programmers won't look at it because they perceive it to be old and out-of-date. But the language hasn't stood still. It's a fast, low-memory language. FreePascal with the Lazarus IDE is one of the best cross-platform development toolkits for building native desktop apps.

Sadly a lot of programmers can never see beyond the verbose (but readable) syntax.

CoreXtreme• 9 years ago

Ruby also has gem.

psyc• 9 years ago

When I started programming in the early 90's, I remember Pascal being a serious contender, with C vs Pascal compilers being a pretty evenly matched contest.

mac01021• 9 years ago

But why should it's memory utilization be so much lower?

Pascal and C give the programmer basically the same control over memory layout, no?

nine_k• 9 years ago

Much of Pascal (or, rather, Modula-2 and Oberon) is revived inside Go.

agumonkey3• 9 years ago

Imaging if the world ran on SBCL instead :)

equalunique2• 9 years ago

SBCL is one lisp that I intend to learn more about. It's apparently the lisp of choice for XMaxima, which is a program that I used a lot in high school.

aaron-lebo• 9 years ago

Where are you getting that Pascal is the most memory efficient (edit: duh, under the "memory efficiency" table, mistakingly read this as "efficient")? It's firmly in the middle of the pack in every result, in their global result table it's slower and less energy efficient than Java (!). It's only advantage is that it's smaller than anything else.

Don't mean to assume you are wrong, but what are you referring to?

shock2• 9 years ago

In the `Normalized Global Results` table, Pascal is used as the reference (for memory use) and everything else is reported relative to it.

copx• 9 years ago

5. I wonder which implementation of Lua they used. Lua can be pretty fast, one of the fastest dynamic languages out there.

I looked at the binary trees benchmark and while the Python version used multiprocessing to parallelize the code, the Lua version was not parallelized at all..

.. not that this is entirely unrealistic. Python comes with MP in the standard library while I am not aware of any "industrial strength" solution for Lua, whether MP or thread-based.

igouy21• 9 years ago

For an interested Lua programmer, there's a kind-of workable approach --

http://benchmarksgame.alioth.debian.org/u64q/program.php?tes...

wruza• 9 years ago

Idk if it is industrial strong, but there is lanes and probably few more listed at lua-users. Do you have any experience to share on these?

The choice of 5.3 is pretty strange, as Lua is known to split in two at 5.2, because LuaJIT was fixed to 5.1 variant with compat backports and is to my opinion an "industrial" Lua standard. 5.x are actually three different (but not absolutely superior to each other) languages with shared 5-like foundation and app-level code compatibility, so speaking about Lua in general is pointless in a sense.

http://lualanes.github.io/lanes/

http://lua-users.org/wiki/MultiTasking

etiene10• 9 years ago

The industry standard in Lua is to have multiple Lua states doing stuff. A different approach, but it is there

shock2• 9 years ago

5. I wonder which implementation of Lua they used.

There is a table with the versions they used¹, but they don't mention the implementation – in can be inferred for some: e.g. for Pascal they used the Free Pascal Compiler.

① - https://sites.google.com/view/energy-efficiency-languages/se...

vvanders• 9 years ago

Ouch, they used Lua 5.3

LuaJIT is at least an order of magnitude(10x) faster.

ashark3• 9 years ago

4. PHP, JRuby, Ruby, Typescript, Perl, Python, being massively slow than the fastest languages, for example Ruby being 59 times slower than Rust or C.

Take a look at some of the TypeScript code and compare it to the Javascript versions. At least some are way different, i.e. the TypeScript uses "modern" JS features and the JS version... looks about as much like C as it can.

jsd19822• 9 years ago

Typescript is not its own language in terms of having a distinct runtime from Javascript so it has no performance difference and does not make sense to appear in this comparison.

I can see comparing different target versions of Javascript against one another (es2015 vs es2017 for instance), but it all depends on the VM chosen as well.

avodonosov2• 9 years ago

And Forth is missing.

etiene10• 9 years ago

5. I wonder which implementation of Lua they used. Lua can be pretty fast, one of the fastest dynamic languages out there.

Apparently 5.3, but I am very skeptical of these results. Also the way the data is organised is very weird. One should not compare a JIT compiled VM (JavaScript) with interpreters (Lua, Python, Ruby...). An honest approach would have used LuaJIT, PyPy etc. or separated the languages into "families" better. Not to mention the choice of algorithms they used. Some of the algorithms in the benchmark game have many flaws.

As it was already stated in this thread, some of the Python ones were multithreaded but the ones in Lua weren't. Why? In a different benchmark I've seen algorithms using C wrappers for Python, but not in Lua. While I understand wrapping C and multithreading reflect the industry use in Python, so they do in Lua. Benchmarks are useless if you are not consistent in the implementations of the things you are comparing with. It disgusts me, to be honest, when I see benchmarks comparing JIT compilers with vanilla Lua just to say "hey we're faster", as LuaJIT does not even exist. It looks like pure dishonesty. In the case of this paper in particular, as they do not represent any language, it strikes me as lack of research / understanding.

igouy21• 9 years ago

… but the ones in Lua weren't. Why?

No mystery: someone has made the effort to contribute all those multi-process Python programs; no one has made the effort to contribute all those multi-process Lua programs.

igouy21• 9 years ago

One should not compare a JIT … with interpreters

Because ?

problemsmith• 9 years ago

It'd be nice if these werent images of the tables ...:-\

knolan• 9 years ago

They're just screen capped from the actual tables in the paper PDF.

mtm• 9 years ago

A real shame that Forth wasn't included in the tested languages. Chuck Moore has been an advocate for more energy efficient computation for a while now.

fnord1233• 9 years ago

The tests come from the benchmark game. If you want to see Forth, implement the tests in Forth:

http://benchmarksgame.alioth.debian.org/

mncharity3• 9 years ago

The tests come from the benchmark game. If you want to see Forth, implement the tests in Forth:

But perhaps archive your code elsewhere. Github? Lots of language communities participated in past versions of the game. Like gforth and bigforth on shootout.alioth.debian.org circa 2008. Now it seems little remains but a few archive.org snapshots without source. Though perhaps all of that code was archived somewhere else that I'm not quickly finding?

mncharity3• 9 years ago

@igouy points out that while the old shootout site is gone, its language files are still available[1]. There are about 70 benchmarks, in about 70 languages. I remember them as interesting multilingual browsing.

Unlike the old shootout, adding languages directly to benchmarksgame.alioth.debian.org is not an option. "Because I know it will take more time than I choose. Been there; done that."[2] :) Instead, the code is available[3], and communities are invited to document their own benchmarksgame comparisons[2], such as Nim''s[4]. The OP used different code[5].

A brief glance at the current benchmarks, suggests there is some overlap with the old ones. Or at least an overlap of names - the requirements may have changed. So it might be possible to get started fairly easily? Perhaps even to do several languages...

[1] https://alioth.debian.org/scm/viewvc.php/shootout/bench/?roo... [2] https://benchmarksgame.alioth.debian.org/play.html [3] https://github.com/Byron/benchmarksgame-cvs-mirror [4] https://github.com/def-/nim-benchmarksgame [5] https://github.com/greensoftwarelab/Energy-Languages

avodonosov2• 9 years ago

Yes, see several minutes starting around 01:18 in this talk:

https://www.infoq.com/presentations/power-144-chip

As for efficiency, he mentions 7 picojoules per instruction in his forth chips.

astrobe_2• 9 years ago

I'd rather say it begun to be a focus with its latest processors because it's the only hope they have to sell Forth processors. I remember he mentioned introducing a lot of complexity in its previous design to achieve very high speed for exactly the same reason (the only hope we have to sell a Forth processor is to make it super-fast), which was a bit in contradiction with the reason why he moved from software to hardware - simplifying globally the hardware/software combo.

kcarnold• 9 years ago

I see they monitor a complete process lifetime, not just the active workload. A quick grep of the paper didn't turn up anything discussing this. And it would seem to hurt dynamic languages and runtime-JIT languages a lot. Perhaps the active workload takes long enough that the transient is washed out. Anyone have insights?

fnord1233• 9 years ago

If you want to tease the startup out, then you can run the game with a null or extremely small input set and then discount the time it takes to run a small/null workload from the actual run.

But really, if they hurt, then they get hurt. There's only so many knobs you can turn and so many runs you can run before you have to assemble a table and submit your paper. They put all the code online so you can try it yourself.

igouy21• 9 years ago

Perhaps the active workload takes long enough that the transient is washed out.

That does seem to be the case with some-of the benchmarks game programs --

http://benchmarksgame.alioth.debian.org/sometimes-people-jus...

goatlover2• 9 years ago

No Julia?

ram_rar• 9 years ago

I am curious to know, if there is any explanation for why java performs better than Go.

jcelerier• 9 years ago

20 years of optimizing tends to do this

glibgil• 9 years ago

Go was invented in the 1970s and then placed on the self for 40 years

jsd19822• 9 years ago

I personally enjoy Go but your comment gave me a chuckle since it's very true. Have your upvote, friend, for whatever they're worth on HN.

ademarre• 9 years ago

I can't speak to the Java comparison, but the site says they used Go 1.6.3. The SSA back end introduced in Go 1.7 (amd64) came with a nice performance boost.

https://blog.golang.org/go1.7

topspin• 9 years ago

Two decades of optimization probably account for that. From the paper:

    Moreover, the top 5 languages that need less energy and
    time to execute the solutions are: C (57J, 2019ms), Rust (59J,
    2103ms), C++ (77J, 3155ms), Ada (98J, 3740ms), and Java
    (114J, 3821ms); of these, only Java is not compiled.

That is an admirable accomplishment. JRE 1.8 has the most efficient "managed" language runtime tested.

The PHP result is interesting; they used 7.x which is roughly twice as efficient as older (5.x) versions; PHP 7 was primarily a (successful) effort to improve performance. If you extrapolate the PHP result to the old version (multiply by 2) it's nearly as bad as Perl and among the slowest/most costly. That certainly corroborates Facebook's several PHP re-implementations.

jaawn• 9 years ago

The Java performance was the most interesting part of the results for me. I'm glad it got some attention. I have been developing the opinion for awhile that, as an industry, we've become far too avoidant of "pre-mature" optimization. It is astonishing to me how much slower a lot of this stuff is than Java.

Granted, I also think considering Java to be "not compiled" is a slight misrepresentation since it does get compiled to Java bytecode.

mapletree• 9 years ago

Nim compiles to C

cwzwarich• 9 years ago

I can't believe they used the Computer Language Benchmarks Game. Those benchmarks don't reflect real-world workloads at all, and the contest has fairly arbitrary rules about implementations and widely differing implementation quality between languages. This should have been rejected by peer review.

igouy21• 9 years ago

Those benchmarks don't reflect real-world workloads at all…

That's a very definite claim, for which you provide no supporting evidence ;-)

oconnor663• 9 years ago

Maybe we could get away with a couple smaller claims:

- It would be weird to expect programs specifically optimized for a performance benchmark, to also be optimal for energy usage or memory. Maybe less weird for energy, if runtime is the biggest factor in how much energy gets used. But I'd expect there to be huge tradeoffs between runtime and memory usage, once we start really optimizing for memory.

- Running the Benchmarks Game has meant doing a lot of work to figure out what the relationship between "optimal" and "representative" is, and what the rules of the game should be as a result. Even if we take it for granted that we've answered those questions in a way that gives us useful benchmarks for performance, it would be weird to expect the answers to be exactly the same in benchmarks for energy or memory.

rbehrends3• 9 years ago

One of the worst offenders is probably the binary-trees benchmark.

1. It features an atypically high allocation rate compared to real-world programs.

2. As a synthetic micro benchmark that only ever allocates objects of one size, it makes pool allocators look disproportionately good. But in real-world applications with varying object sizes, overuse of pool allocators creates a serious risk for fragmentation.

3. The rules allow programs that use manual memory management to use pretty much any pool allocator library that they can get their hands on, but forbid garbage-collected languages from adjusting their GC parameters (last I checked, you could easily improve the performance of OCaml and Dart for that benchmark by a factor of 2-3 simply by adjusting GC settings, especially the size of the minor heap).

jackmott4• 9 years ago

they reflect a number of different real world workloads.

devveppev2• 9 years ago

This criticism is flawed. The fact that the code is imperfect or sub optimal actually makes their conclusions strong and more useful for real world comparisons.

The code guidelines in their benchmarks represent real programming practices in those languages. They promote "idiomatic" code because it is meant to represent the typical quality of code in the real world. Obviously many of the examples could be made more performant, but in doing so you would made the code less representative of the real world. Example: the typescript code outputs wanky ES6 features that are slower than their old plain js counterparts (classes, arrow functions, let, etc). You could abuse the ts code until its output is identical to js, but you would have a pointless benchmark now.

What would the benchmark honestly represent if the code looked nothing like code in the real world? The theoretical speed of the language just doesn't matter.

In fact, JS engines have been optimizing their performance around numerical benchmarks for decades. The benchmark problems (nbody, etc) are actually highly unrepresentative of real javascript performance because real world javascript is touching strings and awful DOM apis and messing with dictionaries all day.

Your 'real-world workload' should be stuff like text editor operations, a domain where JS's alleged 6x slowdown compared to C has not been remotely approached by current editors.

Veedrac2• 9 years ago

The code guidelines in their benchmarks represent real programming practices in those languages. They promote "idiomatic" code because it is meant to represent the typical quality of code in the real world.

This could not be further from my observations when I tried contributing. These results are neither controlled nor the result of idiomatic programs.

pmontra• 9 years ago

The results are what I have expected but yes, mandelbrot and n-body in Ruby? Ruby's typical workload is getting some strings from a web server, building a SQL query, sending it to a database, get a resultset, creating a bunch of objects to represent the result and send back another string to the web server. That would be more energy efficient if done in C but not as much as doing a mandelbrot.

Then factor in the energy of the developer and its supporting environment for the extra development time (commuting, air conditioning / heating, laptop, monitors, etc.) The less the load on the app, the more important it gets.

astrobe_2• 9 years ago

Then factor in the energy of the developer and its supporting environment for the extra development time (commuting, air conditioning / heating, laptop, monitors, etc.) The less the load on the app, the more important it gets.

No, it's almost always negligible because it's essentially a one-time cost, whereas the program's inefficiencies are multiplied by an unknown number of users over an unknown period of time (cf. Cobol programs still being in use today) - but for sure your goal is often to maximize both of these unknown numbers.

This fact is very obvious when you develop hardware/firmware combos. The firmware is often designed to minimize the hardware costs (RAM size, mass storage size,...) because the cost of producing the hardware is per piece, while the development cost is mostly a one-time cost that one rarely includes in the total cost of the product.

BenoitEssiambre• 9 years ago

Why is TypeScript so much less efficient than JavaScript?

kerpele• 9 years ago

It seems to only be significantly different with fannkuch-redux. Looking at the CLBG page (http://benchmarksgame.alioth.debian.org/u32/performance.php?...) for that algorithm there doesn't seem to be very big difference at all.

I cannot explain the difference and if this is indeed an error by the authors of the paper it casts a fairly big shadow on the reliability of the rest of the data.

jackmott4• 9 years ago

less optimal implementation is all

ashark3• 9 years ago

Guess: its output has more lines of JS than the vanilla JS versions, and since a lot of the variation between scripting languages in these things often come down to how fast they can stop running themselves and start running some underlying C, those extra lines hurt it.

This may also explain why JS is really fast in a couple of the examples. Those script likely do almost nothing but call some built-in function that does all its work in C (or C++, or whatever).

[EDIT] OK, if I'm reading these[1][2] files correctly, it has more to do with the the TS version being written in "modern" "idiomatic" "beautiful" JS with promises and crap, and the vanilla JS one being written in "ugly" "bad" old-school 90's-lookin' C-ish Javascript.

[1] https://github.com/greensoftwarelab/Energy-Languages/blob/ma...

[2] https://github.com/greensoftwarelab/Energy-Languages/blob/ma...

stewbrew• 9 years ago

So go (a garbage collected language) is about as memory efficient as pascal, the most memory efficient language. That comes surprising.

brlewis2• 9 years ago

Not really. "Manual" memory allocation is usually malloc/free, which leaves a lot of gaps.

bbatha• 9 years ago

Go's garbage collector isn't compacting.

trhway• 9 years ago

For a bunch of years i was writing in Java and driving V8. Now i'm back to native (C++) and am driving Prius these days. I guess that it is my eco-consciousness that is making me shudder in disgust every time i look at Python... pretty much the same way like when looking at Hummers. While of course i love Perl, an M1 Abrams which gets things done despite anything :)

danieltillett2• 9 years ago

On of the things with the C set is they didn't use Intel's icc. On our main application (which involved a lot of heavy duty number crunching) it speeds up our application by 100% over the best we can get out gcc.

fnord1233• 9 years ago

I've seen 20% speedups with icc over gcc; but not a doubling. Is this including mkl over some other numeric library as well?

danieltillett2• 9 years ago

No nothing extra. The code does vectorise well. But even if the result is 20%, it would make quite a bit of difference to the results presented in this paper.

winter_blue• 9 years ago

Note: The PDF of the paper has the a lot more detailed listing of the results (with graphs, explanation, etc) than the results web page: http://greenlab.di.uminho.pt/wp-content/uploads/2017/09/pape...

pjmlp5• 9 years ago

Only while printing it, I noticed my university name there! :)

br1n0• 9 years ago

brief: C is the winner of all the races ( time, energy and memory)

milcron• 9 years ago

Pascal won on memory.

boznz• 9 years ago

Considering most CPU cores (Arm/X86/etc) are optimized for C/GCC its an unsurprising result

nur0n2• 9 years ago

Can someone elaborate. I always thought that the compiler was optimized for the architecture, not the other way around. Why would an architecture be optimized for a specific compiler?

matt_wulfeck• 9 years ago

Higher-level scripting languages. Fast for programmer's time, slow for run times.

pjmlp5• 9 years ago

Not really, if you look into Lisp results.

Apocryphon• 9 years ago

One wonders how Kotlin (or other popular JVM languages) would compare. And also Objective-C or Smalltalk.

phkahler• 9 years ago

Rust is fairly competitive with C, but it's memory use was 50 percent higher. I bet if they focus on getting the memory usage down it will perform better too.

steveklabnik• 9 years ago

Rust uses jemalloc by default; I wonder what it would look like if you swapped the allocator.

int_19h2• 9 years ago

I'm kinda surprised to see Go score so low on some of these, given that it's AOT-compiled to native code, and that its memory model and safety guarantees are optimization-friendly.

I'm also kinda surprised that OCaml scored that high, considering how high-level it is.

tyoverby• 9 years ago

OCaml has an incredible optimizer, and many of the abstractions that one would use in OCaml are transparent to the optimizer.

You could say the opposite about Go.

int_19h2• 9 years ago

Kinda makes me wonder why OCaml didn't become more popular for Unix systems programming, the way Go seems to be becoming now. Looking at the two languages, it seems like pretty much everything that Go has to offer in that department, OCaml has as well - and if it also optimizes better, why is this even a contest?

flukus• 9 years ago

I'd love to see some scatter plots of CPU-time and energu-time, I expected a fairly linear correlation but there looks to be quite a lot of outliers, some but not all are explained with parallelism. Memory seems all over the place too.

nur0n2• 9 years ago

I have just started looking into Ada and it is cool to see up there with C. Does anyone know a good resource for writing fast Ada?

gwenzek• 9 years ago

I'm suprised at the Java-C# comparaison. When are this numbers from?

The following blogspot explains how the author improved C# implementations after seeing defavorable C# results.

https://anthonylloyd.github.io/blog/2017/08/15/dotnetcore-pe...

equalunique2• 9 years ago

Haskell did not shine here. Ocaml did surprisingly well.

Energy Efficiency Across Programming Languages

Related Stories