A Look at How Traders and Economists Are Using the Julia Programming Language

http://waterstechnology.com/waters/feature/2476518/the-infancy-of-julia-an-inside-look-at-how-traders-and-economists-are-using-the-julia-programming-language

by ViralBShah • 10 years ago

399 195 10 years ago

waterstechnology.com

sebg• 10 years ago

If you want to learn more about Julia, here are Julia Conf videos / slides from past years:

* http://juliacon.org/2014/

* http://juliacon.org/2015/

* Julia Con 2015 India - https://www.youtube.com/playlist?list=PL279M8GbNseuhXmZk9rWM...

optimali• 10 years ago

2016 talks are also up - http://juliacon.org

Overview - http://julialang.org/blog/2016/09/juliacon2016

e12e• 10 years ago

Speaking of talks on Julia, I'll again recommend:

https://www.youtube.com/watch?v=dK3zRXhrFZY

"Julia: to Lisp or not to Lisp?" from the European Lisp Symposium in May 2016. I found it was rather illuminating on Julia in particular and lisps and programming languages in general.

agounaris• 10 years ago

Julia is a very interesting language but big data processing is not a matter of the programming language itself, but how your platform and your architecture is put together.

And nearly always your bottleneck is not the language itself..

staticassertion3• 10 years ago

I think that if this were true, the data science libraries in Python wouldn't all be written in C under the hood, linking to Fortran, etc. Language can have a huge impact on performance with big data.

Julia is just a small step forward. It still has performance problems where the JIT can't help much - like the DataFrame structure, which is effectively a black box and therefor hard to optimize.

VodkaHaze3• 10 years ago

If you need performance on these kind of structures, you can use structured arrays in python.

The main reason I like the idea of Julia is that I don't see the problem of parallelism being addressed in python, at least not without cumbersome libraries (disclaimer: I haven't tried dask yet). If python had an equivalent to

    #pragma omp parallel for

I wouldn't have Julia ready in mind for my next project that requires creating a high performance algorithm.

hadsed• 10 years ago

Python numerical libraries links to LAPACK/BLAS/etc because that's how people used to do scientific computing before Python. Also for scientific computing you're mostly CPU bound which means that no one would use Python if the libraries were slower than Fortran or C. But big data problems are different in the sense that you're typically bound by IO, so while it certainly helps to have fast code it wouldn't be the first problem you'd want to solve.

quasarken• 10 years ago

It depends on what kind of data you're working with and what kind of operations you perform on it. If protein folding, then that's computationally expensive and the language will matter, doing mapreduce jobs, language matters little as the network overhead is going to be your bottleneck in a distributed environment.

johnmyleswhite• 10 years ago

This is not quite accurate as stated: the problem is the combination of a data structure that is not amenable to type inference with an API that would only be efficient if type inference had perfect knowledge of the contents of a DataFrame. There is a lot of work being done to develop a new API that has none of these problems and provides very high performance, which demonstrates that Julia's JIT is up to the task so long as you choose an appropriate API. Julia, like any language, has intrinsic limitations, but this is not a good example: it's an example of how good API design for Julia differs from good API design for other systems.

GeneralMayhem• 10 years ago

Depends on whether you're optimizing for latency or throughput - and of course, in some systems, that decision changes on an instant-to-instant basis. For throughput, indexing and shuffling is probably going to be the bottleneck, and that's dependent on network and filesystem I/O. But for the low-latency path, choice of language absolutely matters, so if you want to be able to share code between the two you end up needing both a fast language and fast architecture.

importantbrian3• 10 years ago

I agree that this is generally true, but Finance is sort of a special case. In a world where firms pay the premium for colocated servers and direct data feeds in order to shave off a few milliseconds of latency there is also a strong push to use the vary fastest language possible + have the very best possible architecture, all to shave off as many extra milliseconds as possible.

polskibus• 10 years ago

When raw performance is so important you end up implementing the solution on FPGAs. Switch to Julia does not really matter in such cases.

tikhonj2• 10 years ago

That's only true for a very niche sort of trading (only high-frequency/latency arbitrage type stuff) which makes up a tiny portion of the financial industry[1]—and that tiny portion is certainly not using Julia for production trading systems.

Everyone else has more reasonable performance constraints.

[1]: I can't find my source right now, but I remember seeing a credible estimate that the HFT industry in the US takes in $10 billion in revenue. That's a drop in the bucket of an industry that's measured in trillions just in the US.

Intuitively this makes a lot of sense because HFT strategies make a tiny amount of money per trade and have to do a ton of trades across a ton of different securities—which means the capacity of HFT strategies is relatively small.

lucozade• 10 years ago

I doubt anyone's currently using Julia for the fast path of low latency trading. Usually it goes from FPGAs for sub-milli work, C++ on GPUs then either C++ or Java for multi-milli work. In my experience.

I'd expect most of the Julia usage in this space would be to replace any R or Matlab in the back testing engines if at all.

I didn't get the impression from the article that they were really talking about low latency anyway.

That's not to suggest that Julia doesn't have a place in Finance. I think it's a great language and there are a bunch of places I can see it being very valuable. Just doubt that low latency fast path is one of them in the near term.

pitaj2• 10 years ago

Well, interpreted languages like MATLAB and R can be annoyingly slow at times

arcanus• 10 years ago

Anyone used Julia for something approaching HPC, e.g. in a distributed environment instead of MPI?

I've heard plenty of buzz about it in the HPC community but haven't had a moment to play with it yet.

KenoFischer• 10 years ago

Yes, there's been some scientific workloads that used Julia/MPI to great effect. I don't think the paper is out of review yet, so probably can't link to details, but expect news on that front very soon.

Xcelerate• 10 years ago

I have. I used it for all of the research I did during my PhD over the last few years. Some of my Julia code ran on Titan.

wruza• 10 years ago

I really liked Julia (and Juno) when tried it last year, but it was too math-aware as all [actually good] math-originated programming environments. Hope it will break that circle and get some general-purpose attention in nearest future. For me, it is very promising even without that HPC or whatever.

s_q_b3• 10 years ago

Most financial firms need their quants to worry less about the new programming language hotness, and more about moving entire systems off unbelievably complicated Excel spreadsheets.

markovbling2• 10 years ago

Totally - I used to work as quant for a boutique asset manager and the whole business was running on spreadsheets.

Insane.

Ended up putting it all in a database and developing an excel add-in to pull it from the database as array formulae. Used a great library called Excel DNA to develop the add-in using C# if anyone is interested.

sandGorgon• 10 years ago

Just a question-did you ever consider using Jupiter Notebooks? Or RMarkdown Notebook?

wz1000• 10 years ago

IIRC, Standard Chartered achieved this by initially adding Haskell interop to Excel[0] and then moving to a custom GUI solution to replace Excel altogether.

[0] https://www.youtube.com/watch?v=hgOzYZDrXL0

thearn4• 10 years ago

I work in aerospace engineering, and it's definitely the same here. In scientific research or engineering these days, there are a lot of potential steps up from excel spreadsheet hell or spaghettified MATLAB code.

Bonus points for the facilitation of any type of documentation, automated testing, or version control.

valarauca14• 10 years ago

The reason for this is simple. Excel is

1. An incredibly powerful tool.

2. No bar of entry (cost aside, true in corporate environment).

3. Very gradual learning curve.

4. The efficiency gain vs time invested is exponential.

Power Excel users, much like their VIM/Emacs counter parts don't use a mouse. It is just keyboard short cuts [1][2]. This makes them insanely productive.

[1] https://youtu.be/jFSf5YhYQbw

[2] https://youtu.be/0nbkaYsR94c

walshemj• 10 years ago

There are also many downsides

gravypod3• 10 years ago

I can vouch for this. I thought spreadsheets where stupid and useless until I was forced to use them by my boss at the CS department here for grading. I didn't realize how easy having your computation and data being in the same place.

It makes it slow, but usually for these sorts of things you want your data to be available on the scale of days, not nanoseconds so it works out. We have a complicated grading sheet here that manages all of the students information all one the same sheet in a google drive. Grades, attendance, recitation attendance, and at the end of the class homework.

It also verifies the test answers against the correct ones to make sure we score exams correctly.

Excel is great for fixed sets of data that need simple map/reductions & input verification because that's all we really use it for. After that move to something like Python&Numpy/R/Julia/Matlab.

flamedoge• 10 years ago

does it have regex search though?

gaur4• 10 years ago

4. The efficiency gain vs learning curve is exponential.

Does this actually mean anything?

joelthelion• 10 years ago

Can you version/diff it though? Basically can you have a semi-sane software engineering process when it becomes big?

VMG• 10 years ago

Is there any open source framework that has an excel-like GUI and good integration with standard coding tools?

s_q_b3• 10 years ago

This is digital Stockhold syndrome.

Excel is something managers and executives can understand, so it became the default language for data analysis. Now technologists trapped using it have to create ex post facto justifications for why it's really "just misunderstood."

Excel is massively slow, makes it easy for beginners to make massive mistakes, computes lots of types in very odd manners, performs floating point operations wrong, and leads to spaghetti code that is a rat's nest of incomprehensible cross-references.

Worst of all, the lack of code path visibility usually leads to a bus factor of 1.

Sure, one can learn to operate Excel for data analysis with a decent level of efficiency, in the same manner one can cross the Pacific in a canoe, but both are still terrible tools for the job.

inopinatus• 10 years ago

I know of a retail bank using a huge and complex Excel spreadsheet for their entire loan approvals process.

Did it have bugs? No-one knew.

adakbar• 10 years ago

Couldn't agree more, in Indonesia Oil and Gas Upstream Special Task Force use Excel for almost everything, we try to replace Excel with many good app, but alas, when an app failed we just switch to good old Excel, someone even created Montecarlo simulation in Excel!

JackFr• 10 years ago

Indeed -- the most compelling reasons to move trading desks off Excel are non-technical. Model auditability and traceability are no longer 'nice-to-haves' but are compliance requirements.

ska2• 10 years ago

People really love it because they can do business analysis without hiring programmers or becoming them. And spreadsheets are a great model for a constrained set of problems.

However, it's not all sweetness and light. Excel even gets some basic calculations wrong - and those ignorant of its quirks happily propagate those errors. More problematically, it can easily be pushed to the degree that your modeling is really beyond the tool or spreadsheet's design strengths without knowing it. And debugging is a pain in the ass. As a result lots of erroneous outputs get presented as meaninful.

bsg752• 10 years ago

All at the expense of reproducibility, testing, auditing.

Quick and very, very dirty.

vegabook• 10 years ago

The efficiency gain vs time invested is not exponential. It's linear at best and plateaus after about 5k hours. It is this plateauing of the curve that is the biggest reason power Excel users move to R or Python. That was certainly my experience after eeking everything I could out of Excel in 15 years of trading floor fixed income. A visual 2d paradigm is excellent for quick productivity but is severely limiting as complexity and data size rises. Even with VB.

hyperbovine• 10 years ago

Lol, seriously? Martin Shkreli? I had been wondering what hole he crawled into.

SnowingXIV• 10 years ago

Excel is incredibly useful and powerful. This type of comment screams "I've never used Excel a day in my life for anything other than creating a table." It can handle very complex formulas, that are easy to follow, and the data manipulation and efficiency is amazing. Your argument for moving away from Excel is the same as those who don't develop and say everything should be done through a WYSIWG editor.

goatlover2• 10 years ago

Excel use should have an inverse relationship with complexity. Just because you can make Excel handle complex formulas doesn't mean you should, or that's it's the right tool for the job.

saretired• 10 years ago

I've been around since Lotus 1.0, and worked in financial and engineering firms. I've seen cases where spreadsheets have been the right answer for knowledgeable and relativity sophisticated users, either to build a quick model or as a front-end, and cases where the result is an unauditable mess. Lots of oops when say accounting people don't understand say the math of partial-period NPVs, or are so innumerate that an obviously wrong result looks fine to them. Without the review process that should go along with production code, sometimes you get lucky, sometimes you don't. It all comes down to who is using the tool, I guess.

TuringNYC• 10 years ago

There are many technical arguments on both sides, but the business argument i've been given is that Excel sheets are the most audit-able, especially when they are self-contained. Auditors like this, especially after Sarbanes Oxley. Excel sheets fall into a different audit classification as compared to "systems" (a python script might be considered a "system".)

fma• 10 years ago

I know a software engineer who worked for one of the big quant firms in the north east (forgot the name...it's big) before people even knew what quants were.

He worked for them till he got out on his own. All his backtracking software is written by him and is in C (nice GUI, graphing feature, etc). He uses it to find his edge.

His trading platform is Excel...Obviously he doesn't do HFT...his trades are measured in days.

I know - 1 data point, but if a software engineer who is better than me in both trading & coding is using Excel, I'm not going to knock it.

lordnacho• 10 years ago

I've worked in financial firms my whole career, and I agree.

Excel is useful in one particular case only: when you don't want to build a GUI. It's great as a not-very-pretty interface for functionality written in DLLs.

For any process that's well thought out, you can write a Python script if it's not time critical. And it probably isn't if you were doing it in Excel.

The main problem with Excel is it's too easy to write an ad-hoc fix. Sounds like a weird reason, but in finance they just pile up and up and up. Finance Excel users also tend to know just enough coding to dig a huge hole, and just little enough to not understand this. Soon you have an unauditable mess, and the business is almost never going to spend time paying up technical debt.

There's also the philosophical issue of ever more complex models. If you have some sane coding practices, you will tend to favour more elegant code. Balls of spaghetti are more obvious in something like Python. More elegant code is connected to more elegant models. Inelegant models, such as the ones often bragged about by M&A guys (let's be honest, they're sales tools, not predictions) when written into an ordinary language, will look like the balls of spaghetti that they are.

sgt1013• 10 years ago

It's all part of agility vs. efficiency, but not in the way people think! If people are heavily utilized they turn to Excel because they can get through a few simple things really fast. There's risk to using things like R or Julia because you can't see (literally) how to do it, or what you can do, and trying something different will earn you a rapid sacking at the hands of the super utilizers.

But pretty soon you are mired in spreadsheet hell. Nothing can be seen or understood, everything is invalid or valid - who knows and worst of all when something stops working you don't know why.

And you don't know when it will stop. Goodbye agility!

Any spreadsheet with more that 2 days of work to reproduce it should be counted as IT and put on a formal risk register until it is recoded and removed. But dream on..

arca_vorago• 10 years ago

Does anyone know if the way complicated/advanced formulas are handles in libreoffice would make it less suitable for these tasks? They seem to have ironed out most of the bugs, so I wonder if it would be worth pushing these quants who can program into the libre scene to get them to contribute back to projects? Of course if whatever the backend is handling formulas in something like libreoffice truly is subpar, that will never happen.

branchless• 10 years ago

Spreadsheets where functions are entirely based on cell formulas publishing to bespoke internal data busses with in-house plugins that randomly stop working.

There can be no god.

sdegutis• 10 years ago

Why isn't LuaJIT instead? If they both get their speed from JIT, isn't LuaJIT faster than Julia?

benjaminva• 10 years ago

I have used both in the science context and found Julia much better suited for mathematical problems. Matrix/vector support built into the language is a huge advantage. On top, running code in parallel environments is better supported by Julia (Macros, scheduling, etc...). On top of it, Julia has an excellent Python interface which makes it possible to also use this huge ecosystem of libraries. Makes it easy to use matplotlib out of the box.

cschmidt• 10 years ago

I like Julia as well, but there is an interesting scientific library for LuaJIT:

http://torch.ch/

Anyone have experience with it?

_v7gu• 10 years ago

having worked with it, torch is amazing for neural network stuff (thanks to the nn rock); however it raises the same problems there are with Python. Lua is basically a wrapper around C/C++ and whenever you need to do complex stuff, you find yourself writing C/C++ rather than Lua.

pjmlp• 10 years ago

Isn't LuaJIT development stale?

cryptrash• 10 years ago

Yes. Lua could have been the Julia of its time if they had decided to pursue a more advanced compilation system rather than trying to embed lua everywhere in everything. Luajit is fast because of the nature of optimizations upon the tables structure and single table architecture. Julia is a better, faster lua and I don't find myself missing being able to embed my code when I can just call into what I need FROM julia.

gabrielgoh• 10 years ago

I've used both. Lua is powerful because of torch and it's GPU computing capabilities, but in terms of ease of use and modern syntax, Julia is lightyears ahead.

williamstein• 10 years ago

"Lovas adds that currently, the de facto standard numerical programming language when talking to banks and other finance firms is Python/NumPy."

dx0342• 10 years ago

I doubt that. It's certainly the trend and a lot of new code is produced in Python. But most models will run on R/Matlab/C++/C#. I'd guess that most quant departments have less than 10% of their code in Python.

But Python is the modern way to go, so every shop stresses that they use it and they expect every new hire to master it.

thefastlane• 10 years ago

... master it instead of C++ or in addition to C++?

dekhn5• 10 years ago

it's almost the perfect language. The breaking issue for me was the choice to count from 1 (yes, I know there are pro arguments for counting from 1, but every other language I use doesn't do that).

rrock• 10 years ago

Why limit yourself to zero or one? Julia now lets you use any start point you'd like. A fairly recent change, so you might have missed it.

http://docs.julialang.org/en/latest/devdocs/offset-arrays/

dekhn5• 10 years ago

The problem is that I maintain other people's code, and I don't want the language to add all those extra things. It's a big cognitive load to maintain, and leads to bugs in production software that is hard to debug.

The julia docs mention that Fortran arrays can be indexed with various starting numbers. I've programmed and maintained millions of lines of Fortran over 20 years and I've never seen anybody use anything other than 0 as the starting index.

VodkaHaze3• 10 years ago

Always fun to point out is that the NY federal reserve switched their full scale model of the economy to Julia last year[1]. It's also open source, so data scientists interested in macroeconomics can feel free to play around with it.

Moreover, quant-econ, an academic entity dedicated to scientific programming in the social sciences has a few notebooks in python and Julia [2]

[1]https://github.com/FRBNY-DSGE/DSGE.jl

[2] http://quantecon.org/notebooks.html

amelius• 10 years ago

Interesting. What quantities do economic DSGE models measure for example? And how accurate are these models?

Is there some way to play with these models? Or do you need lots of input data, which isn't freely available?

VodkaHaze3• 10 years ago

DSGE models aim to be explanatory models of an economy. They are not the best at predicting (bayesian VARs do better forecasting, there is a tutorial on it in my quant econ link) but they turn out to do forecasting decently well, too.

The reason that is is that purely statistical/predictive models of macro are subject to the Lucas Critique [1]. It says roughly, if you observe an economic relationship is happening, but you don't have an explanatory reason why it is, it's a bad idea to use it for policy prescription.

The NY fed model has a whitepaper here [2], which should be accessible to the technical data scientist (technical, but not prohibitively so).

Their readme points to a few posts on using it, I think the open source code comes with a csv for example input data. There should be publicly accessible macroeconomic data in a few places for you to play with it, say at FRED or the World Bank. I think Julia has a Stata-style api package for FRED data, making the data processing easier.

Have fun!

[1]https://en.m.wikipedia.org/wiki/Lucas_critique

[2] https://www.google.ca/url?sa=t&source=web&rct=j&url=https://...

economist42• 10 years ago

The foundation DSGE models are nothing like regular statistical or machine learning models. They are general equilibrium models, a type of mathematical model the falls under the microeconomics area of economics. DSGE try to predict how an economical system will evolve over time, given agents preferences, technologies and institutions, how much will be produced, consumed, traded, prices and how those behave over time, taking into account stochastic impacts, like idk oil price fluctuation. They are beautiful theoretically, since they are micro constructed. See, economists don't like to use aggregates only to make predictions and infer policy impacts due to something called Lucas critique ( https://en.wikipedia.org/wiki/Lucas_critique ) which basically means that taking an inflation series create an ARIMA(x,y,z) to predict next weeks CPI is theoretically. invalid.

The estimation however, is usually based on Bayesian hierarchical models, taking the constraints imposed by the theoretical model, I believe.

Here is a good source for that https://www.newyorkfed.org/medialibrary/media/research/staff...

DSGE have been criticized for their low predictive power, but it can be said that in that for mid to long term predictions they are more robust for monetary and fiscal policy use.

dekhn5• 10 years ago

Models are just datastructures and algorithms; presumably, you mean there is an implementation of those, and that the Fed uses that implementation as the 'official' one.

That's fine, but I prefer my models to be language-independent.

gre• 10 years ago

How do you have language-independent data structures and algorithms that you can run on a computer?

hodder2• 10 years ago

As a quant, I can't say I've heard of a single quant I know who uses Julia. It very well could be promising, but firms will be very hesitant to switch from Matlab, R, Python, C++, and VBA

sndean2• 10 years ago

I can't find the talk that Stefan Karpinski gave where he said this, but it was something like:

"We've been spending a lot of time working with firms to replace K code."

Maybe the small percentage that uses K/Q/KDB+ are moving to something more readable?

stuntprogrammer• 10 years ago

Based on random sampling of those that contact me about such transitions, the latter is somewhat true. A couple have mentioned Julia but I wouldn't say it's the majority.

qewbie2• 10 years ago

Do you have a link to the talk? It would not make business sense for Stefan to spend time replacing the small percentage, if it were indeed a small percentage. Actually, kdb+ is dominant everywhere on wall st, and the company mentioned in that article are retiring an old, discontinued version of k, not kdb+; perhaps that is the firm Stefan refers to? The debate about readability of k code has become boring - if you learn the language, you'll find it perfectly readable, elegant even.

hodder2• 10 years ago

Definitely possible. Though I'm not sure why those firms wouldn't switch to a more mainstream stack. Most talented quants I've met prefer to code in Matlab, R, and Python, C++. VBA is of course inescapable in the life of a financier.

When competing for talent and desperate for maintainability, companies are unlikely to introduce tools that are not the industry standard.

Ftuuky• 10 years ago

I'm really interested in learning more about data science. Currently attending moocs to learn R but I don't have much free time and it's a slow progress. Currently I work as analyst for an environmental consulting firm but I always wanted to work in something related to quants. I know I'm probably being naive but I want to change careers. Do you guys recommend that I keep learning R or should I take a closer look to Julia? Or both? I don't have a programming background but I do know statistics and Excel. Edit: fast, concise and useful comments in a matter of minutes, I love HN. Thanks!

strictnein• 10 years ago

Can't answer whether to learn R or Julia, but for a more general education on Data Science, this book might be a really good fit for you:

Data Smart: Using Data Science to Transform Information into Insight

https://www.amazon.com/Data-Smart-Science-Transform-Informat...

All the work is done in Excel. I enjoyed the book quite a bit. Author is Chief Data Scientist for MailChimp.

mi100hael• 10 years ago

Become fully confident & proficient in a single programming language before hopping around. If you get to the point where you can build a full application from scratch and answer other peoples' questions on SO or IRC with one, you'll be able to easily get up to speed with another should the need arise.

cwmoo740• 10 years ago

R is great for producing scientific papers, charts, and doing a vast array of "standard" statistical analyses. I have found python to be much nicer because it's more of a "real" programming language, but the numpy/scipy/pandas stack make it very easy to use for data science and financial analysis also.

There's a (very) quick intro here:

http://lectures.quantecon.org/py/pandas.html

You can also look into quantopian for practice, but I don't recommend actually trading money.

https://www.quantopian.com/

max_• 10 years ago

I always wanted to work in something related to quants

Then learn Python, it is mature and has way more study material.

Not to mention Machine Learning libraries like TensorFlow, Scikit Learn & math libraries like Numpy, Pandas, Matplotlib?

Everything there is just set.

I once tried to learn Julia, but the community is just to small to get help at times. It is a work in progress but I recommend you to keep an eye on It.

Conclusion: Think of Julia as a future successor to Python, but not a current replacement.

SixSigma• 10 years ago

A basic introduction to Julia in Data Science

https://www.coursera.org/learn/julia-programming

ellisv• 10 years ago

There are many more resources for learning R, given that the language is older. These resources have become much better in recent years and I think there has been no better time to learn R. Probably the best introduction to data science in R is Hadley Wickham's latest book: http://r4ds.had.co.nz

Learning your second language is much easier than learning your first. You'll be more familiar with programming concepts and terms that make it easier to search for how to do X in that language.

Julia is still relatively new and not quite ready (in my opinion) for new programmers. Things change relatively quickly and I think that could add a lot of confusion that gets in the way of learning.

That being said, I think Julia is a pretty great language. I really love the way it has grown and I'm excited to see how it matures.

bo1024• 10 years ago

The programming side is only part of the picture - don't forget to learn some math and theory as well!

mark_l_watson• 10 years ago

I am not a quant programmer, but last summer I did a ten day job for an old customer in Julia and I had to learn the language as I worked. I thought the expressiveness and conciseness of the language was good and I liked the way the package manager used github projects; easy learning curve and REPL based development.

edblarney3• 10 years ago

Someone explain to me why we need a new language here?

If they are trying to express ideas - wouldn't any language do just fine?

They can 'compile' in an 'optimized manner' using something else ...

But in terms of algorithmic expression?

Unless they are writing true 'real time' stuff right for the chip ... why a new syntax?

awm2• 10 years ago

They can 'compile' in an 'optimized manner' using something else ...

Thats actually the issue Julia attempts to solve as highlighted in the article. There are many languages that you can use to express ideas, but when you need the idea to run as fast as possible, you have to rewrite in C/Fortran/etc and then add bindings to to the language.

Julia lets you write inline C, which I believe means that you can still express algorithms simply, but with some parts (trivially) optimized. (Not a huge julia user, but thats my take on it)

edblarney3• 10 years ago

I understand that - but wouldn't a really smart compiler be able to do the same thing?

And if you have to be able to write 'some parts' in C for speed ... then why not just use C? And have nice libs for whatever you are doing?

Or intelligently distribute it across processes?

I find it hard to believe that the 'financial industry' has performance requirements that have never been encountered in the entire rest of high tech before ...

mauro3• 10 years ago

Whilst Julia's foreign function interface is indeed good and it is really easy to call into C, the point is that Julia itself is as fast as C. So you don't need to write any C code to get performance, instead just tune the bottlenecks in Julia itself.

For instance, the standard library of Julia [1] is written in Julia itself (and is very performant) and only calls into external C or Fortran libraries where there are well established code-bases (e.g. BLAS, FFTW). Compare this to, e.g., Python or R where much of the standard library is written in C.

[1] https://github.com/JuliaLang/julia/tree/master/base

bwanab2• 10 years ago

One of the ideas behind julia is to get a language that is as expressive as python, with better functional programming capability, that runs at the same order of speed as C.

You can't get something for nothing, but it's a really good start.

jernfrost3• 10 years ago

(1) if every programming language was equally good at everything we would not have a multitude of programming languages. And there are always advances in programming language theory and concepts which have not been captured in earlier languages.

(2) The semantics of the language put restrictions on what sort of optimizations you can do or how easy they are to perform. Julia has been designed to make it very easy to optimize. E.g. something like JavaScript runs quite fast, but only through a herculean effort by the JIT makers. It's design is terrible with respect to making it easy to create high performance compilers/JIT.

(3) What is "new" syntax. All languages have variations in syntax. Wherever Julia differs in syntax from whatever you know it isn't just for the hell of it. There are usually good reasons. Mostly Julia tries to look similar to programming languages familiar to its target group. Where it differs I'd say it is usually to improve on deficiencies in said languages.

I am an iOS developer but I write smaller scripts to do custom processing of source code, assets etc. I wrote my scripts in Julia first, but decided to convert to Python afterwards to make it easier to distribute to other people, given that Julia does not come pre-installed.

I got to say Julia is WAY WAY nicer to program than python for shell script like stuff. I kept hitting on so many issues when converting to Python that I kept a long list of issues with Python. My takeaway from this experience is that a LOT has happened since Python, Ruby etc was conceived. Julia has been able to tap into all the advances that has happened since then. It has a much smoother REPL environment, much more sensible set of built in functions. More sane naming conventions. And the libraries are built to combine effectively in much better way. You simply need to write much less code when writing Julia because stuff combine so well. While in Python I find myself having to write a lot of glue code to make pieces fit together.

So the question is, why stick to a language just because it is well established. I had many more years of experience with Python. But I get the job done much faster with Julia. So why waste extra time on Python just because it is established? That has no inherent value.

edblarney3• 10 years ago

"So the question is, why stick to a language just because it is well established."

You answered the question yourself:

"but decided to convert to Python afterwards to make it easier to distribute to other people"

Languages are platforms. They have vast distribution, incumbency, developers, apis etc. - and that has a lot of value.

Imagine if you did dev on Julia - how do you hire devs?

Julia has a very, very small mind-share - in order for it to reach 'platform status' it really has to cross a threshold, and for that - it has to provide significantly more value.

I'm also curious to know 'how much faster' it is - couldn't those performance problems be solved with more horsepower?

Developers have a terrible habit of over-optimizing for performance, which usually comes at the cost of complexity.

Anyhow - I know little about Julia so I'll hold my tongue, maybe it's the bees-knees ...

It'd be great to see some data points.

blitzd3• 10 years ago

My beef with Julia when I last tried it out was the lack of flexibility within the 'data munging' sphere. The assumption appeared to be that all of your data would come in to as UTF-8, whereas in my org we need strong support for Latin1/cp1252, and UTF-16LE BOM. Is this something that has changed recently, or are these traders are economists just working with more sane data?

z92• 10 years ago

Pipe the data through a converter before feeding it into Julia.

blitzd3• 10 years ago

Well - that's my point. I do use a converter, as it's built into python at every intersection that I need it, reading CSV, XLSX, database calls, etc.

Having to stick python scripts in front of all my Julia scripts everywhere that it would perform data access doesn't really seem viable. Thus, my beef with Julia.

jernfrost3• 10 years ago

Not a language issue per say then. It shouldn't a problem to add support for this. Julia is even designed from the start to work with Strings of different types. Something not common in most other languages I've worked with.

Most string functions operate on AbstractString types. You can create whatever subtype you like implementing alternative string encodings.

elcritch2• 10 years ago

Building on this comment, it's actually surprisingly straightforward to add your own encodings into the core language. Just add the appropriate types and extend the necessary base methods.

In this case, it appears Julia 0.5 has a transcode method supporting UTF8/16/32. [1]

1: http://docs.julialang.org/en/release-0.5/stdlib/strings/

blitzd3• 10 years ago

I'd venture a guess that implementing any string encoding support in a language I'm not that familiar with would likely be quite an undertaking.

If not, then it also begs the question - if it's just that simple, why's it not already there in the core alongside UTF-8?

Anyone new to the language working through data problems is going to pick it up and start at the data collection/wrangling stage, and immediately run into the 'you can implement that yourself/run it through a converter' mentality, and likely go back to whatever they were using before. I know we did.

StefanKarpinski• 10 years ago

AbstractStrings (the generic string abstraction) can have any encoding – only the standard built-in String type is UTF-8. If you've got data in another encoding, use the StringEncodings [1] package to read it.

[1] https://github.com/nalimilan/StringEncodings.jl

sgt1013• 10 years ago

When Julia has an environment as good as Rstudio and a dashboard as good as Shiny, and a Spark interface that is as good as Sparkler and R-Spark then I'll be up for it.

Writing good code in R is an effort, it's possible but the language doesn't make it easy. Julia makes me write good code off the bat and facilitates writing excellent, elegant code.

claytonjy2• 10 years ago

This is how I tend to feel about all non-R things; I've been using R for long enough to dislike a lot about the language, but the environment and tooling is too good to give up.

Letting R be the first language I've gotten pretty good at is really spoiling me.

pitaj2• 10 years ago

Yep. In my short time working on an R project, I really enjoyed how everything was right there. I'd love to see more REPLs include package managers.

astrochymist• 10 years ago

Rstudio is nice, no doubts there; however, I find myself avoiding language-centric IDEs. If one uses R most of the time, it's fine, but as someone who usually writes in multiple languages in a day, I find myself appreciating emacs' modes more (of which the Julia one is fine). It's also why I like the fact that the semi-official Julia IDE, Juno, isn't a standalone application (it runs in Atom and is nice, if one doesn't mind the performance of Atom...)

edit: spelling

sgt1013• 10 years ago

I liked the Julia mode in emacs, but dear god, plots, files, environment, help and viewer.. (what is viewer for?)

IndianAstronaut2• 10 years ago

It is the libraries which really make R. Easy decision trees, SEM, robust regressoon, etc.

fredcy• 10 years ago

"The importance of data and speed to information can't be understated in the capital markets ..."

That says, it's less important than you could possibly state.

rafinha• 10 years ago

deja vu

jefferson6662• 10 years ago

just like pandas in python, I want to give a quick impl in go lang for my backtest strategy system.

now it is open sourced in github: https://github.com/qingtiandalaoye/GoDataframe

farright• 10 years ago

I've never used Julia but it reminds me of Go in that it attempts to replace Frankenstein systems that mix low level languages and high level languages/scripts, with a single modern, high level language with sufficient low level features and performance.

However, in order to do so, they have to give up the generality of either language, and focus on a particular domain (numerical computation and web servers respectively). And because of this, the language never develops a big enough ecosystem. Because of this, I think Python combined with native C libraries is ultimately the better way forward and will be more successful.

jernfrost3• 10 years ago

I really love Julia and see a bit potential outside of the scientific computing. I used to do all sorts of automations, smaller throw away programs etc in Python and Ruby, but I find Julia much more effective to work with.

It also feels like a smaller language in many ways. You don't have to keep as much in your head, which makes it more effective to keep up with.

The killer features for me, automating a lot of tasks is the awesome integration with the shell environment. Running and combining Unix commands is very elegant and getting hold of the output is done really smoothly.

Secondly what I think saves a lot of time is that it really has ALL BATTERIES included. With Python that feels more theoretical. With Julia you just start writing the name of a function you think should exist in the REPL and it pops up. With Python you got to start guessing which package it might be located in. And that goes for stuff you use all the time like string manipulations, regular expressions, file reading and writing, process input and output etc.

With Python functions often end up having unnatural names because you can't have naming conflicts. Julia handles functions with the same name but different argument types elegantly.

Also there is no schizophrenia wondering if something is a function or method on an object. Everything is a function so that is easy.

leethargo• 10 years ago

I would also like to use Julia more for "throw away" scripts and general automation. Unfortunately, startup time (and sometimes compilation) make this a less fun endeavor than with Python etc.

cutler• 10 years ago

Given Julia's pre-1.0 status, general immaturity and good Python interop wouldn't a saner approach be to use Python as your primary language and Julia for whatever needs to be fast?

tekknik• 10 years ago

So how come not golang instead of Julia? I don't have anything against Julia myself but it seems like go with its maturity, easy cross compilation, static compilation and easy IPC/multithreading it would fit the bill quite well. Granted the GC is a bit slow but should be improved with the next release.

whyrt12• 10 years ago

Golang is far far inferior in many ways, for scientific programming. For starters, it doesn't have generics.

jefferson6662• 10 years ago

just like pandas in python, I want to give a quick impl in go lang for my backtest strategy system. now it is open sourced in github: https://github.com/qingtiandalaoye/GoDataframe

could anyone give a performance test of it? many thanks

A Look at How Traders and Economists Are Using the Julia Programming Language

Related Stories