Haskell in Production: Standard Chartered

https://serokell.io/blog/haskell-in-production-standard-chartered

by NaeosPsy • 3 years ago

186 160 3 years ago

Mu can easily take several minutes to compile a single large application. We will circumvent those problems by switching our front-end to use GHC directly

What does he mean by this?

Vosporos• 3 years ago

From what I understand: GHC takes care of the parsing of the source file, error reporting, etc. Then the AST (or an intermediate representation) is passed to the of the Mu compiler, which performs whole-program optimisations.

nequo4• 3 years ago

Gergo Erdi gave a talk on the GHC efforts of Standard Chartered last year:

https://www.youtube.com/watch?v=fZ66Pz7015Q

wathef• 3 years ago

I may be wrong, but if I remember correctly Standard Charter uses their own “dialect” called Mu[1].

1 - https://icfp21.sigplan.org/details/hiw-2021-papers/14/Haskel...

grumpyprole7• 3 years ago

He probably means that GHC will parse and type check faster. Switching the front-end doesn't sound like it will circumvent the whole program nature of their compilation. But I'm just guessing.

misja1112• 3 years ago

I'm also interested. Just guessing, but I guess it means that while they use their own (slow compiling) 'Mu' dialect for most applications, they have switched to using 'standard' GHC for their front-end apps.

tuukkah4• 3 years ago

Frontend here refers to the first stage of the three-stage structure of compilers, ie. the functionalities such as parsing and type checking that come before the optimization and target code generation phases: https://en.wikipedia.org/wiki/Compiler#Three-stage_compiler_...

mtl773• 3 years ago

Gergo and I are working on a new implementation of the Mu compiler that uses GHC up to Core; from which we translate to the legacy Mu bytecode format.

The original Mu compiler is not designed for separate compilation, our new version is, based on GHC's binary representation of compiled interfaces with unfoldings.

cosmic_quanta• 3 years ago

While Haskell jobs may appear scarce, Standard Chartered is often recruiting for Haskell devs:

https://discourse.haskell.org/search?expanded=true&q=Standar...

dreixel• 3 years ago

Thanks for finding these! More directly, here's my latest jobs post: https://discourse.haskell.org/t/haskell-jobs-at-standard-cha...

nemoniac• 3 years ago

So many questions left unasked! In particular they have "recursion disabled by default" but "we have all the usual recursive combinators". Does this mean that they can't explicitly write recursion buy may achieve recursive effects only through the combinators? I'd like to know more about this. Is it as expressive as recursion only through "properly optimised" combinators?

Laaas• 3 years ago

By combinators they mean things like folding/reducing over lists, i.e. combinators/higher order functions for operating on potentially infinite structures.

It's not as expressive as raw recursion since many uses of recursion are hard to express through combinators, albeit a great deal of research has been done in this area (look up recursion schemes).

ackfoobar• 3 years ago

It has been argued that unrestricted recursion is like GOTO. I have some limited experience with "recursion schemes", they're quite expressive. But I feel like it's trying to give every possible pattern of recursion a name, to the point that it's almost becoming a joke - e.g. "zygohistomorphic prepromorphism".

mtl773• 3 years ago

The reason is that we don't have a general solution for tail-call optimisation for computations made of a mixture of Mu code and C++ primitives. We have higher order functions defined in C++ that can be called from Mu, and can call back Mu functions passed to them. This back and forth between languages has occurred naturally in recursive definitions, and we don't have a way of reusing stack frames between the Mu interpreter and the native C++ code for these higher-order functions. We decided a long time ago to disable recursion by default, encouraging programmers to use recursion operators. We do however have a Mu language extension that enables recursion, which is used when the recursion depth is known to be bounded.

boxed4• 3 years ago

Haskell in Production

Does not use Haskell.

Is it just me or is this... weird?

tuukkah4• 3 years ago

Sure, it's a bit weird but not that weird as they have a huge codebase in a specific domain. Also, they still end up using a lot of Haskell code (and semantics) although they write in a thin layer of "DSL" on top of Haskell and GHC (which itself is a lot of "Haskell in production").

Mu has a strict runtime, which makes program performance easier to analyse and predict – but prevents us from just copy-pasting some Haskell libraries that rely crucially on lazy evaluation.

boxed4• 3 years ago

From what I understood they don't use Haskell underneath at all, but they copy paste code from Haskell and sometimes have to modify it.

yakshaving_jgt3• 3 years ago

Mu is essentially Haskell. It's not clear to me where you got "does not use Haskell" from.

boxed4• 3 years ago

It's quite clearly different in many important ways. Not being lazy is a pretty huge one!

gorjusborg• 3 years ago

Should probably be 'Dialect of Haskell in Production'

koromak3• 3 years ago

I write TypeScript in production but I'm happy to call it JS

mark_l_watson3• 3 years ago

Interesting interview. I applied for a Haskell job at Standard Charter in Singapore about eight years ago. I didn’t really expect to get the job because the only Haskell experience I had was my own Haskell projects and I had written a short book on Haskell. The guy interviewing me and reviewing my take home programming test had done a PhD studying aspects of Haskell. It was an interesting experience for me.

weavie• 3 years ago

Did you get the job?

mark_l_watson3• 3 years ago

No. The interviewer suggested that I get one year experience elsewhere, then reapply for the job. I started to do that, but a month into a Haskell gig, I received an offer to manage a deep learning team at Capital One, and pivoted.

grumpyprole7• 3 years ago

You won't need a Haskell PhD to work at a bank, even one that uses Haskell.

eru• 3 years ago

I worked in that team from 2012 to 2014. I don't have any degree and didn't publish any books.

Perhaps they made the interviews more difficult since, or you got unlucky?

mark_l_watson3• 3 years ago

I just don’t think I had enough Haskell experience.

kerkeslager5• 3 years ago

I've talked to a lot of people using Haskell in production over the years, and one of the things I've consistently noticed is a disconnect between the Haskell features that are often touted, and the Haskell features that people use in production.

This isn't always a quiet omission, like "we didn't end up using this" either. The best example is that a lot of people end up doing some significant work to disable lazy evaluation. Simon Peyton Jones is reported to have said that lazy evaluation has been kept "to keep us honest about side effects", i.e., that size effects become immediately painful with pervasive lazy evaluation. But in a production system, functional purity (no side-effects) is a means to an end, and lazy evaluation comes at a pretty extreme performance cost. Not everyone disables lazy evaluation in production Haskell, because not everyone thinks they need to for performance, but I've definitely heard about it enough times that it seems like a relevant pattern.

Production users of Haskell do seem to be happy about types, years into projects, so I'd take away that types are a benefit, but I do wonder if one might get those same benefits in another language without the problems which seem to crop up with Haskell, such as laziness causing performance issues, monadic complexity, and difficulty hiring.

kqr• 3 years ago

Small distinction that does not detract from your excellent point: the question is not about laziness vs. strictness since any reasonably large program uses both extensively. Rather, it is about the properties of laziness or strictness as a default evaluation strategy.

yodsanklai2• 3 years ago

OCaml may be seen as more pragmatic solution.

thesz10• 3 years ago

It cannot.

Arbitrary side effects allowed by strict-by-default language without types constraining these ruin easeness of software development provided by Haskell.

When your Haskell program works and walks, add bangs and it will run. Lately I heard from my colleagues it will run even without bangs as ghc has improved.

thesz10• 3 years ago

If you disable lazy evaluation, you would not be able to write arbitrary "where" sections, with definitions ordered as you want them, including tying-the-knot definitions, which are not exceptionally rare.

So no one disables lazy evaluation.

It is strict evaluation that is selectively enabled.

I also have to say that focusing on the types in the compiler development was the right thing, but what was not right is not focusing on the automatic analysis and removal of thunk creation for lazy evaluation. There is a project now to implement GRIN (graph reductin intermediate notation) [1] which supports such analysis and GRIN is early 90-th.

[1] https://grin-compiler.github.io/

GRIN also automatically gives you many benefits of supercompilation.

We could have a much better Haskell code now if ghc decided to support several backends back then, not only STG/C--.

kerkeslager5• 3 years ago

So no one disables lazy evaluation.

That's objectively incorrect.

foldr7• 3 years ago

So no one disables lazy evaluation. It is strict evaluation that is selectively enabled.

True of vanilla Haskell, but from the article:

So we ended up with Mu, a whole-program compiler to a bytecode representation with a small runtime interpreter... Mu has a strict runtime, which makes program performance easier to analyse and predict – but prevents us from just copy-pasting some Haskell libraries that rely crucially on lazy evaluation.

If S&C is to count as a Haskell success story (which it's often touted as), then I think it's fair to point out that it's a codebase where strict evaluation is used throughout.

yodsanklai2• 3 years ago

Regarding hiring difficulty, some say it's a feature rather than a bug. There are always good candidates. And good engineers can learn and be proficient in any language. Haskell is certainly not harder than C++.

I see some issues though.

Some niche languages enthusiasts can be very picky and reluctant to work on other languages, and would pick Haskell when a simple python script would be more appropriate.

People from Academia interested in Haskell may be good in PL theory, and writing compilers or static analysers, but may lack in other fields such as system programming, which incidentally is harder in Haskell too. So you also need to find devops, system guys and so on who can also be productive in Haskell, and those are more rare.

surement• 3 years ago

Some niche languages enthusiasts can be very picky and reluctant to work on other languages, and would pick Haskell when a simple python script would be more appropriate.

in my experience, these people also refuse to learn about any software engineering best practices as they are "for OO languages" and "not needed" in haskell

rebeccaskinner• 3 years ago

Hiring doesn’t really seem to be a problem for Haskell teams in my experience. Onboarding can be a little slower, but there are a lot of developers out there who want to work with Haskell. If anything, I would consider Haskell a hiring advantage.

nh23• 3 years ago

Hiring for Haskell has never been a problem anywhere I've seen.

I've worked profesisonally in Haskell for 9 years now.

When working as a consultancy, whenever we hired for ourselves or our clients, we got 5x more applicants than we were looking for. Of those, around 80% got a hire recommendation (unfortunately we couldn't hire them all).

When hiring 1 role for our startup, we got 40 good applicants immediately, with a single post on Reddit.

In all cases we had the luxury of picking the best-fitting among many excellent engineers.

Maybe you'll face issues if you want to hire 100 people on the spot. But most companies don't have that problem.

dllthomas• 3 years ago

If you want to hire five developers, Haskell makes that easier. If you want to hire five hundred developers, Haskell probably makes that harder, although it's not something I have experience with.

tome4• 3 years ago

A few minor corrections

Simon Peyton Jones is reported to have said that lazy evaluation has been kept "to keep us honest about side effects"

Rather, one of the main benefits of laziness was to force purity and thereby to instigate the development of typed effects. By contrast, the reason that laziness has been "kept" is simply that switching to strictness now would break (almost) all Haskell programs!

lazy evaluation comes at a pretty extreme performance cost

Not really. Haskell's implementation of laziness is very high performance and the compiler can easily determine when creating thunks is not necessary. The reason that lazy evaluation leads to performance problems is that programmers don't know how to use it correctly. Whether that's the fault of the language or the programmers is up for debate.

logicchains2• 3 years ago

The reason that lazy evaluation leads to performance problems is that programmers don't know how to use it correctly

It's because it's objectively harder to reason about performance with lazy evaluation as performance becomes non-local. You can no longer reason about a function's performance just by looking at its body, instead you have to know the nature of the inputs passed to it (whether they're thunks or materialised values).

Each single uses of lazy evaluation also imposes a latency overhead as the thunk/value must be behind a pointer. Yes the compiler can remove unnecessary thunks, but that's "unnecessary" in the sense of "unnecessary given the semantics of the program", not "unnecessary given the problem the program's trying to solve". I.e. lazy by default will generally result in someone writing code that's lazier than necessary unless they're very careful with annotating to the compiler everywhere laziness isn't needed.

chriswarbo• 3 years ago

By contrast, the reason that laziness has been "kept" is simply that switching to strictness now would break (almost) all Haskell programs!

Also, laziness is a big part of why Haskell exists in the first place. "Strict Haskell" would arguably feel more like ML, Agda, Idris, etc. than (lazy) Haskell. In which case, just use one of those; they're also decent languages ;)

time_to_smile• 3 years ago

Personally I think this is a strength of Haskell that many people don't really recognize and appreciate. It's a language the provides a lot of flexibility. If you want it to be a pure research language, it's happy to do that. If you need to make some sacrifices in regards to purity so you can get stuff into production, you can do that too.

As someone with a lot of time spent experimenting with programming languages, I can't think of any that can be so excellent from a research/experimentation perspective that also share the real production usage that Haskell sees. Take for example Racket, an amazing and also extremely flexible research language. It's probably easier to get started with than Haskell, but has never seen the real world usage that I've seen from Haskell.

scns3• 3 years ago

Simon Peyton Jones is reported to have said that lazy evaluation has been kept "to keep us honest about side effects", i.e., that size effects become immediately painful with pervasive lazy evaluation.

IIRC he wrote or said that lazy evaluation, was a mistake in hindsight.

nequo4• 3 years ago

Do you have a source for that? From what I recall, SPJ says that the grass is always greener on the other side. So if Haskell was strict by default, some of us would be pining for a lazy version of it.

aranchelk3• 3 years ago

Subtle correction: as far as I’m aware if you were to disable lazy evaluation (which you can now do via compiler pragma) on any Haskell program of significant complexity it would blow up. Common practice is to force evaluation of critical sections of code. That’s something the compiler can sometimes do for you.

Laziness in Haskell is a great way to write declarative code, you really are describing a concept rather that telling the computer what to do. That can lead to very clear and orderly source code.

Having said that, the drawbacks are significant, performance, memory usage, and debugging can all be a real pain. The last one sucks because it affects development of all kinds of programs, not just performance critical (and challenged) ones.

Shout out for PureScript, which is essentially a strictly evaluated Haskell that runs on the browser, servers, and PaaS.

Difficulty hiring on a small scale in my experience is complete BS, there are still many passionate users looking for professional gigs. Once that pool is exhausted, I think a company should probably look outside the language for people with some type of FP experience and expect language-specific training as part of onboarding, whether that’s worth the time and cost is entirely settled for me but certainly a topic for debate.

ParetoOptimal15• 3 years ago

Subtle correction: as far as I’m aware if you were to disable lazy evaluation (which you can now do via compiler pragma) on any Haskell program of significant complexity it would blow up. Common practice is to force evaluation of critical sections of code. That’s something the compiler can sometimes do for you.

Anecdotally I can confirm this is true.

Laziness in Haskell is a great way to write declarative code, you really are describing a concept rather that telling the computer what to do. That can lead to very clear and orderly source code.

100% agree.

Having said that, the drawbacks are significant, performance, memory usage, and debugging can all be a real pain. The last one sucks because it affects development of all kinds of programs, not just performance critical (and challenged) ones.

I think this largely depends on domain. Then, it's not clear if the best tradeoff is to go fully strict or develop better intuition for lazy evaluation in many of those cases.

Difficulty hiring on a small scale in my experience is complete BS

I have a theory this meme is repeated by people who don't know where to find the many functional programming candidates.

ParetoOptimal15• 3 years ago

But in a production system, functional purity (no side-effects) is a means to an end, and lazy evaluation comes at a pretty extreme performance cost.

In theory yes, in practice... not in my experience. Or at least, things are almost always fast enough in web backends.

Not everyone disables lazy evaluation in production Haskell, because not everyone thinks they need to for performance, but I've definitely heard about it enough times that it seems like a relevant pattern.

Most people don't disable lazy evaluation in the way I think you mean it, but they use either BangPatterns selectively or StrictData.

Maybe there's widespread usage of `-XStrict` as well I don't know about.

Anecdotally I can say that trying to enable `-XStrict` in production caused performance issues that prevented us from trialing it further.

Laziness is frequently talked about as a negative, but it has big advantages in composition and code ordering like another reply talked about with regards to where bindings to enable truly top-down program design.

whateveracct2• 3 years ago

Anecdotally I can say that trying to enable `-XStrict` in production caused performance issues that prevented us from trialing it further.

hah this is in many ways indirect proof that laziness-by-default is improving performance. It may even be that a rote translation to strict (which `-XStrict` does) is causing _time_ leaks, which are actually pervasive in production strict-by-default code.

ParetoOptimal15• 3 years ago

monadic complexity

In production Haskell I've seen it's typical you only need to understand how to use the common Maybe, Either, and IO monads. Maybe... sometimes... you need to create your own.

I actually think it's a bit of an anti-pattern, but this anti-pattern leads to a reality where the monadic complexity you alledge does not exist.

jsbg3• 3 years ago

In production Haskell you need to combine monads using e.g. monad transformers and this is significantly more complicated than just understanding how bind works. It also requires a lot of boilerplate code and any errors there are not obvious to track down.

whateveracct2• 3 years ago

People take advantage of laziness without even realizing it in production all the time. In fact, people often get free performance gains due to laziness all the time!

hardwaresofton• 3 years ago

Production users of Haskell do seem to be happy about types, years into projects, so I'd take away that types are a benefit, but I do wonder if one might get those same benefits in another language without the problems which seem to crop up with Haskell, such as laziness causing performance issues, monadic complexity, and difficulty hiring.

Rust. Rust is that language.

Rust's type system is good enough and close enough to Haskell without the footguns of other options (Java, C++, etc), with new paradigms that Haskell hasn't yet fully adopted (borrowing, etc) and consistency in build tooling, etc.

I hate to say it, but it's incredibly hard to suggest people run Haskell in production when Rust exists and you basically trade the difficulty of learning monads + mtl + ... for the difficulty of learning to live with the borrow checker.

ParetoOptimal15• 3 years ago

Rust is good, but writing the experience writing Rust isn't like the experience writing Haskell.

Here is a good post on it:

https://serokell.io/blog/rust-vs-haskell

icrbow• 3 years ago

Rust's type system is good enough

Good enough for what?

close enough to Haskell

Close enough for what?

with new paradigms that Haskell hasn't yet fully adopted (borrowing, etc)

Honestly, I don't want to think about borrowing. For my gamedev/rendering tasks I have resource regions (as a library) and trust the runtime to do its thing.

grumpyprole7• 3 years ago

I do wonder if one might get those same benefits in another language without the problems which seem to crop up with Haskell

Not until other languages raise their game. Nothing compares to Haskell for productivity and I've tried many different languages professionally over the years (including Haskell). But yes, reasoning about what happens operationally is the big problem.

ParetoOptimal15• 3 years ago

Not until other languages raise their game. Nothing compares to Haskell for productivity and I've tried many different languages professionally over the years (including Haskell).

100% agree.

But yes, reasoning about what happens operationally is the big problem.

What types of issues do you experience?

garrisonj• 3 years ago

Really? I can’t imagine people turning off the lazy evaluation. It’s a deep part of the language. You can force things to evaluate strictly, but that’s not disabling it.

brundolf• 3 years ago

lazy evaluation comes at a pretty extreme performance cost

This is surprising to me. I understand it making code harder to debug because timing/ordering is less predictable, but I would expect lazy evaluation to help with performance if anything ("the code doesnt rely on a specific 'when', so the compiler/runtime are free to re-arrange stuff for optimization")

What am I missing?

tildeheimat• 3 years ago

Should probably look into Haskell. Have absolutely no clue what's it best suited for?

Pirate-of-SV• 3 years ago

Writing parsers and tokenizers probably.

ParetoOptimal15• 3 years ago

That's what it's best for, but personally I use it for everything. If I ever get into low-level code I'll probably use Rust though.

You can confirm that parsers/tokenizers is ranked "best in class" here though:

https://github.com/Gabriella439/post-rfc/blob/main/sotu.md

unhammer• 3 years ago

Here are some stories of what it's actually used for: https://old.reddit.com/r/haskell/comments/12vhz1c/genuine_qu...

irmgardweber• 3 years ago

Ha! I was also curious. Thank you for sharing.

gilmi• 3 years ago

Please allow me to share a blog post and a learning resource:

- Consider Haskell - https://gilmi.me/blog/post/2020/04/28/consider-haskell

- Learn Haskell by building a blog generator - https://lhbg-book.link

ParetoOptimal15• 3 years ago

You might find this survey of Haskell's maturity of different use cases very useful:

https://github.com/Gabriella439/post-rfc/blob/main/sotu.md

koromak3• 3 years ago

I have yet to find a place in my own work where Haskell would strictly be a better choice, but god damn its fun to write.

Thats not a knock on Haskell either, I have a more bog standard job than it likes.

jsbg3• 3 years ago

It's a general purpose language, I wouldn't say it's more or less suited to this or that, other than the libraries available for it.

stonemetal12• 3 years ago

It is a garbage collected language so it is generally less suited for real time applications. As far as I am aware there aren't any implementations designed to run on very low memory embedded platforms.

As far as I am aware GHC's collector is not as good as Java's.

usgroup2• 3 years ago

I'm struggling to understand how possibly inventing your own language -- one not even compatible with the Haskell eco-system -- was ever a good idea in production.

tome4• 3 years ago

It was written to replace an existing functional DSL called "Lambda", and to target Lambda's runtime.

usgroup2• 3 years ago

Reminds of an epistemology joke.

The reason the moon is made of cheese is because it is made of yellow cheese.

grumpyprole7• 3 years ago

Why is it any different to "bank python" used by e.g. JPMorgan or BAML? Or Google writing Dart?

cuteboy19• 3 years ago

I don't think they ever touched the interpreter. It's more like a huge framework on top of regular py.

mtl773• 3 years ago

Main reasons at the time: * strictness was preferred for consistency 1) with the C++ execution model (a lot of our code base is written in C++ by quants, and a typical Mu program will make extensive use of these external calls; they can even be higher-order, calling back into C++); and 2) because we have a lot of end-users who are not Haskell experts and are much more confortable with a strict execution model * code mobility is essential to what we do, and was not directly available in Haskell: we often capture computations on our users' desktops running Windows, and stream them to a Unix compute farm. * GHC was not openly available as an API years ago; this is different now which is why we have started a new implementation that is GHC-based.

liendolucas• 3 years ago

I'm shocked, in the first paragraph they assert that they handle 6 million lines of code. Is it common for the fintech industry to have codebases of that magnitude? I'm aware that for example OSes and browsers can achieve similar sizes (if I'm not wrong certainly more) but I'd have never thought that a fintech company could approach such size. It just seems way too much...

Edit: Question out of ignorance for people that regularly program in Haskell in the industry: I want to believe that the language has nothing to do with this and is just their business logic, right?

great_wubwub• 3 years ago

I don't know about Standard Chartered specifically, but I remember one of the big US banks (I forget which one. JPMC or Goldman Sachs, probably - it was a while ago) telling us that they had more developers than Microsoft.

Don't sleep on these finance companies, they're big development shops.

postexitus• 3 years ago

Jamie Dimon said that they have enough engineers to build their own Windows, back in 2007/2008.

mhh__• 3 years ago

6 million is easy, in fact its probably a small-ish fraction of their total code as a firm. Especially for a bank where their processes are a bit more defined than (say) a hedge fund.

What do you think fintech companies do? (I'm not entirely sure either but I can potentially give a flavour of finance coding in general)

If you want to come up with prices for things then you'll need a bunch of mathematical optimization code, interpolation code etc, then an enormous chunk of market conventions (Quick! How do you calculate the yield to maturity for a Brazilian treasury bill).

Having established this you will then need to build a system for getting market data to the right places to be used - and probably snapping data for historical record too. This could be as simple as drinking from the Bloomberg firehose, or reading from an SFTP server in the morning (no really), or pulling in trading data from an exchange (https://www.eurex.com/ex-en/data/trading-files/efpi for example). It all gets very messy.

And now, if you are a bank rather than merely a player in a market you also probably have your fingers in many fiddly boring pies too (e.g. retail banking, mortgages stuff like that).

Agingcoder2• 3 years ago

That’s a fairly accurate overview. If you’re a bank, there is also code to handle market parameters/fits, software to handle positions/booking, exotic payoffs ( and the accompanying compiler to compile the dsl ), regulation, GUIs and custom tools for trading/sales, risks, etc.

6 million sloc is indeed a fraction of what you’ll find in an investment bank. It’s also (I think) what makes it very interesting: an IB is a huge software system, with a mixture of extremely legacy (30y) and bleeding edge tech, and very strange human made rules.

mrkeen• 3 years ago

Yes it's hard to imagine what all that code's doing.

But I'm currently at a very small fintech, and I've got about half a million lines of code checked out. There is a lot of copy-pasted code, but I also have nowhere near all the repositories checked out.

twic• 3 years ago

I work on a trading desk at a financial firm. I just checked out all our applications, and they total 388444 non-comment lines of Java, 73889 of JavaScript, 13619 of Python, 9160 of C++, and some other things. The JS probably includes some vendored libraries. Also, this is just apps - we have a bunch of libraries used across our apps which will add tens or low hundreds of thousands more lines, but i have no systematic way to check them out, so i can't tell.

That's a single trading desk!

game_the0ry• 3 years ago

I'm shocked, in the first paragraph they assert that they handle 6 million lines of code.

I work at a bank.

To answer your question - yes. The consumer website for the bank I work at was 9 million lines of code (at least as of 2019). A lot of it was bad code, some of it was dead code that was never deleted.

swyx• 3 years ago

I guess I have the relatively unique distinction of having worked at Standard Chartered and used Haskell/Mu in anger from 2012-14, as a currency/interest rates options trader using it for volatility surface mapping and making pricing/market data tools. Offering thoughts:

I’m the head of the Core Strats team at Standard Chartered Bank. Core Strats consists of over 40 developers using Haskell (well, actually Mu) as their main programming language.

wow - this team used to be 8-10 people? amazing to see its grown so much. our old head used to be https://donsbot.com/about/ . i have a pet theory that Don and gang ensured Haskell's viability by building so much stuff at SCB that SCB was kind of pot-committed to Haskell, thereby ensuring that they would keep investing and hiring until the end of SCB. Similar move was played by Cheng Lou over at Facebook, for Reason/Rescript. if you want to grow a programming language at a bigcorp, embed it in a biz critical application and you have job security for life

Our core financial analytics library, Cortex, consists mostly of Mu/Haskell code (over 6.5 million lines) and forms the basis of our price and risk engine across all asset classes. It’s also pretty full-stack; we use it for everything, from server-side long-running batches handling millions of trades a day to front-end desktop graphical user interfaces.

yep i can attest to this - i will say that Mu was the best developer experience i've had for a long time. It comes with its own storage/persistence layer (simple KV store iirc) but the joy of not having to choose or maintain or scale was great. also the entire internal package ecosystem was searchable via function signature (via Neil Mitchell's Hoogle) and versioned daily (so if any regressions happened you could just jump back day by day - we didnt use git in my day). i also missed if there was a testing suite.

We can seamlessly call C++ functions from Mu and vice-versa, and pure Mu functions can be used in Excel formulae via our Excel add-in

yeah this was used a bunch. however the traders rarely touched this code so it probably required a lot of support from the devs and probably some misunderstandings where had.

What benefits do you see for using Haskell in banking and fintech areas? One great advantage is static typing..

no no no. correct me if i'm wrong but dont large fintechs like stanchart and jane st use functional languages because its so much better to parallelize for risk scenarios and complex option pricing?

Does Standard Chartered have an in-house training program for upskilling Haskell developers? For training absolute beginners in Haskell we typically engage an external training partner.

ha. in my day i was just given Learn You A Haskell and told good luck!

SCB people reading along, i hope some of my FXO apps are still around. Coriolis, Corona, and i'm blanking on the main pricing one. 10 years wipes away a lot.

koromak3• 3 years ago

I'm just starting to learn Haskell and I honestly can't fathom how it gets written in production. Everything requires so much more consideration, which is fun and valuable in some situations, but not when you just need to get the damn thing done in time for a deadline.

I'm not experienced enough to know, but it does feel like Haskell prioritizes "clever" code. Very beautiful, but hard to understand at a glance.

jsbg3• 3 years ago

You can get features done very quickly in Haskell after the initial learning curve that most languages have. The type system lets you catch a lot of mistakes before runtime and in my experience things are more likely to work the first time in Haskell.

Testability does require some forethought but if your codebase always takes that into account then it's as easy as other languages.

alex_lav• 3 years ago

It's sad you're getting downvoted only because you've commented (ever so slightly) negatively about people's language de jour.

I would argue that "Can people understand a unit of code without experience in that language/domain" is valuable, even if not the _most_ valuable aspect of a language. I also think dismissing this desire is usually an act of gatekeeping. "_I_ understand it so it's fine."

grumpyprole7• 3 years ago

Sounds like you just need more practice. Haskell is more expressive than most other languages and offers much more consistent and powerful abstractions. This does result in more productivity once the learning curve is overcome.

ParetoOptimal15• 3 years ago

I'm just starting to learn Haskell and I honestly can't fathom how it gets written in production. Everything requires so much more consideration, which is fun and valuable in some situations, but not when you just need to get the damn thing done in time for a deadline.

There are ways to write quick n dirty Haskell. Actually a super-power of Haskell is how easy it is to refactor from "quick n dirty" to "reasonable quality" without introducing defects.

spopejoy• 3 years ago

Actually a super-power of Haskell is how easy it is to refactor from "quick n dirty" to "reasonable quality" without introducing defects.

I call haskell "the language of maintainable spaghetti" in this spirit. It really is incredible how types can take 99% of the risk out of refactoring.

Indeed while I miss machine refactoring tools from Java, I've seen vanishingly few bugs result from manual refactoring in haskell.

mrose112• 3 years ago

We can seamlessly call C++ functions from Mu and vice-versa, and pure Mu functions can be used in Excel formulae via our Excel add-in.

wtf

ykonstant• 3 years ago

It makes perfect sense; non-technical users and employees with different specializations (like actuaries) use lots of Excel, especially in fintech; and if you can share function implementations verbatim between your codebase and Excel, you get higher confidence for coherence.

WJW• 3 years ago

What's weird about that? Excel is absolutely everywhere in finance, so if you have a special programming language that you consider to be part of your secret sauce it makes sense to integrate it into excel as well.

A friend who works as a at risk manager at a major bank sometimes shares horror stories about the giant spreadsheet they have for some part of their holdings. They have disabled the "automatic recalculation" feature because it takes over 48 hours to recalculate everything. This is after running it through a special excel compiler to speed it up.

mrose112• 3 years ago

Excel is everywhere used everywhere and for very important things.

I just never... thought about writing a Haskell plugin for it. It makes sense! Better than using some other language. Functional for Excel makes sense.

I love hearing horror stories about small (and large!) governments and companies abusing and misusing Excel. https://sheetcast.com/articles/ten-memorable-excel-disasters

xboxnolifes• 3 years ago

It's not very different from using python to hook C/Fortran libraries together.

Haskell in Production: Standard Chartered

Related Stories