Building a Forex Trading Platform Using Kafka, Storm and Cassandra

http://insightdataengineering.com/blog/Building_a_Forex_trading_platform_using_Kafka_Storm_Cassandra.html

by jecs321 • 12 years ago

139 39 12 years ago

fasteo• 12 years ago

I am no expert in forex platforms, but this statement seems a bit off:

"individual investors only have a few simple tools at their disposal, e.g., Meta Trader or Ninja Trader"

Both meta trader and Ninja trader have powerful event-based scripting languages (MQL5 [1] and NinjaScript [2]) with a solid library of charts, strategies, indicators and order execution. More than that, they have a huge community providing all kinds of services around those platforms[3][4]

Other than that, Wolf seems like a nice piece of software.

[1] http://www.mql5.com/en/docs

[2] http://www.ninjatrader.com/support/helpGuides/nt7/

[3] http://www.mql5.com/en/signals/mt5

[4] http://www.metatrader5.com/en/automated-trading/mql5market

dclusin• 12 years ago

Lots of brokers also offer vanilla FIX connections to their systems as well, so a GUI isn't necessary.

monkeyboysteve• 12 years ago

Virtual Brokers comes to mind.

kolencherry• 12 years ago

Interactive Brokers offers a FIX interface to their system.

reduce• 12 years ago

Simple... versus the tools inside the top investment banks. I'd estimate that the publicly available tools are at least 10 to 15 years behind.

spitfire• 12 years ago

More than that. Further still, the publicly available tools trend towards only covering the "algorithm" part of the problem - that's what's sexy. Missing out is things like risk management, money management, allocation, OMS vs EMS, etc.

Trading for most prosumers is a vanity hobby, though they don't know it.

mattfrommars2• 12 years ago

Anyone know how I can run this? I was into FOREX few months back and this tool looks very interestnig. I need to exchange large sum and want the most of the volatile FOREX market.

pas4• 12 years ago

https://github.com/slawekj/wolf#getting-started

It suggests to deploy modules on different machines, but you could probably try this out on 3-4 VMs (use SSDs).

There are rather simple deployment scripts (it uses apt-get, so get Ubuntu VMs), for some of the modules.

Set up a Hadoop cluster (HDFS, Hive, Camus) use YARN, you can run Storm on it, but there is a deployment (install & run) script for running Storm standalone (with a standalone Zookeeper node).

So, it's a lot of work, but sounds really interesting, fun and immensely didactive!

jjHey5• 12 years ago

A pilot installation of Wolf was made on 10 ec2 boxes, mainly m1.medium and m1.large if I remember correctly. 3 boxes for C*, 3 boxes for Hadoop, 1 for Kafka, 1 for Storm, 2 for everything else. You might be able to squeeze it in less machines as Amazon is upgrading specs all the time. Deployment scripts are very simple at the moment, just a bunch of shell commands. That might be upgraded in the future to some magic automation. Nevertheless, scripts were tested on Ubuntu 12.04, but are not specific to Ubuntu, other then the package manager (apt-get). Probably if you sed s/apt-get/yum the scripts, they will work on CentOS/Fedora. If you have a really strong machine you might even be able to squeeze everything in one box. I wouldn't recommend it though, keeping the state of all the clusters in one Zookeeper is a very risky business, plus reliability issues. Unfortunately, it's not so simple to get Wolf from nothingness to a working/trading system. To actually trade you will also need a data feed, and a brokerage account.

pas4• 12 years ago

It is a truly global marketplace that only sleeps on weekends.

Sure, there are 4 trillion reasons why this is enough and yes, I know it's very much about traders themselves, and that if there were an incentive to change it probably would change pretty fast, but anything that stops on the weekend seems very silly in the 21st century. (Even EVE Online's daily downtime is a bit ridiculous.)

xxs2• 12 years ago

You need liquidity on Saturday and the banks don't work.

quantisan• 12 years ago

yep, you actually can execute trades on saturdays on some retail brokers (they usually act as the counter-party) if you're willing to take the humongous spreads during those hours.

pas4• 12 years ago

Why is liquidity for settlement so strongly linked to making contracts (trading)?

fxthrowaway• 12 years ago

This is a somewhat related question about MetaTrader, which is the most popular FX platform.

It could be considered paranoid and jingoistic, so I apologize in advance to all the great Russian hackers out there.

Does anyone know anything about MetaQuotes (the maker of MetaTrader)? They're a trusted solution provider to many of the biggest banks in the world, and yet in my experience it's difficult to find anything about them online (the company, not the software).

I know it's a broad stereotype, but many Russian companies don't have the best reputations when it comes to business ethics. It's an open secret that many of the top oligarchs formed more or less a kleptocracy with the Putin administration, and that the rule of law is pretty shoddy when it comes to Western companies or individuals seeking justice.

Traders all over the world plug in their algorithms in plain text right into MetaTrader. Realistically, do they have anything to worry about, or am I being completely paranoid?

I don't mean to disparage MetaQuotes. As I said, I don't know anything about them, and they partner with big financial institutions all over the world. Many of these companies provide customized versions of MetaTrader, so I would imagine they might have access to some or all of the MetaTrader source code. I guess I'm just looking to be reassured.

What steps could a trader take on any platform in order to reduce exposure to potential bad actors? Is using an API from a broker the only solution?

Anyway, again my apologies if I offended anyone.

Update: their Wikipedia page has a few more articles on them than I remembered. Also, technically they are registered in Cyprus, but were founded in and are primarily based in Russia. Still, I'd love to hear people's thoughts on this issue:

http://en.wikipedia.org/wiki/MetaQuotes_Software

notastartup• 12 years ago

I used to trade on Metatrader back in 2007~2009.

There was a huge scandal because it was revealed that MetaQuotes implemented a function to forcefully cause disruption in the orders that were coming in. This function would be used by brokers to forcefully widen the spread at any given time. I remembered a lot of traders, myself included, just completely lost trust in retail forex. It was nothing but a sham.

cseelus• 12 years ago

If you do still trade, which Application do you use today?

twelve40• 12 years ago

Just like Intellij! Omg the Russians own our codes

random3• 12 years ago

JetBrains (the company behind IntelliJ) is headquartered in Prague - Czech Republic.

genericacct2• 12 years ago

I can't think of a single bank using metatrader, you're thinking of brokers.

genericacct2• 12 years ago

i fail to see the point in using hadoop if your universe is limited to foreign exchange... you can probably fit all ticks of all pairs in existence since the 70s in less than a TB

dodders2• 12 years ago

Assuming 40 pairs, 2 ticks per second and 1kb to store a tick (including timestamps, indexes etc), I reckon that's about 70TB for 40 years worth of data, not counting weekends.

I've assumed only outright ticks, hence the 40 pairs - in reality most banks store the crosses as well, which can be up to 1,600 pairs (40 x 40), which will get you into the PB range for the 40 years.

btn• 12 years ago

One kilobyte per tick seems quite generous. All of the EURUSD tick data for 2013 from histdata.com (the source mentioned in the article) is only 515MB (~20GB for 40 years, ~824GB for 40 pairs).

xxs2• 12 years ago

Not the surprise: you need less than 3bytes on average to store a tick...

jjHey5• 12 years ago

That is a very good point. An initial idea of Wolf was to create a platform that could trade virtually anything (Craigslist, Amazon, Ebay, Stock market, Forex, etc) as long as there's a trading API, and could make use of data from just about anywhere. Not just ticks. Imagine you can write a trading rule like this: "if Mr. X invested $ in company A and Ms. Y recommends buying stocks of company B or an epidemy spreads in country C or product D received great consumer reviews then trade E for F". That would require a lot of data: news articles, financial data, customer reviews, geopolitical events, etc. One of the Insight data engineer found some cool correlations between geopolitical events from GDELT and Forex. Wolf was created end-to-end in 6 weeks so... I focused on Forex. P.S. One year of compressed Forex ticks from HistData.com is about 100GB.

scottlocklin2• 12 years ago

"The foreign exchange market, or forex, is the biggest and the most liquid exchange service in the world with over $4 trillion worth of trades made every day."

Yes, and you are shut out of that market unless you're on the interbank (which generally means you are an actual bank). Retail forex "markets" are bucket shops that get first look at all their customer bids. The actual forex market is OTC.

http://www.ecnforex.co.uk/interbank-forex-markets-explained/

Trading retail forex is a mug's game.

threeseed• 12 years ago

I would disagree that it is entirely a mug's game.

There are some EAs for MetaTrader that will check if your broker is manipulating your SL/TP and stop the trade. Also if you're a medium/long term trader then it doesn't really matter what your broker does.

scottlocklin2• 12 years ago

In that case: you should be trading futures.

itaifrenkel• 12 years ago

Were you actually able to get 1ms latency from Kafka? How long does it take an event to enter then exit Kafka?

jjHey5• 12 years ago

No. I only had boxes with SATA drives. I don't see how you could get below 1ms latency with spinning disks. You would have to use in-memory queue to get this sort of latency. Secondly, I had Storm consuming off Kafka every millisecond, so in the worst case scenario a message would have to wait one millisecond in the queue anyway. Here's a great article about throughput/latency of Kafka https://engineering.linkedin.com/kafka/benchmarking-apache-k....

badname• 12 years ago

Yes. Because that's what was missing in this world. Yet another foreign exchange lottery platform for the rich kids to gamble around.

Edit: Oh boy - the all-I-wanna-be-is-rich egotistical nerds downvoted me. My soul is crushed.

yanev• 12 years ago

The library definitely deserves attention considering the fact that the current MetaTrader and NinjaTrader platforms are like dinosaurs, they feel very old. In the article you said that the stream of information is essential to the operation of Wolf. I also think that this is the fundamental part of a forex platform - the data should be accurate because you have to validate the forex strategy as much as you can during the backtesting phase. Have you considered replacing the current data provider (HistData.com) with something better? Actually we are on the same track because I am also implementing a forex trading platform and I have experienced a lot of problems just because the HistData database is not accurate enough for backtesting - it has a lot of empty records, sometimes a whole day of tick data can be missing.

jjHey5• 12 years ago

Thank you for the comment. I agree that having sound data is essential to building any trading algorithm. Probably that's why they are so expensive. I won't complain about HistData.com, it's a free service, they only charge a small fee for a convenient ftp access to their site. At the moment, that's the most you can get for free with a millisecond resolution. Despite it's drawbacks, works well for backtesting, IMHO.

A great alternative is finance.yahoo.com. It is reasonably rate limited, see http://stackoverflow.com/questions/9346582/what-is-the-query.... You won't be able to do any high-frequency trading anyway without dedicated hardware in a dedicated data center.

sorincos• 12 years ago

Because running a trading algorithm on a non-idempotent stack is such a good idea.

jjHey5• 12 years ago

Very good remark! Storm by default implements at-least-once semantics. If a bolt process executed an event and failed right away without acknowledging, an execution would get replayed (so you would buy/sell twice). However, there's a remedy! By sacrifacing latency you can have exactly-once-semantics, see Trident, Summingbird.

Building a Forex Trading Platform Using Kafka, Storm and Cassandra

Related Stories