Shortwave Trading: The West Chicago Tower Mystery

https://sniperinmahwah.wordpress.com/2018/05/07/shortwave-trading-part-i-the-west-chicago-tower-mystery/

by TheAlchemist • 8 years ago

242 113 8 years ago

sniperinmahwah.wordpress.com

dx0347• 8 years ago

I guess low bandwidth is no problem. You only need to send a stream of price changes of the most common instruments (currencies, stocks). And only if there are meaningful movements. I guess a few kbit/s can already provide a lot of value.

How fast would such a connection be? Is there an easy way to calculate the distance the signal takes compared to great circle line?

coob• 8 years ago

Wouldn't you just send whatever is needed to place the order?

Do all of the processing at the source and just send the final instruction?

(honest question I have no idea how this works)

dmeeker2• 8 years ago

Generally speaking, folks use the CME futures as a leading indicator, so you may have several different trading strategies that trade against the European markets and just use the CME pricing as one of their inputs. In that case, the most bandwidth-efficient play is to send your market data over once from Chicago, replicate it / make it available to all your trading strategies, and let them do their work locally.

212• 8 years ago

The price on the other side also moves, so it's more complicated.

dx0347• 8 years ago

You can send samples. E.g. the USD/EUR rate (or the price of an equities future) if it changes by more than x within the last 50ms. The difference of that rate to the "local" rate in London is enough to generate the trade. Same with equity indices. You'll know that if there's no movement, there's no opportunity. So it's enough to send a signal if there has been a significant move that's "young" enough so that the other side can't know it yet.

Your trades only make money on the difference of the public information in London and what you get faster from Chicago. Therefore sending trade instructions isn't helpful as Chicago doesn't know London prices. But just giving the London algorithm that bit of inside can help a lot.

I assume trade strategies are mostly very simple as well. Basically just anticipating a change in price and quickly buying/selling to make some money off it.

N.B.: I only talk about Chicago->London here but given that Chicago hosts one of the largest Futures markets, I'd assume a lot is actually done in reverse. Using local prices from London of an underlying instrument (e.g. London stock market) to "predict" the value of the future traded in Chicago. You can make a lot of money if you know the price of the underlying instrument of a future before others do.

edf13• 8 years ago

It may well be signals going over the stream... or significant movements only.

chrisseaton• 8 years ago

Mystery #6: What could be interesting across an ocean?

Err.... the rest of the world?

ndespres• 8 years ago

The question is asked in the context of the tower being owned by U.S. Cellular, who would not need shortwave antennae pointed across the pond to service local cell customers.

tomglynch• 8 years ago

So intriguing. Bringing out the old tech for new use cases

madengr3• 8 years ago

I wouldn’t call it old tech. It’s just another portion of the spectrum. There is some pretty advanced stuff going on with HF.

badosu• 8 years ago

Given the limitations of such a medium I wonder if the TCP/IP stack is still viable or another transport protocol is used.

tomalpha2• 8 years ago

It's been a few years since I was involved with any of this, but even for higher-bandwidth microwave networks you want to squeeze every last bit out of the protocol.

I'd guess in this instance that they'll skip both the TCP and IP layers completely and essentially be sending minimally encapsulated payloads out almost raw. Minimally encapsulated at this level might actually mean no encapsulation. Maybe some error-detection/CRC type stuff as these links tend to be super lossy.

gaius• 8 years ago

Why IP over HF Radio should be Avoided https://www.isode.com/whitepapers/ip-over-stanag-5066.html

212• 8 years ago

You probably want to combine in a single raw packet the data, compressed with a prediction codec with the error detection code. Since you also have a lower-latency high-speed link (the transatlantic cable), you can continuously update the probabilistic model used by the compression to squeeze the last bit.

JustSomeNobody2• 8 years ago

I can't imagine they'd need the TCP/IP stack. You might do better with something like:

SOH<header data>EOH STX<data>ETX CRC EOT

Then the receiver would send an ACK or NACK.

Repeat as necessary.

dx0347• 8 years ago

Would you even acknowledge? The data is worthless after one round-trip, so you wouldn't send it twice. The receiver just needs to make sure that they read information correctly and can deal with losses.

gpderetta• 8 years ago

I haven't worked with microwave feeds, but from the war stories I have heard, not even IP is used but data is sent using whatever low level protocol (Ethernet or something similar) is used by the transport.

In fact the payload might be empty or mostly empty and data can be encoded in protocol fields (for example encoding symbol names in address fields as the link is otherwise point to point).

lallysingh2• 8 years ago

"Garden Leave" is a paid non-compete time (often a year) used in the financial industry when someone switched jobs. The prior employer invokes it at their option.

dx0347• 8 years ago

That's not always correct. It can be shorter (3 months are common in London) and is often a fixed part of your contract (notice period). If both agree you can shorten it but it's not just invoked at the discretion of your employer.

golergka• 8 years ago

I wasn't aware that "paid non-compete" is a thing, thanks for clarifying it. Next time when I see a non-compete in a contract (which is usual for developers), instead of negotiating it out, I'll try to negotiate something like this. What do these agreements usually look like?

lallysingh2• 8 years ago

The employer has the option to pay you your salary (not bonus, which is a big part of your income!) to not work at all instead of you going to a competitor.

I think that the comp part is necessary to make it enforceable at all.

madengr3• 8 years ago

They are probably also monitoring HF propagation paths, as there can be ground wave and sky wave, with varying dispersion.

The X300 has some latency too. I’ll be they are doing the majority of processing directly on the FPGA to avoid SDR latency. The X310 has a larger FPGA, and would be tell-tale.

Bring on the comms with neutrinos or entangled particles.

andai• 8 years ago

Up next: Tachyon trading

cameldrv2• 8 years ago

Maybe not, but almost as exotic: https://www.forbes.com/sites/brucedorminey/2012/04/30/neutri...

While all of the other chumps are trying to bounce their signals around the surface in fibers or various layers of the atmosphere, you just blast straight through the earth with (anti)neutrinos!

We live in a weird, weird world, and there's real money behind this idea. It might even affect the strategic nuclear balance, because submarines are supposed to be silent and invisible. Their nuclear reactors do generate extremely large numbers of antineutrinos though...

tomfanning• 8 years ago

Would love to know what licensing regime / category they are using for this. It definitely isn't ham, because that's non-commercial only. The fact that log period antennas are in use is interesting too - low gain but directional and useful across a broad frequency range - maybe some spread spectrum technology?

madengr3• 8 years ago

Probably licensed on several bands, as propagation conditions change quickly. They’d need to hop around, if not simultaneously transmitting on multiple bands.

Jommi3• 8 years ago

https://apps.fcc.gov/els/GetAtt.html?id=205708&x=

jhpankow2• 8 years ago

Wow, that would do it.

Animats• 8 years ago

That one, WI2XER, has been heard, and seen on a spectrum analyzer.[1] They send their call sign in Morse code, send various test tones, and send what look like modem-type signals. 100 KW, so they will have no problem reaching Europe. Not much bandwidth, though, because they don't have enough spectrum space. Maybe a few hundred kilobits/second.

As a way to get info from markets faster, it's limited. You avoid some switching and path delays, but the data rate is slow, so it's only faster for short messages.

[1] http://i56578-swl.blogspot.com/2018/03/wi2xer-skycast-experi...

sailfast• 8 years ago

This is a wonderful article for so many reasons. Outlines resources for doing this yourself, uses industry knowledge to outline hypotheses and then tests them with data, and reveals a really interesting possibility for market communication. Also, a bit of local Chicago geography :) An enjoyable read!

Edit: the “is this a troll” comment makes this even more enjoyable for some reason. Now it’s an even bigger treasure hunt.

Jommi3• 8 years ago

Weirdest thing I've read in a long while.

The Author's comments refer to an internet forum thread[0] in German which discusses a German news article from this April.[1]However, it turns out this article is an April Fools text.

Just digging deeper into the whole blog you can find something that can only be described as a fanfiction representation of "Flashboys", but in the context of this "Shortwave trading".[2]

Is this whole thing a troll? Is this real? What on earth is this?

[0]http://funkbasis.de/viewtopic.php?f=39&t=45900

[1]https://www.heise.de/newsticker/meldung/Boersenhandel-besche...

[2]https://sniperinmahwah.wordpress.com/2017/06/26/network-effe...

Jommi3• 8 years ago

Seems like it is real: https://www.six-group.com/en/home/media/releases/2018/201805...

dx0347• 8 years ago

This is microwave based. Those lines are well known by now and there are known routes within Europe (most notable Frankfurt<->London) and the US (NYC<->Chicago). However, microwave only works over short distances (FRA<->LON has multiple towers to cover the distance). Crossing the atlantic isn't possible. Shortwave could deliver a similar speedup over long distances.

I'm a bit sceptical though. Don't the waves have to bounce several times to get to the target? I'm not sure how much of the ~30% potential speed up vs. fibre you still get.

antb123• 8 years ago

Yeah it started in Chicago. At my old shop we had a semi-exclusive for Toronto - Chicago route which is used for ETF market making.

The main issue is that the network has a lot of packet loss due to weather condititions.

nielsole• 8 years ago

The linked article [1] appears to be built around a word-play, where "Shortwave"-trading and "High frequency"-trading are ambiguously used as "Hochfrequenz"handel

baxtr2• 8 years ago

German speaker here. The sentiment on heise.de comments is that this was an April’s Fools joke

ubiyubix• 8 years ago

The majority of Heise forum participants are wrong then.

avip• 8 years ago

Long-time "sniper in mahwah" follower here. The author is without the slightest doubt an elite tech reporter with in-depth understanding of RF tech. He always provides an overwhelming body of evidence to back his writing.

wglb3• 8 years ago

But why not then listen and figure out what frequencies are being used? That skill would be well within the "elite" category, no?

social_quotient• 8 years ago

If this article was an intro to a fiction novel, I’d be interested in reading the entire thing. Some sort of HFT heist with a good mix of factual engineering and finance.

yangcanvas• 8 years ago

Rougue Code by Mark Russinovich was a novel I enjoyed about a HFT heist.

Ftuuky3• 8 years ago

Can I read it without reading the first two books?

a-dub• 8 years ago

All along everyone thought that numbers stations were how governments send messages to their spooks, in reality they were just poor bastards holed up in tiny shacks manually reading the tape for the S&P e-mini...

lettergram• 8 years ago

Interesting... I actually know someone working on a project similar to this (who works at an HFT firm) - if not actually this.

His team was simulating and testing how various weather conditions could affect trade placement times. For instance, does a high-pressure system, a heat wave, a blizzard, etc. impact the trade placement times, if so how does that impact the algorithms.

He works out of Chicago and I've heard of similar things before, so this isn't unknown to people in the area.

Ftuuky3• 8 years ago

From what I understand (and I'm no expert in HFT or electric engineering), this kind of sub-second trading is done in very specific corridors: Chicago-NY, NY-London, London-Frankfurt, London-Paris, Paris-Frankfurt. I wonder if the same applies in Asia, with Singapore, HK, Tokyo, Sidney...

sniperinmahwah6• 8 years ago

Some fun comments here, thanks! For all those who doubt, here are some public documents (to be detailed on the blog soon) about antennas in UK, filed by a well-known "HFT" firm: https://publicaccess.southbucks.gov.uk/online-applications/a...

discussedbefore• 8 years ago

The secret world of microwave networks https://news.ycombinator.com/item?id=12862789 (2016)

HFT in My Backyard https://news.ycombinator.com/item?id=8354278 (2014)

cjsawyer• 8 years ago

A microwave radio setup to the nearest populated area gave my rural-living parents modern internet a few years ago. It’s not the same scale but the same idea! It’s far and away better than satellite.

aoki2• 8 years ago

i am not sure i would agree that this has been discussed before. this is not simply about use of RF (microwave, mmwave) for HFT. as i understand it, that kind of thing typically involves COTS radios and conventional dish antennas.

this instance appears to involve longer-range links and SDRs (in this case, the ettus USRPs).

bobvan139• 8 years ago

Yes, well said. I think you're making an important distinction. Shortwave is old technology, but combined with SDR, this is a novel application of it AFAIK.

ropeadopepope3• 8 years ago

Does anybody know what happened to Windy Apple Technologies? They promised the lowest latency microwave connection from NY to Chicago, then promptly disappeared...

sniperinmahwah6• 8 years ago

I know

ropeadopepope3• 8 years ago

Are you willing to say here, or is there another way I can contact you?

alexpil• 8 years ago

I'm still around and in the industry.

Never promised the lowest latency, but I was the first one by two years to build NY<>Chicago.

sniperinmahwah6• 8 years ago

True.

ropeadopepope3• 8 years ago

Thank you for replying! I'm glad to hear you're still around. I was always impressed with your work.

nimbius• 8 years ago

Could it possibly be bulk data from sensors?

I worked on a project in several states for IBM that handled traffic shaping and detection on highways and onramps. This system was used to identify gridlock and adjust lights at offramps or give people an ETA until theyre downtown. Most adjacent cities were staunchly NIMBY and didnt care to have the state government trenching cable to and fro for traffic control, so the data ended up hopping from town to town on microwave subcontracted from the cellular company that handled the large electric billboards on the highway.

the benefit of this was also the ability to route signal control data to large intersections without having to shut them down to dig cable trenches from the IBM network.

jlgaddis3• 8 years ago

Not at the frequencies that those LP antennas are tuned for.

vasilipupkin4• 8 years ago

i have one question: why expend some much energy on unearthing obscure stuff that has relevance only to a very small subset of a very small industry and literally nobody else? seems like a waste of effort.

jlgaddis3• 8 years ago

Because it's interesting to someone.

I don't understand why my girlfriend likes to watch Youtube videos of other girls doing their make-up and hair but, then again, she doesn't understand why I need all the computers that I have.

vasilipupkin4• 8 years ago

sure, that's a trivial response that doesn't really address my point. I am suggesting that the authors could expend the same energy and find or create something that's much more interesting to many more people.

joezydeco• 8 years ago

Perhaps because that very small subset of a very small industry can have a major impact on the entire global economy when something goes off plan?

It also hints at how much money is actually in play with these "very small" companies. If they're spending hundreds of millions to get an edge (sometimes in picoseconds) on the competition, what does that say about the magnitude of the money they are working with and willing to risk?

vasilipupkin4• 8 years ago

"Perhaps because that very small subset of a very small industry can have a major impact on the entire global economy when something goes off plan?"

this is a red herring. this specific article in no way discusses or addresses how this would happen, nor does it tie in relevance of shortwave to this specific point

FearNotDaniel• 8 years ago

Short answer: because this is Hacker News, not Middle Manager In A Large Organization Serving A Well Established Market News.

The author went to the effort simply because he himself found it interesting. He published it on a hunch that at least a few other people would find it as interesting as he did, even if they are not working in that industry, just because technology is interesting in its own right. He was right.

vasilipupkin4• 8 years ago

that's obviously a tautological answer. I am saying, he is doing a lot of work, if he redirected this work towards something more relevant to more people, he would produce more results for himself and for everyone else.

wglb3• 8 years ago

So why not listen in the vicinity of the towers and deduce what frequency this is on?

sniperinmahwah6• 8 years ago

Some are doing it, obviously.

bobvan139• 8 years ago

Listening is the easy part. But the sender would be foolish not to strongly encrypt what they send. Are you really "listening" if all you get after decoding is pseudo-random bits?

wglb3• 8 years ago

Listening will at least you know the frequency and type of modulation so you can at least understand who it is. And if the signal is always on or at certain times of the day only can help identify purpose

jlgaddis3• 8 years ago

I'm not even remotely involved with the HFT world, but I can't think of any good reason why they'd want to receive "shortwave" signals from Europe in Chicago.

Short-haul microwave links for low latency and high bandwidth, yes, but at shortwave frequencies you are literally talking about BITS per second. TFA says "Think dialup speeds" but this is lower than even dialup!

They certainly aren't receiving any meaningful amount of data (which I presume you need for HFT) with that shortwave receiver.

IndexicalDemon• 8 years ago

The crucial information is up and down ticks of the ES-mini futures at the Chicago Mercantile Exchange. These lead the S&P500 (SPY) index funding in NYC (by 4 milliseconds), and the ES-mini ticks up and down about once per second. This few bits of information carries huge value since much of the world's equity market responds to it. See https://arxiv.org/abs/1302.5966

mturmon• 8 years ago

Great link, thank you. Abstract in case it piques people's interest:

"High frequency trading has led to widespread efforts to reduce information propagation delays between physically distant exchanges. Using relativistically correct millisecond-resolution tick data, we document a 3-millisecond decrease in one-way communication time between the Chicago and New York areas that has occurred from April 27th, 2010 to August 17th, 2012. We attribute the first segment of this decline to the introduction of a latency-optimized fiber optic connection in late 2010. A second phase of latency decrease can be attributed to line-of-sight microwave networks, operating primarily in the 6-11 GHz region of the spectrum, licensed during 2011 and 2012. Using publicly available information, we estimate these networks’ latencies and bandwidths. We estimate the total infrastructure and 5-year operations costs associated with these latency improvements to exceed $500 million."

mathgenius• 8 years ago

It's not amount of data but latency. A signal to buy is just one bit, but you need to get it before everyone else does.

tudorconstantin• 8 years ago

Could an adversary send buy/sell fake signals and profit?

Given the low bandwidth I assume they can't use very sophisticated authentication mechanisms

reaperducer• 8 years ago

This was covered pretty extensively by Crain's Chicago Business. Some companies are so obsessed with this sort of thing that they'll pay half a million dollars to move a receive site across the street. They measure results in picoseconds. Too much pressure for me.

sandworm101• 8 years ago

One has to wonder if this cutthroat competition will ever turn aggressive. Someone with a SDR and a bit of knowhow could shutdown/jam these systems very easily. The apparatus is in public view and they don't seem to be guarding their trash. A well-timed attack on a couple of these towers would create a few seconds of instability on the markets, enough time for another HFT platform to take advantage.

A blast from a 1-watt transmitter parked across the street would overwhelm any signal coming from across an ocean. Try calling the cops about to complain that your antenna reception is being interfered with. The days of cops hunting rogue radio stations are long gone.

jandrese3• 8 years ago

They wouldn't call the cops, they would call the FCC. If you keep doing it they will catch you.

johnohara3• 8 years ago

It showed a direct link between the West Chicago tower and another tower right by the Chicago Mercantile Exchange!

It's pointing at the top of 10 South Canal, one of downtown Chicago's main tombstones for data communications. The building has 3-6 foot concrete walls, no windows for the first 23 stories and was reportedly designed to withstand an overhead nuclear blast. The standing joke used to be that if Chicago were nuked you'd still get a phone bill because of that place.

There's a lot of underground fiber in that area too. Including 24-count bundles mounted near the ceiling of the old underground tunnel delivery system -- the one that accidentally flooded during the 90's. They were installed by Metropolitan Cable back in the late eighties/early nineties during the "politically connected" fiber gold rush to replace dedicated copper with light.

It would not be speculation to think there's fiber running directly from the MERC to 10 South Canal or that the MERC's location near 10 South is a coincidence.

The line-of-sight out to Aurora is interesting too. There are high spots out there with tower clusters reminiscent of East Texas in the 1930's. I believe there are backup trading centers out there as well built after 9/11.

johnohara3• 8 years ago

Correction: Metropolitan Fiber not Metropolitan Cable. Ultimately purchased by WorldCom.

retzkek• 8 years ago

The map shows he meant the CME Group's data center in Aurora: http://www.cmegroup.com/trading/colocation/co-location-servi...

johnohara3• 8 years ago

The other thing to note is the proximity of the BNSF rail line. There are two 4-inch continuous PVC pipes next to the track, each with a gel-sealing 24-count fiber bundle inside. They run from downtown Chicago to Hinsdale to Aurora and beyond. They were installed about a year before Bershire-Hathaway purchased them.

dghughes• 8 years ago

I recall seeing I think on 60 Minutes HFT that wanted worse latency but I can't recall why. They have massive spools of fiber optics using them as a "resistor" (my term) to slow down the transmission.

It seems to be the opposite of this but for the life of me I can't remember the advantage of doing it.

Found it: https://www.theatlantic.com/business/archive/2016/06/iex-app...

Ftuuky3• 8 years ago

I think you're referring to IEX, an exchange that adds a delay with those spools of fiber optics. According to wiki, "350 microsecond delay adds a round-trip delay of 0.0007 seconds and is designed to negate the certain speed advantages utilized by some high-frequency traders".

https://en.wikipedia.org/wiki/IEX Edit: didn't saw your edit.

neomantra• 8 years ago

Unrelated to the 60 Minutes piece, but fiber is also spooled at network aggregation points within an exchange colo environment (at least NYSE Mahwah and NASDAQ Carteret) so that access is fair (equal latency) for all the customer cabinets.

jandrese3• 8 years ago

The reason is that "the market" is made up multiple exchanges and to make a trade you need to do it on a specific exchange, but stocks are spread across all of the exchanges so if you need to make a big trade you have to send the trade order to several exchanges at once.

The problem is that HFT firms will see the trade hit the closest exchange (in terms of network latency), and then rush out to buy out all of the remaining shares of that stock at that price on the other exchanges (beating you because they HFT better) and relisting them at a higher price.

HFT firms will tell you that this is perfectly acceptable behavior because if you then buy the stock from them that the original person simply wasn't charging enough for it. They will also tell you that it improves liquidity, but that doesn't make sense either. It's not like they're sitting on the shares keeping the markets moving. All it does is add a tax to trades.

IEX delays the transactions just enough that the nanosecond advantages that HFT enjoys can't be used to front run trades.

slr555• 8 years ago

As a Ham I am certainly intrigued by this posting.

There are a number of obstacles I can see to effectively using shortwave for financial services. Propagation conditions vary by time of day, solar activity and other exogenous factors. Another post hypothesized a 100 Kw transmitter power but even without running the numbers it seems like the FCCs Maximum Permissible Exposure Limits (MPE) would dictate a larger exclusion zone than the author encountered, however proper antenna modeling would be needed to be completely accurate. Another confounding factor to potential efficacy of the site is the surrounding terrain. The photos show nearby structures and trees which are less conducive to effective take off angles and propagation associated with the Yagi type antennas depicted. Most hardcore Ham sites with big towers have well groomed earth (or even better, water) around them. Additionally, the data would obviously have be encrypted and transmitted with extremely robust error correction.

It would be fun to grab a couple of boxes from AOR and see what is coming off those antennas.

cameldrv2• 8 years ago

From the McKay brothers' website: "It's better to be fast 99% of the time than slow 99.999% of the time" - Bob Meade and Stéphane Tyč, Co-Founders

If your trading strategy is simply arbitraging Chicago/London, it's probably ok if your link is even 50% reliable, if for that 50% of the time, it's significantly faster than your competition's data. London is going to get the information 10ms later anyhow, so if a packet drops, you can still execute the offsetting transaction, just at the same time as everyone else.

aoki2• 8 years ago

a meta-comment:

it's weird - i submitted this a couple of days ago (https://news.ycombinator.com/item?id=17018883), but it still doesn't show up in the default HN search view. (we even edited the blog headline the same way!) but it does show up in the "by date" view.

mhurd• 8 years ago

some radio stuff here that may or may not be of interest FWIW: https://meanderful.blogspot.com/2017/05/lines-radios-and-cab...

sniperinmahwah6• 8 years ago

Shortwave Trading: The West Chicago Tower Mystery

Related Stories