High Frequency Trading

August 5, 2009 4:06 AM

High Frequency TradingThe Algorithmic Trading animals have been let loose. (via)

A look at the various strategies used in High frequency trading and the hardware used.

"it's ironic that the hardware that HFT platforms are using to battle it out over stocks, bonds, commodities, and other assets is essentially the same as the technology that PC gamers are using to play their own games with much lower stakes. "

posted by manny_calavera (39 comments total)

yeah, interesting:

Press reports of trading days that end with big gains or losses are typically accompanied by shots of a trading floor where young traders are either euphorically throwing papers into the air (up days) or staring dejectedly at a stock ticker with hand pressed to forehead, shoulders slumped in defeat (down days). These guys, you think, are "the market," and if you looked up the New York Stock Exchange (NYSE) on Investopedia you'd find nothing in the "Stocks Basics: How Stocks Trade" entry to disabuse you of this widely held notion ....

... Only about three percent of the trading volume on the NYSE is actually carried out by means of traditional "open outcry" trading, where flesh-and-blood humans gather to buy and sell securities. The other 97 percent of NYSE trades are executed via electronic communication networks (ECNs) ... So the ECNs are the markets in 2009, and those pit traders who pose for the cameras are mainly there for the cameras ...

... Experts guess that between 60 and 75 percent of the NYSE's daily trading volume is just computers trading against one another using a variety of strategies ... What the vast majority of these algos have in common is that they are not long-term, buy-and-hold "investors" in the classic sense. Rather, they focus on executing as many trades per second as possible and on turning a small profit (often pennies or fractions of a penny) on each trade.

posted by memebake at 4:38 AM on August 5 [2 favorites]

Hey, what could go wrong?
posted by Thorzdad at 5:05 AM on August 5 [2 favorites]

The real issue is that when the average retail investor gets an E*Trade account and tries to play the stock market, she typically has no idea that she's going up against the market equivalent of IBM's chess grandmaster-thumping supercomputer, Deep Blue.

According to the article, what the writer calls Deep Blue is scamming the average retail investor for a penny or two. The last time I bought (or sold) on E-Trade I was shocked at how much E-Trade was willing to charge me. And it is certainly better for the average retail investor than it use to be, not too long ago, when spreads were what, and eighth? (12 cents) And wasn't there a gentleman's agreement to keep the spread at a quarter?

Experts guess that between 60 and 75 percent of the NYSE's daily trading volume is just computers trading against one another using a variety of strategies.

I would believe this if volume was more consistant. People go to lunch and people like to trade between 3:00 and 4:00, the volume data shows this.

Yes, there is money to be made here (I've spent the past while working on my "algo"), but since volatility is low between noon and 1:00 (when people go to lunch leaving the computer traders to trade among themselves), is computer trading bad for markets? The first link sites Black Monday which was 22 years ago, but besides that? I might make the argument that people shouldn't be allowed to trade, causes too much volatility, let me think about that.
posted by rakish_yet_centered at 5:12 AM on August 5 [1 favorite]

It's all just a game until BizNet achieves sentience -- and the next thing you know, robots are travelling back in time to short sell the airlines on 9/10.
posted by localroger at 5:21 AM on August 5 [2 favorites]

Ban on Flash Trading is Considered by the SEC
posted by gagglezoomer at 5:24 AM on August 5 [3 favorites]

I would believe this if volume was more consistant. People go to lunch and people like to trade between 3:00 and 4:00, the volume data shows this.

It might be used something like a plane's autopilot and only turned on when someone is there to watch it.
posted by prak at 5:26 AM on August 5

It's great living in the future and by "the future" I mean "a post-apocalyptic wasteland".
posted by DU at 5:26 AM on August 5 [2 favorites]

I wonder why this is only just starting to appear in the press? It's been going on for at least 5 years if not longer.

"it's ironic that the hardware that HFT platforms are using to battle it out over stocks, bonds, commodities, and other assets is essentially the same as the technology that PC gamers are using to play their own games with much lower stakes. "

Why is this ironic?
posted by sunkzero at 5:34 AM on August 5 [2 favorites]

I recently read JK Galbraith's classic The Great Crash for the first time. It's a brief history of the 1929 stock market crash. One of the fascinating aspects of that amazing story is the role played by information technology. Even then, the stock market was dependent on what was then high-tech - telephone networks, telegrams, and the ticker, which conveyed stock price information across the world, even to ocean liners.

When the bubble burst and the crash began, the trading volume began to overwhelm the information technology that was available. Telephone exchanges jammed up, and the ticker developed a huge lag, dragging hours behind actual prices, which were in free-fall. These breakdowns began to exacerbate the situation. Brokers making margin calls or facing stop-loss barriers couldn't reach their clients to find out what to do. Late in the day, when the banks stepped in to try and stabilise the situation, people outside the stock exchange building itself - in the rest of NYC, and across the world - heard that corrective measures were being taken, but didn't have up-to-date ticker information, they were still seeing losses that had actually occurred hours before. All these technologies that had made trading easier, and had meant that you didn't actually have to be in the room to trade, contributed to the disaster. It wasn't the fault of the technology, of course, it had been built to facilitate human complacency and greed. Galbraith is brilliant on the blindness that affects people caught up in a boom - the blind faith that It's Different This Time, we have the tools or the safeguards or the brains that mean crash can't happen, or it can be headed off, or the machines will correct it ... little has changed in 80 years.
posted by WPW at 5:34 AM on August 5 [3 favorites]

When the bubble burst and the crash began, the trading volume began to overwhelm the information technology that was available.

Not just 1929. The causes of Black Monday are still debated, but automatic program trading is one of the candidates, and it definitely was the accelerant that took a bad day down and turned it into the largest single day down, percentage wise, in the history of most of the equity markets out there. Trading restriction had to be put in place on many of the markets, because the technology of the time simply couldn't handle the orders -- I don't think the NYSE cleared until after close on Tuesday, and other markets took longer.

So, a very similar setup to the 1929 crash -- you can trade faster than you can get the information about your trades.
posted by eriko at 6:20 AM on August 5

Just so we're clear: flash trading and high frequency trading aren't the same thing.

I'd be okay with a ban on immediate-or-cancel orders though.
posted by anotherpanacea at 6:20 AM on August 5

I liked this better when it was a book called Accelerando. On the plus side, maybe I'll get to travel to Saturn.
posted by Mister_A at 6:33 AM on August 5 [1 favorite]

So basically this is Corewars with the possibility of bankrupting large swaths of the population.
posted by PenDevil at 6:43 AM on August 5 [4 favorites]

The third largest exchange by volume operates here in Kansas City, BATS Exchange, and caters to low latency, high frequency trading.
posted by geoff. at 6:44 AM on August 5

I would believe this if volume was more consistant. People go to lunch and people like to trade between 3:00 and 4:00, the volume data shows this.

That's when the fish show up for the circling sharks.
posted by Malor at 6:47 AM on August 5 [1 favorite]

So basically this is Corewars with the possibility of bankrupting large swaths of the population.

More fun than the standard rock-papers-scissor pattern it normally falls into.
posted by Leon at 6:57 AM on August 5

Goldman Sachs clarifies its high frequency trading position, not.
posted by Xurando at 7:48 AM on August 5

automatic program trading is one of the candidates, and it definitely was the accelerant that took a bad day down

Was that the same thing though? The automatic program trading was not high frequency trading. It was portfolio insurance, "sell if we go below this price," that was practiced by the buy-side investors. No one was on the left to bid on the orders.

HFT is not like portfolio insurance. They are not investors; they want to have a net neutral book at the end of a day. They are different from market makers in that they only show up when other people show a desire to trade.

I do think that it's possible to see volatility if, say, HFT programs start thinking other HFT orders are investors. But frankly, it's just nitpicking timeframes. HFTs have operated together, day traders have operated together, quick-buck retail investors have operated together, along with the traditional long-term investors.

Why limit ourselves to HFT? Why not ban day-trading for the risk they pose to the system?
posted by FuManchu at 7:52 AM on August 5

I used to have conversations about statistical arbitrage with a hedge fund quant friend of mine. One of my takes on this is that the existence of hedge funds and their obscene profits (which are either obscenely large for investors, or more likely, for the people running the funds) is one of the most convincing counterproofs to the efficient market hypothesis.

One thing that I found funny was that he used to claim that hedge funds (and HFT in particular) provided a useful function to society by making markets more efficient. If there's some "correct" price, and HFT gets the market price to that "correct" price while making a little bit of money through arbitrage, then that's a service to society, right?

The one thing that I have become convinced of is that in order to come up with a definition of "correct" in the preceding paragraph, it seems (to me) pretty far removed from what I perceive to be the "value" of the stocks people are investing in. (Unless, like my friend, you define "value" as "what people will pay for something." Which seems a bit like circular reasoning to me.)

Then again, I'm a pretty long-term investor, so of course my definition of value should be different: I can't make money on statistical arbitrage as an individual in many cases because in a lot of these cases, the transaction costs would eat up any profit I would make. Hedge funds have to do lots of transactions, so that the transaction costs are a miniscule part of the cost of any given transaction.

That quant friend of mine? He's no longer at the fund, since he decided he was tired of thinking about money all the time. I'm sure he could have continued on and made money (he was doing fine with his strategies in late '08). But at one point last year, he turned to me and asked, "How much money would I need to retire?"
posted by grae at 7:59 AM on August 5 [2 favorites]

I do think that it's possible to see volatility if, say, HFT programs start thinking other HFT orders are investors. But frankly, it's just nitpicking timeframes. HFTs have operated together, day traders have operated together, quick-buck retail investors have operated together, along with the traditional long-term investors.

The big problem with HFT programs reacting to each other is that they can cycle out of control pretty quickly. And part of the problem there is the automatic nature of it: when you are trying to beat the other guy by fractions of a second, it has to be automatic. Unless you're really careful about always putting in controls to prevent things from spiraling out of control, if you get on the wrong side of some other automated program, you both are going to keep betting until one of you runs out of money. And since HFTs are usually run by hedge funds, which typically use leverage to effectively gamble more money than they have, if this strategy goes bad, you can lose a lot of money very quickly.

And you might actually want to spiral it up: in most of these cases, one side of the trade is going to end up making money. If you put a limit on it, then you're putting a limit on your upside. As far as the incentives are structured for hedge fund operators, they want to avoid limits on their upside, because the typical fee structure to a hedge fund guarantees them a pretty substantial payment even if they lose money. Why bother limiting the upside when your downside is already limited?

(It's the fact that HFTs exist in this environment with these incentives that makes this such a bad situation. The incentives aren't set up for investors to make money, they're set up for hedge fund operators to make money, and while that can be correlated with investors making money, it doesn't have to be.)
posted by grae at 8:13 AM on August 5 [1 favorite]

I think it's fascinating the folks who have 50ms execution are making money off the folks who only have 400ms execution.
posted by Nelson at 8:34 AM on August 5 [1 favorite]

I'm interested in hearing how HFT could be good for the economy.
posted by garlic at 8:35 AM on August 5

I'm interested in hearing how HFT could be good for the economy.

All of the money being extracted from the system will eventually trickle down (back?) to the rest of us.
posted by ryoshu at 9:10 AM on August 5

The article linked in this FPP from a few days ago, about automatically-generated data considered as a "third replicator", was pretty overblown (as people pointed out in comments). And the examples presented, like Google Pagerank data, were pretty uninspiring. But (as noted by Glee) activities like HFT, which are necessarily automated and non-humanly-monitorable, and which are highly influential at a large scale, illustrate the point that Blackmore was reaching for: in a sufficiently high-tech civilization, not every idea that sweeps and re-shapes the world will begin life in a human brain.

Now, it could certainly be objected (as people did in that thread) that what we're looking at here is nothing but 'memes' on a new substrate, created and exchanged by computers rather than brains. But with the orders-of-magnitude increases in the speed and volume with which these memes (or 'temes', or whatever) are created and spread, its seems like there's a pretty good case for thinking about them as a separate phenomenon from human-culture memes. When a billion speculative buy orders can be flung out in the time it takes for a single amusing picture to hit the Digg front page -- well, it makes 'Internet time' look pretty geological.
posted by logopetria at 9:52 AM on August 5 [1 favorite]

I can confirm they've been doing this for more than a few years. I've had a few interviews at, well, Goldman Sachs only for some stupid reason, and this was important: C, valgrind, and gdb. I've heard they even go to assembler language, something gaming programming has given up on since Quake.
posted by sleslie at 10:05 AM on August 5

I think it's fascinating the folks who have 50ms execution are making money off the folks who only have 400ms execution.

i wonder if the NYSE has installed a liquid nitrogen source yet for their clients that like to overclock...
posted by sexyrobot at 10:51 AM on August 5

One of my takes on this is that the existence of hedge funds and their obscene profits (which are either obscenely large for investors, or more likely, for the people running the funds) is one of the most convincing counterproofs to the efficient market hypothesis.

Would you like to join my hedge fund? Here's how it works: you give me $100,000. I go to the nearest casino and bet it all on black. If I win, I give you back $125,000 (a 25% profit!) and keep $75,000 for myself. If I lose, well you knew the risks when you invested. Statistically, I'll make approximately $37,500 on every $100,000 invested, a 37% yield for me, which is pretty obscene.
posted by Pyry at 10:55 AM on August 5 [2 favorites]

This helps explain the need for big, fast datacenters in NJ just across the river from the NYSE, as discussed previously.
posted by exogenous at 11:05 AM on August 5 [1 favorite]

A fairly typical payment structure for hedge funds is "2 and 20". Here's how it works:

The fund picks a benchmark that they're trying to beat. I believe I've seen some that are something like S&P500, or even S&P500+x%.

No matter what, you pay 2% of what you've invested (that's the "2" of "2 and 20"). Typical minimums for investment are in the millions of dollars, so if I've invested $10million, I automatically end up paying $200k even if I don't make any money.

Every time it's time to pay fees, we take a look at how well the fund has performed. If it hasn't performed as well as the benchmark, the hedge fund doesn't make any additional money. But they don't get penalized for their performance. If, on the other hand, they do well, they get to keep 20% of the money over the benchmark (this is where the "20" of "2 and 20" comes from.) Sometimes this calculation is complicated by having a "high water mark", where the hedge fund doesn't get performance fees unless the hedge fund goes above its previous high.

So, Pyry, in the roulette example, if you used a more typical payment structure, you'd get $2000 no matter what, and you'd get to keep $19600 more if you won (you'd only be betting $98000). Your expected value is still $10800 (I'm using the same simplifying approximation you used that you have a 50% chance of winning, which isn't actually true because of the 0 and possibly 00.) And in this case, you get to keep $2000 even if you lose all my money.

In Pyry's suggested scenario, he doesn't get any money if he loses anything, so his incentives are aligned with mine as an investor. In the 2 and 20 scenario, the fund gets to keep something even if it loses all of the investor's money. It's that difference that most people are complaining about when they complain about hedge fund's incentives not being aligned.
posted by grae at 11:17 AM on August 5 [3 favorites]

Simply require an order to be truly exposed to the market by making a minimal time commitment to the length of the order has to be in the market before it is canceled. I would make it 1 second.

If these flash orders are driving prices away from one "investor", it is giving a better price to his other side. If I as a trader think that the price is "not right", I simply step in and offer my order to the market.
posted by JohnnyGunn at 11:28 AM on August 5

I liked this better when it was a book called Accelerando. On the plus side, maybe I'll get to travel to Saturn.

I was thinking the same thing! Economics 2.0!
posted by brundlefly at 11:35 AM on August 5

The problem with the casino examples are that the games of chance are driven by deterministic factors and we can determine their probabilities trough some deduction and running tests on them over and over again. Comparisons to games of chance are bad analogs for the market.

From a mathematical perspective I find it fascinating that markets seem to show self-similarity. As you get to smaller time sequences you begin to approximate brownian motion. The further out you go the less it looks like brownian motion. Except time scales keep shrinking, I would suppose that the algorithms that make bets on statistical inference using different variations of Gaussian randomness need to seek shorter and shorter time scales.

Given that there is a physical limit to the speed at which information can travel, at least without some sort of quantum magic, I begin to wonder how much quicker these algorithms can process before they begin to hit a sort of constant.
posted by geoff. at 12:11 PM on August 5

Given that there is a physical limit to the speed at which information can travel, at least without some sort of quantum magic.

Even with 'quantum magic' you aren't going to be able to transmit information faster than the speed of light. The point of the casino examples is that a reward structure that lets hedge funds keep winnings while shrugging off losses simply encourages them to take ridiculous risks. It's easy to make tons of money when you're betting with someone else's cash.
posted by Pyry at 12:31 PM on August 5

Given that there is a physical limit to the speed at which information can travel, at least without some sort of quantum magic, I begin to wonder how much quicker these algorithms can process before they begin to hit a sort of constant.

That's going to be a factor more of proximity to data sources than anything else. Which is the point that exogenous made up-thread.
posted by grae at 1:12 PM on August 5

Pyry, kind of getting off topic, but 'quantum magic' (aka Quantum entanglement) does claim that transmitting data faster than the speed of light is possible, theoretically of course.
posted by ShadowCrash at 2:49 PM on August 5

Quantum entanglement doesn't claim that data can be transmitted faster than the speed of light. It does claim that the same event can be seen arbitrarily far apart... Just because you know that the spin of a particle is opposite of its paired particle halfway across the galaxy doesn't mean that you can transmit data as a result.

I think Pyry's physics are accurate here. (And besides the point, at any rate.)
posted by grae at 3:10 PM on August 5

The potential problems relating to misaligned incentives are, as demonstrated above, not unique to HFT. The more worrying problems presumably arise from the inability to monitor and control the algorithms' behaviour in real time. If automated systems just spotted and proposed trades for human operators to place, or made their own unsupervised trades at roughly the same speed as human operators, then any error or unexpected behaviour by the algorithm could (in principle) be intercepted or corrected before it's able to spiral out of control. But if the bidding process is accelerated by a few orders of magnitude, while the human supervision necessarily performs at the same old plodding pace, the first warning you'd get of a runaway spiral might be, say, an entire sector of investments crashing in fifteen seconds.

Of course, the people who write these algorithms do a lot of work to put in checks and limits to make sure that obvious disaster scenarios like this can't happen -- but that probably leaves plenty of scope for non-obvious scenarios. When you put a bunch of simple agents together and have them interact, surprising behaviour often emerges, even when the agents' code is known. Now make the algorithms in question (a) secret, and (b) basically designed to screw each other over, and how long do you give before someone gets the kind of surprise that'll become a few PhD theses once the economic rubble has settled?
posted by logopetria at 4:07 PM on August 5 [1 favorite]

I once made a web site for a guy with his own 'buy low, wait a day or two, sell high' heuristic.

I figured he was mistaken or maybe lying about its efficacy, but I watch him consistently pick only winners (even if sometimes small winners) every day for month after month.

He finally took the site down. The program wasn't the hit he'd hoped it would be. But as far as I know he still trades that way himself.
posted by jfrancis at 1:00 PM on August 6

Nasdaq to stop offering flash trading Sept. 1

Unilateral move ahead of the SEC; they're asking other electronic markets to join them.
posted by Nelson at 1:39 PM on August 6