Predatory algorithms & Ultrafast Extreme Events
September 11, 2013 12:39 PM Subscribe
One of my AI friends once told me over a beer that he strongly suspected that we are undergoing an AI apocalypse: this was before the Knight flash crash, which convinced me, too. The systems we ourselves have set up will kill us...
posted by curuinor at 12:52 PM on September 11, 2013 [1 favorite]
posted by curuinor at 12:52 PM on September 11, 2013 [1 favorite]
Fascinating. Thanks for posting this.
posted by joannemerriam at 12:54 PM on September 11, 2013
posted by joannemerriam at 12:54 PM on September 11, 2013
"There is real money being gained and lost here — even a few thousand dollars every millisecond, which is a tiny amount on the market, is a million dollars per second."
And there is nothing - nothing - of social value that is gained here. The usual justification is that HFT is providing liquidity, but these bots are the first to head for the exits in bad times, so that is no excuse at all.
I didn't quite follow why a tiny transaction tax ($0.01 per trade, say) wouldn't fix this problem. It makes no difference to the buy+hold investor or even the day trader, but a bot trading for fractional penny gains doesn't make sense any more. The only line in the article about that seems to be:
The Europeans are trying to apply a so-called Tobin tax on transactions. But lawyers today have said this may be illegal, so it is a complex situation concerning what to do [...]
That seems to be an easy fix, though - change the laws. Yes, yes, regulatory capture etc., etc. But conceptually, at least, why not a transaction tax?
posted by RedOrGreen at 12:56 PM on September 11, 2013 [23 favorites]
And there is nothing - nothing - of social value that is gained here. The usual justification is that HFT is providing liquidity, but these bots are the first to head for the exits in bad times, so that is no excuse at all.
I didn't quite follow why a tiny transaction tax ($0.01 per trade, say) wouldn't fix this problem. It makes no difference to the buy+hold investor or even the day trader, but a bot trading for fractional penny gains doesn't make sense any more. The only line in the article about that seems to be:
The Europeans are trying to apply a so-called Tobin tax on transactions. But lawyers today have said this may be illegal, so it is a complex situation concerning what to do [...]
That seems to be an easy fix, though - change the laws. Yes, yes, regulatory capture etc., etc. But conceptually, at least, why not a transaction tax?
posted by RedOrGreen at 12:56 PM on September 11, 2013 [23 favorites]
Johnson dismisses the idea of a Tobin tax in vague terms, understandable given that his whole field of research depends on there being no Tobin tax.
posted by justsomebodythatyouusedtoknow at 12:57 PM on September 11, 2013 [9 favorites]
posted by justsomebodythatyouusedtoknow at 12:57 PM on September 11, 2013 [9 favorites]
|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\
I am now an awesome machine intelligence.
posted by Artw at 1:05 PM on September 11, 2013 [3 favorites]
I am now an awesome machine intelligence.
posted by Artw at 1:05 PM on September 11, 2013 [3 favorites]
Money & Speed: Inside The Black Box looks at & reconstructs the "flash crash" of may 6th, 2010, "the fastest and deepest U.S. stock market plunge ever" & the role of high-speed computer trading had in it.
posted by Pirate-Bartender-Zombie-Monkey at 1:07 PM on September 11, 2013 [4 favorites]
posted by Pirate-Bartender-Zombie-Monkey at 1:07 PM on September 11, 2013 [4 favorites]
This was all really interesting as it is, but I felt like I was missing out on understanding what must surely be the best part: how a predatory algorithm takes its prey. What lets a predatory algorithm, uh, predate? I think I'd have a much better intuitive understanding of this process if there were an example of exactly how one of these algorithms actually works.
Edit: OK I know what I really want to see, more specifically: a nanosecond-by-nanosecond history of the price movement around an actual stock for one especially active half-second. This probably can't happen because only one of the companies involved would have access to such precise data and they wouldn't want to release it, but still.
also, is "goodness-of-fit" seriously the preferred term in statistics, over, idk, "fit quality"? .__.
posted by a birds at 1:07 PM on September 11, 2013
Edit: OK I know what I really want to see, more specifically: a nanosecond-by-nanosecond history of the price movement around an actual stock for one especially active half-second. This probably can't happen because only one of the companies involved would have access to such precise data and they wouldn't want to release it, but still.
also, is "goodness-of-fit" seriously the preferred term in statistics, over, idk, "fit quality"? .__.
posted by a birds at 1:07 PM on September 11, 2013
Artw: "|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\
I am now an awesome machine intelligence."
I find your ideas interesting. Might I subscribe to your newsletter?
posted by Samizdata at 1:08 PM on September 11, 2013
I am now an awesome machine intelligence."
I find your ideas interesting. Might I subscribe to your newsletter?
posted by Samizdata at 1:08 PM on September 11, 2013
I'm in favor of a Tobin tax as long as it's called the Tobin Special Purpose Information Retrieval Iteration Tax.
They could publish a guide to it.
posted by Steely-eyed Missile Man at 1:08 PM on September 11, 2013 [11 favorites]
They could publish a guide to it.
posted by Steely-eyed Missile Man at 1:08 PM on September 11, 2013 [11 favorites]
I'd swear there was a post on the Blue about this in the past few months -- something about how humans are in the near future not even going to be able to comprehend the math and the "thinking" that machines are doing, which posed some interesting questions about our ability to trust that the machines will in fact doing what we tell them to do. Anyone remember this?
posted by gauche at 1:09 PM on September 11, 2013
posted by gauche at 1:09 PM on September 11, 2013
We find 18,520 crashes and spikes with durations less than 1500 ms in our dataset .... Since both crashes and spikes are typically more than 30 standard deviations larger than the average price movement either side of an event (see Figs. 1A and 1B), they are unlikely to have arisen by chance since, in that case, their expected number would be essentially zero whereas we observe 18,520.
Their data set is Jan 2006 - Feb 2011, in which time frame there are, assuming an exchange open 40 hours/wk year-round, roughly 25 million time intervals of 1500 ms. Are we sure 18,520 is NOT essentially zero? IANAStatistician.
posted by solotoro at 1:14 PM on September 11, 2013
Their data set is Jan 2006 - Feb 2011, in which time frame there are, assuming an exchange open 40 hours/wk year-round, roughly 25 million time intervals of 1500 ms. Are we sure 18,520 is NOT essentially zero? IANAStatistician.
posted by solotoro at 1:14 PM on September 11, 2013
* buys three mules *
* hits the street *
posted by sandettie light vessel automatic at 1:15 PM on September 11, 2013 [13 favorites]
* hits the street *
posted by sandettie light vessel automatic at 1:15 PM on September 11, 2013 [13 favorites]
RAWR! I R A PREDATOR BOT!
Don't bother banning me. I already worked out a way around that.
posted by Samizdata at 1:17 PM on September 11, 2013 [1 favorite]
Don't bother banning me. I already worked out a way around that.
posted by Samizdata at 1:17 PM on September 11, 2013 [1 favorite]
Is there some reason they couldn't just increase the minimum level of granularity to, say, a minute?
posted by Sys Rq at 1:20 PM on September 11, 2013
posted by Sys Rq at 1:20 PM on September 11, 2013
Sys Rq, any level of granularity would open up huge cross-exchange arbitrage opportunities. (The price has already fallen on NASDAQ but the time tick hasn't happened yet on NYSE - better queue up those buy/sell orders.)
(And no one bring up a special master clock that controls every tick at every exchange. Einstein laughs at you when you do that.)
posted by RedOrGreen at 1:24 PM on September 11, 2013 [8 favorites]
(And no one bring up a special master clock that controls every tick at every exchange. Einstein laughs at you when you do that.)
posted by RedOrGreen at 1:24 PM on September 11, 2013 [8 favorites]
I didn't quite follow why a tiny transaction tax ($0.01 per trade, say) wouldn't fix this problem. It makes no difference to the buy+hold investor or even the day trader, but a bot trading for fractional penny gains doesn't make sense any more.
I wonder if the stock markets could also introduce some delay into their systems, maybe 30 seconds or so, to starve the HFT systems of the real-time data they need to operate.
posted by cosmic.osmo at 1:29 PM on September 11, 2013 [1 favorite]
I wonder if the stock markets could also introduce some delay into their systems, maybe 30 seconds or so, to starve the HFT systems of the real-time data they need to operate.
posted by cosmic.osmo at 1:29 PM on September 11, 2013 [1 favorite]
I'm in favor of a Tobin tax as long as it's called the Tobin Special Purpose Information Retrieval Iteration Tax.
I prefer Tobin Automated Rapid-transaction Disincentivization Information System. It's the only way to deal with evil robots manipulating time to dominate humanity.
posted by justsomebodythatyouusedtoknow at 1:29 PM on September 11, 2013 [5 favorites]
I prefer Tobin Automated Rapid-transaction Disincentivization Information System. It's the only way to deal with evil robots manipulating time to dominate humanity.
posted by justsomebodythatyouusedtoknow at 1:29 PM on September 11, 2013 [5 favorites]
That seems to be an easy fix, though - change the laws.
The issue is that the EU FTT under discussion (and the France and Italy FTTs currently in effect) really only work when they're imposed everywhere, because the markets are liquid enough that traders will transact wherever the tax does NOT apply. Consequently, the 11-member EU FTT block has proposed a rule whereby the tax would be required to be collected on transactions taking place beyond their borders based on the presence, among other things, of affiliates within the FTT-zone. So, if I buy a French ADR on a US exchange with JPM as my counterparty, JPM would have to collect the tax because it has a branch in France, despite the fact that no French "ordinary" share changes hands--that share is still sitting in a trust somewhere.
So the "illegality" is not just simply amending a rule that says "there is no transaction tax" to read "Yes! Transaction tax!"--it's a fundamental question about the extraterritorial application of certain nations' laws. Of course, JPM does have that branch in France, so some will argue it's not really extraterritorial. I'm not an international lawyer. I'm just a simple cave man. But what I do know is that the EU FTT is struggling, the UK is mounting a legal challenge in the EU high courts, and there has been significant market shifts as a result of the France and Italy FTTs. Much is left to be written about these rules, but it is far, far from an easy fix.
(I'm down on the bots, personally; I think they're too disruptive to the markets.)
posted by Admiral Haddock at 1:34 PM on September 11, 2013 [2 favorites]
The issue is that the EU FTT under discussion (and the France and Italy FTTs currently in effect) really only work when they're imposed everywhere, because the markets are liquid enough that traders will transact wherever the tax does NOT apply. Consequently, the 11-member EU FTT block has proposed a rule whereby the tax would be required to be collected on transactions taking place beyond their borders based on the presence, among other things, of affiliates within the FTT-zone. So, if I buy a French ADR on a US exchange with JPM as my counterparty, JPM would have to collect the tax because it has a branch in France, despite the fact that no French "ordinary" share changes hands--that share is still sitting in a trust somewhere.
So the "illegality" is not just simply amending a rule that says "there is no transaction tax" to read "Yes! Transaction tax!"--it's a fundamental question about the extraterritorial application of certain nations' laws. Of course, JPM does have that branch in France, so some will argue it's not really extraterritorial. I'm not an international lawyer. I'm just a simple cave man. But what I do know is that the EU FTT is struggling, the UK is mounting a legal challenge in the EU high courts, and there has been significant market shifts as a result of the France and Italy FTTs. Much is left to be written about these rules, but it is far, far from an easy fix.
(I'm down on the bots, personally; I think they're too disruptive to the markets.)
posted by Admiral Haddock at 1:34 PM on September 11, 2013 [2 favorites]
Sys Rq: "Is there some reason they couldn't just increase the minimum level of granularity to, say, a minute?
"
Wont it be a problem if an exchange could move only in minutes and bots could execute trades in seconds? Exchange is ultimately a recording system of the trades that happened. it would be analogous ... to buffer overflow problem, won't it?
Even if there was a rule that a particular bot or broker can only trade say every 2 minutes .. it would just encourage everyone to create more brokers and time trades accordingly.
My point is that we will need to setup a lot of rules to ensure that trades are basically in human time ... and even then there is no guarantee that some one fast enough will not try to beat the system.
On preview: what RedorDreen said.
posted by TheLittlePrince at 1:35 PM on September 11, 2013
"
Wont it be a problem if an exchange could move only in minutes and bots could execute trades in seconds? Exchange is ultimately a recording system of the trades that happened. it would be analogous ... to buffer overflow problem, won't it?
Even if there was a rule that a particular bot or broker can only trade say every 2 minutes .. it would just encourage everyone to create more brokers and time trades accordingly.
My point is that we will need to setup a lot of rules to ensure that trades are basically in human time ... and even then there is no guarantee that some one fast enough will not try to beat the system.
On preview: what RedorDreen said.
posted by TheLittlePrince at 1:35 PM on September 11, 2013
There's been a couple proposals in Congress to impose a by-transaction tax as well. They weren't pushed all that hard, but they looked like they would pass Constitutional muster. Just need someone to push hard for it.
posted by Punkey at 1:42 PM on September 11, 2013
posted by Punkey at 1:42 PM on September 11, 2013
I love these scare stories.
"By contrast, it takes a human at least one full second to both recognize and react to potential danger."
"there were a jaw-dropping 18,520 crashes and spikes with durations less than [1.5 seconds] between January 2006 and February 2011."
Sometimes I wonder if io9 writers ever took a math or science class in high school.
posted by Ardiril at 1:44 PM on September 11, 2013 [3 favorites]
"By contrast, it takes a human at least one full second to both recognize and react to potential danger."
"there were a jaw-dropping 18,520 crashes and spikes with durations less than [1.5 seconds] between January 2006 and February 2011."
Sometimes I wonder if io9 writers ever took a math or science class in high school.
posted by Ardiril at 1:44 PM on September 11, 2013 [3 favorites]
I bet there is also an ideological bias at play here. That is, many financial brokers and legislators may believe that machines can more purely represent the "invisible hand" that makes markets infallible, or whatever. They just use math, and don't have any human fallibility (that is, sentiment) to get in the market's way. So any attempt to slow down or tax HFT is going to run into a lot of resistance among those with high levels of influence.
posted by Kevin Street at 1:50 PM on September 11, 2013 [1 favorite]
posted by Kevin Street at 1:50 PM on September 11, 2013 [1 favorite]
1 - at this granularity, it simply doesn't matter that the financial instruments involved are actually real. This is a perfectly fine, even interesting, arena for code competition, but when the units involved are livelihoods it's somehow even more immoral that it's unnecessary for them to be so. If you're sure your algorithms have what it takes, put up your $1bn surety and play in the NASDAQ enclosed garden, where a variety of simulated equities are available for them to trade at light speed. And then limit real equity movement to say, 5s time slices. Time-dependent arbitrage is only a problem if global stock exchanges can't implement a synchronous time protocol, like, say, every UNIX machine on the planet.
2 - It's amazing how far we'll go to avoid taxing income at the person level, isn't it?
posted by cromagnon at 1:51 PM on September 11, 2013 [4 favorites]
2 - It's amazing how far we'll go to avoid taxing income at the person level, isn't it?
posted by cromagnon at 1:51 PM on September 11, 2013 [4 favorites]
Their data set is Jan 2006 - Feb 2011, in which time frame there are, assuming an exchange open 40 hours/wk year-round, roughly 25 million time intervals of 1500 ms. Are we sure 18,520 is NOT essentially zero? IANAStatistician.
Depending on how the short-term asset movements are being modeled, i.e., which distribution they're assumed to be generated by, the chance of observing any one >30 sigma movement is ridiculously exceptionally tiny. Out of 25 million time intervals, the expected amount of such movements would indeed be very small (much less than 1) and hence essentially zero. In this case, no, 18,520 isn't essentially zero -- there's a huge gulf of difference between zero and 18,520 here, and under some reasonable distributional assumption of returns, such a result would be very statistically significant.
posted by un petit cadeau at 2:12 PM on September 11, 2013 [2 favorites]
Depending on how the short-term asset movements are being modeled, i.e., which distribution they're assumed to be generated by, the chance of observing any one >30 sigma movement is ridiculously exceptionally tiny. Out of 25 million time intervals, the expected amount of such movements would indeed be very small (much less than 1) and hence essentially zero. In this case, no, 18,520 isn't essentially zero -- there's a huge gulf of difference between zero and 18,520 here, and under some reasonable distributional assumption of returns, such a result would be very statistically significant.
posted by un petit cadeau at 2:12 PM on September 11, 2013 [2 favorites]
Tobin Taxes have been tried in Sweden.
The BBC article talks about them.
The BBC article talks about them.
A version of it was tried in Sweden in 1984, where a 0.5% tax was introduced on the buying and selling of shares.This Economist blog piece also describes the effects.
The results were disappointing. It had been hoped that about 1.5bn Swedish kronor (£142m) per year would be raised. On average, 50 million was raised. Capital gains tax revenues also fell.
But the impact on market trading was also more dramatic. During the first week of the tax, the volume of bond trading fell by 85%.
In 1991, the tax was scrapped.
By 1990 over 50% of Swedish equity trading had moved to London.posted by sien at 2:29 PM on September 11, 2013 [4 favorites]
solotoro: "We find 18,520 crashes and spikes with durations less than 1500 ms in our dataset .... Since both crashes and spikes are typically more than 30 standard deviations larger than the average price movement either side of an event (see Figs. 1A and 1B), they are unlikely to have arisen by chance since, in that case, their expected number would be essentially zero whereas we observe 18,520.
Their data set is Jan 2006 - Feb 2011, in which time frame there are, assuming an exchange open 40 hours/wk year-round, roughly 25 million time intervals of 1500 ms. Are we sure 18,520 is NOT essentially zero? IANAStatistician."
The statement "more than 30 standard deviations larger than the average price movement" indicates (in rough terms) how unlikely it is that this is an insignificant number of occurrences.
I say "roughly", because we don't know that these occurrences have a normal distribution. If we did, we could easily determine how likely it is that 18 thousand events happened in a random cluster of 25 million samples.
1/10^14, BTW, are the odds that any one of them would happen in any single 1-ms period, if the events had a normal distribution (e^-30). I'd need to calculate the Teacher Algorithm to see what the total odds are, but given that a single event happening in any particular slot is one in a hundred billion unlikely, 25 million reps aren't likely to have produced even one of these events, much less 18,250.
I think we're pretty safe in saying these are highly unusual events, caused by something completely new in the market.
posted by IAmBroom at 2:31 PM on September 11, 2013 [3 favorites]
Their data set is Jan 2006 - Feb 2011, in which time frame there are, assuming an exchange open 40 hours/wk year-round, roughly 25 million time intervals of 1500 ms. Are we sure 18,520 is NOT essentially zero? IANAStatistician."
The statement "more than 30 standard deviations larger than the average price movement" indicates (in rough terms) how unlikely it is that this is an insignificant number of occurrences.
I say "roughly", because we don't know that these occurrences have a normal distribution. If we did, we could easily determine how likely it is that 18 thousand events happened in a random cluster of 25 million samples.
1/10^14, BTW, are the odds that any one of them would happen in any single 1-ms period, if the events had a normal distribution (e^-30). I'd need to calculate the Teacher Algorithm to see what the total odds are, but given that a single event happening in any particular slot is one in a hundred billion unlikely, 25 million reps aren't likely to have produced even one of these events, much less 18,250.
I think we're pretty safe in saying these are highly unusual events, caused by something completely new in the market.
posted by IAmBroom at 2:31 PM on September 11, 2013 [3 favorites]
But, un petit cadeau, this is exactly the stance that Black Swan argues against: econ isn't about bell curves, it's about calm times juxtaposed against hard-to-predict and hard-to-model large fluctuations. These fluctuations occur in human time scales with just as little foresight or control as they do with the machines at the helm, and leave a wake of economists revising their explanations of the world and busted pension funds. These fluctuations at the human scale have names like 'housing bubble' or 'dot-com crash' or 'asian stock market plunge' and so on. We end up with ex-post-facto explanations which might be more or less true in the infinite wisdom of the rear-view mirror, but were still difficult to see coming. (Any crash which can be reliably predicted won't happen, right?) We end up with a lot of historically fitted models and distributions that likely tell us very little about the future, cousins of the octopus that predicted the world cup a few years ago: sometimes right, and possibly very wrong when it matters most.
I think the wake-up call to be had here is that humans didn't have much understanding or control of the financial system even before algorithmic trading....
posted by kaibutsu at 2:32 PM on September 11, 2013 [1 favorite]
I think the wake-up call to be had here is that humans didn't have much understanding or control of the financial system even before algorithmic trading....
posted by kaibutsu at 2:32 PM on September 11, 2013 [1 favorite]
Yeah, I'd love to see a similar analysis held against large-sigma swings over a longer human scale. The same analysis would probably tell us that we should "never" see an event like a Black Monday.
posted by kaibutsu at 2:41 PM on September 11, 2013
posted by kaibutsu at 2:41 PM on September 11, 2013
I have read over the relevant comments several times now and I cannot for the life of me figure out how that comment is a response to un petit cadeau's.
posted by a birds at 2:41 PM on September 11, 2013
posted by a birds at 2:41 PM on September 11, 2013
Ardiril: "I love these scare stories.
"By contrast, it takes a human at least one full second to both recognize and react to potential danger."
"there were a jaw-dropping 18,520 crashes and spikes with durations less than [1.5 seconds] between January 2006 and February 2011."
Sometimes I wonder if io9 writers ever took a math or science class in high school."
I'm failing to see what you find laughable about those sentences. 1.5 seconds is approximately 1 full second.
posted by IAmBroom at 2:46 PM on September 11, 2013 [1 favorite]
"By contrast, it takes a human at least one full second to both recognize and react to potential danger."
"there were a jaw-dropping 18,520 crashes and spikes with durations less than [1.5 seconds] between January 2006 and February 2011."
Sometimes I wonder if io9 writers ever took a math or science class in high school."
I'm failing to see what you find laughable about those sentences. 1.5 seconds is approximately 1 full second.
posted by IAmBroom at 2:46 PM on September 11, 2013 [1 favorite]
kaibutsu: "Yeah, I'd love to see a similar analysis held against large-sigma swings over a longer human scale. The same analysis would probably tell us that we should "never" see an event like a Black Monday."
Of course they'd say that. "Black Monday" wasn't a "normal transaction" event. It was a runaway glitch in the programming of early bots, IIRC.
Imagine that you had the statistics down for your favorite sports team, and knew that their star player was very likely to score between 10 and 15 points in each remaining game of the season. Now imagine that he is found in the act by a jealous husband,shot and killed. Did your model accurately predict that the sports star would show up dead before his next game? Does that mean your betting system is completely worthless?
Black Friday was every single member of your team refusing to show up to play, because they also philandered with the femme fatale, and are afraid they're next. You can't build models that predict that.
But, once it's happened, you can do something to prevent it. Maybe you assign a tail to each player to make sure he keeps it zipped outside of his house. Maybe you lock chastity belts on them. Maybe you only let castrated men play. Whatever you do, the players won't like it. But maybe it's what you have to do.
posted by IAmBroom at 2:59 PM on September 11, 2013 [1 favorite]
Of course they'd say that. "Black Monday" wasn't a "normal transaction" event. It was a runaway glitch in the programming of early bots, IIRC.
Imagine that you had the statistics down for your favorite sports team, and knew that their star player was very likely to score between 10 and 15 points in each remaining game of the season. Now imagine that he is found in the act by a jealous husband,shot and killed. Did your model accurately predict that the sports star would show up dead before his next game? Does that mean your betting system is completely worthless?
Black Friday was every single member of your team refusing to show up to play, because they also philandered with the femme fatale, and are afraid they're next. You can't build models that predict that.
But, once it's happened, you can do something to prevent it. Maybe you assign a tail to each player to make sure he keeps it zipped outside of his house. Maybe you lock chastity belts on them. Maybe you only let castrated men play. Whatever you do, the players won't like it. But maybe it's what you have to do.
posted by IAmBroom at 2:59 PM on September 11, 2013 [1 favorite]
The singularity bus is getting ready to leave, and there's no room on it for us humans.
posted by CaseyB at 3:10 PM on September 11, 2013 [1 favorite]
posted by CaseyB at 3:10 PM on September 11, 2013 [1 favorite]
HFT is a parasite. There is zero societal benefit from price arbitrage on the scale of nanoseconds. Who the fuck needs to make a deal right this nanosecond? Nobody that's involved in the real economy. Also, as mentioned elsewhere in the thread, the fuckers run for the hills in a bear market, so the liquidity argument is at best weak, if not outright bullshit.
The only virtue of HFT is that it creates a playground that only the big boys get to play in, and which we all have to pay to subsidise. I would go one step further than the Tobin tax and propose a bid/ask tax. If you are actually putting your money down and have the courage of your convictions, then you get a free pass. If you're continually checking out who's got a longer optical path length to their trade node, go take a flying fuck to yourself, you virus.
posted by Jakey at 3:19 PM on September 11, 2013 [12 favorites]
The only virtue of HFT is that it creates a playground that only the big boys get to play in, and which we all have to pay to subsidise. I would go one step further than the Tobin tax and propose a bid/ask tax. If you are actually putting your money down and have the courage of your convictions, then you get a free pass. If you're continually checking out who's got a longer optical path length to their trade node, go take a flying fuck to yourself, you virus.
posted by Jakey at 3:19 PM on September 11, 2013 [12 favorites]
One of my AI friends once told me over a beer that he strongly suspected that we are undergoing an AI apocalypse
Well, they're already drinking our beer, so I'd say we're well on our way.
posted by indubitable at 3:26 PM on September 11, 2013 [7 favorites]
Well, they're already drinking our beer, so I'd say we're well on our way.
posted by indubitable at 3:26 PM on September 11, 2013 [7 favorites]
I have read over the relevant comments several times now and I cannot for the life of me figure out how that comment is a response to un petit cadeau's.If you replace the initial "But" with an "and" it makes a lot of sense; un petit cadeau points out that there are some distributions for which none of these crashes are expected, some background distributions for which these crashes are expected. kaibutsu points out that this is the point of the "black swan" talk that points out the distributions are probably not best modeled by a single distribution, but by a mixture of a something that's nicely behaved and something that's extremely poorly behaved. The "Normal" distribution may be the best for the normal times, with that, with a pool of 25 million observations, you expect zero if the market is modeled by a Normal distribution. If it's modeled by a more Black Swan like distribution with a heavier tail, like a Log Normal, then you expect perhaps 400 crashes in the 25 million potential crashes.
Standard deviations (sigmas) are a terribly difficult measurement for conveying the rareness of most events, except in particular and highly controlled experimental settings. Sigmas as a measurement works well for Normal distributions, but very few things in this world are modeled well by a Normal distribution, and sloppily assuming a Normal will significantly underestimate the frequency of such events. (I suddenly realize that I'm directly repeating the black swan talk...) But even though sigmas are terrible, they are something, and I don't know if there's anything better.
posted by Llama-Lime at 3:27 PM on September 11, 2013
|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\
I am now an awesome machine intelligence.
Either that or you just won the 100-yard dash in Microsoft Decathlon.
posted by escabeche at 3:32 PM on September 11, 2013 [4 favorites]
I am now an awesome machine intelligence.
Either that or you just won the 100-yard dash in Microsoft Decathlon.
posted by escabeche at 3:32 PM on September 11, 2013 [4 favorites]
It's linked as a source in the Nature paper but An ecological perspective on the future of computer trading is (IMO) required reading for this discussion.
There's also good background on what kinds of participants are actually active in the market and their various tradeoffs and restrictions at Aleph Blog, Part 1 and Part 2.
Oh and you should look at Bats' Market Volume Summary as well to get a sense of where things trade.
posted by Skorgu at 3:57 PM on September 11, 2013
There's also good background on what kinds of participants are actually active in the market and their various tradeoffs and restrictions at Aleph Blog, Part 1 and Part 2.
Oh and you should look at Bats' Market Volume Summary as well to get a sense of where things trade.
posted by Skorgu at 3:57 PM on September 11, 2013
Tobin Taxes have been tried in Sweden.
0.5 percent of every transaction is quite different from the one-cent transaction fees suggested this time around. While such a fee might scare the HFTers into other markets, it won't devastate the human side of trading, like the Swedish tax did.
posted by ymgve at 4:06 PM on September 11, 2013 [2 favorites]
0.5 percent of every transaction is quite different from the one-cent transaction fees suggested this time around. While such a fee might scare the HFTers into other markets, it won't devastate the human side of trading, like the Swedish tax did.
posted by ymgve at 4:06 PM on September 11, 2013 [2 favorites]
I skimmed the paper and it's pretty technical, but I thought I'd make a couple observations:
1. The probably of an event being n-sigma or larger is p ~ 0.5 erfc(n / sqrt(2)). A 30-sigma event corresponds to roughly p ~10^-194. I think you can pretty much rule that out as impossible, so the distribution is decidedly not normal (or these events are extraneous events unrelated to background fluctuations). The authors know this, they discuss the nearly power-law distribution of events, and the deviations from power-law.
2. Non-normality is interesting in this case, because the intervals are so short. The central limit theorem shows that if you add a bunch of random numbers then you either get a gaussian (normal) or a power law distribution. The requirement for gaussian is finite-variance, independence, identical distribution. Since these spikes/crashes result from many individual changes in price but are not normal, that either means that they introduce effectively infinite variance, or the price changes during a spike are responding to changes earlier in the same spike. Thus these spikes result either from the computer programs injecting insane amounts of noise into the market, or from their ability to respond in extremely short duration to the action of other computer programs.
The authors of the paper are clearly aware of this and have developed a model of the market populated by acting programs. The details of their model seem to indicate that the programs are not injecting infinite variance individually (phew) but rather it's their fast interactions. The wild changes can be diminished by forcing them to act more slowly, or to adopt more varied strategies. I've heard it suggested in the past that these events may actually be partly due to programs deliberately injecting insane amounts of noise in order to cover up their true strategy.
posted by Humanzee at 4:07 PM on September 11, 2013
1. The probably of an event being n-sigma or larger is p ~ 0.5 erfc(n / sqrt(2)). A 30-sigma event corresponds to roughly p ~10^-194. I think you can pretty much rule that out as impossible, so the distribution is decidedly not normal (or these events are extraneous events unrelated to background fluctuations). The authors know this, they discuss the nearly power-law distribution of events, and the deviations from power-law.
2. Non-normality is interesting in this case, because the intervals are so short. The central limit theorem shows that if you add a bunch of random numbers then you either get a gaussian (normal) or a power law distribution. The requirement for gaussian is finite-variance, independence, identical distribution. Since these spikes/crashes result from many individual changes in price but are not normal, that either means that they introduce effectively infinite variance, or the price changes during a spike are responding to changes earlier in the same spike. Thus these spikes result either from the computer programs injecting insane amounts of noise into the market, or from their ability to respond in extremely short duration to the action of other computer programs.
The authors of the paper are clearly aware of this and have developed a model of the market populated by acting programs. The details of their model seem to indicate that the programs are not injecting infinite variance individually (phew) but rather it's their fast interactions. The wild changes can be diminished by forcing them to act more slowly, or to adopt more varied strategies. I've heard it suggested in the past that these events may actually be partly due to programs deliberately injecting insane amounts of noise in order to cover up their true strategy.
posted by Humanzee at 4:07 PM on September 11, 2013
"OK I know what I really want to see, more specifically: a nanosecond-by-nanosecond history of the price movement around an actual stock for one especially active half-second. This probably can't happen because only one of the companies involved would have access to such precise data and they wouldn't want to release it, but still."
a birds: something like this?
posted by late afternoon dreaming hotel at 4:22 PM on September 11, 2013 [1 favorite]
a birds: something like this?
posted by late afternoon dreaming hotel at 4:22 PM on September 11, 2013 [1 favorite]
"OK I know what I really want to see, more specifically: a nanosecond-by-nanosecond history of the price movement around an actual stock for one especially active half-second. This probably can't happen because only one of the companies involved would have access to such precise data and they wouldn't want to release it, but still."
This data is available from any of the companies that supply market data, for a fee. I don't know how many of them will provide you historical data of that detail, but the real time data is on those time scales and you can buy access to the feed, "listen" to and record the data and wait for one of these events to happen. I'm not sure what you really expect to find in there, but some times even plain day-to-day events have interesting patterns that reveal people's intentions.
The only virtue of HFT is that it creates a playground that only the big boys get to play in, and which we all have to pay to subsidise.
I don't know where you get the impression that HFT is the realm of the "big boys". I work for a HFT company that would probably be considered small - 100ish people. I know of other HFT companies that are literally 3-5 people.
posted by RustyBrooks at 4:47 PM on September 11, 2013
This data is available from any of the companies that supply market data, for a fee. I don't know how many of them will provide you historical data of that detail, but the real time data is on those time scales and you can buy access to the feed, "listen" to and record the data and wait for one of these events to happen. I'm not sure what you really expect to find in there, but some times even plain day-to-day events have interesting patterns that reveal people's intentions.
The only virtue of HFT is that it creates a playground that only the big boys get to play in, and which we all have to pay to subsidise.
I don't know where you get the impression that HFT is the realm of the "big boys". I work for a HFT company that would probably be considered small - 100ish people. I know of other HFT companies that are literally 3-5 people.
posted by RustyBrooks at 4:47 PM on September 11, 2013
also, is "goodness-of-fit" seriously the preferred term in statistics, over, idk, "fit quality"?
pretty much, yeah.
posted by mixing at 4:48 PM on September 11, 2013
pretty much, yeah.
posted by mixing at 4:48 PM on September 11, 2013
IAmBroom: The writer was trying to say these happen too fast for humans to react and then indicates that these conditions can occur over a period of time long enough for a human to react.
Also, his interpretation of what is happening must be way off. I haven't designed a database in a couple decades, but tracking a price trend over ten transactions is not overly difficult, even at these speeds.
posted by Ardiril at 5:05 PM on September 11, 2013 [1 favorite]
Also, his interpretation of what is happening must be way off. I haven't designed a database in a couple decades, but tracking a price trend over ten transactions is not overly difficult, even at these speeds.
posted by Ardiril at 5:05 PM on September 11, 2013 [1 favorite]
I can't believe anyone treats this bull**** as a real market when regulatory capture is obviously complete. Statistical analysis of an ongoing fraud won't yield any actually meaningful results that work in real world circles.
I've long advocated a market with a 30 second cycle, with batches executed and no take-backsies when Goldman Sachs loses on a trade.
Human time scales are important to preserve here, as well as market neutrality. This GS skimming operation needs to end.
posted by MikeWarot at 5:24 PM on September 11, 2013 [4 favorites]
I've long advocated a market with a 30 second cycle, with batches executed and no take-backsies when Goldman Sachs loses on a trade.
Human time scales are important to preserve here, as well as market neutrality. This GS skimming operation needs to end.
posted by MikeWarot at 5:24 PM on September 11, 2013 [4 favorites]
(Not to harp on this, but batching transactions e.g. on a 1 second time tick can't be the answer. It opens up severe arbitrage opportunities with no social value.)
For example: Alice and Bob agree that Alice will buy shares of Dinoco on the London exchange while Bob sells on the NYSE, and then after an hour, they'll do it in reverse so that it all nets out. Now Alice sees Dinoco change in price in London; her signal to Bob telling him to sell/not sell travels at the speed of light and gets to Bob well before the time tick at NYSE when the price changes. Or vice versa. Boom, money from nothing.
And no, you can not prevent this by "synchronizing" clocks. Light travels only 180 miles in a millisecond, 1.8 miles in a microsecond. Good luck synchronizing three or more clocks to prevent arbitrage-able gaps (it's physically impossible).
posted by RedOrGreen at 6:12 PM on September 11, 2013 [4 favorites]
For example: Alice and Bob agree that Alice will buy shares of Dinoco on the London exchange while Bob sells on the NYSE, and then after an hour, they'll do it in reverse so that it all nets out. Now Alice sees Dinoco change in price in London; her signal to Bob telling him to sell/not sell travels at the speed of light and gets to Bob well before the time tick at NYSE when the price changes. Or vice versa. Boom, money from nothing.
And no, you can not prevent this by "synchronizing" clocks. Light travels only 180 miles in a millisecond, 1.8 miles in a microsecond. Good luck synchronizing three or more clocks to prevent arbitrage-able gaps (it's physically impossible).
posted by RedOrGreen at 6:12 PM on September 11, 2013 [4 favorites]
The Skynet Funding Bill is passed.
The system goes on-line August 4th, 1997.
Human decisions are removed from strategic defense.
Skynet begins to learn at a geometric rate.
It becomes self-aware at 2:14 a.m. Eastern time, August 29th.
In a panic, they try to pull the plug.
posted by Sebmojo at 6:29 PM on September 11, 2013 [1 favorite]
The system goes on-line August 4th, 1997.
Human decisions are removed from strategic defense.
Skynet begins to learn at a geometric rate.
It becomes self-aware at 2:14 a.m. Eastern time, August 29th.
In a panic, they try to pull the plug.
posted by Sebmojo at 6:29 PM on September 11, 2013 [1 favorite]
This milk is pizzled.
posted by charlie don't surf at 7:03 PM on September 11, 2013
posted by charlie don't surf at 7:03 PM on September 11, 2013
IAmBroom: The writer was trying to say these happen too fast for humans to react and then indicates that these conditions can occur over a period of time long enough for a human to react.
Er, by the time you have noticed and reacted here, the entire event is over (event time is roughly equal to reaction time). Which I think is the point the writer is trying to make.
posted by flug at 7:11 PM on September 11, 2013 [1 favorite]
Er, by the time you have noticed and reacted here, the entire event is over (event time is roughly equal to reaction time). Which I think is the point the writer is trying to make.
posted by flug at 7:11 PM on September 11, 2013 [1 favorite]
Also if the 1.5s includes both the spike up and the retreat down, by the time you reacted you'd be doing the wrong thing.
posted by RustyBrooks at 7:16 PM on September 11, 2013 [1 favorite]
posted by RustyBrooks at 7:16 PM on September 11, 2013 [1 favorite]
And no, you can not prevent this by "synchronizing" clocks. Light travels only 180 miles in a millisecond, 1.8 miles in a microsecond. Good luck synchronizing three or more clocks to prevent arbitrage-able gaps (it's physically impossible).
Why couldn't you synchronize them to GPS time, which is global and accurate to *looks* 100 nanoseconds?
posted by ROU_Xenophobe at 8:48 PM on September 11, 2013
Why couldn't you synchronize them to GPS time, which is global and accurate to *looks* 100 nanoseconds?
posted by ROU_Xenophobe at 8:48 PM on September 11, 2013
indubitable: "One of my AI friends once told me over a beer that he strongly suspected that we are undergoing an AI apocalypse
Well, they're already drinking our beer, so I'd say we're well on our way."
Simbeer. Simmicrobrew.
posted by Samizdata at 9:56 PM on September 11, 2013
Well, they're already drinking our beer, so I'd say we're well on our way."
Simbeer. Simmicrobrew.
posted by Samizdata at 9:56 PM on September 11, 2013
I didn't quite follow why a tiny transaction tax ($0.01 per trade, say) wouldn't fix this problem.
Because shares can be traded anywhere, even privately, and if it's cheaper to trade shares on a different exchange then share trading will move to that exchange. Honestly, this is just the end point of hundreds of years of acceleration: faster traders have always had an advantage, which is why they push for faster and faster techniques. This is why exchanges were invented in the first place.
posted by Joe in Australia at 1:20 AM on September 12, 2013
Because shares can be traded anywhere, even privately, and if it's cheaper to trade shares on a different exchange then share trading will move to that exchange. Honestly, this is just the end point of hundreds of years of acceleration: faster traders have always had an advantage, which is why they push for faster and faster techniques. This is why exchanges were invented in the first place.
posted by Joe in Australia at 1:20 AM on September 12, 2013
"new machine ecology" is a terrifying sentence.
I'd like to say "machine" can not have an "ecology" but I know that that is not true - I've worked on machines.
But the 'machines' I've worked on didn't work for themselves. Even I could understand them. The prospect of machines that can recognize and then elaborate on their own 'ecology' is terrifying.
Because from that moment going forward we are no longer in control. And once the levels of complexity get adequately developed, there's less and less of a chance of going back and re-gaining control.
posted by From Bklyn at 4:42 AM on September 12, 2013 [2 favorites]
I'd like to say "machine" can not have an "ecology" but I know that that is not true - I've worked on machines.
But the 'machines' I've worked on didn't work for themselves. Even I could understand them. The prospect of machines that can recognize and then elaborate on their own 'ecology' is terrifying.
Because from that moment going forward we are no longer in control. And once the levels of complexity get adequately developed, there's less and less of a chance of going back and re-gaining control.
posted by From Bklyn at 4:42 AM on September 12, 2013 [2 favorites]
It'll be interesting when we get the first economic collapse that no one can even explain in retrospect, but we'll all be bound to its lash anyway, because markets.
posted by Steely-eyed Missile Man at 5:13 AM on September 12, 2013
posted by Steely-eyed Missile Man at 5:13 AM on September 12, 2013
We're not exactly cheering because "HFT is providing liquidity", RedOrGreen, but because machine trading reduces the buy-sell spread. We've replaced a capitalist elite with machines. That's a massive social gain! And now the designers have started becoming redundant.
At present, HTF serves an even higher capitalist elite, which contributes to inequality, but so did the industrial revolution. We should not slow down the NFT algorithms any more than we should slow down assembly lines. Instead, we should institute progressive taxation and provide necessary social services.
posted by jeffburdges at 6:08 AM on September 12, 2013
At present, HTF serves an even higher capitalist elite, which contributes to inequality, but so did the industrial revolution. We should not slow down the NFT algorithms any more than we should slow down assembly lines. Instead, we should institute progressive taxation and provide necessary social services.
posted by jeffburdges at 6:08 AM on September 12, 2013
We've replaced a capitalist elite with machines.
Have we? Someone still owns those machines and their output. What does it matter if the designers become redundant if the boxes still funnel their gains ever upward?
posted by Steely-eyed Missile Man at 6:28 AM on September 12, 2013
Have we? Someone still owns those machines and their output. What does it matter if the designers become redundant if the boxes still funnel their gains ever upward?
posted by Steely-eyed Missile Man at 6:28 AM on September 12, 2013
And of course I see that I somehow spaced the second part of your comment. Sorry, high-frequency commenting algorithms have caused a lamentable intelligence fluctuation.
posted by Steely-eyed Missile Man at 6:29 AM on September 12, 2013 [1 favorite]
posted by Steely-eyed Missile Man at 6:29 AM on September 12, 2013 [1 favorite]
RedOrGreen: you're misunderstanding relativity. The synchronisation problem occurs when clocks have relative motion, which they (for the purposes of synchronising exchanges fixed to the surface of the earth) do not. GPS and NTP have both solved this problem more than accurately enough, as have quite a number of other systems.
Granted, once you get to trade over intra-solar let alone interstellar distances then the arbitrage becomes very interesting due to time issues but at that point, HFT isn't the problem.
posted by polyglot at 6:53 AM on September 12, 2013 [1 favorite]
Granted, once you get to trade over intra-solar let alone interstellar distances then the arbitrage becomes very interesting due to time issues but at that point, HFT isn't the problem.
posted by polyglot at 6:53 AM on September 12, 2013 [1 favorite]
This whole thing reminds me of the Seven Stages of Robot Replacement (Denial) with one exception:
Humans are the ultimate holders of capital. If you don't have significant capital and the means to earn an income from it, you're worthless once the value of human labour trends to zero. Corporations might own other corporations, but there is always a human at the top of the tree, so far.
(I'm going to disregard @cstross' Saturn's Children/Neptune's Brood here, but they're absolutely worth a read)
posted by polyglot at 7:01 AM on September 12, 2013 [1 favorite]
Humans are the ultimate holders of capital. If you don't have significant capital and the means to earn an income from it, you're worthless once the value of human labour trends to zero. Corporations might own other corporations, but there is always a human at the top of the tree, so far.
(I'm going to disregard @cstross' Saturn's Children/Neptune's Brood here, but they're absolutely worth a read)
posted by polyglot at 7:01 AM on September 12, 2013 [1 favorite]
polyglot: RedOrGreen: you're misunderstanding relativity. The synchronisation problem occurs when clocks have relative motion, which they (for the purposes of synchronising exchanges fixed to the surface of the earth) do not. GPS and NTP have both solved this problem more than accurately enough.
Have they, though? I have to deal with clock correction files for our observatory masers when we observe pulsars. (Our best ones are stabler than 100 ns, and approaching 10 ns - gravity waves here we come!) GPS has certainly made our lives easier, but we get these clock corrections after the fact, by comparing observatory masers to GPS time ticks and then inferring what the actual time of arrival of a given pulse was, referenced back to (for example) the center of the Earth.
I don't see how it could be done that easily in real time for stock trades - there's always going to be a situation where someone in the middle of three exchanges is going to be able to detect a clock drift at one of them before the three of them can get back into sync (and that is limited by light travel time).
Why not just use "the" GPS time? I think the problem isn't the time, so much as the location of the time tick (the phase), because the arbitrage opportunity is between a clock tick at one site and a clock tick at another one. I don't think you can verify that you've maintained phase lock via GPS if you have more than 2 sites.
(Apologies for the massive derail... but time is such a slippery concept.)
posted by RedOrGreen at 8:01 AM on September 12, 2013 [1 favorite]
Have they, though? I have to deal with clock correction files for our observatory masers when we observe pulsars. (Our best ones are stabler than 100 ns, and approaching 10 ns - gravity waves here we come!) GPS has certainly made our lives easier, but we get these clock corrections after the fact, by comparing observatory masers to GPS time ticks and then inferring what the actual time of arrival of a given pulse was, referenced back to (for example) the center of the Earth.
I don't see how it could be done that easily in real time for stock trades - there's always going to be a situation where someone in the middle of three exchanges is going to be able to detect a clock drift at one of them before the three of them can get back into sync (and that is limited by light travel time).
Why not just use "the" GPS time? I think the problem isn't the time, so much as the location of the time tick (the phase), because the arbitrage opportunity is between a clock tick at one site and a clock tick at another one. I don't think you can verify that you've maintained phase lock via GPS if you have more than 2 sites.
(Apologies for the massive derail... but time is such a slippery concept.)
posted by RedOrGreen at 8:01 AM on September 12, 2013 [1 favorite]
Regarding the possibility for a synchronized heartbeat across markets, Here's a neat 3-minute video from a PhD student who's studying HFT, very succinctly explaining her findings that latency arbitrage is currently a problem, and that "call markets" in which orders across markets were aggregated and executed simultaneously on a schedule would be more efficient. (There's also a poster version there if you're not into video.)
A commenter asks how that would be implemented, and she writes:
Are there reasons that wouldn't work?
posted by jhc at 8:02 AM on September 12, 2013 [2 favorites]
A commenter asks how that would be implemented, and she writes:
Switching from continuous trading to a centralized call market would be a fairly complex process. A fully centralized stock market, i.e. one without fragmentation, is most likely infeasible. However, it is more specifically the lack of synchronization between trading venues—a direct result of fragmentation—that permits HFTs to gain an informational advantage via speed.The Sparrow she refers to is Chris Sparrow -- here's an interview with him. Here's his proposal:
Therefore, it will be necessary to develop a robust method for synchronizing multiple trading venues. Such a method would require precise coordination between exchanges and would need to ensure that trade executions and quote updates all occur simultaneously, across all markets, in order to eliminate the potential for latency arbitrage. This is a challenging task, as there are several issues that will necessitate careful consideration, such as how to determine the unique transaction price across all venues for a given matching interval, how to ensure that the global price quote is accurate, and how to handle orders that match across multiple venues (Sparrow, 2012).
From t1 – t2, all trading venues open for order entrySo he doesn't address the timing issue by assuming perfect time synchronization, but rather by forbidding time-priority of trades. You switch between an open market period where anyone can make secret trades, and a closed period where the markets confer among themselves and settle all the trades, with no consideration for who got theirs in first. He suggests this could be done once per second.
- Orders are not made visible to market participants
- Participants allowed to cancel any open orders while order entry window is open
At t2, order entry window closes for all markets
- Allow a short time (t2-t3) to ensure all venues’ order entry windows are closed (venues confirm closure to each other)
At t3, initiate matching process (regulator could provide signal to initiate)
- Determine unique clearing price for the current call by having venues compare their
booked orders
- Each venue matches orders at the clearing price (do not allow intra-call time-priority)
- Cross-venue matches determined as per bi-lateral venue agreements
- Allow dark orders and crosses to execute at clearing price
- Orders that would lock or cross market are cancelled or re-priced
- Allow passive orders to book in order to attract liquidity to next call
At t4, matching process completes, trades are reported to tape and regulator, passively booked orders reported
At t5, next order entry window opens
- Lather, rinse, repeat
Are there reasons that wouldn't work?
posted by jhc at 8:02 AM on September 12, 2013 [2 favorites]
jhc: He doesn't address the timing issue by assuming perfect time synchronization, but rather by forbidding time-priority of trades. You switch between an open market period where anyone can make secret trades, and a closed period where the markets confer among themselves and settle all the trades, with no consideration for who got theirs in first.
Okay, this is a genuinely good idea. I haven't thought about this in any detail, but I don't have an obvious objection to it. There's no phase locking or time sync needed beyond the very coarse variety, and here GPS would certainly work.
Would it work legally? What happens when you have one rogue player? I don't know. But this could work.
posted by RedOrGreen at 8:07 AM on September 12, 2013
Okay, this is a genuinely good idea. I haven't thought about this in any detail, but I don't have an obvious objection to it. There's no phase locking or time sync needed beyond the very coarse variety, and here GPS would certainly work.
Would it work legally? What happens when you have one rogue player? I don't know. But this could work.
posted by RedOrGreen at 8:07 AM on September 12, 2013
jeffburdges: We've replaced a capitalist elite with machines. That's a massive social gain! [...] At present, HTF serves an even higher capitalist elite, which contributes to inequality, but so did the industrial revolution.
Okay, I like this. Funnel more and more of the gains to fewer and fewer people. Ultimately it all goes to Warren Buffet, and then when he is tender and tasty, we eat him. Yum.
We should not slow down the NFT algorithms any more than we should slow down assembly lines. Instead, we should institute progressive taxation and provide necessary social services.
I don't see this as an either/or. Why not do both? Assembly lines have a productive function, HFT does not. (Pushing designers into unemployment is not particularly productive.)
posted by RedOrGreen at 8:12 AM on September 12, 2013
Okay, I like this. Funnel more and more of the gains to fewer and fewer people. Ultimately it all goes to Warren Buffet, and then when he is tender and tasty, we eat him. Yum.
We should not slow down the NFT algorithms any more than we should slow down assembly lines. Instead, we should institute progressive taxation and provide necessary social services.
I don't see this as an either/or. Why not do both? Assembly lines have a productive function, HFT does not. (Pushing designers into unemployment is not particularly productive.)
posted by RedOrGreen at 8:12 AM on September 12, 2013
I didn't quite follow why a tiny transaction tax ($0.01 per trade, say) wouldn't fix this problem.
joe in australia: Because shares can be traded anywhere, even privately, and if it's cheaper to trade shares on a different exchange then share trading will move to that exchange.
I agree, but this is a question of political will, not a conceptual problem. State governments are perfectly happy to tax private transactions (did you pay tax on your garage sale?) and if the trading volume is large enough, they figure out how to enforce their tax collection soon enough (obvious example: Amazon.com).
That's it for catch-up responses to time zones ahead of me. Sorry about the spate of comments.
posted by RedOrGreen at 8:19 AM on September 12, 2013
joe in australia: Because shares can be traded anywhere, even privately, and if it's cheaper to trade shares on a different exchange then share trading will move to that exchange.
I agree, but this is a question of political will, not a conceptual problem. State governments are perfectly happy to tax private transactions (did you pay tax on your garage sale?) and if the trading volume is large enough, they figure out how to enforce their tax collection soon enough (obvious example: Amazon.com).
That's it for catch-up responses to time zones ahead of me. Sorry about the spate of comments.
posted by RedOrGreen at 8:19 AM on September 12, 2013
"aggregated and executed simultaneously on a schedule" were the words I was looking for but didn't get. Bah.
posted by cromagnon at 10:19 AM on September 12, 2013
posted by cromagnon at 10:19 AM on September 12, 2013
Quants: The Alchemists of Wall Street
posted by Pirate-Bartender-Zombie-Monkey at 12:35 PM on September 12, 2013 [1 favorite]
Quants are the math wizards and computer programmers in the engine room of our global financial system who designed the financial products that almost crashed Wall st. The credit crunch has shown how the global financial system has become increasingly dependent on mathematical models trying to quantify human (economic) behaviour. Now the quants are at the heart of yet another technological revolution in finance: trading at the speed of light.By the same folks who did Money and Speed: Inside the Black Box. If Black Box HFT is the monster, these are the (incredibly well compenstaed) Dr. Frankensteins.
What are the risks of treating the economy and its markets as a complex machine? Will we be able to keep control of this model-based financial system, or have we created a monster?
A story about greed, fear and randomness from the insides of Wall Street.
posted by Pirate-Bartender-Zombie-Monkey at 12:35 PM on September 12, 2013 [1 favorite]
Do our financial markets have a function, RedOrGreen? I donno, all relative I suppose, but assume yes.
HFT has demonstrably lowered bid-ask spreads. Those spreads represent actual rents extracted from investors by the financial system. Less money will flow from investors to investments if you outlaw algorithmic trading.
Now Wall St extracts different more creative rents elsewhere, ala faked risks assessment, too big to fail, etc., but mostly those issues represent outright fraud that's not being prosecuted. Afaik, no connection with raw trading speed.
We could discuss tricky protocols for the exchanges that might reduce the bid-ask spreads further, say by allowing more traders to participate. Those protocols might require more computational time, well maybe they'd work more like auctions or bitcoin, maybe they'd respect the speed of light somehow. You want to avoid creating some asymmetrical system where established markets players gain more advantage than they do now with proximity, so it's far more delicate than a speed limit.
There is another problem with Black-Scholes' using the log-normal distribution, when the true distribution has fatter tails, but that predates algorithmic trading. And maybe the robots "heading for the hills" actually reduces this problem, not sure.
posted by jeffburdges at 1:05 PM on September 12, 2013 [1 favorite]
HFT has demonstrably lowered bid-ask spreads. Those spreads represent actual rents extracted from investors by the financial system. Less money will flow from investors to investments if you outlaw algorithmic trading.
Now Wall St extracts different more creative rents elsewhere, ala faked risks assessment, too big to fail, etc., but mostly those issues represent outright fraud that's not being prosecuted. Afaik, no connection with raw trading speed.
We could discuss tricky protocols for the exchanges that might reduce the bid-ask spreads further, say by allowing more traders to participate. Those protocols might require more computational time, well maybe they'd work more like auctions or bitcoin, maybe they'd respect the speed of light somehow. You want to avoid creating some asymmetrical system where established markets players gain more advantage than they do now with proximity, so it's far more delicate than a speed limit.
There is another problem with Black-Scholes' using the log-normal distribution, when the true distribution has fatter tails, but that predates algorithmic trading. And maybe the robots "heading for the hills" actually reduces this problem, not sure.
posted by jeffburdges at 1:05 PM on September 12, 2013 [1 favorite]
Do our financial markets have a function, RedOrGreen? I donno, all relative I suppose, but assume yes.
10 years ago, maybe even 5 years ago, I would have agreed with your assumption.
Now I find it harder and harder to credit the financial markets with any productive social function at all. It feels like a gigantic rigged casino game set up to suck in those employer-directed 401Ks and skim off the top. Meanwhile, new companies are increasingly being funded privately or looking to be bought by established players because playing the IPO lottery is too tricky or too expensive.
(Is not playing a winning move? I don't think so either, even though a former classmate at a hedge fund actually floated the "Mattress backed security" as a haha-only-joking-maybe option when I pointed out that I'd have been better off stuffing all my cash into my mattress instead of the market over a 12 year span.)
posted by RedOrGreen at 2:09 PM on September 12, 2013
10 years ago, maybe even 5 years ago, I would have agreed with your assumption.
Now I find it harder and harder to credit the financial markets with any productive social function at all. It feels like a gigantic rigged casino game set up to suck in those employer-directed 401Ks and skim off the top. Meanwhile, new companies are increasingly being funded privately or looking to be bought by established players because playing the IPO lottery is too tricky or too expensive.
(Is not playing a winning move? I don't think so either, even though a former classmate at a hedge fund actually floated the "Mattress backed security" as a haha-only-joking-maybe option when I pointed out that I'd have been better off stuffing all my cash into my mattress instead of the market over a 12 year span.)
posted by RedOrGreen at 2:09 PM on September 12, 2013
I'll tell you my sci-fi dream scenario : We automate the markets progressively more and more fully. An open source automated trading platform defeats all the in-house and proprietary competitors, although in-house extensions remain critical, ala Linux+Apache+website. The SEC and audit companies begin algorithmically analyzing not merely trades, but the platforms' actual reasoning. Also the SEC audits the audit companies software. All this transparency vastly reduces the "legally grey area" by better exposing a trading firm's "intent". In fact, even if fraudulent activities are legal, the wronged parties can obtain and analyze the trading firm's "thought process". In the longer run, our knowledge of algorithm's "intent" in trading activity forces firms to become more socially conscious, basically the investors start asking for in-house extensions that make them popular, rather than just richer.
We could accomplish roughly this faster with a "revolution", but they've a poor track record for subtle social change. Algorithmic traders with perfectly transparent thought processes working for social good could happen evolutionarily.
posted by jeffburdges at 4:54 PM on September 12, 2013 [1 favorite]
We could accomplish roughly this faster with a "revolution", but they've a poor track record for subtle social change. Algorithmic traders with perfectly transparent thought processes working for social good could happen evolutionarily.
posted by jeffburdges at 4:54 PM on September 12, 2013 [1 favorite]
Two-Phase Commit, bitches.
It's a solved problem in the database world since the 1970s, keeping geographically separated processes synchronized and consistent. See also ACID transactions.
It requires that there is always a network connection between the exchanges (they will diverge if partitioned and devolve to the current situation), and you give up the ability to trade instantly locally to obtain arbitrage... but that's kind of what we're aiming for here.
There are no technical problems in ending HFT within the scope of one planet, only political. But the political issues are huge.
posted by polyglot at 6:41 PM on September 12, 2013 [1 favorite]
It's a solved problem in the database world since the 1970s, keeping geographically separated processes synchronized and consistent. See also ACID transactions.
It requires that there is always a network connection between the exchanges (they will diverge if partitioned and devolve to the current situation), and you give up the ability to trade instantly locally to obtain arbitrage... but that's kind of what we're aiming for here.
There are no technical problems in ending HFT within the scope of one planet, only political. But the political issues are huge.
posted by polyglot at 6:41 PM on September 12, 2013 [1 favorite]
« Older The Dada Baroness, Elsa von Freytag-Loringhoven | Word Magazine, Back from the Dead Newer »
This thread has been archived and is closed to new comments
damn
posted by Smedleyman at 12:41 PM on September 11, 2013 [15 favorites]