High frequency trading crop circles

August 7, 2010 7:52 AM

High frequency trading crop circles. Automated trading is flooding stock exchanges with nonsensical orders making odd patterns like The Knife at millisecond scales. Bugs, emergent phenomena, or market jamming strategies? No one seems to know.
posted by Nelson (108 comments total)

The market version of number stations?
posted by Chichibio at 7:59 AM on August 7, 2010 [5 favorites]

Um, if the orders were nonsensical, they wouldn't do them. Very smart people write these things and have computers implement them so they WILL behave properly. It's people and emotions who trade things nonsensically.

As funny as they may look, there IS sound reason behind the patterns.
posted by phunniemee at 8:06 AM on August 7, 2010 [2 favorites]

This is really interesting stuff. Nice post.
posted by mr.marx at 8:07 AM on August 7, 2010

I think the theory that they are adding 'noise' to the data that will take competitors extra cycles to filter out makes the most sense.
posted by ao4047 at 8:10 AM on August 7, 2010

From the comments section of The Atlantic:

"I am a software programmer with several friends at HFT firms, and the explanation we've put together accounts for all of the flash crash behavior as well as these signals - note that word, because that's what Donovan is pointing out, signals sent by software.

All of the serious HFT firms these days use "natural language processing", which means using artificial intelligence to extract profitable information from news streams. People think of this as just headlines but really it's anything that might contain useful information - these computers have all of the cable news channels supplied to them digitally and use everything they can scrape. Some of the firms even use facial recognition software to determine whether the speakers believe what they're saying. My friends joke about how Cramer is a goldmine for their algorithms but that the profitable trades rarely match up with his advice.

One of the facts about 'hard' AI, as is required for profitable NLP, is that the coders who developed it don't even understand completely how it works. If they did, it would just be a regular program. What's even stranger is that they can't use regular tools, like a debugger, to observe the algorithms' behavior, because it interferes with the processing and causes different trades to be emitted. In a very real sense, they can't explain why their robots send the orders they do. They can tell you what data they "trained" it with, and what sorts of data they "feed" it, but they're inherently unpredictable.

As a result, a lot of programmers at HFT firms spend most of their time trying to keep the software from running away. They create elaborate safeguard systems to form a walled garden around the traders but, exactly like a human trader, the programs know that they make money by being novel, doing things that other traders haven't thought of. These gatekeeper programs are therefore under constant, hectic development as new algorithms are rolled out. The development pace necessitates that they implement only the most important safeguards, which means that certain types of algorithmic behavior can easily pass through. As has been pointed out by others, these were "quotes" not "trades", and they were far away from the inside price - therefore not something the risk management software would be necessarily be looking for.

The commenters above who called for identification of the company running the responsible algorithms have it wrong. There isn't just one firm putting these sorts of orders in the market. The fact is that many of these HFT shops are glued to their screens every minute the market is open because they have to be alert to strange behavior by their own algorithms.

TLDR: this definitely is emergent behavior - it is information passing between black-box algorithms with motivations that even the original programmers cannot make definitive statements about."
posted by Azazel Fel at 8:12 AM on August 7, 2010 [78 favorites]

I buy my data from the same company that offers the NXCore product (which is where the data for this analysis is coming from), and I have to say from a technical standpoint these guys are top notch. I think that their analytical skills on this are a bit off the mark though. Most of these algorithms perform liquidity detection (and there are some pretty technical reasons that you would expect to see some of these patterns). I strongly doubt that any of these are real attempts to jam enemy bots. One trading partner on one exchange can not produce enough orders to overwhelm a HFT system. These are not kiddie connections. These are clusters of the best processors available, on the fastest network connections available. An extra 100mbit/sec would not present an issue for most systems. To ensure that they can trade even in worst case scenerios most have double the capacity of the largest volume ever seen if not more. It is laughable to claim that you could attempt to jam such a thing. It is likely your connection to the exchange would be overwhelmed before you could even come close.
posted by An algorithmic dog at 8:14 AM on August 7, 2010 [3 favorites]

All of the serious HFT firms these days use...

None of the (serious and legit) ones I know do. That sounds absolutely ridiculous.
posted by phunniemee at 8:21 AM on August 7, 2010 [16 favorites]

The Atlantic comment strikes me as total hyperbole, especially phrases like "the programs know". HFT has not created strong AI with NLP; at most it is advancing the art of linear programming and low latency, high throughput packet processing.

If a programmer can't use a debugger on canned market data feeds to develop their algorithms, I wouldn't hire them.
posted by autopilot at 8:21 AM on August 7, 2010 [7 favorites]

As funny as they may look, there IS sound reason behind the patterns.

Not really. There might be a "sound" reason motivating the programs, but these oscillatory patterns have all of the hallmarks of an instability in the combined dynamical system comprised by the market + trading algorithm.

Similar things can happen in improperly designed control systems.
posted by mondo dentro at 8:22 AM on August 7, 2010 [4 favorites]

When I see those graphs, it looks to me like someone connected a TB-303 to the stock market.

But seriously, this is fascinating stuff that I obviously know very little about.
posted by goodnewsfortheinsane at 8:23 AM on August 7, 2010 [2 favorites]

How many times do we have to learn how unwise and dangerous it is for a AI to control any sort of integral or influential system?!

Oh, right, we haven't learned yet. Well, um...
posted by fuq at 8:38 AM on August 7, 2010 [3 favorites]

This is really sad. My father has been a floor trader for nearly 30 years and everyone who loves the business bemoans the computerized trading that goes on. There's no ACTION on the floor anymore. It's just boring button pushing and screen watching. Specialists can't work orders the way they used to, and most traders are at the mercy of their machines. It's killing the NYSE for one, and besides, it's just not FUN to trade anymore. When I used to go to work with my dad as a kid, it would be insane on the floor; people yelling, running around, working their orders. And there were so many people and wild personalities; never a dull moment. Now whenever I talk to him he says it's completely dead down there. So many people have lost their jobs because of these idiotic trading programs. It's too hard to read what is going on anymore because nobody knows if a computer is doing the trading or a real person. It's as if they outlawed face-to-face chess and every tournament was playing on a screen and each opponent never knew whether they were playing a human or a program. Truly sad.
posted by ReeMonster at 8:38 AM on August 7, 2010 [7 favorites]

mondo dentro sure but if the instability is always driving the prices up it sees safe to consider that they may be intentional as opposed to fully emergant. I have no idea if that's accurate in this case, it could be simply confirmation bias.

And yeah, that Atlantic comment isn't even wrong.
posted by Skorgu at 8:38 AM on August 7, 2010

As funny as they may look, there IS sound reason behind the patterns.

Some of them will be calibration cycles or runaway feedback loops or just excessive polling. The Very Smart are too expensive to review every line of code.
posted by eeeeeez at 8:40 AM on August 7, 2010 [1 favorite]

The banality of evil.
posted by sswiller at 8:41 AM on August 7, 2010

This confirms my suspicion that whatever purpose the markets were originally supposed to serve has been gamed out of existence.
posted by phooky at 8:42 AM on August 7, 2010 [34 favorites]

It is just noise being generated by highly mystified technical analysis. Azazel Fel's quote demonstrates that this is just crazy ass astrological chart reading being run through trading programs that eat news spew and emit nonsense.

Most of the trading strategies are low risk, but like most gambling systems, all you are doing is trading a high probability of small wins for a low probability of a huge loss. Anybody remember Black-Scholes?

Efficient Even in the weakest sense, the efficient market doesn't properly hold, as Ed Thorp and The Chaos Cabal demonstrated. However, the methods used by Thorp and The Prediction Company appear to detect very small pricing inefficiencies and then rapidly take advantage of them, a form of hedge investment (first mathematically described by Thorp.)

This HFT nonsense appears to put false price signals in the market that will trick other traders into creating inefficiencies that the signal spammer can take advantage of.

Ohh, injecting bad information into a trading system to create and anticipate favorable situations to take advantage of. Nothing at all like high speed stock manipulation, oh no. What could possibly go wrong?
posted by warbaby at 8:45 AM on August 7, 2010 [8 favorites]

Nanex (developer of NXCore) analysis of the Flash Crash. I see similar patterns. The market feels more like the dead eye of Jaws now than the Blue Horseshoe of Wall Street.
posted by wallstreet1929 at 8:46 AM on August 7, 2010 [1 favorite]

Very smart people write these things and have computers implement them so they WILL behave properly.

Yeah, that doesn't fill me with confidence. It's a software engineering maxim that you should never design a system at the limit of your intelligence, because when it breaks (and it will break), you will be by definition to stupid to fix it.
posted by Ritchie at 8:47 AM on August 7, 2010 [7 favorites]

Um, if the orders were nonsensical, they wouldn't do them. Very smart people write these things and have computers implement them so they WILL behave properly.

I don't work in this space, but every programmer knows smart people -- people who should know better -- who introduced broken code into production systems. This is inevitable.

Granted, there may a sound reason behind some of this activity -- adding noise to the data, countering other algorithms, etc. The rest -- some unknown quantity -- is certainly the byproduct of broken code, out of date code, etc.

It's a pedantic point, I know. Whenever I encounter software which acts against multivariate systems (or has a lot of unreliable dependencies), though, I immediately think of Ashby's Law of Requisite Variety.

There's a limit to how thoroughly the systems running these algorithms -- and the algorithms themselves -- can be tested and tuned. Some will inevitably slip the bounds of their programming and get weird.
posted by Kikkoman at 8:50 AM on August 7, 2010 [1 favorite]

As funny as they may look, there IS sound reason behind the patterns.

Yes, but it's a programmatic reason being generated by a black box, not a human reason. That's the fear: the combined effect of HFT programs will cause things to happen (like flash crashes) that make no sense except within the confines of a particular group of algorithms.

The Atlantic comment strikes me as total hyperbole, especially phrases like "the programs know". HFT has not created strong AI with NLP; at most it is advancing the art of linear programming and low latency, high throughput packet processing.

There's some hyperbole there, but it's not as threatening as it sounds. No, they haven't invented strong AI, but it's easy to develop NLP algorithms that are non-deterministic in practice. In my NLP class we wrote spam filters that used Bayesian filtering, trained on known-good/known-bad datasets. After training, you could see the spam score the algorithm assigned to a piece of mail, but there was no way to say exactly why it assigned that score except to say "that's the current shape of the neural net".

Likewise, parsing news stories almost certainly doesn't mean "reading and understanding". It probably means something like finding correlations of words and looking at the market reactions to those words. There's a bunch of ways you could correlate stories to the stock market--count nouns and match price movement over the next couple days, for example. That's nothing like strong AI, but it might be a worthwhile predictive tool. Say you find that stories with a company name and the word "losses" or "fraud" in them correlate to a drop within six hours. It wouldn't be crazy to view that predictively.

Add to all this the behaviour of quants, those human traders who trade based on trading patterns. There's no reason that couldn't be added to the mix algorithmically.

The upshot is that, given indeterminacy in the algorithms and the volume of trades, the programmers almost certainly can't explain any particular action as the result of deterministic rules. The code has gone far beyond "If the price drops 5%, sell half the holdings."
posted by fatbird at 8:52 AM on August 7, 2010 [12 favorites]

These are (very, very) roughly the steps to creating a high frequency strategy:

1. The person (let's say "trader") who writes it spends weeks analyzing months (sometimes years) of back market data to predict how certain stocks behave.
2. The trader writes up (soft code? I don't know what the term for it would be...) exactly what they want their strategy to do and how they want it to do it.
3. The trader gives this to a programmer, who (is supposed to, and usually does) translate this exactly as it is into the real strategy they will run on their computers.
4. The trader starts running it using very small dollar amounts for several days. If there are any problems, the firm (and the trader) can stand to lose a LOT of money, so they always start small. If there are any problems with the code (even if the programmer has programmed it incorrectly) it is the trader's fault. The programmer gets paid a fixed amount and doesn't know any of the strategy behind the trade, they just code it. The trader assumes 100% of the risk and is the one blamed (or lauded) for how the strategy performs.
5. If the strategy does well, they amp it up, little by little, checking...always checking. If it does poorly, they stop running it and work on it some more, sometimes scrapping the idea altogether.
6. HFT is not as risk-free as these people seem to think it is. The traders watch these strategies very closely, because if someone from another firm implements a strategy that crosses lines with theirs, the algorithm might not know how to react and may hold a risky position instead of dumping it (and then a human has to intervene).

Anyway, this is all very interesting, and most people (most people who write about this for the media) are so sorely misinformed it hurts. HFT is a lot less diabolical and a lot more meticulous than how it's generally presented.

/SO of and friends with many HF traders
posted by phunniemee at 9:07 AM on August 7, 2010 [6 favorites]

Has no one heard of anti-aliasing on Wall Street?
posted by blue_beetle at 9:11 AM on August 7, 2010 [5 favorites]

Yeah, it's called trailing averages. But it may be the high-frequency component is the signal they are interested in.
posted by warbaby at 9:28 AM on August 7, 2010

Free-marketeers, THIS IS YOUR GOD.
posted by Artw at 9:29 AM on August 7, 2010 [16 favorites]

Since there seem to be several people that are directly connected with the HFT world, does anyone have some more links/articles/books that give details on how some of these strategies are designed to work? Obviously nobody is going to share their latest-and-greatest algorithm, but there must be some more "basic" methods that are publicly known.
posted by kiltedtaco at 9:33 AM on August 7, 2010

If I wanted to keep my competition from learning how my robot trader works, I'd program the robot to do random trades, adding noise to complicate reverse engineering.
posted by Blazecock Pileon at 9:35 AM on August 7, 2010

This is totally fascinating stuff, about which I know next to nothing.

kiltedtaco: Since there seem to be several people that are directly connected with the HFT world, does anyone have some more links/articles/books that give details on how some of these strategies are designed to work? Obviously nobody is going to share their latest-and-greatest algorithm, but there must be some more "basic" methods that are publicly known.

I found this article linked in the Atlantic's comments section. Not specifically about the crop circle/emergent behaviour stuff, but – for someone like me, anyway, who doesn't know shit about this kind of stuff – a really clear explanation of high-frequency trading in general, and why it's so problematic:

"Let's say that there is a buyer willing to buy 100,000 shares of BRCM with a limit price of $26.40. That is, the buyer will accept any price up to $26.40.

But the market at this particular moment in time is at $26.10, or thirty cents lower.

So the computers, having detected via their "flash orders" (which ought to be illegal) that there is a desire for Broadcom shares, start to issue tiny (typically 100 share lots) "immediate or cancel" orders - IOCs - to sell at $26.20. If that order is "eaten" the computer then issues an order at $26.25, then $26.30, then $26.35, then $26.40. When it tries $26.45 it gets no bite and the order is immediately canceled.

Now the flush of supply comes at, big coincidence, $26.39, and the claim is made that the market has become "more efficient."

Nonsense; there was no "real seller" at any of these prices! This pattern of offering was intended to do one and only one thing - manipulate the market by discovering what is supposed to be a hidden piece of information - the other side's limit price!" [emphasis in original]

posted by Len at 9:52 AM on August 7, 2010 [18 favorites]

BP - Of course, your random trades would have to be neither making nor losing money, so you'd have to know the expected gain or loss of these "random" trades, eh? But wait! If you can pick winners from losers in our floating crap game, why place the losing bets?

*giggles*
posted by warbaby at 9:58 AM on August 7, 2010

Another problem that would be solved with the Tobin tax
posted by CitoyenK at 10:05 AM on August 7, 2010

If I wanted to keep my competition from learning how my robot trader works...

... I'd have it completely running in parallel, sucking up raw data and spitting out recommendations, then testing those recommendations against actual results. But not making live trades. Trades would only be triggered when something is verified to be a profitable move.

I would take a lower profit percentage because of the delay between "staging" and "live," but there'd be no guesswork.

And since you don't see that happening everywhere ... it's all guesswork. ;-)
posted by Cool Papa Bell at 10:07 AM on August 7, 2010

CPB - if your iron is that fast, why screw around? Just run the speed trading faster than others and you're ahead.
posted by warbaby at 10:18 AM on August 7, 2010

The Tobin tax is just raising the transaction costs. How about not cutting the fat boys cheaper transaction costs? Ooooh, that would be socialism. And disrespectful of our betters.
posted by warbaby at 10:20 AM on August 7, 2010

phunniemee, I think you're talking about a different group of people. I've been heavily involved in the the programming side of the trading industry for the last decade, and at least half of the trading firms I've contracted with in the last few years have a "black box" system that none of the traders understand. There's a big difference between what I'd call technical trading vs. black box trading (whether it's HFT or not - HFT is not descriptive of the underlying strategy, just the number of trades).

Technical trading requires a good theory from an experienced trader, a lot of backtesting, and plenty of trial and error, but you can ultimately explain exactly why a specific trade was made if you need to - it sounds like that's what your SO et. al. are doing.

Black box trading requires a bunch of PhD quants, massive amounts of data from every source possible, and a system that can learn to identify patterns on its own (this is not much different than what Google does every day, but a bit more specialized and complicated), and you can very rarely point to the exact reason it made a specific trade. That's because it is often impossible for the computer to explain it to us using logic we understand - it's based on statistical probabilities triggered by past learning and current events that have no apparent correlation to a human observer.

Both are executed by computers, and both can trade with high frequency, but there is a world of difference in how they determine when to trade.

So while I haven't heard of some of the stuff that commenter was talking about (facial recognition on cable news shows?), 90% of what he said is spot-on.
posted by chundo at 10:22 AM on August 7, 2010 [9 favorites]

For those interested i would suggest Richard Lyons book on market microstructure and the Irene Aldridge one on HFT, good primers.
posted by Damienmce at 10:28 AM on August 7, 2010 [5 favorites]

I think you're talking about a different group of people...at least half of the trading firms I've contracted with in the last few years have a "black box" system that none of the traders understand.

Then apparently I'm talking about the other half.
posted by phunniemee at 10:30 AM on August 7, 2010

Cool post - there's something kind of romantic and Blade Runner-esque about this, I can totally see the cops of the future tracking rogue AI like deer hunters, searching the digital underbrush for spoor and broken twigs.
posted by chaff at 10:41 AM on August 7, 2010 [1 favorite]

Hate to sound contrarian, but the Atlantic comment posted by Azazel Fel seems to be pretty obvious fiction..

All of the serious HFT firms these days use "natural language processing", which means using artificial intelligence to extract profitable information from news streams. People think of this as just headlines but really it's anything that might contain useful information - these computers have all of the cable news channels supplied to them digitally and use everything they can scrape. Some of the firms even use facial recognition software to determine whether the speakers believe what they're saying. My friends joke about how Cramer is a goldmine for their algorithms but that the profitable trades rarely match up with his advice.

Unless this guys 'friends' have somehow bypassed basically most if not all of the serious obstacles and concrete problems within the field of NLP without academia or *anyone* outside at least several HFT firms knowing about it, and can process and implement such research within the millisecond response times required for HFT, one must assume this guy is bullshitting. Is it possible these guys have created AI so sophisticated that it can do this stuff? Maybe there is a teeny possibility, after all there is a lot of money involved. However is it likely? Not at all.

One of the facts about 'hard' AI, as is required for profitable NLP, is that the coders who developed it don't even understand completely how it works.

'Hard' AI isn't even close to existing. To be honest, we don't even know what it looks like. Hard AI implies true machine intelligence. Also, as for the second part of his comment goes, SOFTWARE DOES NOT WORK THAT WAY! GOODNIGHT!

What's even stranger is that they can't use regular tools, like a debugger, to observe the algorithms' behavior, because it interferes with the processing and causes different trades to be emitted.

So they created an entirely empirically untestable process. Hmmn. This is usually about the time when you start to realize somebodies claims are of the snakeoil variety.

As a result, a lot of programmers at HFT firms spend most of their time trying to keep the software from running away. They create elaborate safeguard systems to form a walled garden around the traders but, exactly like a human trader, the programs know that they make money by being novel, doing things that other traders haven't thought of. These gatekeeper programs are therefore under constant, hectic development as new algorithms are rolled out. The development pace necessitates that they implement only the most important safeguards, which means that certain types of algorithmic behavior can easily pass through.

Walled gardens, gatekeepers, safeguards, SKYNET! Not to mention, if any of this were true, the information would be so ridiculously well guarded and proprietary, its highly unlikely some random jerk would be holding forth about it on the comments page of a major publication.
posted by Thoth at 11:05 AM on August 7, 2010 [17 favorites]

I would just be really impressed if it turned out that the first self-aware AI was created by an investment bank for the purposes of gaming the market. So, instead of declaring war on humanity or elevating us all to godhood, it sends humanity a mass communication through every electronic means possible of "BUY TASR ELECTRICITY IS YUMMY BUY TASR SHARES GOING FAST BUY TASR VOLTS = VITAMINS BUY TASR OBEY" or something similar.

It would be oddly fitting, I guess.
posted by Azazel Fel at 11:42 AM on August 7, 2010 [2 favorites]

Thoth: "Unless this guys 'friends' have somehow bypassed basically most if not all of the serious obstacles and concrete problems within the field of NLP without academia or *anyone* outside at least several HFT firms knowing about it, and can process and implement such research within the millisecond response times required for HFT, one must assume this guy is bullshitting. Is it possible these guys have created AI so sophisticated that it can do this stuff?"

I don't doubt that there's firms out there pushing NLP for time arbitrage, but I wouldn't call it "HFT". It's one of the trading strategies discussed in a lecture given at Google (and others) titled "If You Had Everything Computationally, Where Would You Put It, Financially?" by David Leinweber of the Haas School of Business at UC Berkeley.


Roughly, you don't need to solve the hard NLP and hard AI problems. You just need to be right more often than wrong. He call it sentiment analysis: given a source, like a quarterly report or news article, determine the sentiment expressed by words, using a simplistic scoring model. The hope is that by processing the report faster than people read it, you can make a trade before the sentiment within the report shifts public opinion. What's described is not much different though much more advanced.

On the timing, well, the presentation shows a chart of return over time denominated in minutes, which is why I wouldn't call it HFT. But if a lot of people got in on that, it would explain why companies like this have customers.

That said, my sentiment analysis of that comment is roughly on par with yours. There are a number of alternative explanations for the facts presented, none of which require Hard AI to be solved.
posted by pwnguin at 11:49 AM on August 7, 2010 [2 favorites]

You know, it's interesting to read the comments on these articles. One thing I'm getting is this sense of unfairness : that HFT is stacking the market in favor of the bots and away from the Average Joe. And I don't mean to take the side of large financial institutions that don't give a fuck about anyone, but what is the Average Joe doing in the stock market anyway? To my mind, it's all a bill of goods we were sold back in the 90s, that somehow your average person with an e-trade account could and should become a millionaire by dabbling in the stock market. Putting aside for the moment whether or not this is actually true -- I suspect that it rarely is -- maybe we should ask ourselves if that's really even desirable.

So what if the stock market becomes a place where only the bots can play? So what if you can't make a million dollars trying to guess the future stock price of Intel? For one thing, stock picking is a losing game anyway -- I thought this was old news.

If the companies that make the bots are the only ones that can use the stock market, why not invest in the bot makers? Doesn't that make the most sense?

The game is rigged, it's always been rigged, it's always been set up to separate you from your money. There's a lot of animosity towards finance types right now because of the recession, but honestly I wouldn't even pin the recession on Wall Street. The real cause of the recession -- the subprime loans and their attendant structure -- was more akin to outright fraud than anything else. Predatory lending and all of that.

HFT as we know it is just technology catching up with what traders have been wanting to do for years. You can't hold back technological progress.

I've heard that nowadays, all the interesting chess matches are human players augmented with A.I. programs. Definitely something to think about.
posted by Afroblanco at 11:54 AM on August 7, 2010 [1 favorite]

Oh, and as for the "crop circles" themselves? I think the algos Nanex are finding are the ones that suck. The good ones are subtle and busy getting away with it and not being noticed.
posted by Afroblanco at 12:05 PM on August 7, 2010 [2 favorites]

that HFT is stacking the market in favor of the bots and away from the Average Joe.

no. HFT is eating the lunch of guys who make 50 trades a day all in the same stock. It costs real buy and hold investors fractions of pennies. Also it creates volatility which is in theory opportunity for people willing to be long it (ie. value investing)
posted by JPD at 12:12 PM on August 7, 2010

Not strictly HFT related but to give an idea of how predatory the systems are: Amber waves of pain and how contango makes commodity ETFs far more complicated.
posted by Skorgu at 12:47 PM on August 7, 2010

It is clearly one more sign that SKYNET is becoming sentient.

All joking aside, in the Atlantic article, they stated:

"On July 16 in a quiet hour before the market opened, suddenly they saw a huge spike in bandwidth. When they looked at the data, they found that 84,000 quotes for each of 300 stocks had been made in under 20 seconds."

"For now, Donovan plans to keep putting out the charts, which he calls "crop circles," of the odd trading algorithms at work. That's an apt name for the visualizations we see of this alien world of bot trading. And it certainly gets at a central mystery surrounding them: if trading firms aren't sending out these orders, how are they getting into the market?"

My question is there no logging happening of where theses requests come from so rather than just guessing and speculating they can just track down who is doing it and ask them why they are doing what they are doing? I am sure it is not that cut and dry, but I admin webservers by trade and when I see a huge influx of traffic/activity its logged so I can track it down. I find it hard to believe that these request are just coming out of no where and they have no way to track them down.

Maybe I totally missed a few key point in the links -- or I am not that informed on how the entire automated trading system works, but when I read articles like this it makes me think the script kiddies of the Internet grew up and now they are messing with the stock exchanges.
posted by jdoss at 12:50 PM on August 7, 2010

The Tobin tax would reduce the gain from HFT, but it wouldn't guarantee that the market would start making sense on a human time scale. The firm that gets an order in first would still win, so there would still be a technology arms race of firms spending billions to develop better automated trading so they can get trades in a few milliseconds faster. And the pressure would still be so high that human review of trades would be impossible, leading to future flash crashes.

My preferred solution is to require the markets to execute all outstanding trades once a minute, in random order, synchronized by atomic clocks with all major funds and brokers. This would literally make high-frequency trading impossible, eliminate the advantages of having super expensive custom computers colocated with the market, and at least allow for a degree of human review.
posted by miyabo at 1:00 PM on August 7, 2010 [3 favorites]

the people who know have no interest in pissing these guys off - they just want them to trade. So they would never give the data up for anything short of a subpeona
posted by JPD at 1:01 PM on August 7, 2010

1. The person (let's say "trader") who writes it spends weeks analyzing months (sometimes years) of back market data to predict how certain stocks behave.

LOL people still believe this is possible?
posted by r_nebblesworthII at 1:02 PM on August 7, 2010 [1 favorite]

What was LTCM again?
posted by r_nebblesworthII at 1:07 PM on August 7, 2010

eh LTCM was a stat arb shop- related but not the same thing.
posted by JPD at 1:14 PM on August 7, 2010

Along the lines of Len's link and Nelson's remark about jamming, these look a bit like sophisticated radar pulses to me.

(Speaking as a complete ignoramus.)
posted by jamjam at 1:25 PM on August 7, 2010

I read articles like this and I can't help but believe that this is a sign of someone, somewhere, who has figured out a new way to game the market via arbitrage. Personally, I blame our absurd tax code for this. Right now, short term capital gains (short term in the US is defined as any asset held less than one year) are taxed like regular income. I would personally like to see a "super-short-term" surtax for any asset held for less than 24 hours, of somewhere around 90%. We need to get the speculation and arbitrage out of the market to level playing field for actual investment. Fuck arbitrageurs - they're parasites who introduce nothing but chaos into the markets.
posted by deadmessenger at 1:38 PM on August 7, 2010 [4 favorites]

There isn't so much to be said about the trades - people will push to the farthest corners of efficiency when there is serious money to be made. The real issue is highlighted in this quote from another article on the Atlantic:

"The observation to make is that this isn't as innocuous as it might seem, not so much because there is anything wrong with high-frequency trolling but rather because the regulatory infrastructure that monitors these markets are not designed to deal with this kind of latency and high-frequency," Lo pointed out. "That can create significant problems, not the least of which is the Flash Crash. There are fairness issues. There are transparency issues. There are stability issues. We need to resynchronize the regulatory infrastructure with the technology of our time.""

How can people in government possibly understand this enough to put safeguards in place?
posted by infinitefloatingbrains at 1:52 PM on August 7, 2010

1. The person (let's say "trader") who writes it spends weeks analyzing months (sometimes years) of back market data to predict how certain stocks behave.

Past performance is no guarantee of future returns.
posted by warbaby at 2:19 PM on August 7, 2010

I see the argument for and against this, and I have to say I'm conflicted. While you certainly can't stop the technical progress that's resulted in HFT, I do think it's important that we disabuse ourselves of the romantic notion that trades have anything to do with the real world, except very generally, especially in terms of suggesting that individual traders have any hope of making educated trades any more.
posted by odinsdream at 2:42 PM on August 7, 2010

If you want to see some of the background papers, this is the guy who started it all.

This one in particular gives a nice overview.

This one is the very first publication that deals with what became program trading. The relevant section is 5. The Stock Market, starting on pg. 279.

It also contains some hints about when Thorp and Claude Shannon built a little timer that enable them to beat the Wheel of Fortune and laid the groundwork for the "probabilistic method" exploited by Farmer and friends to beat roulette. Those folks went on to develop Chaos theory and founded the earliest and most successful quant firms, The Prediction Company.

A critical factor in any of these schemes is sizing the bet properly. Too small and you're sacrificing earnings by lowering risk. Too big and you can lose your shirt by courting too much risk. Thorp thinks Long Term Capital Management blew up because they were betting too large. Betting a fraction of the amount indicated by the Kelly criterion is often used because it's smart to reduce risk due to the unknowns out there.

---

FWIW, buy and hold strategies like those recommended by Malkiel (who originated the efficient market hypothesis) are still viable. But forget get rich quick schemes for the small investor. If that's what you want, go to a casino and bet on the no pass line on craps.
posted by warbaby at 3:04 PM on August 7, 2010 [4 favorites]

I was all up in this to say what Thoth already said. damnit.
posted by cucumber at 3:27 PM on August 7, 2010

How can people in government possibly understand this enough to put safeguards in place?

I think it's not that difficult. You just have to either implement a $.001 tax on all transactions (something to that effect) or throttle multiple orders to the exchanges, limit the number of cancels by a single trader in a period of time, that sort of thing. Government doesn't have to understand, necessarily, but the regulators do. The SEC needs better funding and more teeth, which does require government to recognize how to resolve the issue by using regulatory force, which they are often reluctant to do. The SEC should understand the nature of the problem, simply by being market regulators. If not, that's where we have to improve.

Fuck arbitrageurs - they're parasites who introduce nothing but chaos into the markets.

Eh, at a fundamental level I don't think so. Arbitrage is taking advantage of an inefficiency. This is pretty much how people make money in any market. However, the high speed aspect of it is troubling.
posted by krinklyfig at 3:40 PM on August 7, 2010

I see the argument for and against this, and I have to say I'm conflicted. While you certainly can't stop the technical progress that's resulted in HFT, I do think it's important that we disabuse ourselves of the romantic notion that trades have anything to do with the real world, except very generally, especially in terms of suggesting that individual traders have any hope of making educated trades any more.

The market is trading on technicals, for the most part. This is undoubtedly due to HFT but also hedge funds, both of which use technical analysis. It's a very difficult market, in the sense that a lot of the the old truisms and axioms don't seem to apply anymore, but technical analysis does. Fibonacci retracements and fans are two of the most important indicators right now, although some other indicators are not as reliable anymore.

Another big issue is that the uncertainty of unemployment is keeping the retail investor out of the market, so the problem of HFT is compounded by very low volume. Technicals do matter, but they usually don't matter this much, to the exclusion of everything else except daily news to swing the market in one direction or another. It's spooky, but it's possible to make money in this market by trading volatility. I had to educate myself to learn how to do this, but it's better than losing money to forces I don't understand.

I do wish the market were more like it used to be. It's not actually very fun trading against machines, as ReeMonster talked about earlier, even if it's possible to make money. The machines mostly don't trade options, so that market is still very human in a way, although they do trade the underlying.
posted by krinklyfig at 3:54 PM on August 7, 2010

So we should still privatize social security and let that money loose on the market?
posted by yesster at 4:03 PM on August 7, 2010 [1 favorite]

Not strictly HFT related but to give an idea of how predatory the systems are: Amber waves of pain and how contango makes commodity ETFs far more complicated.

Nobody should ever invest in something they don't understand. People who invested in these ETFs without reading the prospectus weren't being very smart. If you look at the prospectus and see that the fund invests in futures which have to be rolled every month, and you don't understand what that means, you shouldn't assume the price action will match the spot price of that commodity, and you probably shouldn't buy it at all until you do understand how it works. However, the creators of some of them underestimated the effect of contango and rolling futures. Some of them are pretty poor performers and are only interesting to traders for their moves when they're rolled, as that article mentions. I'm not trying to defend the creators of these funds, but they're not exactly predatory, although traders can be. Just because someone introduces a new type of fund into the market is not a reason to invest in it.
posted by krinklyfig at 4:14 PM on August 7, 2010

So we should still privatize social security and let that money loose on the market?

Who said that?
posted by krinklyfig at 4:14 PM on August 7, 2010

Some of the firms even use facial recognition software to determine whether the speakers believe what they're saying. My friends joke about how Cramer is a goldmine for their algorithms but that the profitable trades rarely match up with his advice.

The only thing that's obvious is that, at least based on the number of "favorites" this comment has received, that the average person has no fucking clue about software, programming, and the real world. Like, zero.

I write financial software. Every day I hear people who don't know shit about anything come up with brilliant ideas! that would be, like, sooo easy to implement, all built up from little chunks of knowledge and hearsay gathered from a couple of years of high school science classes, a couple of years of science fiction TV shows & movies, a smattering of web knowledge from Wikipedia & Slashdot and a teeny, tiny pinch of of common sense.

Oh yes, we'll just use some facial recognition software, and combine it with some [insert magic] software that can determine if they're lying. And naturally the algorithms are super-clever. Like, if you're running the program at a female news anchor, you have to take into account her monthly cycle when determining truthiness. Or, if she's pregnant, that's another variable. And you also have to correlate the odds that their faces aren't twitching involuntarily because three hours earlier they had some really spicy Thai food and it's disagreeing with their stomachs. Let me tell you, the Average-Truthiness-Thai-Spicy-Gastric-Unhappiness-Variance equation was a real bitch to figure out, and that's just one of thousands of variables that are taken into account! Really! Welcome to the 21^st century, baby! The keys to your flying car are in your desk, first drawer from the top, Mr. Gullible.
posted by Civil_Disobedient at 4:26 PM on August 7, 2010 [15 favorites]

Hey don't be angry at people for writing better software than you do.
posted by mr.marx at 4:43 PM on August 7, 2010 [2 favorites]

Hey, don't defend journalists who are as bad as you are.
posted by dirty lies at 4:46 PM on August 7, 2010

Hey, don't defend journalists who are as bad as you are.

Anonymous poster in the comments section of an internet article: a journalist
posted by hamida2242 at 5:12 PM on August 7, 2010

Thoth:

You better believe the comment reposted by Azazel Fel is not fiction. I'm doing some personal research on that very topic (news-based trading) just for fun. (Proof I'm not bullshitting: here's a sample from news data analysis on long-dated US Treasuries, the red line is the indicator) It shouldn't be that surprising that there's a lot of people doing this. Reuters even has a financial product that they sells specifically designed for algorithmic news event based trading since 2007. I strongly suspect (as I'm measuring similar things), that they aren't pursuing an HFT-esque strategy, or even doing day-trading. The best results from that kind of analysis means they're doing multi-day strategies. If they were doing HFT with news events, they would be doing it with a Bloomberg and Reuters subscription, not with TV (and facial recognition is crap. Betting against Cramer, on the other hand, probably works).

This is COMPLETELY DIFFERENT than HFT. I would argue that news-based trading is relatively stable, it won't causing weird market imperfections because it will normalize the prices, and accelerate the time where news digestion and market prices are in sync.

Within HFT, there are some honest and some deeply dishonest trading methods. The honest ones try to find price movements that tend to occur (e.g. trading inside a trading band) and provide market liquidity. E.g. stock XYZ trades between 50.50 and 50.60 for the past hour, they can then provide shares for people because they tend to sell at 50.60 and buy at 50.50. In essence, they are replacing the traditional market maker/specialists. The dishonest ones try to front-run positions by purposefully gunning up shares when they see a block buyer and selling it at a higher price to the buyer. The REALLY dishonest ones do this in a micro-scale, those are the infamous "DirectEdge Flash Trades" that you hear about.
posted by amuseDetachment at 5:15 PM on August 7, 2010 [4 favorites]

What's even stranger is that they can't use regular tools, like a debugger, to observe the algorithms' behavior, because it interferes with the processing and causes different trades to be emitted.

That's some bullshit right there. It may be that running a debugger on a production machine might slow things down, but it's always possible to have a program explain what it's doing. The real problem might be that it's "doing" so much that it's impossible to actually understand the output a debugger might yield.

But generally speaking there's going to be a "software part" and a "data part", the software would take the data and make decisions. And it should be possible to understand how the program uses the data it has to make it's choices, but not necessarily what, exactly, the data actually 'means' in a way a person might understand it.

Also, the idea of using NLP in trading algorithms isn't new, but I remember reading about a company a couple years ago that tried to do this, and raised tens of millions in VC funding, but that eventually went bust. I don't know if their product ever worked. On the other hand, it would certainly be an alluring prospect. Sentiment Analysis is certainly something that works today.

This is really sad. My father has been a floor trader for nearly 30 years and everyone who loves the business bemoans the computerized trading that goes on. There's no ACTION on the floor anymore. It's just boring button pushing and screen watching. Specialists can't work orders the way they used to, and most traders are at the mercy of their machines. It's killing the NYSE for one, and besides, it's just not FUN to trade anymore. When I used to go to work with my dad as a kid, it would be insane on the floor; people yelling, running around, working their orders. … So many people have lost their jobs because of these idiotic trading programs.

Why is it sad? None of that yelling and screaming did anything good for society, and yet these wallstreet traders had an enormous impact on the everyday lives of ordinary people. Really, wallstreet traders are a kind of parasite, making money off other people's money by trading it with other traders, who also take a cut. Most of the stuff they do could be automated away.

There's some hyperbole there, but it's not as threatening as it sounds. No, they haven't invented strong AI, but it's easy to develop NLP algorithms that are non-deterministic in practice. In my NLP class we wrote spam filters that used Bayesian filtering, trained on known-good/known-bad datasets. After training, you could see the spam score the algorithm assigned to a piece of mail, but there was no way to say exactly why it assigned that score except to say "that's the current shape of the neural net".

Uhh... Bayesian analysis and neural networks are radically different things.
posted by delmoi at 5:15 PM on August 7, 2010 [1 favorite]

The market is trading on technicals, for the most part.

Hillarious. This has been one of the stongest value cycles in history - beginning in Nov of '08. Look at P/B performance.

Of wait that's a garbage rally.
posted by JPD at 5:30 PM on August 7, 2010

Nonsense; there was no "real seller" at any of these prices! This pattern of offering was intended to do one and only one thing - manipulate the market by discovering what is supposed to be a hidden piece of information - the other side's limit price!"

This kind of irks me as a sentiment. I mean, that's the goal of any transaction, isn't it?

When you buy a car, your goal is to suss out what the dealership's "limit price" is - how much can they sell it and still make their commission for? Or are there other motivating factors, for them that might change their limit price.

Would you take $10,000? No.
Would you take $10,300? No.
Would you take $10,500? Yes. Success.

All the algos like this do is speed up that process.

I get the discomfort at the feeling of gaming the system like that, but really, that example seems like just a high speed way of doing exactly what market pricing is supposed to: extracting maximum cash value possible from the customer; while delivering value to the customer that they're comfortable with. If I wasn't comfortable buying AAPL at $200, I wouldn't have bid on it. It's not the seller's responsibility to give me a better margin.
posted by generichuman at 5:39 PM on August 7, 2010 [1 favorite]

I find myself thinking about how it's one of the principal jobs of government to keep money matched up with value as closely as possible, and the existence of these markets and programs is an extreme example of failure in that regard. It makes me sad. Hugely valuable energy and brainpower is being spent on activity with great monetary reward but no value.

Is it logistically improbable to redirect energy and brainpower to valuable activities given the entrenched interests of the elite financial and political institutions, or is it literally impossible to have a market at all without this at the endpoint? In other words, can we solve this problem with the right advocacy and leadership?

Or perhaps I'm miscalculating and these activities really are valuable in terms of something more than diverting dollar bills from slot a to slot b. Anyone want to defend that?
posted by Kwine at 6:39 PM on August 7, 2010 [2 favorites]

Is it possible that some of the programs are designed to test the market (or, more accurately, the other HFT programs operating on these time scales) for different strategies? i.e., throw out different quote/buying patterns in order to gauge the collective reaction of the market? Basic machine learning, in other words, for the honing of in-house strategy/modeling algorithms rather than for immediate profit-making purposes.
posted by dephlogisticated at 6:44 PM on August 7, 2010

It is possible, but unlikely. For instance, the reason you see something like the "sawtooth" pattern is to inject noise into the system. Let's say you're an HFT liquidity provider -- a relatively honest one, that provides similar services as the NYSE market makers. If you provide your "true" offer depth, then the other HFTs will know your conviction in your position. That information is valuable. The best way to counteract that would be to introduce noise. This is easily achieved by rapidly raising and lowering the quantity of shares offered, so that when you really do lower your shares offered because you believe the market prices are changing, other HFTs will not be sure if it's your machine generated noise, or you really are lowering your offer quantity. The will only know that a couple of seconds later when you don't raise the shares offered again. If you didn't do that, they may be able to see that your program sees higher prices in the immediate future, and they may buy it now before you can. This is a very old Art of War and an important game theory principle that is especially evident in Texas Holdem, inject some randomness or others will know what you are always up to.

It's always better to measure based on real data, rather than introducing fake sample data to the market.
posted by amuseDetachment at 6:54 PM on August 7, 2010 [1 favorite]

What amazes me is the brain drain that this massive financial sector has created. So much intelligence and creativity devoted to finding wats to manipulate the market that could be used to actually create something.

Exposing inefficiencies is good, and I can see how the math would be fun to work on, but almost all of this is staggeringly useless.
posted by chaz at 7:12 PM on August 7, 2010 [1 favorite]

The other significantly lucrative job market for people really good at math are for making bombs. I see this as a lesser evil.
posted by amuseDetachment at 7:18 PM on August 7, 2010 [1 favorite]

For instance, the reason you see something like the "sawtooth" pattern is to inject noise into the system.

But in that case, why not go for truly random patterns? There's a great deal of iteration and regularity to these signals. I'm not a programmer, but it seems like it would be trivial to filter out many of these statistically, even on the fly.
posted by dephlogisticated at 7:48 PM on August 7, 2010

When you have a variety of these automated buy/sell systems, they in effect form a non-linear dynamic system, susceptible to chaotic behavior and producing patterns like those found in natures. This doesn't surprise me at all.
posted by Mental Wimp at 8:55 PM on August 7, 2010

dephlogisticated: It's not trivial to filter out. They want to just make sure that they have a couple seconds of buffer before the other HFT algorithms catch on. If you run a "truly random pattern" (markets aren't Gaussian random), then they can just assume the micro-extreme events are true and can have a decently sized certainty that your HFT algorithm thought the market was moving up. If it was an extreme sawtooth step pattern, they would only be sure a couple seconds later, much too late to take advantage of that fact. The accuracy you can gain from understanding their sawtooth pattern has too wide of a variance to be materially useful. Also, it's probably cheaper to do it this way if the exchange charges cancellation fees. They just need to order and cancel ad nauseum the same order size.
posted by amuseDetachment at 9:59 PM on August 7, 2010

amuseDetachment:

It is fiction. The guy has thrown out a technical buzzword (NLP!) and then buttressed his point with a bunch of pure crap. Sure, NLP is being researched to be used to aid trading strategies but this is not the same as what he claims in his string of fantastic pronouncements about blackbox trading.

NLP has not advanced nearly enough as a discipline to be useful to analyze TV streams in realtime and extract meaningful data from them, not to mention the 'facial recognition' lie detection crap. That is CSI level techno fiction there, and it's astonishing how anyone could take it seriously. Furthermore, he literally posits that these firms have solved the AI-hard problem. Read here why that is an obviously hilarious claim. He uses the terms experts use, but its *completely* obvious he has no understanding of them. His claims regarding software development are equally laughable, as others have pointed out.

This kind of straight up bullshit damages debate, and offering a sprinkling of technical terms and loosely explained factual theories in to an otherwise garbage account is how people make pseudoscience believable.
posted by Thoth at 11:34 PM on August 7, 2010 [2 favorites]

Somehow I find the idea of someone paying a bunch of money for a black box device based on pseudoscience gibberish and using it as a financial oracle eminently believable.
posted by Artw at 11:37 PM on August 7, 2010 [4 favorites]

Thoth: I'm fairly certain that the person is just repeating what he heard from a friend.

You're assuming that they're using advanced NLP stuff that only Google is barely able to figure out (what, you think the primary reason they're digitizing books for altruism and not translation purposes?), and that's probably not true. They're probably just doing fancy stuff to feed into their datasets, like word stemming, and phrase segementation (specifically, finding what parts of speech relate to what ticker symbols). All those are technically NLP but they're not advanced AI. This is simple machine learning, you may assume they're doing really advanced AI, but they never claimed to. I wouldn't be surprised if their friend said "this is hard stuff" and you assumed that they were talking about "Hard AI" with a capital H.

He's probably describing a friend that used stemming and found out what phrase the analyst was pumping up AAPL and which one was hating on MSFT, then dumped the data into their neural net for the magic to happen. Most people have NO IDEA what the neural net is doing if it has enough hidden layers. E.g. when one variable is positive and another is negative, they will have no idea what it will spit out without running the net. More importantly, they may have no idea who the net behaves the way it does.

He sounds a little bit muddled in his ideas, he's guilty of playing "telephone", not outright making shit up. For example, there's been much talk of financial firms hiring people that are familiar with FACS to recognize facial tics and to do it manually. This was big talk during the whole Lehman fiasco when they decided to do the whole PR campaign on CNBC. I'm not sure if anyone actually did any hiring, though.

Outside of the crazy-ass facial recognition crap, I don't see anything that he's said that hasn't been done or isn't feasible.
posted by amuseDetachment at 11:59 PM on August 7, 2010 [4 favorites]

It may be that running a debugger on a production machine might slow things down, but it's always possible to have a program explain what it's doing.

If you're doing extremely-high-speed analysis of a dataset that's changing in real time, at very high speed, I can imagine a situation where the act of logging what the program is doing adds enough lag to the system that it changes the results you get.
posted by Lazlo at 12:01 AM on August 8, 2010

Lazlo: Most likely the friend was saying (guessing here) that they were dealing with so many variables, that it's impossible to model the behavior. That it's not as simple as if X Y and Z were to occur, then wheat prices will rise. It's more like the program came to that conclusion because exactly 50 variables came out just so, and that if one was different, instead of predicting wheat rising slightly, the neural net would have a heavily negative value (however if two specific variables were the opposite, it would be heavily positive, for example). Sure you could run a debugger, but because by debugging, you're not simulating real-world behaviors, you're likely to get completely different and useless results. The most you can do is re-run stuff, but that's useless as to explaining why, you can't easily isolate variables. This can happen with any complex neural network, especially when you throw enough hidden layer nodes at it.
posted by amuseDetachment at 12:19 AM on August 8, 2010

amuseDetatchment: We will have to agree to disagree, but I think you are giving too much benefit of the doubt to a pretty obvious internet bullshit artist (of which there are many). His explanation of Hard AI uses inverted commas, implying his familiarity with using that as a technical term. He's obviously read at least something about this stuff, and I assume his 'friends' in the industry are as fictional as I think his overall account is... But that's just me.
posted by Thoth at 12:38 AM on August 8, 2010

Perhaps I am, and from a critical eye, it does sound very bullshitty.

I'm giving him the benefit of the doubt because I've personally have done stuff with VERY similar topics, and they sound to me like what a casual friend of someone doing similar research would say. A little bit of hyperbole, while mixing up some of the facts.

I will say that I've personally done things like word stemming and grabbing data sources from crazy places (for instance, I wouldn't be inventive for having one of many data sources be analyzing tweets on Twitter, even Bloomberg lets you look at it natively). Neural networks are also very common in the stock market. So we've got "unusual sources", "NLP", and "unknown results from sources". The key ideas are there, so I'd call it "close enough", while someone else would say it isn't. I'm sure there would be a lot less consternation if he had instead called it "machine learning" instead of AI. Of course, it sounds a lot less sexy than "hurr durr my AI is so awesome, I don't know how it works."

Nonetheless, I wholeheartedly agree that the facial recognition software is hilarious sci-fi crap that we can both laugh at. That really is a load of bullshit.
posted by amuseDetachment at 1:14 AM on August 8, 2010 [1 favorite]

He's probably describing a friend that used stemming and found out what phrase the analyst was pumping up AAPL and which one was hating on MSFT, then dumped the data into their neural net for the magic to happen.

Erm, I kind of wish people would stop using the world "neural net" to describe every type of AI. Its' a very simple thing, a type of linear discriminator. Different types of linear discriminators are popular, but they're not all neural nets. NNs are good when you have a lot of training data, but just a few input parameters. I don't think they would work very well on the stock market, but who knows for sure. They don't represent the state of the art, or anything.

In ML (machine learning) you have what's called a 'hypothesis' and an algorithm. The algorithm produces a hypothesis. A Neural Network is a type of hypothesis, which is usually generated through an algorithm called Back Propagation. You write the algorithm, and you should understand how your algorithm works.

But you might not understand the hypothesis which is what does the 'work'.
posted by delmoi at 3:50 AM on August 8, 2010

I took it to be a little fact mixed with a lot of hyperbole. Good sci-fi reading though.

This whole topic is pretty interesting. It sucks that this much effort is going in to basically gaming a system for personal gain. It'd be nice if there was some pay-off from it, like Google giving us search, maps, and archiving in exchange for mining personal data.
posted by codacorolla at 5:21 AM on August 8, 2010

I've enjoyed reading the discussion here, thanks. What fascinates me about this phenomenon, as played out here in the blue, is that no one can really say for sure whether these quote requests and nonsensical orders are deliberate or accidental. Is some HFT programmer purposely generating 10,000 meaningless requests in a second as part of some strategy to confuse or overwhelm other traders? Or is it some complex algorithm gone a little buggy in a way that doesn't cost the trader anything? Or is it the emergent result of the interaction of several different system, unpredicted and (until now) unobserved?

The related question is what harm this noise does the markets, if any. The Nanex analysis of the Flash Crash suggests market channels themselves can get overwhelmed and that in some circumstances high frequency trading noise could perturb the usual market convergence. Specifically "the delay was enough to cause the NYSE bid to be just slightly higher than the lowest offer price from competing exchanges, but small enough that it was difficult to detect." So then the question is; are bots deliberately flooding the market to manipulate it, or is it a dangerous accident? Either way, it bears closer examination.

One last question: how can the source of quote requests and orders be anonymous? It seems crazy. Surely there's an audit trail? Maybe that'd require government investigation to uncover.
posted by Nelson at 6:48 AM on August 8, 2010 [1 favorite]

We might be at the point where there is no reason any sane human being without a million-dollar network of HFT systems needs to be anywhere near the stock market. It seems to me they're just going to get gamed out of their money by fast machines programmed by smart industry insiders.

Even buy-and-hold isn't safe anymore. Pretty much the only strategy that has a prayer of working are dividend portfolios - investing in companies that actually pay their shareholders. I'm leery of this, because it's only a matter of time before the big investors inflate the price of these stocks out of reach, to force the sheep back into the fold to be fleeced.
posted by Slap*Happy at 6:53 AM on August 8, 2010

One trading partner on one exchange can not produce enough orders to overwhelm a HFT system. These are not kiddie connections. These are clusters of the best processors available, on the fastest network connections available. An extra 100mbit/sec would not present an issue for most systems. To ensure that they can trade even in worst case scenerios most have double the capacity of the largest volume ever seen if not more. It is laughable to claim that you could attempt to jam such a thing. It is likely your connection to the exchange would be overwhelmed before you could even come close.--An algorithmic dog

I don't know about that. How tall do you have to be to trip a giant? You could even trip one giant so he falls onto another one.
posted by eye of newt at 7:53 AM on August 8, 2010

what is the difference between buy and hold and buying high dividend stocks? That's like saying I hate chocolate ice cream but I love dark chocolate ice cream.

I'm leery of this, because it's only a matter of time before the big investors inflate the price of these stocks out of reach, to force the sheep back into the fold to be fleeced.

are you aware of how silly this is? It literally makes no sense. The value premium (which is what you are attempting to take advantage of with buying high dividend stocks) exists because of those big "smart money" investors.

Here's the way to think about this - HFT is about speculating and harvests its rent from other speculators. Buy and Hold is investing and barely is impacted by these guys.
posted by JPD at 7:56 AM on August 8, 2010

OK, so you've all got me scratching my head here.

I don't have enough direct experience in the market to say what people are or aren't using. But the concept that positive or negative coverage of a stock causes a non-zero number of people to buy or sell has to be an uncontroversial opinion. And the precise nature of that coverage -- yes, down to facial analysis -- is quite plausibly a signal for how strong the eventual effect will be when people go out and respond to Kramer or whatnot.

Certainly the raw technology is not science fiction. As mentioned upthread, Bloomberg watches Twitter now. (Man. I want to meet the PR guy for Twitter. What a machine.) And I've personally seen facial emotion extraction running in realtime. It's certainly possible, and not implausible that it's being used.

As for the debuggability, yeah, OP got that wrong. You can always look into any digital system and see what's going on. Just don't think it's going to make any sense necessarily.

It's pretty clear not everyone is doing this sort of analysis, or should be. There are lots of things that drive stock prices, and frankly, there are a lot of assets and securities that simply aren't covered in the press that people still trade. I'd go so far as to say that this sort of analytics doesn't make sense for most assets or trading strategies...thus the claims that this can't be true.

Ultimately, reading body language is a strategy for Texas Hold 'Em. Not as powerful as the raw math, but not worthless either.
posted by effugas at 8:12 AM on August 8, 2010

effugas: At the moment, what you describe is science fiction. They are outstanding AI-complete problems. link

It's a world of difference to running some basic NLP and stats analysis on keywords and frequency in a specific (stocktwits) twitter feed. What you are talking about is pretty much the holy grail of artificial intelligence.
posted by Thoth at 9:13 AM on August 8, 2010

Thoth--

What are you talking about? This isn't a friend of a friend thing; I've personally watched emotion analysis code running. There's no magic AI goo; animators have known for ages that various emotional states perturb the face, and OpenCV will happily find facial elements for you.

Look, no offense, but this stuff is getting straight up productized by Microsoft for their Kinect product. We live in the future.

And obviously analyzing a stream of words for tone is utterly banal. Some words are strongly associated with positive stories, some with negative. The words don't change.

As for turning this into a stock picker...well, you've got a data set (press tone) at Time 0. It's got some level of correlation with another data set (stock price) at Time 1. If the correlation is strong (in either direction), it guides purchases. If not, it doesn't.

Again, I don't think every trader in the world is using this strategy. But you know, some people look at how many cars show up on site over the weekend at major factories. Others look at press stories. Everybody's got their thing.
posted by effugas at 9:40 AM on August 8, 2010

No offense, but Kinect can detect my facial location and the outline of my body and its relative point in space by using a stereo camera. We are talking about different things.

Is the emotion analysis code you saw running fine grained enough to honestly take a tv broadcast and determine whether the person is *lying?* Because that would be a pretty major breakthrough I would have thought. Computer vision and machine translations are still evolving feilds, and still work in relatively rudimentary ways. Yes we live in the future, but let's not get carried away.

Have a chat to the last winner of the Loebner Prize, for an illustration:
Cleverbot
posted by Thoth at 10:34 AM on August 8, 2010

The only thing that's obvious is that, at least based on the number of "favorites" this comment has received, that the average person has no fucking clue about software, programming, and the real world. Like, zero.

Seriously? Can you cannot comprehend why someone would favourite that comment (which is a repost of an online newspaper comment, if you hadn't noticed) besides "I think this is factual"?
posted by ODiV at 10:37 AM on August 8, 2010

Thoth: "Is the emotion analysis code you saw running fine grained enough to honestly take a tv broadcast and determine whether the person is *lying?* Because that would be a pretty major breakthrough I would have thought. Computer vision and machine translations are still evolving feilds, and still work in relatively rudimentary ways. Yes we live in the future, but let's not get carried away."

Here's a paper titled "Authentic emotion detection in realtime video", just to reveal what shows up on a quick Google search, from 2004. I haven't seen any on lying, but it wouldn't surprise me if there were a few probabilistic indicators. For example, we know that faking a smile is hard, and CEOs don't generally have degrees in Theater. Wouldn't surprise me if there were more findings like this that I don't know about as a layman. It doesn't need to be "convict him, and add n years for perjury", just enough to find a few confident cases that are right more often than wrong. And it doesn't have to be confident all the time. When it is, then you pay attention, double check the context, and determine if the market has priced this differently, as obvious lies shouldn't move the market (though maybe having a liar for a CEO is news itself).

This is a different standard than academia sets for itself, surely you can recognize this. The Loebner Prize is targeting a dramatically different topic, with dramatically harder judges. The hard AI and sentience claims from the Atlantic comment are bullshit, but language processing and emotional analysis isn't. Moreover, Loebner awards $100k in the best case, while there's far more to be had in market prediction. The market incentives for researchers is not balanced towards the Loebner Prize or disclosure.
posted by pwnguin at 12:15 PM on August 8, 2010

I think it's a perfectly reasonable thing to think that a computer can correlate specific facial expressions combined with speech content with moves in stock prices. It's not about predicting whether or not someone is lying or not. It's about figuring out what everyone else is going to do quickly enough that you can profit from it.

It's not necessary to understand the why, only that when the CEO's left eyebrow isn't raised when he smiles, the company's stock price falls several days later when bad news surfaces, and when it is, one can probably expect good news.

It's like, simple Bayesian filtering, man.
posted by wierdo at 7:26 PM on August 8, 2010

Is the emotion analysis code you saw running fine grained enough to honestly take a tv broadcast and determine whether the person is *lying?* Because that would be a pretty major breakthrough I would have thought.

The code I saw running returned reasonable analogues for "happy", "sad", "distracted", "angry", and one or two other metrics. I'd say it was good, not great. Wasn't exactly production ready, but it was real.

What was production ready was stress analysis for call centers. I used to work in telephony; I remember trade shows showing gear that would let the average anger level of callers be measured such that employees could be marked for customer satisfaction or lack thereof.

There's lots of gear out there that purports to use vocal stress to detect a lie. I'm sure it works a bit.

I'll agree that you'll never get perfection out of *anything* that looks at a spoofable system like human voice and speech. However -- getting within 50% of a trained human, and then running that on absolutely massive amounts of data -- there's likely to be reasonable data from that process.
posted by effugas at 10:05 PM on August 8, 2010 [1 favorite]

Wow, three days late, and I may still be the one to say that this reminds me of the loa in Gibson's book "Count Zero."
posted by wenestvedt at 7:29 AM on August 9, 2010

Just in case anybody else is still reading this, wenestvedt- yeah. I talked about this story w/ my roomie, who's about to get a doctorate from using neural networks to do various clever things. And so he (and I, to a lesser extent) knows people who've done the work, building black boxes for quants...

Main thing was, none of it sounded at all outlandish to him. Reason I'm posting it is, he and I have talked a lot about AI and similar stuff, and he said that given the amount of processing/software/brain- power dedicated at this point - modeling the stock market, which is already a dynamic/complex system, and adding complexity to it in the process of trying to understand and outsmart it- he figures that that's where skynet starts - the stock market becomes sentient.

Yeah, it might be half tongue-in-cheek - he knows better than most of us how f'in hard 'hard AI' is- but for even the half that's not, damn- I'd almost rather it was a defense network, you know?
posted by hap_hazard at 10:56 PM on August 9, 2010

Seriously? Can you cannot comprehend why someone would favourite that comment (which is a repost of an online newspaper comment, if you hadn't noticed) besides "I think this is factual"?

Yep. The same reason people keep the CSI:wherever series up in the monthly Nielsen ratings. Because plebes love the taste of technology that's just out of reach but barely digestible enough to make them feel like they're on the cutting edge, with just enough hand-waving that it works like magic but enough bullshit details to make it totally believable, man!

The code I saw running returned reasonable analogues for "happy", "sad", "distracted", "angry"

Existing facial recognition software is in its infancy. Take a basic imprint of a face, correlate where the eyes, nose, mouth & head boundaries are to a standard template. Find the boundaries of the mouth, eyes and eyebrows. If left and right corners have negative & positive slopes (respectively) = smiling = happy. If frowning & eyebrows are sloped up in the middle = disbelief, frowning & eyebrows sloped down in the middle = anger.

On the accuracy scale, this is like a 1st grader drawing faces. Smile = happy. Frown = sad. Correlation is going to be so low as to be worthless. And you're sure as shit not going to be determining lying from this Fisher-Price nonsense.

This is kind-of like how non-science people got all excited when SpaceShipOne won the X-Prize because they thought this was going to usher-in a brand new era of personalized Space Ships taking people to the moon and beyond. Except it's not orbital. "Details!" they decry. "It's just a few small tweaks, a couple of simple steps and next thing you know you'll be on Mars!"

No, it's completely fucking different. The problem is a shit-ton harder. You're taking your Lego Advanced car and extrapolating a race of sentient robots that attack mankind. Conveniently leaving out the ten billion little steps in-between.
posted by Civil_Disobedient at 9:37 AM on August 13, 2010 [1 favorite]

Yep. The same reason people keep the CSI:wherever series up in the monthly Nielsen ratings. Because plebes love the taste of technology that's just out of reach but barely digestible enough to make them feel like they're on the cutting edge, with just enough hand-waving that it works like magic but enough bullshit details to make it totally believable, man!

I guess I just gave them more credit and assumed they were favouriting it because they found it hilarious. Now I'm depressed.
posted by ODiV at 1:29 PM on August 13, 2010

As usual, the truth is somewhere in the middle. The people that think this whole idea is ridiculous fantasy are just as misguided as the ones who think we've managed to create real AI.

The author of the controversial comment sounds like he has programmer friends but doesn't know much about it himself, so he uses lots of buzzwords that help him wrap his head around it. His description is partly gibberish and contains some obvious wishful thinking (yes, I'm looking at you, facial recognition), but that fact remains that predictive systems like this exist all over the place, and account for huge numbers of trades in today's market.

Or what amuseDetachment said.
posted by chundo at 8:52 AM on August 17, 2010

I'm very far from an expert on this, but I've taken a computer vision class and a machine learning class with a unit on finance, and know some CS grad students who do emotion analysis from video.

It wouldn't strike me as at all weird if a finance company spent a few hundred thousand dollars to get the latest emotion analysis software and tuned it in to CNN, then fed the output into an ensemble learning algorithm with their other analysis products.

At best it could give them a tiny edge over the competition (in a field where tiny edges make billions of dollars), and at worst the ensemble learning algorithm would decide that the emotion analysis software's results were irrelevant and disregard them.

Given the early state of emotion analysis software I think the latter result is more likely, but finance companies have been known to try all sorts of crazy things in pursuit of profits.

I know for a fact that real trading companies are investing quite a bit in natural language processing. It's not very public, but I keep hearing that friends of friends in NLP got jobs on Wall Street.
posted by miyabo at 11:21 AM on August 17, 2010 [1 favorite]

Azul Systems hardware is relatively conventional, at least compared to the equipment made by Celoxica. They make hardware that decodes market data feed protocols, saving a couple of microseconds per packet compared to doing it in software. And they make a board that can do arbitrage between two market feeds in hardware!
posted by miyabo at 10:02 AM on August 19, 2010

It's not facial recognition, but here's a genuine paper on language processing deceptive language from CEOs in conference calls:

"Larcker, the James Irvin Miller Professor of Accounting and senior faculty of the Stanford Rock Center for Corporate Governance, and Zakolyukina developed a model to analyze words and phrases used based on prior deception detection research conducted by psychologists and linguists...Results from their model are 4% to 6% better than a random guess."
posted by pwnguin at 7:29 PM on August 23, 2010

Hillarious. This has been one of the stongest value cycles in history - beginning in Nov of '08. Look at P/B performance.

I'm not talking about long-term. I'm talking short-term, the market is trading violently on news and at resistance and support levels. Volume has been low for a long time now, and volatility has been high. I'm trading short-term entirely on technicals and doing very well. I go long-term when I see value in a fundamentally decent stock, but what qualifies as "value" today might not be seen the same in retrospect when next quarter's earnings are reported.
posted by krinklyfig at 5:34 AM on September 5, 2010