Input: Some Intelligent comments mixed with plausible deny ability....comments from the linked article by readers learning can occur from within this framework.
anon3213 2 days ago
Competition between hedgefunds is fierce. The trading algorthmn models are always changing. So lets say you develop a model, and it makes money. Great! It doesnt last long, because your model must evolve or else it becomes obsolete.
I am a trader. I go head to head with these algorythmns. adapting is crucial. because the patterns are changing I notice that the profitable trades i made last month stop working.
to succeed i think it crucial for these hft system to spoof orders. for example everyday 95% of the orders that enter the market are canceled before they are filled. why are they canceled? because they are intended to read where the buyers and sellers "are", not to get filled. its like they are using them as a flashlight to light up short term market moves.
the action doesnt occur where the stock is trading, the action is occuring below and above the last price. where fake orders are being used to play games to see where the buyers and sellers are.
(Edited by author 2 days ago)
Flag 27 people liked this. Like ReplyReply Brandon 2 days ago in reply to anon3213
Has anyone run a pattern analysis to see if there is repetition? It could be far more interesting then any comments I've seen. Let's say for a second that you don't trust traditional communication channels. Mail, email, phone, etc. A great way to send a message would be hide a signal in noise. If done correctly, you could scan the fluctuations for a known pattern. And anyone that knows to look for the pattern could easily decode it. And since the American stock market is largely unconcerned with time slices this small, no one notices. You could send your message anywhere and no one would think to look at it.
Flag 4 people liked this. Like ReplyReply Lance Devin 2 days ago in reply to anon3213
if they are sniffing around the edges of the market order and an event occurs (say a fat finger order" that shocks the system, can't these probes actually get executed? And if so, do they provide an amplification effect to the "chaos".
Flag 1 person liked this. Like ReplyReply n_torenvliet 2 days ago
I enjoyed reading the comments on this one... lots of theories... so its not random noise, emerging events, or anything based on agent interaction. I'd buy quite quickly the idea that the patterns are simply someone testing some newly conceived high speed loops -- I'd definitely be testing my new HFT code in a production environment this way.
I'm surprised no one has conceived of some kind of price fixing, or volatility fixing scheme on the part of major market makers, do a transform on the patterns and find embedded the proposed "market maker/HFT/quant" game plan e.g. the parameters for the day's neutral risk portfolio... you could trip a lot of retail stop loss orders in a very predictable fashion if enough money had the same marching orders, or shall we say market parameterization.
Lets say x new retail investors per time slice... you do the math...
Please take me away in your beautiful ship.
Flag 5 people liked this. Like ReplyReply alexismadrigal 2 days ago
I've got to say that this is my favorite comment thread ever on our tech coverage. Many thanks to all of you and I hope that you stick around. We've got a lot more HFT/algo/financial tech coverage coming up over the next several days (and weeks).
Flag 8 people liked this. Like ReplyReply coldweathervacation 2 days ago in reply to alexismadrigal
Great article, keep us posted.
Flag 1 person liked this. Like ReplyReply MartinKenneth 2 days ago
It's not testing. It's market manipulation.
market-ticker.org.
Flag 4 people liked this. Like ReplyReply Lissa Kl 1 day ago in reply to MartinKenneth
Fantastic explanation at the link! If it cannot be stopped it can at least be charged for. Every communication with the system should have a cost and that cost should make 'messing around' too expensive.
Flag Like ReplyReply Aaron Wirtz 2 days ago
Emergant, not directly - time scale is too small to include enough interaction between actors.
Intentional noise, no - too statistically consistent - would be filtered as DC bias.
Badly written code, unlikely - internal standards for such code are very strict as algorithmic errors could be financially fatal.
In my opinion, the most likely explanation is that this is a carefully-engineered attempt by one HFT system to probe the internal state of other automated trading systems: Dangle a lure, look for a reaction, adjust the lure slightly, repeat. Since these are not really whole-hearted trade attempts, it doesn't matter (and is perhaps preferable) if they are never long-lived enough to resolve - as long as they elicit some (perhaps delayed, but still observable) response from the other automated systems. Once the internal state of other trading systems is known, the HFT system can bridge any thus-revealed market gaps, taking a profit in the process.
Flag 12 people liked this. Like ReplyReply gameDevNYC 2 days ago
I am a software programmer with several friends at HFT firms, and the explanation we've put together accounts for all of the flash crash behavior as well as these signals - note that word, because that's what Donovan is pointing out, signals sent by software.
All of the serious HFT firms these days use "natural language processing", which means using artificial intelligence to extract profitable information from news streams. People think of this as just headlines but really it's anything that might contain useful information - these computers have all of the cable news channels supplied to them digitally and use everything they can scrape. Some of the firms even use facial recognition software to determine whether the speakers believe what they're saying. My friends joke about how Cramer is a goldmine for their algorithms but that the profitable trades rarely match up with his advice.
One of the facts about 'hard' AI, as is required for profitable NLP, is that the coders who developed it don't even understand completely how it works. If they did, it would just be a regular program. What's even stranger is that they can't use regular tools, like a debugger, to observe the algorithms' behavior, because it interferes with the processing and causes different trades to be emitted. In a very real sense, they can't explain why their robots send the orders they do. They can tell you what data they "trained" it with, and what sorts of data they "feed" it, but they're inherently unpredictable.
As a result, a lot of programmers at HFT firms spend most of their time trying to keep the software from running away. They create elaborate safeguard systems to form a walled garden around the traders but, exactly like a human trader, the programs know that they make money by being novel, doing things that other traders haven't thought of. These gatekeeper programs are therefore under constant, hectic development as new algorithms are rolled out. The development pace necessitates that they implement only the most important safeguards, which means that certain types of algorithmic behavior can easily pass through. As has been pointed out by others, these were "quotes" not "trades", and they were far away from the inside price - therefore not something the risk management software would be necessarily be looking for.
The commenters above who called for identification of the company running the responsible algorithms have it wrong. There isn't just one firm putting these sorts of orders in the market. The fact is that many of these HFT shops are glued to their screens every minute the market is open because they have to be alert to strange behavior by their own algorithms.
TLDR: this definitely is emergent behavior - it is information passing between black-box algorithms with motivations that even the original programmers cannot make definitive statements about.
Flag 57 people liked this. Like ReplyReply Anthony Gallard 2 days ago in reply to gameDevNYC
Hmm....
Flag 4 people liked this. Like ReplyReply gofigureit 2 days ago in reply to gameDevNYC
is there any rebating going on here? Or testing of algos for rebating in certain markets? participation on some exhchanges will pay premiums to those participating even if they are not in the current trading limits.
Flag Like ReplyReply Herostratus 1 day ago in reply to gofigureit
"All of the serious HFT firms these days use "natural language processing" "
Some of the firms even use facial recognition software to determine whether the speakers believe what they're saying"
Complete B.S. Should have left that bit out, otherwise your claim of novel insider info was almost halfway believable.
Flag 3 people liked this. Like ReplyReply Hal Grossman 1 day ago in reply to Herostratus
If he doesn't know what he's talking about, please refute him with some evidence. He had me convinced (and frightened).
Flag 3 people liked this. Like ReplyReply Ezra Van Dort 1 day ago in reply to gameDevNYC
It may not be just one firm playing these sorts of orders, but identifying the source(s) would probably differentiate between simple iterative algorithms and the more interesting possibilities of "behavior" (in the sense of one system responding to the actions of another).
Yes, lots of people might be doing very similar stuff - they are probably drinking at the same bars surrounding the Exchange. The important question is are their systems interacting outside the intended purpose (interesting), is this some kind of market-influencing shenanigans (interesting to market authorities), or are these patterns just "sonar" from the battleships of the market wars (interesting only to market geeks).
Applying Occam's Razor, I would put my money (figuratively) on the last version, perhaps the shenanigans if anyone can figure out how these orders could have any effect (maybe they are simply some form of stealth trap - semi-randomly hoping to be in the optimum position should the target shares make a sudden deviation, not hanging around long enough for anyone to notice and tag along).
I will try to keep an eye on this discussion, because personally I enjoy the Skynet possibility - that so much complex power, comms and data has been poured into these systems they have indeed evolved an emergent life form - and despite the largely artificial and self-serving nature of the modern "market", these systems still have influence on real-world activity, which might be the stimulus for such a being to manifest in ways recognizable to our poor anthro-centric selves.
Flag 3 people liked this. Like ReplyReply Username456 1 day ago in reply to gameDevNYC
Nice fiction. Trying to be the next Gibson?
Flag Like ReplyReply Mark Huckabee 5 hours ago in reply to gameDevNYC
This is not sci-fi. It is really happening. See this recent article at New Scientist magazine: newscientist.com When the scientists met with several hedge funds they found that they were already using language and speech processing software to gain an edge.
Flag 1 person liked this. Like ReplyReply ElPlumo 2 days ago
(Excuse my english)
I've have been working for a decade now in Space industry softwares (satellite instruments operating softwares), and of course one wants them to be bug-free, hence huge amounts of manpower and times spent testing and validating them. Even so, after years, for those algorithms that are not so complex, some bugs remains... I happen to also have one friend (met during studies) that works in market trading, and who sometimes asks me programming question, and that also showed me some articles about trading algorithm, they are horribly complex, I also see in which conditions those folks work in market programming : huge pressure, short time delay, few sleeping.. Therefore, I cannot imagine that market bots are bug free, and most probably they are full of bugs, mostly harmless, but still doing things that were not intended to, and they are problably equivalent to a small amount of random activity that averages to harmless residue within the size and complexity of orders. Yet, some "Bug Order Freak Wave" may happen.
Flag 4 people liked this. Like ReplyReply Brian Hall 17 hours ago in reply to ElPlumo
Ist English correction: there is no such word as "softwares" (plural). Software is a generic, collective term. Several pieces of software, or programs, are the specific nouns that can be plural. (Similar: "informations" would not be a valid word.)
Flag Like ReplyReply StockAvoider_2010 2 days ago
Don't put your money in the hands of stock brokers!
They find a way to transfer it into their own pockets. In this example, they buy expensive computer equipment in order to preempt you when you buy or sell stocks of your retirement savings. If the price goes up or down, the owner if the stock should benefit from it, not the millisecond trader. With millisecond trading, the regular trader always looses money. This is the whole purpose of it in the first place. Why else should you operate such expensive equipment at banks with profit?!
Flag 6 people liked this. Like ReplyReply hybrid bike 2 days ago
Thats crazy, millisecond charts..
thekpv.com
Flag 4 people liked this. Like ReplyReply Jean-Pierre Gilgamesh 2 days ago
Hum,
A very simple explanation could be that what this guy "discovers" is simply it's own sampling of data both by selecting (picking) convenient samples and by sampling it at an unreasonable rate and/or phase and removing other discriminating information.
For example if there are two repetitive streams of orders always at the same value, one high and the other low, if you sample at a convenient rate you see the saw pattern that is highlighted in the article. But it doesn't really exists.
Flag 6 people liked this. Like ReplyReply coffeedragon1 2 days ago
If the radio telescopes at SETI had picked up signals like this their conclusions would have been an interesting read. Perhaps those guys should have a look at these, if anyone understands the difference between noise and other phenomena, they do.
Flag 2 people liked this. Like ReplyReply c 2 days ago in reply to coffeedragon1
I think you'd find the most qualified people to extract this information to be employed at the very firms producing these patterns. Or at the NSA. Maybe a couple of them turned down the money to publish freely at Stanford, etc.
Flag Like ReplyReply 1tootall 2 days ago
Stockavoider.....Im wondering if you could elaborate your explanation. I am missing some of the connections you are trying to make. BTW, I AM a stockbroker, and have been for 26 years. I'd love to learn more about this...so I can take advantage, of course!!! (JK)
Flag 1 person liked this. Like ReplyReply duckyinfo 2 days ago
Three thoughts come to mind. 1) Someone's testing their algorithms and thus the look of it just being noise. 2) It's the Rorschach syndrome, trying to make sense out of something that's ambiguous and not really there. 3) Our new overlord, Skynet, is gaining consciousness.
Flag 7 people liked this. Like ReplyReply Kapitano 2 days ago
This is hardly mysterious. It's just evidence that most algorithms are badly designed, most programmers don't know what they're doing, and most traders couldn't spot a winning strategy if they fell in one.
So long as you realise incompetence is the norm, masses of unproductive activity is unsurprising.
Flag 6 people liked this. Like ReplyReply gofigureit 2 days ago in reply to Kapitano
is there any rebating going on here? Or testing of algos for rebating in certain markets? participation on some exhchanges will pay premiums to those participating even if they are not within the current trading limits.
Flag Like ReplyReply Douglas M Carter 1 day ago
There should be some sort of law against it
its just to introduce noise your right
but disruptive to the whole market place unnecessarily
creating the need for unnecessary bandwidth
Flag Like ReplyReply JG35 1 day ago
The high frequency quotes are not transactions. They are well away from the current bid/ask spread so won't get executed. A transaction tax is irrelevant here. The quotes would appear in the level II book which shows buy and sell orders away from the current bid and ask price. Traders (human and bots) often look at this to judge supply and demand. So this might be used to mess with the head of any bot trying to use the level II book. The cure is simple -- ban quotes on the same instrument from the same source in excess of say, 100 quotes per second. I can't imagine that even a high frequency trader would need more legitimate quotes that that.
Flag 2 people liked this. Like ReplyReply Lissa Kl 1 day ago in reply to JG35
True if transaction means financial transaction - buy or sell
Not true if you mean data processing transaction as in communication. Every communication should have a cost.
Flag 3 people liked this. Like ReplyReply JG35 1 day ago in reply to Lissa Kl
Every communication already has a cost, determined by the current level of technology and market competition. If you mean that the government ought to artificially increase the cost of communication with higher taxes, you've lost me on that one. Maybe the government should quadruple the cost of your phone bill until you change your mind.
Flag Like ReplyReply Brian Hall 17 hours ago in reply to JG35
Not the government, the exchange.
Flag Like ReplyReply pi_resident 1 day ago
Does HFT really reflect tangible value? Of course not.
Indeed while I was oversea in October 2008 I marveled at the swings in the "market." No economic news of "value" had occurred, but the net up and down swings would be more than a 1000 points in half and hour.When I think of how "we" saved capitalism and the market (read the rich) I wince and still do.
HFT essentially manipulates the markets - this I believe.
Flag 2 people liked this. Like ReplyReply JG35 1 day ago in reply to pi_resident
Done right high frequency traders act as market makers, providing liquidity to the market, which does serve a purpose. However, generating thousands of quotes a second to mess with the competition serves no good purpose and should be banned.
Flag 2 people liked this. Like ReplyReply DoctorJay 1 day ago
On the quantitative trading forum, Nuclear Phynance, the consensus on the patterns seemed to be that they simply just emerged. They were the result of "a dynamical system that can enter oscillatory/unstable modes of behaviour," as one member put it.
"Crop circles" are the perfect name for them, since they are self-evidently no more the result of emergent behavior than crop circles are the result of windstorms or other natural phenomenon. And the HFT programmers would of course be the first to tell us if they were actually doing this stuff.
Flag Like ReplyReply DoctorJay 1 day ago in reply to DoctorJay
You know, the description "a dynamical system that can enter oscillatory/unstable modes of behavior" seems to me to describe the overall system, consisting of the electronic exchange plus HFT trading bots and the people that run them, perfectly.
Kind of a geek version of "hate the game, not the player".
Flag Like ReplyReply Mike Caprio 1 day ago
Have we learned nothing from "Superman 3" and "Office Space"?
Flag Like ReplyReply Mike Caprio 1 day ago
Have we learned nothing from "Superman 3" and "Office Space"?
Flag Like ReplyReply Brett Dalton 1 day ago
Has any one considered that this could a great way to transmit encrypted data in the open and still have it hidden in the noise. No one even knew the patterns were there until recently and even after finding them the number of explanations are many. Some of the patterns remind me of various forms of modulation and encoding.
Flag 4 people liked this. Like ReplyReply meglodon 1 day ago
Transaction Tax penalizes everyone, instead create a tax that is used on cancelled orders, if these clowns only purpose is to game the system then make them pay for it... Any and all cancelled orders are accessed a fee. 2ndly limit the rebate that exachanges can provide HFT traders... guess what this could be implemented tomorrow and the games stop the following day..
Flag 2 people liked this. Like ReplyReply a_olegoff 1 day ago
Have you ever seen how professional thieves work in groups? One or more are destructing the mark, another one takes the moment when the victim is occupied with destruction and pulls out the wallet or takes the purse. Very often there is another participant to whom the catch is passed almost instantaneously and who immediately disappears before the victim discovers the loss.
What I can see in the “knife pattern” looks like a series of defined strategies. First the price is tested for resistance (knife handle), then distraction is applied for a short period to introduce volatility (wide portion of the blade), then the quote is dropped for the lowest level (still within the range of previous disturbance). At this moment another program may make on actual order. As soon it is achieved the disturbance is introduced again gradually fading until initial condition is reached. If you want to nail the thief, you have to look who makes trades during the period of thinness oscillation (center of the blade). If this activity is occurring outside of the trading hours it could be the testing /synchronization of two different algorithms in preparation for action. To nail perpetrators you would have to know their strategy, observe them in action and capture the proof.
Flag 5 people liked this. Like ReplyReply Brian Hall 17 hours ago in reply to a_olegoff
The problem with your theory is that the quotes are "well outside" the price ranges where actual trades are occurring.
Flag |
