Did *bufferbloat* cause the 2010 flashcrash?
This guy seems to think so, and his arguments seem pretty convincing to me, but I don't understand the financial system as well as I might. yarchive.net/blog/computers/flash_crash.html Gettys is namechecked in the piece. Cheers, -- jra -- Sent from my Android phone with K-9 Mail. Please excuse my brevity.
On Sun, 02 Aug 2015 23:19:02 -0400 Jay Ashworth <jra@baylink.com> wrote:
This guy seems to think so, and his arguments seem pretty convincing to me, but I don't understand the financial system as well as I might.
Interesting Jay, thanks for forwarding that. I'm not convinced, but I could be. Interesting hypothesis that, at least for me, raises more questions that I'm not qualified to answer either. Perhaps most fundamentally, what and where exactly is the "queue" in the transmission system that Nanex refers to? It would seem surprising that delays in general due to long queues would not have been noticed before, since or would have caused other more far reaching problems. If there is serious debate about the cause of the problem, then it may be necessary to invite an independent, neutral party to fully investigate. One has to ask, wouldn't it be convenient for Nanex if queueing delay from others into their systems were the cause? John
On Thu, Aug 6, 2015 at 12:51 PM, John Kristoff <jtk@cymru.com> wrote:
It would seem surprising that delays in general due to long queues would not have been noticed before, since or would have caused other more far reaching problems.
bufferbloat is the boogieman... of late. I think that's foolish :( I think this comment from jtk is really on point though! 'why only then?' that sure seems convenient, eh?
On Thu, 6 Aug 2015, Christopher Morrow wrote:
bufferbloat is the boogieman... of late. I think that's foolish :( I think this comment from jtk is really on point though! 'why only then?' that sure seems convenient, eh?
Failures almost never have a single cause. Transport networks are never perfect, i.e. delays, dropped packets, data corruption, etc. They may be contributing factors, or a combination of rare events. The hard question that SEC and the industry has been wrestling with is "Why?" not so much "How?" The apparent condititions didn't change, but the system reacted differently during those seconds. Why? What was different?
On 8/6/15 9:58 AM, Christopher Morrow wrote:
On Thu, Aug 6, 2015 at 12:51 PM, John Kristoff <jtk@cymru.com> wrote:
It would seem surprising that delays in general due to long queues would not have been noticed before, since or would have caused other more far reaching problems.
bufferbloat is the boogieman... of late. I think that's foolish :( I think this comment from jtk is really on point though! 'why only then?' that sure seems convenient, eh?
The queuing like the RBC dudes were doing was in order transmission not on the wire. given wires of various lengths having the request arrive on different exchanges at different times based on distance was considered unedesirable (by people loooking to reduce the opportunity for arbitrage on latency). I have have minimal experience with trading platforms but what switch vendors were selling us as a latency sensitive customer (and HFT shops at time) were broadcom or fulcrum asics which by virtue of being cut-through are essentially minimally buffered.
Various technical issues may have made things worse, but the central cause of the flash crash was due to lack of regulator and/or procedures for the now distributed markets (exchanges, ecns, dark-pools, etc...) on a market imbalance. What started the whole thing was a selloff of a large quantity of e-mini futures, which caused a broad run on the market. This is a feature, not a bug. The thundering herd responded, as normal, by starting their own sell-off Before the decentralized markets, when there was a market imbalance (all buyers or all sellers), the "specialist" or "market maker" would halt trading in a symbol, and work with the buyers/sellers to determine a new price and re-open with the new price. The problem is without coordination, when the NYSE/Nasdaq market makers halted trading, the distributed exchanges weren't halted. Since the market makers in those exchanges are required to always have quotes, they put out "stub" quotes of 0.01/$10000.00. Since there weren't valid quotes on the regular exchanges and because of RegNMS, those stub quotes got disseminated as BBO. This was a cosmetic issue and didn't effect trading. Who really got screwed were people that had a stop order on their stocks and didn't realize there were no guarantees of trading through that price. For example, if you bought a stock at $100, and put a stop order to sell at $90, and there was a market imbalance, the price could trade discontinuously. For example the last valid quote could have been at $95.90, then halted, then re-opened at $82.50. The stop order would sell immediately at $82.50, not the $90 people thought. Then the stock could recover and be trading at $95.05 and you could really feel you were screwed. But that's how it is supposed to work. ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-694-5669 -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of joel jaeggli Sent: Thursday, August 6, 2015 1:31 PM To: Christopher Morrow <morrowc.lists@gmail.com>; John Kristoff <jtk@cymru.com> Cc: nanog list <nanog@nanog.org> Subject: Re: Did *bufferbloat* cause the 2010 flashcrash? On 8/6/15 9:58 AM, Christopher Morrow wrote:
On Thu, Aug 6, 2015 at 12:51 PM, John Kristoff <jtk@cymru.com> wrote:
It would seem surprising that delays in general due to long queues would not have been noticed before, since or would have caused other more far reaching problems.
bufferbloat is the boogieman... of late. I think that's foolish :( I think this comment from jtk is really on point though! 'why only then?' that sure seems convenient, eh?
The queuing like the RBC dudes were doing was in order transmission not on the wire. given wires of various lengths having the request arrive on different exchanges at different times based on distance was considered unedesirable (by people loooking to reduce the opportunity for arbitrage on latency). I have have minimal experience with trading platforms but what switch vendors were selling us as a latency sensitive customer (and HFT shops at time) were broadcom or fulcrum asics which by virtue of being cut-through are essentially minimally buffered.
On Sun, Aug 2, 2015 at 11:19 PM, Jay Ashworth <jra@baylink.com> wrote:
This guy seems to think so, and his arguments seem pretty convincing to me, but I don't understand the financial system as well as I might.
Hi Jay, My read is that the author got it upside down. The intermediate cause of the problem was propagation delay (including buffer bloat) which induced an oscillating set of states in the trading software. The root cause was a flipping jassack trying to out-time his competitors by assuming a degree of instantaneity which proved untrue. Don't do that. Don't make assumptions about network timing. You can count on being wrong. If timing matters to your application, find a way to continuously measure. Regards, Bill Herrin P.S. Recruiters: No, I do NOT want to move to New York City and engineer another half millisecond out of your network. I would, however, welcome a law which bans both buying and selling instruments of the same stock or commodity within 24 hours. -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
In 8/6/15 10:44 AM, William Herrin wrote:
The intermediate cause of the problem was propagation delay (including buffer bloat) which induced an oscillating set of states in the trading software.
The root cause was a flipping jassack trying to out-time his competitors by assuming a degree of instantaneity which proved untrue. Don't do that. Don't make assumptions about network timing. You can count on being wrong. If timing matters to your application, find a way to continuously measure.
Similar things happen when folks decide they are going to twiddle the knobs of NTP's behavior. NTP works locally, and gets/provides information globally. More or less. When folks decide to make a change in its core behavior, the usually don't consider how those changes will affect anybody else. I know enough about this to know I don't know anywhere near enough about it, so I leave the knobs alone. -- Harlan Stenn <stenn@nwtime.org> http://networktimefoundation.org - be a member!
participants (8)
-
Christopher Morrow -
Harlan Stenn -
Jay Ashworth -
joel jaeggli -
John Kristoff -
Matthew Huff -
Sean Donelan -
William Herrin