Nanosecond Trading Could Make Markets Go Haywire http://www.wired.com/wiredscience/2012/02/high-speed-trading/ "Below the 950-millisecond level, where computerized trading occurs so quickly that human traders can't even react, no fewer than 18,520 crashes and spikes occurred." Anyone who has managed a network knows that when you look at your MRTG/Cacti graphs at 5min, 10min ,15min intervals - all looks well. Start looking at 1sec intervals and you will see spikes that hit 100% of capacity - even on networks running at 25% average utilization. I guess trading and networking do have many unseen similarities. -Hank
On Feb 16, 2012, at 8:03 AM, Hank Nussbacher <hank@efes.iucc.ac.il> wrote:
Nanosecond Trading Could Make Markets Go Haywire http://www.wired.com/wiredscience/2012/02/high-speed-trading/
"Below the 950-millisecond level, where computerized trading occurs so quickly that human traders can't even react, no fewer than 18,520 crashes and spikes occurred."
Anyone who has managed a network knows that when you look at your MRTG/Cacti graphs at 5min, 10min ,15min intervals - all looks well. Start looking at 1sec intervals and you will see spikes that hit 100% of capacity - even on networks running at 25% average utilization
I've had great success using appliances from network monitoring vendor Corvil - http://www.corvil.com/ and TS-A's TipOff - www.ts-a.com/TipOff/tipoff.html Both can decode most market data feeds formats and drill down to provide a slew of details when trying to debug jitter and latency on networks where sub-millisecond thresholds are essential for analysis. Disclaimer: I'm no way affiliated with any of these companies or products.
I guess trading and networking do have many unseen similarities.
-Hank
On Thu, 16 Feb 2012, Hank Nussbacher wrote:
Nanosecond Trading Could Make Markets Go Haywire http://www.wired.com/wiredscience/2012/02/high-speed-trading/
"Below the 950-millisecond level, where computerized trading occurs so quickly that human traders can't even react, no fewer than 18,520 crashes and spikes occurred."
Anyone who has managed a network knows that when you look at your MRTG/Cacti graphs at 5min, 10min ,15min intervals - all looks well. Start looking at 1sec intervals and you will see spikes that hit 100% of capacity - even on networks running at 25% average utilization.
I guess trading and networking do have many unseen similarities.
Tieing the two together, this post shows how a lot of 'conventional' network thinking needs to be turned on its head when it comes to networks for trading floors: http://www.fragmentationneeded.net/2011/12/pricing-and-trading-networks-down... Jethro. . . . . . . . . . . . . . . . . . . . . . . . . . Jethro R Binks, Network Manager, Information Services Directorate, University Of Strathclyde, Glasgow, UK The University of Strathclyde is a charitable body, registered in Scotland, number SC015263.
On Thu, Feb 16, 2012 at 03:03:55PM +0200, Hank Nussbacher wrote:
Anyone who has managed a network knows that when you look at your MRTG/Cacti graphs at 5min, 10min ,15min intervals - all looks well. Start looking at 1sec intervals and you will see spikes that hit 100% of capacity - even on networks running at 25% average utilization.
As sampling rate approaches zero, so will the "spikyness" of the graph--ultimately an interface is either sending a frame (100%) or it's not (0%). -cjp
On 2/16/12 5:03 AM, Hank Nussbacher wrote:
Nanosecond Trading Could Make Markets Go Haywire http://www.wired.com/wiredscience/2012/02/high-speed-trading/
"Below the 950-millisecond level, where computerized trading occurs so quickly that human traders can't even react, no fewer than 18,520 crashes and spikes occurred."
Anyone who has managed a network knows that when you look at your MRTG/Cacti graphs at 5min, 10min ,15min intervals - all looks well. Start looking at 1sec intervals and you will see spikes that hit 100% of capacity - even on networks running at 25% average utilization.
I guess trading and networking do have many unseen similarities.
Some complementary information I read a few weeks ago: http://www.homelandsecuritynewswire.com/critical-cyber-vulnerabilities-found... http://www.cpacket.com/latency http://www.cpacket.com/download/Introduction%20to%20Network%20Latency%20Engi... Regards, --Jason
On Thu, Feb 16, 2012 at 12:59 PM, Jason Chambers <jchambers@ucla.edu> wrote:
On 2/16/12 5:03 AM, Hank Nussbacher wrote:
Nanosecond Trading Could Make Markets Go Haywire http://www.wired.com/wiredscience/2012/02/high-speed-trading/
"Below the 950-millisecond level, where computerized trading occurs so quickly that human traders can't even react, no fewer than 18,520 crashes and spikes occurred."
Anyone who has managed a network knows that when you look at your MRTG/Cacti graphs at 5min, 10min ,15min intervals - all looks well. Start looking at 1sec intervals and you will see spikes that hit 100% of capacity - even on networks running at 25% average utilization.
I guess trading and networking do have many unseen similarities.
Some complementary information I read a few weeks ago:
http://www.homelandsecuritynewswire.com/critical-cyber-vulnerabilities-found...
http://www.cpacket.com/latency
http://www.cpacket.com/download/Introduction%20to%20Network%20Latency%20Engi...
Regards,
--Jason
This all is very familiar to anyone who's looked at ethernet (or other networks) for real-time control purposes, such as flight control of aircraft or rockets or for autos or other ground vehicles. Though the finance people are pushing it a lot more than the rocket and aircraft control people I know... I guess markets crash faster than rockets! -- -george william herbert george.herbert@gmail.com
On 2/16/12 05:03 , Hank Nussbacher wrote:
Nanosecond Trading Could Make Markets Go Haywire http://www.wired.com/wiredscience/2012/02/high-speed-trading/
"Below the 950-millisecond level, where computerized trading occurs so quickly that human traders can't even react, no fewer than 18,520 crashes and spikes occurred."
Anyone who has managed a network knows that when you look at your MRTG/Cacti graphs at 5min, 10min ,15min intervals - all looks well. Start looking at 1sec intervals and you will see spikes that hit 100% of capacity - even on networks running at 25% average utilization.
given a serialized interface the network is 100% utilized everytime a packet is sent, if you're measuring at the microsecond level it's far more germain what the queue depth is rather than whether a packet is currently on the wire or not. A 5 minute 1 minute or 1 second average is a pretty good measure of how much of the time it wasn't being utilitized during the same interval. The fm4000 based Aristas have hardware hooks to stream the latency data in the form of queue thresholds at you via LANZ, that can take your visibility down to 800 or so usec snapshot of the queue. The broadcom devices doesn't have a comparable functionality.
I guess trading and networking do have many unseen similarities.
I'd be careful about analogizing them to much. high speeding trading strategies to a very large extent depend on the high speed parties being about to make more granular decisions faster that other participants can complete transactions, forwarding engines are mostly looking to get rid of packets as expeditiously as possible.
-Hank
On 2/16/2012 3:03 AM, Hank Nussbacher wrote:
Nanosecond Trading Could Make Markets Go Haywire http://www.wired.com/wiredscience/2012/02/high-speed-trading/
"Below the 950-millisecond level, where computerized trading occurs so quickly that human traders can't even react, no fewer than 18,520 crashes and spikes occurred."
Anyone who has managed a network knows that when you look at your MRTG/Cacti graphs at 5min, 10min ,15min intervals - all looks well. Start looking at 1sec intervals and you will see spikes that hit 100% of capacity - even on networks running at 25% average utilization.
I guess trading and networking do have many unseen similarities.
-Hank
Anecdotally, I had an interview years ago for a small-ish futures trading company based in London. The interviewer had to pause the interview part way through whilst he investigated a 10ms latency spike that the traders were noticing on a short point-to-point fiber link to the London Stock Exchange. He commented that the traders were far better at 'feeling' when an connection was showing even a trace of lag compared to normal than anything he'd set up by way of monitoring (not sure how good his monitoring was, though.) Paul
----- Original Message -----
From: "Paul Graydon" <paul@paulgraydon.co.uk>
Anecdotally, I had an interview years ago for a small-ish futures trading company based in London. The interviewer had to pause the interview part way through whilst he investigated a 10ms latency spike that the traders were noticing on a short point-to-point fiber link to the London Stock Exchange. He commented that the traders were far better at 'feeling' when an connection was showing even a trace of lag compared to normal than anything he'd set up by way of monitoring (not sure how good his monitoring was, though.)
This was my experience in a callcenter as well; network type problem reports always came in from the floor managers before Nagios came forth with an opinion. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
On Fri, Feb 17, 2012 at 10:30:33AM -0500, Jay Ashworth wrote:
----- Original Message -----
From: "Paul Graydon" <paul@paulgraydon.co.uk>
Anecdotally, I had an interview years ago for a small-ish futures trading company based in London. The interviewer had to pause the interview part way through whilst he investigated a 10ms latency spike that the traders were noticing on a short point-to-point fiber link to the London Stock Exchange. He commented that the traders were far better at 'feeling' when an connection was showing even a trace of lag compared to normal than anything he'd set up by way of monitoring (not sure how good his monitoring was, though.)
This was my experience in a callcenter as well; network type problem reports always came in from the floor managers before Nagios came forth with an opinion.
When I used to run an ISP network, our NOC always talked about "that porn guy" who would call the *exact* *momment* the NNTP server had any type of stutter... I guess there's always a canary for the coal mine -- eh?
On Feb 17, 2012, at 10:30 AM, Jay Ashworth <jra@baylink.com> wrote:
----- Original Message -----
From: "Paul Graydon" <paul@paulgraydon.co.uk>
Anecdotally, I had an interview years ago for a small-ish futures trading company based in London. The interviewer had to pause the interview part way through whilst he investigated a 10ms latency spike that the traders were noticing on a short point-to-point fiber link to the London Stock Exchange. He commented that the traders were far better at 'feeling' when an connection was showing even a trace of lag compared to normal than anything he'd set up by way of monitoring (not sure how good his monitoring was, though.)
This was my experience in a callcenter as well; network type problem reports always came in from the floor managers before Nagios came forth with an opinion.
This has nothing to do with a gut feeling or instinct. Trading companies today monitor P&L near realtime and traders will begin to experience low fill rates or worse be rejected by trading counter parties when prices are too far off or out of the money. The longer a system takes to responds to market quotes the lower fills rates they begin to notice and higher execution costs. Trades today in the equity markets must be within the national best bid, best offer price range or companies can be fined by the SEC which is why latency an jitter can be problematic in financial networks.
Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
On Fri, 17 Feb 2012 13:01:36 EST, Rodrick Brown said:
Trades today in the equity markets must be within the national best bid, best offer price range or companies can be fined by the SEC which is why latency an jitter can be problematic in financial networks.
Am I the only one who thinks that if network jitter can make you fall outside the acceptable price window, maybe, just maybe, the market is just too damned volatile for its own good?
----- Original Message -----
From: "Valdis Kletnieks" <Valdis.Kletnieks@vt.edu>
On Fri, 17 Feb 2012 13:01:36 EST, Rodrick Brown said:
Trades today in the equity markets must be within the national best bid, best offer price range or companies can be fined by the SEC which is why latency an jitter can be problematic in financial networks.
Am I the only one who thinks that if network jitter can make you fall outside the acceptable price window, maybe, just maybe, the market is just too damned volatile for its own good?
You are not the only one. Cheers, -- jr 'hold any purchased stock for a minimum of 7 days' a -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
In a message written on Fri, Feb 17, 2012 at 01:36:35PM -0500, Valdis.Kletnieks@vt.edu wrote:
Am I the only one who thinks that if network jitter can make you fall outside the acceptable price window, maybe, just maybe, the market is just too damned volatile for its own good?
I've had an interesting discussion with some financial heads about a simple idea. What if the exchange, on every inbound trade, inserted a random delay, say between 0 and 60 seconds, before processing it? Almost all of this computer based, let's be closer to the exchange stuff becomes junk, immediately. Anyone "long" (where long is probably more than 10 minutes, with a 60 second jitter) in a security wouldn't notice. I mean, if the general public has to get 15 minute delayed quotes so they don't manipulate the market, shouldn't the big guys? :) -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
Why not just simultaneously settle all trades at the same time? Once every minute, or every 5 minutes, or per day? There are many solutions to the problem. I'm sure those that can take advantage of the latency don't want the solution. Kiriki Delany -----Original Message----- From: Leo Bicknell [mailto:bicknell@ufp.org] Sent: Friday, February 17, 2012 10:54 AM To: NANOG Subject: Re: Hi speed trading - hi speed monitoring In a message written on Fri, Feb 17, 2012 at 01:36:35PM -0500, Valdis.Kletnieks@vt.edu wrote:
Am I the only one who thinks that if network jitter can make you fall outside the acceptable price window, maybe, just maybe, the market is just too damned volatile for its own good?
I've had an interesting discussion with some financial heads about a simple idea. What if the exchange, on every inbound trade, inserted a random delay, say between 0 and 60 seconds, before processing it? Almost all of this computer based, let's be closer to the exchange stuff becomes junk, immediately. Anyone "long" (where long is probably more than 10 minutes, with a 60 second jitter) in a security wouldn't notice. I mean, if the general public has to get 15 minute delayed quotes so they don't manipulate the market, shouldn't the big guys? :) -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
Some longer term players, will use delayed data as they are trading longer term, and dont care too much so if the orders were delayed a bit more, these players most likely wouldn't care/notice. But also you have to consider, there are a large degree of shorter term players, who are in/out of the market and play both sides, these do have real-time data feeds, and do care about latency. Some shops go as far as to only use a certain length patch cables from their trading PC to the switch port they are connected to. Also consider when news releases are announced, the markets often do move quite fast, and a LOT of money can be made/lost in seconds, so delaying the orders, could and would affect the outcome of the trades. Also consider that a vast majority of the trades are automated by computers, and some want their servers setup as close to the exchange as possible, in fact the CME group released that they will offer/lease data center space: "One such project is a 428,000-square-foot data center in the western suburbs of Chicago opened by the CME Group, which owns the Chicago Mercantile Exchange<http://topics.nytimes.com/top/reference/timestopics/organizations/c/chicago_mercantile_exchange/index.html?inline=nyt-org>. It houses the exchange’s Globex electronic futures and options trading platform and space for traders to install computers next to the exchange’s machines, a practice known as co-location — at a cost of about $25,000 a month per rack of computers." http://www.nytimes.com/2011/01/02/business/02speed.html?pagewanted=all http://www.datacenterknowledge.com/archives/2010/08/23/cme-group-opens-chica... On Fri, Feb 17, 2012 at 2:47 PM, Kiriki Delany <kiriki@streamguys.com>wrote:
Why not just simultaneously settle all trades at the same time? Once every minute, or every 5 minutes, or per day?
There are many solutions to the problem. I'm sure those that can take advantage of the latency don't want the solution.
Kiriki Delany
-----Original Message----- From: Leo Bicknell [mailto:bicknell@ufp.org] Sent: Friday, February 17, 2012 10:54 AM To: NANOG Subject: Re: Hi speed trading - hi speed monitoring
In a message written on Fri, Feb 17, 2012 at 01:36:35PM -0500, Valdis.Kletnieks@vt.edu wrote:
Am I the only one who thinks that if network jitter can make you fall outside the acceptable price window, maybe, just maybe, the market is just too damned volatile for its own good?
I've had an interesting discussion with some financial heads about a simple idea.
What if the exchange, on every inbound trade, inserted a random delay, say between 0 and 60 seconds, before processing it?
Almost all of this computer based, let's be closer to the exchange stuff becomes junk, immediately. Anyone "long" (where long is probably more than 10 minutes, with a 60 second jitter) in a security wouldn't notice.
I mean, if the general public has to get 15 minute delayed quotes so they don't manipulate the market, shouldn't the big guys? :)
-- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
----- Original Message -----
From: "Craig" <cvuljanic@gmail.com>
But also you have to consider, there are a large degree of shorter term players, who are in/out of the market and play both sides, these do have real-time data feeds, and do care about latency. Some shops go as far as to only use a certain length patch cables from their trading PC to the switch port they are connected to. Also consider when news releases are announced, the markets often do move quite fast, and a LOT of money can be made/lost in seconds, so delaying the orders, could and would affect the outcome of the trades.
Sure. We're simply asserting that those people serve no useful social function, and we don't *care* whether their needs are served or not. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
On 2/17/12 11:47 , Kiriki Delany wrote:
Why not just simultaneously settle all trades at the same time? Once every minute, or every 5 minutes, or per day?
There are many solutions to the problem. I'm sure those that can take advantage of the latency don't want the solution.
Ask yourself where the incentives are that drive the observed behavior.
Kiriki Delany
-----Original Message----- From: Leo Bicknell [mailto:bicknell@ufp.org] Sent: Friday, February 17, 2012 10:54 AM To: NANOG Subject: Re: Hi speed trading - hi speed monitoring
In a message written on Fri, Feb 17, 2012 at 01:36:35PM -0500, Valdis.Kletnieks@vt.edu wrote:
Am I the only one who thinks that if network jitter can make you fall outside the acceptable price window, maybe, just maybe, the market is just too damned volatile for its own good?
I've had an interesting discussion with some financial heads about a simple idea.
What if the exchange, on every inbound trade, inserted a random delay, say between 0 and 60 seconds, before processing it?
Almost all of this computer based, let's be closer to the exchange stuff becomes junk, immediately. Anyone "long" (where long is probably more than 10 minutes, with a 60 second jitter) in a security wouldn't notice.
I mean, if the general public has to get 15 minute delayed quotes so they don't manipulate the market, shouldn't the big guys? :)
On 02/17/2012 08:36 AM, Valdis.Kletnieks@vt.edu wrote:
Trades today in the equity markets must be within the national best bid, best offer price range or companies can be fined by the SEC which is why latency an jitter can be problematic in financial networks. Am I the only one who thinks that if network jitter can make you fall outside
On Fri, 17 Feb 2012 13:01:36 EST, Rodrick Brown said: the acceptable price window, maybe, just maybe, the market is just too damned volatile for its own good? https://www.google.com/finance?client=ob&q=NASDAQ:AAPL <https://www.google.com/finance?client=ob&q=NASDAQ:AAPL>
See what happened on Wednesday with Apple's stock. With no good cause it looks like various parties started to try and short it. You can see the initial result from 12pm->1pm, the 'quick buck' 1pm-1:30pm rise, then the start of some more shorting at which point you can see the pattern emerge where the automatic trading algorithms started doing their thing. Definitely too volatile. Paul
participants (14)
-
Christopher J. Pilkington -
Craig -
George Herbert -
Hank Nussbacher -
Jason Chambers -
Jay Ashworth -
Jethro R Binks -
Joel jaeggli -
John Osmon -
Kiriki Delany -
Leo Bicknell -
Paul Graydon -
Rodrick Brown -
Valdis.Kletnieks＠vt.edu