fractional gigabit ethernet links?
Hello, I'm trying to troubleshoot a problem with a fractional (311 mbit/second) gigabit-ethernet line provided to me by a metro access provider. Specifically, it is riding a gig-e port of a 15454. The behavior we are seeing is an occasional loss of packets, adding up to a few percent. When doing a cisco-type ping across the link, we were seeing a consistent 3 to 4 percent loss. For fun, the provider brought it up to 622 mbit/second, and loss dropped considerably, but still hangs at about 1 to 2 percent. There is no question in my mind the issue is with the line, as we've done a wide variety of tests to rule out the local equipment (MSFC2s, FYI). Any clues would be exceptional. -- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben -- -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
Hello Alex, I'd say this sounds obvious, but may be deceptively so... If you are taking a pipe capable of 1000 mbit, and rate-limiting it to 311 mbit, the logic used may be: In the last 1000 msec have there been more than 311mbits? If yes: drop. What you want is to shape the traffic, so the rule would be: In the last 1000 msec have there been more than 311 mbits? If yes: store until the msec period is up, then transmit. If you are pushing 100 mbits over this link, it is entirely likely that there will be a few sub-second burts up to 1000 mbit, and a few sub-second drops to 0mbit. An option for you would be to just figure out what the exact rate-limiting rules are, and then shape it into those rules on your side of the link -- assuming they wont change it to a shaping rule. --Phil -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Alex Rubenstein Sent: Monday, July 15, 2002 10:48 PM To: nanog@merit.edu Subject: fractional gigabit ethernet links? Hello, I'm trying to troubleshoot a problem with a fractional (311 mbit/second) gigabit-ethernet line provided to me by a metro access provider. Specifically, it is riding a gig-e port of a 15454. The behavior we are seeing is an occasional loss of packets, adding up to a few percent. When doing a cisco-type ping across the link, we were seeing a consistent 3 to 4 percent loss. For fun, the provider brought it up to 622 mbit/second, and loss dropped considerably, but still hangs at about 1 to 2 percent. There is no question in my mind the issue is with the line, as we've done a wide variety of tests to rule out the local equipment (MSFC2s, FYI). Any clues would be exceptional. -- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben -- -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
On Mon, 15 Jul 2002, Phil Rosenthal wrote:
Hello Alex,
I'd say this sounds obvious, but may be deceptively so... If you are taking a pipe capable of 1000 mbit, and rate-limiting it to 311 mbit, the logic used may be:
In the last 1000 msec have there been more than 311mbits? If yes: drop.
Except, we're at the levels of 100 kbit/second in our tests. I did just find CSCdr94172, which might be related. -- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben -- -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
This may sound a bit ridiculous, but say the timer is every 0.25ms. 100kbit per 0.25ms = 400,000kbit or 400 mbit. It is remotely possible to hit a 300 mbit limit with only 100kbits of traffic, if the timer is sufficiently short, and your traffic is sufficiently bursty. Unless your traffic is Mcast, I doubt that issue is related. Can you ask your provider how exactly they are limiting the pipe? When dealing with 300 or so megs, I doubt they will be shaping with a policy friendly to you, as the logistics of doing so are a bit difficult. --Phil -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Alex Rubenstein Sent: Monday, July 15, 2002 11:06 PM To: Phil Rosenthal Cc: nanog@merit.edu Subject: RE: fractional gigabit ethernet links? On Mon, 15 Jul 2002, Phil Rosenthal wrote:
Hello Alex,
I'd say this sounds obvious, but may be deceptively so... If you are taking a pipe capable of 1000 mbit, and rate-limiting it to
311 mbit, the logic used may be:
In the last 1000 msec have there been more than 311mbits? If yes: drop.
Except, we're at the levels of 100 kbit/second in our tests. I did just find CSCdr94172, which might be related. -- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben -- -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
On Mon, 15 Jul 2002, Phil Rosenthal wrote:
This may sound a bit ridiculous, but say the timer is every 0.25ms. 100kbit per 0.25ms = 400,000kbit or 400 mbit. It is remotely possible to hit a 300 mbit limit with only 100kbits of traffic, if the timer is sufficiently short, and your traffic is sufficiently bursty.
A cisco ping is not bursty, to the extent of hundreds of mb/s. Also, cisco ping doesn't offer 4,000 pings/sec.
Unless your traffic is Mcast, I doubt that issue is related.
Read on; EIGRP, CDP, etc.
Can you ask your provider how exactly they are limiting the pipe? When dealing with 300 or so megs, I doubt they will be shaping with a policy friendly to you, as the logistics of doing so are a bit difficult.
By strapping 1 or more STS-1's to the GE iface. -- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben -- -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
Is there a Pipeline 75 guru out there tonight that might be willing to help me offline? This is my first experience with the Pipeline and I am feeling really stupid about now! Please excuse the minor operation content, but I am stumped and really need a hand. I am replacing my fading (but still functional) Flowpoint 128 ISDN router with a used Pipeline 75 that I got off eBay. It appears to be fine. I have upgraded the firmware and have spent all day trying to get this @!<>**~!! thing to dial a call to the ISDN WAN. Both POTS work in & out bound. Show if stat indicates the all the WAN ports are down except 'wanidle0' which is up. The system options status window says dynamic bandwith allocation is not installed and I can't fiqure out how to turn it on or install it. Is this my problem? In diagnostics, bridisplay occasionally reports ---> parameter 1,128.1 could not be loaded" ---> parameter 1,128.2 could not be loaded" but I am unable to locate any cross reference between these parameter numbers and the associated configuration item. Your kind assistance is greatly appreciated. Dennis ...
: A cisco ping is not bursty, to the extent of hundreds of mb/s. Also, cisco : ping doesn't offer 4,000 pings/sec. No, but you can start 6 simultaneous sessions to the router and have 5 of them pinging the other side of the circuit while looking at the 6th session to watch for traffic levels and errors. You'd be surprised how much traffic you can push when you set the packets to as big as you can and as fast as you can. I don't know how bursty a traffic pattern this would create, though. scott
I may be missing something but.. presumably their rate-limiting involves some form of queuing/buffering.. in which case assuming the ping is the only thing occuring, when the rate hits the limit it will queue, delay and slow down the echo/reply and no packets should be lost? on the other hand, if as is i think suggested below theres no buffer ie when it hits the limit it starts dropping then i dont think thats a good way of rate limiting as it only works for tcp and the network really needs to provide a way to slow the ip layer down. i cant see how that will provide a usable service... Steve -- Stephen J. Wilcox IP Services Manager, Opal Telecom http://www.opaltelecom.co.uk/ Tel: 0161 222 2000 Fax: 0161 222 2008 On Mon, 15 Jul 2002, Phil Rosenthal wrote:
Hello Alex,
I'd say this sounds obvious, but may be deceptively so... If you are taking a pipe capable of 1000 mbit, and rate-limiting it to 311 mbit, the logic used may be:
In the last 1000 msec have there been more than 311mbits? If yes: drop.
What you want is to shape the traffic, so the rule would be: In the last 1000 msec have there been more than 311 mbits? If yes: store until the msec period is up, then transmit.
If you are pushing 100 mbits over this link, it is entirely likely that there will be a few sub-second burts up to 1000 mbit, and a few sub-second drops to 0mbit.
An option for you would be to just figure out what the exact rate-limiting rules are, and then shape it into those rules on your side of the link -- assuming they wont change it to a shaping rule.
--Phil
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Alex Rubenstein Sent: Monday, July 15, 2002 10:48 PM To: nanog@merit.edu Subject: fractional gigabit ethernet links?
Hello,
I'm trying to troubleshoot a problem with a fractional (311 mbit/second) gigabit-ethernet line provided to me by a metro access provider. Specifically, it is riding a gig-e port of a 15454.
The behavior we are seeing is an occasional loss of packets, adding up to a few percent. When doing a cisco-type ping across the link, we were seeing a consistent 3 to 4 percent loss.
For fun, the provider brought it up to 622 mbit/second, and loss dropped considerably, but still hangs at about 1 to 2 percent.
There is no question in my mind the issue is with the line, as we've done a wide variety of tests to rule out the local equipment (MSFC2s, FYI).
Any clues would be exceptional.
-- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben -- -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
On Mon, 15 Jul 2002 22:48:12 -0400 (Eastern Daylight Time) Alex Rubenstein <alex@nac.net> wrote:
Hello,
I'm trying to troubleshoot a problem with a fractional (311 mbit/second) gigabit-ethernet line provided to me by a metro access provider. Specifically, it is riding a gig-e port of a 15454.
The behavior we are seeing is an occasional loss of packets, adding up to a few percent. When doing a cisco-type ping across the link, we were seeing a consistent 3 to 4 percent loss.
Over what averaging time ? Could this be an example of the "65 second problem", where the router stops dead for one or two seconds out of every 65 seconds ? Regards Marshall Eubanks
For fun, the provider brought it up to 622 mbit/second, and loss dropped considerably, but still hangs at about 1 to 2 percent.
There is no question in my mind the issue is with the line, as we've done a wide variety of tests to rule out the local equipment (MSFC2s, FYI).
Any clues would be exceptional.
-- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben -- -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
Might want to query your provider as to where the rate limitting is being done. In some cases, if rate limit is being done egress from the layer 3 infracture towards the MAN layer 2 equipment, there might be a lack of processing power on that device, causing the drops. Of course this will depend on the type of device and whether the rate limiting is being done on hardware or not too. Sush
Hello,
I'm trying to troubleshoot a problem with a fractional (311 mbit/second) gigabit-ethernet line provided to me by a metro access provider. Specifically, it is riding a gig-e port of a 15454.
The behavior we are seeing is an occasional loss of packets, adding up to a few percent. When doing a cisco-type ping across the link, we were seeing a consistent 3 to 4 percent loss.
For fun, the provider brought it up to 622 mbit/second, and loss dropped considerably, but still hangs at about 1 to 2 percent.
There is no question in my mind the issue is with the line, as we've done a wide variety of tests to rule out the local equipment (MSFC2s, FYI).
Any clues would be exceptional.
-- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben -- -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
Since this is being done with the 15454s this is not true rate limiting, more its a matter of STS channels being made available for use on the Vlan assigned to the two GigE ports. We have experienced this problem when sending traffic in excess of 250mbps through either: a) 4 or more 15454 hops that tired to achieve timing off of eachother rather than a stratum one source: <timing>===<15454>===<15454>===<15454>===<15454>===<15454> OR b) a 15454 that "uplinked" its OC48 backhaul through an active DWDM device (in our case a Sycamore SN8000) when both the SN8000 and 15454 were again not on the same stratum one source : <15454>===<sn8000>===<sn8000>===<15454> By not allowing a timing source to be more than 4 hops from any chassis and by ensuring that all DWDM devices in between are also timed the same, we have pushed up to 605mbps over a 15454 GigE link with no recorded packet loss. Hope this helps Alex, Vin PS As another aside we have seen problems when simultaneously using both ports on the 2 x GigE card for more then 200mbps a piece, but not with any regularity. -vb ----- Original Message ----- From: "Sush Bhattarai" <netnews@sush.org> To: "Alex Rubenstein" <alex@nac.net> Cc: <nanog@merit.edu> Sent: Monday, July 15, 2002 11:57 PM Subject: Re: fractional gigabit ethernet links?
Might want to query your provider as to where the rate limitting is being done. In some cases, if rate limit is being done egress from the layer 3 infracture towards the MAN layer 2 equipment, there might be a lack of processing power on that device, causing the drops. Of course this will depend on the type of device and whether the rate limiting is being done
on
hardware or not too.
Sush
Hello,
I'm trying to troubleshoot a problem with a fractional (311 mbit/second) gigabit-ethernet line provided to me by a metro access provider. Specifically, it is riding a gig-e port of a 15454.
The behavior we are seeing is an occasional loss of packets, adding up
to
a few percent. When doing a cisco-type ping across the link, we were seeing a consistent 3 to 4 percent loss.
For fun, the provider brought it up to 622 mbit/second, and loss dropped considerably, but still hangs at about 1 to 2 percent.
There is no question in my mind the issue is with the line, as we've done a wide variety of tests to rule out the local equipment (MSFC2s, FYI).
Any clues would be exceptional.
-- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben -- -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
In other words..intermittent intergap delay? At 01:33 AM 7/16/2002 -0400, Vincent J. Bono wrote:
Since this is being done with the 15454s this is not true rate limiting, more its a matter of STS channels being made available for use on the Vlan assigned to the two GigE ports.
We have experienced this problem when sending traffic in excess of 250mbps through either:
a) 4 or more 15454 hops that tired to achieve timing off of eachother rather than a stratum one source:
<timing>===<15454>===<15454>===<15454>===<15454>===<15454>
OR
b) a 15454 that "uplinked" its OC48 backhaul through an active DWDM device (in our case a Sycamore SN8000) when both the SN8000 and 15454 were again not on the same stratum one source :
<15454>===<sn8000>===<sn8000>===<15454>
By not allowing a timing source to be more than 4 hops from any chassis and by ensuring that all DWDM devices in between are also timed the same, we have pushed up to 605mbps over a 15454 GigE link with no recorded packet loss.
Hope this helps Alex,
Vin
PS
As another aside we have seen problems when simultaneously using both ports on the 2 x GigE card for more then 200mbps a piece, but not with any regularity.
-vb
----- Original Message ----- From: "Sush Bhattarai" <netnews@sush.org> To: "Alex Rubenstein" <alex@nac.net> Cc: <nanog@merit.edu> Sent: Monday, July 15, 2002 11:57 PM Subject: Re: fractional gigabit ethernet links?
Might want to query your provider as to where the rate limitting is being done. In some cases, if rate limit is being done egress from the layer 3 infracture towards the MAN layer 2 equipment, there might be a lack of processing power on that device, causing the drops. Of course this will depend on the type of device and whether the rate limiting is being done
on
hardware or not too.
Sush
Hello,
I'm trying to troubleshoot a problem with a fractional (311 mbit/second) gigabit-ethernet line provided to me by a metro access provider. Specifically, it is riding a gig-e port of a 15454.
The behavior we are seeing is an occasional loss of packets, adding up
to
a few percent. When doing a cisco-type ping across the link, we were seeing a consistent 3 to 4 percent loss.
For fun, the provider brought it up to 622 mbit/second, and loss dropped considerably, but still hangs at about 1 to 2 percent.
There is no question in my mind the issue is with the line, as we've done a wide variety of tests to rule out the local equipment (MSFC2s, FYI).
Any clues would be exceptional.
-- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben -- -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
Regards, -- Martin Hannigan hannigan@fugawi.net
Sush, Are you thinking of rate-limiting or traffic shaping ? I'd expect rate-limiting of bursty traffic to lose some packets irrespective of the L3 hardware/CPU capacity -- Rafi ## On 2002-07-15 23:57 -0400 Sush Bhattarai typed: SB> SB> Might want to query your provider as to where the rate limitting is being SB> done. In some cases, if rate limit is being done egress from the layer 3 SB> infracture towards the MAN layer 2 equipment, there might be a lack of SB> processing power on that device, causing the drops. Of course this will SB> depend on the type of device and whether the rate limiting is being done on SB> hardware or not too. SB> SB> Sush SB>
--On Monday, July 15, 2002 10:48 PM -0400 Alex Rubenstein <alex@nac.net> wrote:
I'm trying to troubleshoot a problem with a fractional (311 mbit/second) gigabit-ethernet line provided to me by a metro access provider. Specifically, it is riding a gig-e port of a 15454.
The behavior we are seeing is an occasional loss of packets, adding up to a few percent. When doing a cisco-type ping across the link, we were seeing a consistent 3 to 4 percent loss.
I would definitely recommend using something like iperf, instead of ping if you want to precisely measure your link performance. Ideally you would have two machines on each side of the link to isolate the problem. Is this link in production? We are using a gigabit ethernet to our provider. We are limited on our traffic going to Commodity traffic, but have free reign on our Internet 2 traffic. We found that we get the best results when we shape/police our traffic to stay within our contractual limits, on our side of the link. Since we are using a 6509 with a Sup1A, we had to do some tricky things to police traffic on only one vlan of an 802.1q trunk on the gigE connection. It works though. We see insignificant losses on the link. good luck! Peter Hill Network Engineer Carnegie Mellon University
Is this link in production? We are using a gigabit ethernet to our provider. We are limited on our traffic going to Commodity traffic, but have free reign on our Internet 2 traffic. We found that we get the best results when we shape/police our traffic to stay within our contractual limits, on our side of the link. Since we are using a 6509 with a Sup1A, we had to do some tricky things to police traffic on only one vlan of an 802.1q trunk on the gigE connection. It works though. We see insignificant losses on the link.
can you share how you are doing this? Hybrid or integrated?
good luck!
Peter Hill Network Engineer Carnegie Mellon University
-- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben -- -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
It is very messy, but until we get a Supervisor/MSFC/PFC that can police on egress vlan... We have two bgp sessions with our provider, one that distributes I2 routes, and the other, default. Each points to the other end of their respective /30 subnets on their own vlan on an 802.1q trunk (set up manually, not using VTP. This box is connected to two separate layer 2 cores via two /29s (via 802.1q trunks). Once /29 is the "Commodity" (ie. we must pay) vlan, and has the default-information-originate. Then, the other /29 has its own ospf process (like i said, messy). We redistribute the bgp I2 routes into that ospf process, using as-path filters. Now at the two cores, when traffic for the commodity traffic comes in, it goes to the border router via one vlan, research/I2 traffic the other. We are then able to filter on the ingress vlan on the border router. Now, for you, if don't have a situation where some of your external traffic needs rate-limited, while some can flow as fast and free as it wants, then you just need to do in-bound rate limiting, coming from your internal network to your border router. What sucks for us, is that since we are not experiencing any congestion on our internal network, we can't take advantage of wfq or wred. That was as simple as i could put it, if you have other questions, please ask. --On Tuesday, July 16, 2002 11:41 AM -0400 Alex Rubenstein <alex@nac.net> wrote:
Is this link in production? We are using a gigabit ethernet to our provider. We are limited on our traffic going to Commodity traffic, but have free reign on our Internet 2 traffic. We found that we get the best results when we shape/police our traffic to stay within our contractual limits, on our side of the link. Since we are using a 6509 with a Sup1A, we had to do some tricky things to police traffic on only one vlan of an 802.1q trunk on the gigE connection. It works though. We see insignificant losses on the link.
can you share how you are doing this?
Hybrid or integrated?
good luck!
Peter Hill Network Engineer Carnegie Mellon University
-- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben -- -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
participants (12)
-
Alex Rubenstein
-
Dennis L. Shreve
-
Marshall Eubanks
-
Martin Hannigan
-
Paul Vixie
-
Peter John Hill
-
Phil Rosenthal
-
Rafi Sadowsky
-
Scott Weeks
-
Stephen J. Wilcox
-
Sush Bhattarai
-
Vincent J. Bono