AT&T MPLS / BIB Routers
We just put in a AT&T MPLS and are having a pretty negative experience with the "Business in a Box" routers they are using for our smaller sites. We are seeing extremely high latency under load. Anyone have any experience with these devices that could shed some light on this? Are they really this bad?
On 02/16/2011 05:44 PM, Mikeal Clark wrote:
We just put in a AT&T MPLS and are having a pretty negative experience with the "Business in a Box" routers they are using for our smaller sites. We are seeing extremely high latency under load. Anyone have any experience with these devices that could shed some light on this? Are they really this bad?
There is excessive buffering in all sorts of devices all over the Internet. This causes high latency under load (along with higher packet losses, and lots of other problems. It's what I've been blogging about on http://gettys.wordpress.com. These buffers fill; and they are so large they have defeated TCP congestion avoidance to boot, with horrifying consequences. So far, I've found this problem (almost) everywhere I've looked: o ICSI has good data that bufferbloat is endemic in DSL, Cable, and FIOS. Delays are often measured in seconds (rather than milliseconds). o some corporate and ISP networks run without AQM, in circumstances that they should. o Windows, Mac OSX and Linux all have bufferbloat in their network stacks, at a minimum on recent network device drivers, and often elsewhere. o Every home router I've tested is horrifyingly bad. o 3g networks & 802.11 have this in spades. Why should AT&T's MPLS be any different? My next topic will be "transient" bufferbloat, having to do with defeating slowstart. Come start helping fix this: please join us at bufferbloat.net, as we try to get people to fix it. Already there are some experimental patches for the Linux Intel wireless driver. Jim Gettys Bell Labs
I'm building up to 3000-4000ms latency with these BIB routers. We never had this issue on the old point to points using Cisco gear. On Wed, Feb 16, 2011 at 5:09 PM, Jim Gettys <jg@freedesktop.org> wrote:
On 02/16/2011 05:44 PM, Mikeal Clark wrote:
We just put in a AT&T MPLS and are having a pretty negative experience with the "Business in a Box" routers they are using for our smaller sites. We are seeing extremely high latency under load. Anyone have any experience with these devices that could shed some light on this? Are they really this bad?
There is excessive buffering in all sorts of devices all over the Internet. This causes high latency under load (along with higher packet losses, and lots of other problems.
It's what I've been blogging about on http://gettys.wordpress.com. These buffers fill; and they are so large they have defeated TCP congestion avoidance to boot, with horrifying consequences.
So far, I've found this problem (almost) everywhere I've looked: o ICSI has good data that bufferbloat is endemic in DSL, Cable, and FIOS. Delays are often measured in seconds (rather than milliseconds). o some corporate and ISP networks run without AQM, in circumstances that they should. o Windows, Mac OSX and Linux all have bufferbloat in their network stacks, at a minimum on recent network device drivers, and often elsewhere. o Every home router I've tested is horrifyingly bad. o 3g networks & 802.11 have this in spades.
Why should AT&T's MPLS be any different?
My next topic will be "transient" bufferbloat, having to do with defeating slowstart.
Come start helping fix this: please join us at bufferbloat.net, as we try to get people to fix it. Already there are some experimental patches for the Linux Intel wireless driver. Jim Gettys Bell Labs
On 02/16/2011 06:16 PM, Mikeal Clark wrote:
I'm building up to 3000-4000ms latency with these BIB routers. We never had this issue on the old point to points using Cisco gear.
Bufferbloat is getting more and more common, as memory has gotten cheaper, and braindead people claim that 0 packet loss is a good idea and only test for bandwidth. So push back on AT&T, and start routinely testing latency under saturating load on each and every piece of hardware you bring in the door. You don't see the queues build unless the gear is the bottleneck of a path, so it's easy to have the problem move on you: ergo test everything independently. For example, as soon as your broadband bandwidth exceeds that of your wireless link in your house, your suffering will move from the broadband link to your 802.11 hop. There your home router (and your laptop both) will have even more bufferbloat than most broadband connections, and you suffer even worse. Ironically, the cleaner your 802.11 environment is, the worse you'll suffer, as some packet loss will keep the insanely huge buffers from filling. Most current OS hardware seems to put queues of around 256 packets in the driver, and there may be additional buffering above that. - Jim
On Wed, Feb 16, 2011 at 5:09 PM, Jim Gettys <jg@freedesktop.org <mailto:jg@freedesktop.org>> wrote:
On 02/16/2011 05:44 PM, Mikeal Clark wrote:
We just put in a AT&T MPLS and are having a pretty negative experience with the "Business in a Box" routers they are using for our smaller sites. We are seeing extremely high latency under load. Anyone have any experience with these devices that could shed some light on this? Are they really this bad?
There is excessive buffering in all sorts of devices all over the Internet. This causes high latency under load (along with higher packet losses, and lots of other problems.
It's what I've been blogging about on http://gettys.wordpress.com. These buffers fill; and they are so large they have defeated TCP congestion avoidance to boot, with horrifying consequences.
So far, I've found this problem (almost) everywhere I've looked: o ICSI has good data that bufferbloat is endemic in DSL, Cable, and FIOS. Delays are often measured in seconds (rather than milliseconds). o some corporate and ISP networks run without AQM, in circumstances that they should. o Windows, Mac OSX and Linux all have bufferbloat in their network stacks, at a minimum on recent network device drivers, and often elsewhere. o Every home router I've tested is horrifyingly bad. o 3g networks & 802.11 have this in spades.
Why should AT&T's MPLS be any different?
My next topic will be "transient" bufferbloat, having to do with defeating slowstart.
Come start helping fix this: please join us at bufferbloat.net <http://bufferbloat.net>, as we try to get people to fix it. Already there are some experimental patches for the Linux Intel wireless driver. Jim Gettys Bell Labs
From: Mikeal Clark Sent: Wednesday, February 16, 2011 3:16 PM To: Jim Gettys Cc: nanog@nanog.org Subject: Re: AT&T MPLS / BIB Routers
I'm building up to 3000-4000ms latency with these BIB routers. We never had this issue on the old point to points using Cisco gear.
Something I might try, assuming that the BIB unit plugs into a switch port, is to try bandwidth limiting that port to whatever the CIR is of the MPLS link. If buffering in that path is the problem, limiting the input bandwidth to the box to the maximum of the output bandwidth should eliminate any buffering in the path or the BIB box. Assuming your old Cisco gear was using the same network infrastructure, that might rule out excessive buffering in the MPLS path as the cause, unless AT&T can't actually deliver the advertized bandwidth across the path they are selling. What is the CIR? If you have a 10Meg path and have a GigE jacked into the box, yeah, it's going to get into buffers pretty quick. Maybe even taking the ethernet port down to 10Meg might help, depending on what you are expecting the bandwidth of the path to be.
On 02/17/2011 01:02 AM, George Bonser wrote:
From: Mikeal Clark Sent: Wednesday, February 16, 2011 3:16 PM To: Jim Gettys Cc: nanog@nanog.org Subject: Re: AT&T MPLS / BIB Routers
I'm building up to 3000-4000ms latency with these BIB routers. We never had this issue on the old point to points using Cisco gear.
Something I might try, assuming that the BIB unit plugs into a switch port, is to try bandwidth limiting that port to whatever the CIR is of the MPLS link. If buffering in that path is the problem, limiting the input bandwidth to the box to the maximum of the output bandwidth should eliminate any buffering in the path or the BIB box. Assuming your old Cisco gear was using the same network infrastructure, that might rule out excessive buffering in the MPLS path as the cause, unless AT&T can't actually deliver the advertized bandwidth across the path they are selling.
What is the CIR? If you have a 10Meg path and have a GigE jacked into the box, yeah, it's going to get into buffers pretty quick. Maybe even taking the ethernet port down to 10Meg might help, depending on what you are expecting the bandwidth of the path to be.
Yes, bandwidth limiting is something to try. It's how you can deal with your home broadband connection to inject sanity. Note that you can have bufferbloat just upstream as well. For example, if you plug a GigE ethernet into a 100Mbps switch, if there is buffering upstream, it will fill. http://gettys.wordpress.com/2010/11/29/home-router-puzzle-piece-one-fun-with... In the test case in that post, the bloating is in the laptop plugged into the 100Mbps switch (in the device driver ring, and possibly transmit queue). - Jim
What type of hardware are they using for this BIB router? Cheers Ryan -----Original Message----- From: Mikeal Clark [mailto:mikeal.clark@gmail.com] Sent: Wednesday, February 16, 2011 6:16 PM To: Jim Gettys Cc: nanog@nanog.org Subject: Re: AT&T MPLS / BIB Routers I'm building up to 3000-4000ms latency with these BIB routers. We never had this issue on the old point to points using Cisco gear. On Wed, Feb 16, 2011 at 5:09 PM, Jim Gettys <jg@freedesktop.org> wrote:
On 02/16/2011 05:44 PM, Mikeal Clark wrote:
We just put in a AT&T MPLS and are having a pretty negative experience with the "Business in a Box" routers they are using for our smaller sites. We are seeing extremely high latency under load.
Anyone have any experience with these devices that could shed some light on this? Are they really this bad?
There is excessive buffering in all sorts of devices all over the Internet. This causes high latency under load (along with higher packet losses, and lots of other problems.
It's what I've been blogging about on http://gettys.wordpress.com. These buffers fill; and they are so large they have defeated TCP congestion avoidance to boot, with horrifying consequences.
So far, I've found this problem (almost) everywhere I've looked: o ICSI has good data that bufferbloat is endemic in DSL, Cable,
and FIOS. Delays are often measured in seconds (rather than milliseconds). o some corporate and ISP networks run without AQM, in circumstances that they should. o Windows, Mac OSX and Linux all have bufferbloat in their network stacks, at a minimum on recent network device drivers, and often elsewhere. o Every home router I've tested is horrifyingly bad. o 3g networks & 802.11 have this in spades.
Why should AT&T's MPLS be any different?
My next topic will be "transient" bufferbloat, having to do with defeating slowstart.
Come start helping fix this: please join us at bufferbloat.net, as we try to get people to fix it. Already there are some experimental patches for the Linux Intel wireless driver. Jim Gettys Bell Labs
The routers are Edgemarc. P/N EM-4608T4 http://www.edgewaternetworks.com/edgemarc_overview_page.htm On Thu, Feb 17, 2011 at 8:19 AM, Ryan Finnesey < ryan.finnesey@harrierinvestments.com> wrote:
What type of hardware are they using for this BIB router?
Cheers Ryan
-----Original Message----- From: Mikeal Clark [mailto:mikeal.clark@gmail.com] Sent: Wednesday, February 16, 2011 6:16 PM To: Jim Gettys Cc: nanog@nanog.org Subject: Re: AT&T MPLS / BIB Routers
I'm building up to 3000-4000ms latency with these BIB routers. We never had this issue on the old point to points using Cisco gear.
On Wed, Feb 16, 2011 at 5:09 PM, Jim Gettys <jg@freedesktop.org> wrote:
On 02/16/2011 05:44 PM, Mikeal Clark wrote:
We just put in a AT&T MPLS and are having a pretty negative experience with the "Business in a Box" routers they are using for our smaller sites. We are seeing extremely high latency under load.
Anyone have any experience with these devices that could shed some light on this? Are they really this bad?
There is excessive buffering in all sorts of devices all over the Internet. This causes high latency under load (along with higher packet losses, and lots of other problems.
It's what I've been blogging about on http://gettys.wordpress.com. These buffers fill; and they are so large they have defeated TCP congestion avoidance to boot, with horrifying consequences.
So far, I've found this problem (almost) everywhere I've looked: o ICSI has good data that bufferbloat is endemic in DSL, Cable,
and FIOS. Delays are often measured in seconds (rather than milliseconds). o some corporate and ISP networks run without AQM, in circumstances that they should. o Windows, Mac OSX and Linux all have bufferbloat in their network stacks, at a minimum on recent network device drivers, and often elsewhere. o Every home router I've tested is horrifyingly bad. o 3g networks & 802.11 have this in spades.
Why should AT&T's MPLS be any different?
My next topic will be "transient" bufferbloat, having to do with defeating slowstart.
Come start helping fix this: please join us at bufferbloat.net, as we try to get people to fix it. Already there are some experimental patches for the Linux Intel wireless driver. Jim Gettys Bell Labs
AT&T and Edgemarc now believe they might have shipped routers with firmware that is causing the issues. We will upgrade the firmware on Tuesday and see if that is the case. Thanks everyone for the input. On Thu, Feb 17, 2011 at 9:04 AM, Mikeal Clark <mikeal.clark@gmail.com>wrote:
The routers are Edgemarc. P/N EM-4608T4
http://www.edgewaternetworks.com/edgemarc_overview_page.htm
On Thu, Feb 17, 2011 at 8:19 AM, Ryan Finnesey < ryan.finnesey@harrierinvestments.com> wrote:
What type of hardware are they using for this BIB router?
Cheers Ryan
-----Original Message----- From: Mikeal Clark [mailto:mikeal.clark@gmail.com] Sent: Wednesday, February 16, 2011 6:16 PM To: Jim Gettys Cc: nanog@nanog.org Subject: Re: AT&T MPLS / BIB Routers
I'm building up to 3000-4000ms latency with these BIB routers. We never had this issue on the old point to points using Cisco gear.
On Wed, Feb 16, 2011 at 5:09 PM, Jim Gettys <jg@freedesktop.org> wrote:
On 02/16/2011 05:44 PM, Mikeal Clark wrote:
We just put in a AT&T MPLS and are having a pretty negative experience with the "Business in a Box" routers they are using for our smaller sites. We are seeing extremely high latency under load.
Anyone have any experience with these devices that could shed some light on this? Are they really this bad?
There is excessive buffering in all sorts of devices all over the Internet. This causes high latency under load (along with higher packet losses, and lots of other problems.
It's what I've been blogging about on http://gettys.wordpress.com. These buffers fill; and they are so large they have defeated TCP congestion avoidance to boot, with horrifying consequences.
So far, I've found this problem (almost) everywhere I've looked: o ICSI has good data that bufferbloat is endemic in DSL, Cable,
and FIOS. Delays are often measured in seconds (rather than milliseconds). o some corporate and ISP networks run without AQM, in circumstances that they should. o Windows, Mac OSX and Linux all have bufferbloat in their network stacks, at a minimum on recent network device drivers, and often elsewhere. o Every home router I've tested is horrifyingly bad. o 3g networks & 802.11 have this in spades.
Why should AT&T's MPLS be any different?
My next topic will be "transient" bufferbloat, having to do with defeating slowstart.
Come start helping fix this: please join us at bufferbloat.net, as we try to get people to fix it. Already there are some experimental patches for the Linux Intel wireless driver. Jim Gettys Bell Labs
participants (4)
-
George Bonser -
Jim Gettys -
Mikeal Clark -
Ryan Finnesey