Can someone shed some technical light on the details of how two T1's are bonded (typically). We've got two sets of T's at two different location with vendor 'X' (name starts w/ an 'A') and it appears that we're really only getting about 1 full T's worth of bandwidth and maybe 20% of the second. Seems like they're bonded perhaps using destination IP? It's a vendor managed solution and I need to get some answers faster than they're coming in. Thanks. Matt
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Matt Bazan wrote:
Can someone shed some technical light on the details of how two T1's are bonded (typically). We've got two sets of T's at two different location with vendor 'X' (name starts w/ an 'A') and it appears that we're really only getting about 1 full T's worth of bandwidth and maybe 20% of the second.
Seems like they're bonded perhaps using destination IP? It's a vendor managed solution and I need to get some answers faster than they're coming in. Thanks.
Matt
More than likely they are not bonded t1's they are just load balanced by the router which by default on Cisco is per session. Meaning pc1 to t1#1, pc2to t1#2, pc3 to t1#1. If they are truly bonded with some sort of MUX for a 3 meg port then you would not see the results you are seeing. - -- http://www.digitalrage.org/ The Information Technology News Center -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFD1sXyt06NWq3hlzkRAvi4AJ0R4RVii+Wrxzs5WI5es+FYhxHD0ACgioFW /UHUMapXnmuPFSpKrXzD3JU= =MqxV -----END PGP SIGNATURE-----
If you're treating them as two separate links (e.g. two POPs, etc.) then that's correct, it'll be done by the routers choice of load-balancing (L3). If you are going to the same POP (or box potentially) you can do MLPPP and have a more effective L2 load balancing. Otherwise, it's possible to get an iMux DSU (Digital Link is a vendor as I recall, but there may be others) that allow that magical bonding to occur prior to the router seeing the link. At that point, the router just sees a bigger line coming in (some do 6xT-1 and have a 10meg ethernet output to your router). If you're seeing the balancing the way that you are, most likely that vendor (I have no specific knowledge about the A-vendor) is doing usage-based aggregation which isn't exactly a balancing act. The ones at some of my sites are MLPPP which is a vendor-agnostic approach for the most part. Scott -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Elijah Savage Sent: Tuesday, January 24, 2006 7:28 PM To: Matt Bazan Cc: nanog@merit.edu Subject: Re: T1 bonding -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Matt Bazan wrote:
Can someone shed some technical light on the details of how two T1's are bonded (typically). We've got two sets of T's at two different location with vendor 'X' (name starts w/ an 'A') and it appears that we're really only getting about 1 full T's worth of bandwidth and maybe 20% of the second.
Seems like they're bonded perhaps using destination IP? It's a vendor managed solution and I need to get some answers faster than they're coming in. Thanks.
Matt
More than likely they are not bonded t1's they are just load balanced by the router which by default on Cisco is per session. Meaning pc1 to t1#1, pc2to t1#2, pc3 to t1#1. If they are truly bonded with some sort of MUX for a 3 meg port then you would not see the results you are seeing. - -- http://www.digitalrage.org/ The Information Technology News Center -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFD1sXyt06NWq3hlzkRAvi4AJ0R4RVii+Wrxzs5WI5es+FYhxHD0ACgioFW /UHUMapXnmuPFSpKrXzD3JU= =MqxV -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Scott Morris wrote:
If you're treating them as two separate links (e.g. two POPs, etc.) then that's correct, it'll be done by the routers choice of load-balancing (L3). If you are going to the same POP (or box potentially) you can do MLPPP and have a more effective L2 load balancing.
Otherwise, it's possible to get an iMux DSU (Digital Link is a vendor as I recall, but there may be others) that allow that magical bonding to occur prior to the router seeing the link. At that point, the router just sees a bigger line coming in (some do 6xT-1 and have a 10meg ethernet output to your router).
If you're seeing the balancing the way that you are, most likely that vendor (I have no specific knowledge about the A-vendor) is doing usage-based aggregation which isn't exactly a balancing act. The ones at some of my sites are MLPPP which is a vendor-agnostic approach for the most part.
Scott
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Elijah Savage Sent: Tuesday, January 24, 2006 7:28 PM To: Matt Bazan Cc: nanog@merit.edu Subject: Re: T1 bonding
Matt Bazan wrote:
Can someone shed some technical light on the details of how two T1's are bonded (typically). We've got two sets of T's at two different location with vendor 'X' (name starts w/ an 'A') and it appears that we're really only getting about 1 full T's worth of bandwidth and maybe 20% of the second.
Seems like they're bonded perhaps using destination IP? It's a vendor managed solution and I need to get some answers faster than they're coming in. Thanks.
Matt
More than likely they are not bonded t1's they are just load balanced by the router which by default on Cisco is per session. Meaning pc1 to t1#1, pc2to t1#2, pc3 to t1#1. If they are truly bonded with some sort of MUX for a 3 meg port then you would not see the results you are seeing.
-- http://www.digitalrage.org/ The Information Technology News Center Remember he said both t1's are coming from different vendors, which would only leave the Mux route which is why I said what I said :)
http://www.digitalrage.org/ The Information Technology News Center -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFD1tJWt06NWq3hlzkRApDsAJ9nq+J+26EKYy9cwlFRmN3zhT/EFQCfdf2v IX2wkyZvsGM1sPvcEMSyK+0= =WINE -----END PGP SIGNATURE-----
I'm re-reading it, and slowly, but I don't see mention of having two different vendors. Perhaps I need to put the beer a bit further away, but he talks about generic vendor 'x' and notes that it starts with letter 'A' as further definition, not as two separate vendors. *shrug* Scott -----Original Message----- From: Elijah Savage [mailto:esavage@digitalrage.org] Sent: Tuesday, January 24, 2006 8:20 PM To: swm@emanon.com Cc: 'Matt Bazan'; nanog@merit.edu Subject: Re: T1 bonding -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Scott Morris wrote:
If you're treating them as two separate links (e.g. two POPs, etc.) then that's correct, it'll be done by the routers choice of load-balancing (L3). If you are going to the same POP (or box potentially) you can do MLPPP and have a more effective L2 load balancing.
Otherwise, it's possible to get an iMux DSU (Digital Link is a vendor as I recall, but there may be others) that allow that magical bonding to occur prior to the router seeing the link. At that point, the router just sees a bigger line coming in (some do 6xT-1 and have a 10meg ethernet output to your router).
If you're seeing the balancing the way that you are, most likely that vendor (I have no specific knowledge about the A-vendor) is doing usage-based aggregation which isn't exactly a balancing act. The ones at some of my sites are MLPPP which is a vendor-agnostic approach for the most part.
Scott
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Elijah Savage Sent: Tuesday, January 24, 2006 7:28 PM To: Matt Bazan Cc: nanog@merit.edu Subject: Re: T1 bonding
Matt Bazan wrote:
Can someone shed some technical light on the details of how two T1's are bonded (typically). We've got two sets of T's at two different location with vendor 'X' (name starts w/ an 'A') and it appears that we're really only getting about 1 full T's worth of bandwidth and maybe 20% of the second.
Seems like they're bonded perhaps using destination IP? It's a vendor managed solution and I need to get some answers faster than they're coming in. Thanks.
Matt
More than likely they are not bonded t1's they are just load balanced by the router which by default on Cisco is per session. Meaning pc1 to t1#1, pc2to t1#2, pc3 to t1#1. If they are truly bonded with some sort of MUX for a 3 meg port then you would not see the results you are seeing.
-- http://www.digitalrage.org/ The Information Technology News Center Remember he said both t1's are coming from different vendors, which would only leave the Mux route which is why I said what I said :)
http://www.digitalrage.org/ The Information Technology News Center -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFD1tJWt06NWq3hlzkRApDsAJ9nq+J+26EKYy9cwlFRmN3zhT/EFQCfdf2v IX2wkyZvsGM1sPvcEMSyK+0= =WINE -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Scott Morris wrote:
I'm re-reading it, and slowly, but I don't see mention of having two different vendors. Perhaps I need to put the beer a bit further away, but he talks about generic vendor 'x' and notes that it starts with letter 'A' as further definition, not as two separate vendors.
*shrug*
Scott
-----Original Message----- From: Elijah Savage [mailto:esavage@digitalrage.org] Sent: Tuesday, January 24, 2006 8:20 PM To: swm@emanon.com Cc: 'Matt Bazan'; nanog@merit.edu Subject: Re: T1 bonding
Scott Morris wrote:
If you're treating them as two separate links (e.g. two POPs, etc.) then that's correct, it'll be done by the routers choice of load-balancing (L3). If you are going to the same POP (or box potentially) you can do MLPPP and have a more effective L2 load balancing.
Otherwise, it's possible to get an iMux DSU (Digital Link is a vendor as I recall, but there may be others) that allow that magical bonding to occur prior to the router seeing the link. At that point, the router just sees a bigger line coming in (some do 6xT-1 and have a 10meg ethernet output to your router).
If you're seeing the balancing the way that you are, most likely that vendor (I have no specific knowledge about the A-vendor) is doing usage-based aggregation which isn't exactly a balancing act. The ones at some of my sites are MLPPP which is a vendor-agnostic approach for the most part. Scott
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Elijah Savage Sent: Tuesday, January 24, 2006 7:28 PM To: Matt Bazan Cc: nanog@merit.edu Subject: Re: T1 bonding
Matt Bazan wrote:
Can someone shed some technical light on the details of how two T1's are bonded (typically). We've got two sets of T's at two different location with vendor 'X' (name starts w/ an 'A') and it appears that we're really only getting about 1 full T's worth of bandwidth and maybe 20% of the second.
Seems like they're bonded perhaps using destination IP? It's a vendor managed solution and I need to get some answers faster than they're coming in. Thanks.
Matt
More than likely they are not bonded t1's they are just load balanced by the router which by default on Cisco is per session. Meaning pc1 to t1#1, pc2to t1#2, pc3 to t1#1. If they are truly bonded with some sort of MUX for a 3 meg port then you would not see the results you are seeing.
-- http://www.digitalrage.org/ The Information Technology News Center Remember he said both t1's are coming from different vendors, which would only leave the Mux route which is why I said what I said :) -- http://www.digitalrage.org/ The Information Technology News Center Uh Scott I think it is I whom by the way is getting up right now and going to put the rest of the beer back in the fridge. OOOOPS
http://www.digitalrage.org/ The Information Technology News Center -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFD1tPCt06NWq3hlzkRAqTUAJ44ss3rZxpxv20zXab94GbIbRoudgCaA1J9 3dTi8Msj+xp6qkJvfrSylsY= =CTM7 -----END PGP SIGNATURE-----
Is it AT&T? If so, they only use Cisco Express Forwarding on the router, or so that's at least what I was told by the level 1 techs. If packet order reassembly is a an issue and the link is oversubscribed (IE: Heavy VoIP/gaming use), this method isn't the greatest over others like MLPPP, or per-flow CEF, but in 99% of circumstances it works great (and has other advantages). Can you max out the T-1 with two or three separate "flows" (IE: simultaneous transfers?) If so, it is possible that they are doing per flow and not per-packet load balancing. It should be per packet. Call them up. Once you get screened and transferred to a Cisco guy, fire away with your questions -- they know their stuff in my experience. Or if is your equipment, log into the router and see if ip load-sharing per-packet is set (assuming it is CEF), and confirm they did the same. Off topic, but in my experience MLPPP usually does a better job of getting 190% of a T-1's speed with two of them. CEF usually tops out at around 160-170% with a single flow, but will max out with as little as two flows. I don't know why though, and haven't cared since I've never really had a dual T-1 all to myself without any other users. 2.5 megabit seems to be the single flow norm on our AT&T Circuits at 3 AM with no usage., 2.8-2.9 with two or three flows. As for the technical details, here is some reading material that explains it quite nicely. http://www.cisco.com/en/US/products/hw/modules/ps2033/products_white_paper09... http://www.swcp.com/~jgentry/cisco/cisco-load.html http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft/120... Test file here for speed tests: ftp://ftp1.optonline.net/test64 Matt Bazan wrote:
Can someone shed some technical light on the details of how two T1's are bonded (typically). We've got two sets of T's at two different location with vendor 'X' (name starts w/ an 'A') and it appears that we're really only getting about 1 full T's worth of bandwidth and maybe 20% of the second.
Seems like they're bonded perhaps using destination IP? It's a vendor managed solution and I need to get some answers faster than they're coming in. Thanks.
Matt
Matt Bazan wrote:
Can someone shed some technical light on the details of how two T1's are bonded (typically). We've got two sets of T's at two different location with vendor 'X' (name starts w/ an 'A') and it appears that we're really only getting about 1 full T's worth of bandwidth and maybe 20% of the second.
Seems like they're bonded perhaps using destination IP? It's a vendor managed solution and I need to get some answers faster than they're coming in. Thanks.
Matt
They can be bonded via MLPPP or IMA, as stated previously. Also they can be load-balanced via EIGRP. What are you using to test your bandwidth (IPerf is pretty handy)? I'm kinda assuming that the T1's are point to point, how far apart are the offices? -Wil Matt Bazan wrote:
Can someone shed some technical light on the details of how two T1's are bonded (typically). We've got two sets of T's at two different location with vendor 'X' (name starts w/ an 'A') and it appears that we're really only getting about 1 full T's worth of bandwidth and maybe 20% of the second.
Seems like they're bonded perhaps using destination IP? It's a vendor managed solution and I need to get some answers faster than they're coming in. Thanks.
Matt
participants (5)
-
Elijah Savage -
Matt Bazan -
PC -
Scott Morris -
Wil Schultz