Whoops! 2 hours to find routers w/o an IGP tsk tsk. Dear AT&T IP Services Customer, Please be advised of the following: Date: 8/28/02 Customer Care TT#: 1070909 Start of Impairment: 14:56 ET End of Impairment: 16:52 ET On behalf of AT&T, we would like to extend our apologies for any inconveniences to your business caused by the service impairment and delay in restoring your service. AT&T IP Services customers with traffic routed through our Chicago backbone routers may have been affected by this impairment. Our Network Engineers found that OSPF* network statements were missing from two backbone routers, causing the routing issues that customers were experiencing. AT&T Network Engineers manually reloaded the OSPF network statements and then proceeded to reboot the routers in order for the changes to take effect. The network statements were mistakenly deleted during a routine configuration update to our backbone routers. Management has been made aware of these findings and appropriate actions will be taken to ensure this situation does not re-occur. Prior reported information was leading the AT&T Network Engineers to believe that there were issues with the route reflectors being out of synch. However, this was only symptomatic of the problem stated above. In other words, due to the missing network statements, the route reflectors were not advertising certain networks; therefore traffic was not properly routed. It is AT&T’s goal to provide the highest level of service to our customers. We appreciate your business and look forward to continuing our relationship in the future. *Routing protocol (Open Shortest Path First) Depending on the particular IP access services to which you are subscribed you may also find additional information as it becomes available at: AT&T MIS: https://mis-att.bus.att.com/ AT&T VPNS: http://www.vpn.att.net If you require further information, please feel free to contact AT&T at: AT&T MIS: 1-888-613-6330 AT&T VPNS: 1-888-613-6501 Thank you for using AT&T. Sincerely, The AT&T Customer Care Team -----Original Message----- From: Matt Levine [mailto:matt@deliver3.com] Sent: Wed 8/28/2002 6:21 PM To: Mike Tancsa Cc: Wes Bachman; nanog@merit.edu Subject: Re: AT&T NYC On Wednesday, August 28, 2002, at 04:17 PM, Mike Tancsa wrote:
route-server.ip.att.net is not currently reachable, but AS15290's router server is for those who want a view on things...
Interestingly enough, ATT is announcing 12.0.0.0/23 to BBN (and nobody else, including AS7018 internal)..
route-server.east.attcanada.com. and route-server.west.attcanada.com.
which come in handy :-)
---Mike
At 04:11 PM 28/08/2002 -0400, Mike Tancsa wrote:
I am seeing this as well. One of my upstreams (AT&T Canada- 15290) has connections with AT&T US (7018) in Chicago and Vancouver. Chicago seems to have disappeared for me and all traffic bound via that path is going via Vancouver now.
---Mike
At 02:52 PM 28/08/2002 -0500, Wes Bachman wrote:
Bryan,
There is a known AT&T outage in Chicago currently. Could this be effecting you in some way?
-Wes
On Wed, 2002-08-28 at 14:44, Bryan Heitman wrote:
Anyone seeing any problems with ATT in new york?
Best regards,
Bryan Heitman Interland, Inc.
-- Wes Bachman System & Network Administration, Software Development Leepfrog Technologies, Inc. wbachman@leepfrog.com
-- Matt Levine @Home: matt@deliver3.com @Work: matt@eldosales.com ICQ : 17080004 AIM : exile GPG : http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x6C0D04CF "The Trouble with doing anything right the first time is that nobody appreciates how difficult it was." -BIX
On Wed, 28 Aug 2002, Frank Scalzo wrote:
Whoops! 2 hours to find routers w/o an IGP tsk tsk.
I think it's worth full credit that they actually say what was wrong and why it took so long to fix it. A lot of other companies would mumbo-jumbo a human error like this in order to not lose face. -- Mikael Abrahamsson email: swmike@swm.pp.se
On Thu, Aug 29, 2002 at 06:42:04AM +0200, Mikael Abrahamsson wrote:
On Wed, 28 Aug 2002, Frank Scalzo wrote:
Whoops! 2 hours to find routers w/o an IGP tsk tsk.
I think it's worth full credit that they actually say what was wrong and why it took so long to fix it. A lot of other companies would mumbo-jumbo a human error like this in order to not lose face.
Massive stupidity shouldn't win you points just because you admit to it (note that I consider the *massive* stupidity part to start at the inability to troubleshoot the problem, not the failure in change management). But to be fair, when a NOC or Customer Service person writes an RFO, it usually comes out sounding very very bad, regardless of the actual complexity of the situation. I'm certain all the AT&T customers hope that was the case here. :) -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
Massive stupidity shouldn't win you points just because you admit to it (note that I consider the *massive* stupidity part to start at the inability to troubleshoot the problem, not the failure in change management).
I don't think anyone can point to massive stupidity w/o knowing exactly what happened and the exact steps taken to fix it. Even though the RFO note indicated the "problem" and the "solution", my guess is that things were a bit more complicated than finding a couple config statements, putting them back in and watching the network magically start working again. Of course that is just my opinion, I could be wrong :)
But to be fair, when a NOC or Customer Service person writes an RFO, it usually comes out sounding very very bad, regardless of the actual complexity of the situation. I'm certain all the AT&T customers hope that was the case here. :)
Agreed, and sometimes the RFO tends to be a bit simplistic compared to the actual trouble. -sean Spoon!
It would also be interesting to know which backbone/core product requires a reboot to activate OSPF configuration changes. Sounds like something one should stay away from. Pete Frank Scalzo wrote:
Whoops! 2 hours to find routers w/o an IGP tsk tsk.
Dear AT&T IP Services Customer,
Please be advised of the following:
Date: 8/28/02 Customer Care TT#: 1070909 Start of Impairment: 14:56 ET End of Impairment: 16:52 ET
On behalf of AT&T, we would like to extend our apologies for any inconveniences to your business caused by the service impairment and delay in restoring your service.
AT&T IP Services customers with traffic routed through our Chicago backbone routers may have been affected by this impairment.
Our Network Engineers found that OSPF* network statements were missing from two backbone routers, causing the routing issues that customers were experiencing. AT&T Network Engineers manually reloaded the OSPF network statements and then proceeded to reboot the routers in order for the changes to take effect.
The network statements were mistakenly deleted during a routine configuration update to our backbone routers. Management has been made aware of these findings and appropriate actions will be taken to ensure this situation does not re-occur.
Prior reported information was leading the AT&T Network Engineers to believe that there were issues with the route reflectors being out of synch. However, this was only symptomatic of the problem stated above. In other words, due to the missing network statements, the route reflectors were not advertising certain networks; therefore traffic was not properly routed.
It is AT&T?s goal to provide the highest level of service to our customers. We appreciate your business and look forward to continuing our relationship in the future.
*Routing protocol (Open Shortest Path First)
Depending on the particular IP access services to which you are subscribed you may also find additional information as it becomes available at:
AT&T MIS: https://mis-att.bus.att.com/
AT&T VPNS: http://www.vpn.att.net
If you require further information, please feel free to contact AT&T at:
AT&T MIS: 1-888-613-6330 AT&T VPNS: 1-888-613-6501
Thank you for using AT&T.
Sincerely,
The AT&T Customer Care Team
-----Original Message----- From: Matt Levine [mailto:matt@deliver3.com] Sent: Wed 8/28/2002 6:21 PM To: Mike Tancsa Cc: Wes Bachman; nanog@merit.edu Subject: Re: AT&T NYC
On Wednesday, August 28, 2002, at 04:17 PM, Mike Tancsa wrote:
route-server.ip.att.net is not currently reachable, but AS15290's router server is for those who want a view on things...
Interestingly enough, ATT is announcing 12.0.0.0/23 to BBN (and nobody else, including AS7018 internal)..
route-server.east.attcanada.com. and route-server.west.attcanada.com.
which come in handy :-)
---Mike
At 04:11 PM 28/08/2002 -0400, Mike Tancsa wrote:
I am seeing this as well. One of my upstreams (AT&T Canada- 15290) has connections with AT&T US (7018) in Chicago and Vancouver. Chicago seems to have disappeared for me and all traffic bound via that path is going via Vancouver now.
---Mike
At 02:52 PM 28/08/2002 -0500, Wes Bachman wrote:
Bryan,
There is a known AT&T outage in Chicago currently. Could this be effecting you in some way?
-Wes
On Wed, 2002-08-28 at 14:44, Bryan Heitman wrote:
Anyone seeing any problems with ATT in new york?
Best regards,
Bryan Heitman Interland, Inc.
-- Wes Bachman System & Network Administration, Software Development Leepfrog Technologies, Inc. wbachman@leepfrog.com
-- Matt Levine @Home: matt@deliver3.com @Work: matt@eldosales.com ICQ : 17080004 AIM : exile GPG : http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x6C0D04CF "The Trouble with doing anything right the first time is that nobody appreciates how difficult it was." -BIX
This is impressive. It's very nice to see a carrier providing this level of technical analysis to customers after an outage. Many carriers would be embaressed or try to gloss over what has happened. Sprintlink, in the old days, was also very good about this. Customers really appreciate honesty and being kept in the loop on these sorts of things. Sure, this was a dumb mistake, but everyone has made mistakes - no carrier or engineer is perfect. I think AT&T gets big points on this one. It's a sad statement about our industry that being honest with your customers is the mark of outstanding customer service, rather than average customer service. - Daniel Golding
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Frank Scalzo Sent: Wednesday, August 28, 2002 11:21 PM To: Matt Levine; Mike Tancsa Cc: Wes Bachman; nanog@merit.edu Subject: RE: AT&T NYC
Whoops! 2 hours to find routers w/o an IGP tsk tsk.
Dear AT&T IP Services Customer,
Please be advised of the following:
Date: 8/28/02 Customer Care TT#: 1070909 Start of Impairment: 14:56 ET End of Impairment: 16:52 ET
On behalf of AT&T, we would like to extend our apologies for any inconveniences to your business caused by the service impairment and delay in restoring your service.
AT&T IP Services customers with traffic routed through our Chicago backbone routers may have been affected by this impairment.
Our Network Engineers found that OSPF* network statements were missing from two backbone routers, causing the routing issues that customers were experiencing. AT&T Network Engineers manually reloaded the OSPF network statements and then proceeded to reboot the routers in order for the changes to take effect.
The network statements were mistakenly deleted during a routine configuration update to our backbone routers. Management has been made aware of these findings and appropriate actions will be taken to ensure this situation does not re-occur.
Prior reported information was leading the AT&T Network Engineers to believe that there were issues with the route reflectors being out of synch. However, this was only symptomatic of the problem stated above. In other words, due to the missing network statements, the route reflectors were not advertising certain networks; therefore traffic was not properly routed.
It is AT&T’s goal to provide the highest level of service to our customers. We appreciate your business and look forward to continuing our relationship in the future.
*Routing protocol (Open Shortest Path First)
Depending on the particular IP access services to which you are subscribed you may also find additional information as it becomes available at:
AT&T MIS: https://mis-att.bus.att.com/
AT&T VPNS: http://www.vpn.att.net
If you require further information, please feel free to contact AT&T at:
AT&T MIS: 1-888-613-6330 AT&T VPNS: 1-888-613-6501
Thank you for using AT&T.
Sincerely,
The AT&T Customer Care Team
-----Original Message----- From: Matt Levine [mailto:matt@deliver3.com] Sent: Wed 8/28/2002 6:21 PM To: Mike Tancsa Cc: Wes Bachman; nanog@merit.edu Subject: Re: AT&T NYC
On Wednesday, August 28, 2002, at 04:17 PM, Mike Tancsa wrote:
route-server.ip.att.net is not currently reachable, but AS15290's router server is for those who want a view on things...
Interestingly enough, ATT is announcing 12.0.0.0/23 to BBN (and nobody else, including AS7018 internal)..
route-server.east.attcanada.com. and route-server.west.attcanada.com.
which come in handy :-)
---Mike
At 04:11 PM 28/08/2002 -0400, Mike Tancsa wrote:
I am seeing this as well. One of my upstreams (AT&T Canada- 15290) has connections with AT&T US (7018) in Chicago and Vancouver. Chicago seems to have disappeared for me and all traffic bound via that path is going via Vancouver now.
---Mike
At 02:52 PM 28/08/2002 -0500, Wes Bachman wrote:
Bryan,
There is a known AT&T outage in Chicago currently. Could this be effecting you in some way?
-Wes
On Wed, 2002-08-28 at 14:44, Bryan Heitman wrote:
Anyone seeing any problems with ATT in new york?
Best regards,
Bryan Heitman Interland, Inc.
-- Wes Bachman System & Network Administration, Software Development Leepfrog Technologies, Inc. wbachman@leepfrog.com
-- Matt Levine @Home: matt@deliver3.com @Work: matt@eldosales.com ICQ : 17080004 AIM : exile GPG : http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x6C0D04CF "The Trouble with doing anything right the first time is that nobody appreciates how difficult it was." -BIX
Has anybody mentioned the benefits of ISIS as an IGP to them. Kesva -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Daniel Golding Sent: Thursday, August 29, 2002 10:57 AM To: Frank Scalzo; Matt Levine; Mike Tancsa Cc: Wes Bachman; nanog@merit.edu Subject: RE: AT&T NYC This is impressive. It's very nice to see a carrier providing this level of technical analysis to customers after an outage. Many carriers would be embaressed or try to gloss over what has happened. Sprintlink, in the old days, was also very good about this. Customers really appreciate honesty and being kept in the loop on these sorts of things. Sure, this was a dumb mistake, but everyone has made mistakes - no carrier or engineer is perfect. I think AT&T gets big points on this one. It's a sad statement about our industry that being honest with your customers is the mark of outstanding customer service, rather than average customer service. - Daniel Golding
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Frank Scalzo Sent: Wednesday, August 28, 2002 11:21 PM To: Matt Levine; Mike Tancsa Cc: Wes Bachman; nanog@merit.edu Subject: RE: AT&T NYC
Whoops! 2 hours to find routers w/o an IGP tsk tsk.
Dear AT&T IP Services Customer,
Please be advised of the following:
Date: 8/28/02 Customer Care TT#: 1070909 Start of Impairment: 14:56 ET End of Impairment: 16:52 ET
On behalf of AT&T, we would like to extend our apologies for any inconveniences to your business caused by the service impairment and delay in restoring your service.
AT&T IP Services customers with traffic routed through our Chicago backbone routers may have been affected by this impairment.
Our Network Engineers found that OSPF* network statements were missing from two backbone routers, causing the routing issues that customers were experiencing. AT&T Network Engineers manually reloaded the OSPF network statements and then proceeded to reboot the routers in order for the changes to take effect.
The network statements were mistakenly deleted during a routine configuration update to our backbone routers. Management has been made aware of these findings and appropriate actions will be taken to ensure this situation does not re-occur.
Prior reported information was leading the AT&T Network Engineers to believe that there were issues with the route reflectors being out of synch. However, this was only symptomatic of the problem stated above. In other words, due to the missing network statements, the route reflectors were not advertising certain networks; therefore traffic was not properly routed.
It is AT&T’s goal to provide the highest level of service to our customers. We appreciate your business and look forward to continuing our relationship in the future.
*Routing protocol (Open Shortest Path First)
Depending on the particular IP access services to which you are subscribed you may also find additional information as it becomes available at:
AT&T MIS: https://mis-att.bus.att.com/
AT&T VPNS: http://www.vpn.att.net
If you require further information, please feel free to contact AT&T at:
AT&T MIS: 1-888-613-6330 AT&T VPNS: 1-888-613-6501
Thank you for using AT&T.
Sincerely,
The AT&T Customer Care Team
-----Original Message----- From: Matt Levine [mailto:matt@deliver3.com] Sent: Wed 8/28/2002 6:21 PM To: Mike Tancsa Cc: Wes Bachman; nanog@merit.edu Subject: Re: AT&T NYC
On Wednesday, August 28, 2002, at 04:17 PM, Mike Tancsa wrote:
route-server.ip.att.net is not currently reachable, but AS15290's router server is for those who want a view on things...
Interestingly enough, ATT is announcing 12.0.0.0/23 to BBN (and nobody else, including AS7018 internal)..
route-server.east.attcanada.com. and route-server.west.attcanada.com.
which come in handy :-)
---Mike
At 04:11 PM 28/08/2002 -0400, Mike Tancsa wrote:
I am seeing this as well. One of my upstreams (AT&T Canada- 15290) has connections with AT&T US (7018) in Chicago and Vancouver. Chicago seems to have disappeared for me and all traffic bound via that path is going via Vancouver now.
---Mike
At 02:52 PM 28/08/2002 -0500, Wes Bachman wrote:
Bryan,
There is a known AT&T outage in Chicago currently. Could this be effecting you in some way?
-Wes
On Wed, 2002-08-28 at 14:44, Bryan Heitman wrote:
Anyone seeing any problems with ATT in new york?
Best regards,
Bryan Heitman Interland, Inc.
-- Wes Bachman System & Network Administration, Software Development Leepfrog Technologies, Inc. wbachman@leepfrog.com
-- Matt Levine @Home: matt@deliver3.com @Work: matt@eldosales.com ICQ : 17080004 AIM : exile GPG : http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x6C0D04CF "The Trouble with doing anything right the first time is that nobody appreciates how difficult it was." -BIX
On Thu, 29 Aug 2002, NAIDOO Kesva FTLD/IAP wrote: : :Has anybody mentioned the benefits of ISIS as an IGP to them. : Of course, ISIS is no more resilient against the deletion of igp configuration than OSPF. cheers, brian
On Thu, Aug 29, 2002 at 01:09:54PM -0400, alex@yuriev.com wrote:
Has anybody mentioned the benefits of ISIS as an IGP to them. Link-state protocols are evil, and when they break, they *really* break. I still do not see a compeling argument for not using BGP as your IGP.
Slow convergence. Greetz, Peter -- MegaBIT - open air networking event - http://www.megabit.nl/
On Thu, 29 Aug 2002, Peter van Dijk wrote:
On Thu, Aug 29, 2002 at 01:09:54PM -0400, alex@yuriev.com wrote:
Has anybody mentioned the benefits of ISIS as an IGP to them. Link-state protocols are evil, and when they break, they *really* break. I still do not see a compeling argument for not using BGP as your IGP.
Slow convergence.
As well there is the issues of running a full iBGP mesh. I've actually been doing it, and now that I'm about to add my 5th router, OSPF is looking a lot better than configuring 4 more BGP sessions. I've heard some people recommend a route-reflector, but that would mean if the route-reflector goes down you're screwed. -Ralph
On Thu, Aug 29, 2002 at 03:07:59PM -0400, Ralph Doncaster wrote:
On Thu, 29 Aug 2002, Peter van Dijk wrote:
On Thu, Aug 29, 2002 at 01:09:54PM -0400, alex@yuriev.com wrote:
Has anybody mentioned the benefits of ISIS as an IGP to them. Link-state protocols are evil, and when they break, they *really* break. I still do not see a compeling argument for not using BGP as your IGP.
Slow convergence.
As well there is the issues of running a full iBGP mesh. I've actually been doing it, and now that I'm about to add my 5th router, OSPF is looking a lot better than configuring 4 more BGP sessions. I've heard some people recommend a route-reflector, but that would mean if the route-reflector goes down you're screwed.
Planning on doing away with that pesky IBGP mesh and just redistributing BGP into OSPF are we Ralph? There is so much wrong with the above post that I can't do anything except hang my head in shame for people running networks everywhere around the world. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
On Thu, Aug 29, 2002 at 01:09:54PM -0400, alex@yuriev.com wrote:
Has anybody mentioned the benefits of ISIS as an IGP to them. Link-state protocols are evil, and when they break, they *really*
break.
I still do not see a compeling argument for not using BGP as your IGP.
Slow convergence.
As well there is the issues of running a full iBGP mesh. I've actually been doing it, and now that I'm about to add my 5th router, OSPF is looking a lot better than configuring 4 more BGP sessions. I've heard some people recommend a route-reflector, but that would mean if the route-reflector goes down you're screwed.
What about a well choosen (wrt topo) pair of them...
Planning on doing away with that pesky IBGP mesh and just redistributing BGP into OSPF are we Ralph?
There is so much wrong with the above post that I can't do anything except hang my head in shame for people running networks everywhere around the world.
mh
-- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
I wish this was a joke, but I know it's not. Ralph, they are talking about running BGP as an IGP, not if they are going to run BGP at all. Most large carriers run BGP everywhere. They also run an IGP for next-hop reachability within their networks (loopbacks, interface /30s, etc). The issue was whether you can get away with not running the IGP, and just running BGP. The problem is, of course, BGP handles many routes well, and converges relatively slowly. IGPs converge quickly, but only handle a relatively small number of routes. If you are offering transit to folks, you need to run BGP pretty much everywhere. If you are peering, or have multiple transits in multiple locations, with a network between them, BGP makes a lot of sense. If not, just run BGP on your edge routers. As far as an IGP - you need one. You don't need route reflectors for 5 routers. You probably do need something at double that number. Use two route reflectors, so if one goes down, you're ok. Redundancy is wonderful. Or use confederation BGP, which doesn't usually have single points of failure, but many be a bit too complex for some networks. Finally, if cutting and pasting a configuration is making you a "sad clown", learn Expect or Perl, and write a script. That way, more routers can be broken in a shorter period of time, leading to greater efficiency. We now return to our regularly scheduled, low level of signal to noise. - Daniel Golding
Ralph Doncaster Said....
On Thu, 29 Aug 2002, Peter van Dijk wrote:
On Thu, Aug 29, 2002 at 01:09:54PM -0400, alex@yuriev.com wrote:
Has anybody mentioned the benefits of ISIS as an IGP to them. Link-state protocols are evil, and when they break, they *really* break. I still do not see a compeling argument for not using BGP as your IGP.
Slow convergence.
As well there is the issues of running a full iBGP mesh. I've actually been doing it, and now that I'm about to add my 5th router, OSPF is looking a lot better than configuring 4 more BGP sessions. I've heard some people recommend a route-reflector, but that would mean if the route-reflector goes down you're screwed.
-Ralph
Daniel, I'm talking about using OSPF as my IGP instead of an iBGP mesh. Even if I were crazy enough to want to use OSPF as my EGP, there's no transit provider crazy enough to use OSPF as their EGP to customers (at least that I know of). -Ralph On Thu, 29 Aug 2002, Daniel Golding wrote:
Ralph, they are talking about running BGP as an IGP, not if they are going to run BGP at all. Most large carriers run BGP everywhere. They also run an
daniel, why would you return to that state? -- dima.
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Daniel Golding Sent: Thursday, August 29, 2002 3:27 PM To: Ralph Doncaster; Peter van Dijk Cc: nanog@merit.edu Subject: RE: AT&T NYC
We now return to our regularly scheduled, low level of signal to noise.
- Daniel Golding
Dmitri, Absolutely unavoidable. I think it's called Dalph Roncaster's Law of Impropability. Derek
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Dmitri Krioukov Sent: Thursday, August 29, 2002 5:10 PM To: Daniel Golding Cc: nanog@merit.edu Subject: RE: AT&T NYC
daniel, why would you return to that state? -- dima.
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Daniel Golding Sent: Thursday, August 29, 2002 3:27 PM To: Ralph Doncaster; Peter van Dijk Cc: nanog@merit.edu Subject: RE: AT&T NYC
We now return to our regularly scheduled, low level of signal to noise.
- Daniel Golding
Um. Set up more than one reflector.... On Thu, 29 Aug 2002, Ralph Doncaster wrote:
On Thu, 29 Aug 2002, Peter van Dijk wrote:
On Thu, Aug 29, 2002 at 01:09:54PM -0400, alex@yuriev.com wrote:
Has anybody mentioned the benefits of ISIS as an IGP to them. Link-state protocols are evil, and when they break, they *really* break. I still do not see a compeling argument for not using BGP as your IGP.
Slow convergence.
As well there is the issues of running a full iBGP mesh. I've actually been doing it, and now that I'm about to add my 5th router, OSPF is looking a lot better than configuring 4 more BGP sessions. I've heard some people recommend a route-reflector, but that would mean if the route-reflector goes down you're screwed.
-Ralph
It depends a lot on your topgraphy. For example, if you have a specific city/region aggregation center, your redundant boarder routers for that city are probably also RRs. Those boarders peer with your boarders in other cities as well as your intra-region routers. Of course, this is just one way. Trying not to start a religious war. On Thu, 29 Aug 2002, Ralph Doncaster wrote:
On Thu, 29 Aug 2002, Robert A. Hayden wrote:
Um. Set up more than one reflector....
So how many is enough? I would think 3 is a minimum to come close to the reliability/redundancy of OSPF.
-Ralph
Figure out how to do reverse route reflecting.
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Ralph Doncaster Sent: Thursday, August 29, 2002 3:46 PM To: Robert A. Hayden Cc: Peter van Dijk; nanog@merit.edu Subject: routing architectures ( was Re: AT&T NYCrouting )
On Thu, 29 Aug 2002, Robert A. Hayden wrote:
Um. Set up more than one reflector....
So how many is enough? I would think 3 is a minimum to come close to the reliability/redundancy of OSPF.
-Ralph
Uhh, come to think of it, the term reverse route reflecting probably won't get you much help -- client to client route reflecting is probably an easier term to understand.. My bad.
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Ralph Doncaster Sent: Thursday, August 29, 2002 3:46 PM To: Robert A. Hayden Cc: Peter van Dijk; nanog@merit.edu Subject: routing architectures ( was Re: AT&T NYCrouting )
On Thu, 29 Aug 2002, Robert A. Hayden wrote:
Um. Set up more than one reflector....
So how many is enough? I would think 3 is a minimum to come close to the reliability/redundancy of OSPF.
-Ralph
On Thu, 29 Aug 2002, Robert A. Hayden wrote:
Um. Set up more than one reflector....
So how many is enough? I would think 3 is a minimum to come close to the reliability/redundancy of OSPF.
-Ralph
My two cents.. BGP beats OSPF in the reliability stakes easily in my experience. I'd put as little as possible (loopbacks/internal links only) into OSPF. Two (or more) BGP route reflectors deployed to reflect the physical topology/area structure is explicitly redundant in a way that OSPF isn't. Ross
Um. Set up more than one reflector....
So how many is enough? I would think 3 is a minimum to come close to the reliability/redundancy of OSPF.
I fail to see what reflectors have to do with stability (except for the obicous human errors and configuration mistakes). Reflectors and confederations is more to help with scaling. And I would then expect the number of BGP speaking routers to be up around 10-15-20. It doesn't make much sense below that (unless you are playing with the number of routes on boxes). Last I fail to see what the number of reflectors have to do with correlation to a IGP? - kurtis -
Um. Set up more than one reflector....
yes... and align your setup with your physical topology(so making it useful); use other proto for mapping your infra, etc, etc,.. mh On Thu, 29 Aug 2002, Ralph Doncaster wrote:
On Thu, 29 Aug 2002, Peter van Dijk wrote:
On Thu, Aug 29, 2002 at 01:09:54PM -0400, alex@yuriev.com wrote:
Has anybody mentioned the benefits of ISIS as an IGP to them. Link-state protocols are evil, and when they break, they *really*
break.
I still do not see a compeling argument for not using BGP as your IGP.
Slow convergence.
As well there is the issues of running a full iBGP mesh. I've actually been doing it, and now that I'm about o add my 5th router, OSPF is looking a lot better than configuring 4 more BGP sessions. I've heard some people recommend a route-reflector, but that would mean if the route-reflector goes down you're screwed.
-Ralph
Yup. I like using OSPF to set up the mesh to the loopbacks and then ibgp as the IGP. On Thu, 29 Aug 2002, Michael Hallgren wrote:
Um. Set up more than one reflector....
yes... and align your setup with your physical topology(so making it useful); use other proto for mapping your infra, etc, etc,..
mh
On Thu, 29 Aug 2002, Ralph Doncaster wrote:
On Thu, 29 Aug 2002, Peter van Dijk wrote:
On Thu, Aug 29, 2002 at 01:09:54PM -0400, alex@yuriev.com wrote:
Has anybody mentioned the benefits of ISIS as an IGP to them. Link-state protocols are evil, and when they break, they *really*
break.
I still do not see a compeling argument for not using BGP as your IGP.
Slow convergence.
As well there is the issues of running a full iBGP mesh. I've actually been doing it, and now that I'm about o add my 5th router, OSPF is looking a lot better than configuring 4 more BGP sessions. I've heard some people recommend a route-reflector, but that would mean if the route-reflector goes down you're screwed.
-Ralph
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Robert A. Hayden Sent: Thursday, August 29, 2002 3:53 PM To: Michael Hallgren Cc: Ralph Doncaster; Peter van Dijk; nanog@merit.edu Subject: RE: AT&T NYC
Yup. I like using OSPF to set up the mesh to the loopbacks and then ibgp as the IGP.
On Thu, 29 Aug 2002, Michael Hallgren wrote:
Um. Set up more than one reflector....
yes... and align your setup with your physical topology(so making it useful); use other proto for mapping your infra, etc, etc,..
mh
On Thu, 29 Aug 2002, Ralph Doncaster wrote:
On Thu, 29 Aug 2002, Peter van Dijk wrote:
On Thu, Aug 29, 2002 at 01:09:54PM -0400, alex@yuriev.com wrote:
Has anybody mentioned the benefits of ISIS as an IGP to
Link-state protocols are evil, and when they break, they *really* break. I still do not see a compeling argument for not using BGP as your IGP.
Slow convergence.
As well there is the issues of running a full iBGP mesh. I've actually been doing it, and now that I'm about o add my 5th router, OSPF is looking a lot better than configuring 4 more BGP sessions. I've heard some people recommend a route-reflector, but that would mean if
I personally prefer using IS-IS for loopback/infrastructure routes, and I use confederations for my IBGP. If a confederation ever gets to large, I can always add a route-reflector inside the confederation. Ralph, you have never failed to amaze me with your love for WCP (Worst Current Practices.) Derek them. the
route-reflector goes down you're screwed.
-Ralph
On Thu, 29 Aug 2002, Derek Samford wrote:
I personally prefer using IS-IS for loopback/infrastructure routes, and I use confederations for my IBGP. If a confederation ever gets to large, I can always add a route-reflector inside the confederation. Ralph, you have never failed to amaze me with your love for WCP (Worst Current Practices.)
OK, then hand me a clue and explain why ruing an iBGP mesh with 3-4 routers is so bad (seeing as Bassam Halabi didn't in his book). -Ralph
Ralph, Okay, no one ever said an IBGP mesh was bad. We were all upset by the mention of an IGP distributed into an EGP. Let's do a little math here. The formula for IBGP sessions goes as follows. n*(n-1)/2 2=1 3=3 4=6 5=10 So you've only got 4 routers? That's fine, 6 sessions is not too hard to maintain. However, one more router, annd10 can get to be cumbersome and 10 Routers in your network, you have to maintain 45 BGP sessions. My personal favorite approach (And this may, or may not, start a religious war.) is confederations. The great part is, if your IBGP mesh inside a Sub-as gets to large, you can add a route-reflector, and have a hybrid RR/Confederation approach. This is very scalable, although there are some issues with being able to follow shortest path out of a confederation, so you need to have a little skill at traffic engineering. Building networks is easy Ralph, building SCALEABLE networks, is not. Derek
-----Original Message----- From: Ralph Doncaster [mailto:ralph@istop.com] Sent: Thursday, August 29, 2002 4:44 PM To: Derek Samford Cc: 'Robert A. Hayden'; 'Michael Hallgren'; 'Peter van Dijk'; nanog@merit.edu Subject: RE: AT&T NYC
On Thu, 29 Aug 2002, Derek Samford wrote:
I personally prefer using IS-IS for loopback/infrastructure routes, and I use confederations for my IBGP. If a confederation ever gets to large, I can always add a route-reflector inside the confederation. Ralph, you have never failed to amaze me with your love for WCP (Worst Current Practices.)
OK, then hand me a clue and explain why ruing an iBGP mesh with 3-4 routers is so bad (seeing as Bassam Halabi didn't in his book).
-Ralph
On Thu, 29 Aug 2002, Derek Samford wrote:
Ralph, Okay, no one ever said an IBGP mesh was bad. We were all upset by the mention of an IGP distributed into an EGP.
Derek, I think we're both confused now. Your example seems to have nothing to do with what I'm talking about. I'm currently using an iBGP mesh in my network, with no OSPF or IS-IS. In other words I have internal routers not connected to external peers that are running iBGP. Specifically I have 2 routers that are ruinning EBGP and iBGP, and 2 routers that are running iBGP only. Now that I'm adding a 5th router to my network, I'm considering running OSPF for my IGP. I would still run iBGP between my 2 peering routers, as well as EBGP to my peers. Now if there really is something horrible about that, and someone will politely explain it to me, I'll happily take the dunce cap and sit in the corner for 5 minutes. -Ralph
"Ralph" == Ralph Doncaster <ralph@istop.com> writes:
Ralph> I think we're both confused now. Your example seems to Ralph> have nothing to do with what I'm talking about. I'm Ralph> currently using an iBGP mesh in my network, with no OSPF or Ralph> IS-IS. In other words I have internal routers not Ralph> connected to external peers that are running iBGP. Ralph> Specifically I have 2 routers that are ruinning EBGP and Ralph> iBGP, and 2 routers that are running iBGP only. Now that Ralph> I'm adding a 5th router to my network, I'm considering Ralph> running OSPF for my IGP. I would still run iBGP between my Ralph> 2 peering routers, as well as EBGP to my peers. Ralph, there is nothing wrong with doing that. Just make sure that you don't have any routers without full tables in your transit path -- i.e. between your two peering routers -- and it will be fine as long as you don't do anything wacky like redistributing BGP routes into OSPF. Be careful with redistributing in the other direction also. There's no problem. It won't ruin anything. It is a pretty standard setup. Cheers, -w
-----Original Message----- From: Ralph Doncaster [mailto:ralph@istop.com] Sent: Thursday, August 29, 2002 5:11 PM To: Derek Samford Cc: 'Robert A. Hayden'; 'Michael Hallgren'; 'Peter van Dijk'; nanog@merit.edu Subject: RE: AT&T NYC
On Thu, 29 Aug 2002, Derek Samford wrote:
Ralph, Okay, no one ever said an IBGP mesh was bad. We were all upset by the mention of an IGP distributed into an EGP.
Derek,
I think we're both confused now. Your example seems to have nothing to do with what I'm talking about. I'm currently using an iBGP mesh in my network, with no OSPF or IS-IS. In other words I have internal routers not connected to external peers that are running iBGP. Specifically I have 2 routers that are ruinning EBGP and iBGP, and 2 routers that are running iBGP only. Now that I'm adding a 5th router to my network, I'm considering running OSPF for my IGP. I would still run iBGP between my 2 peering routers, as well as EBGP to my peers.
Now if there really is something horrible about that, and someone will politely explain it to me, I'll happily take the dunce cap and sit in
Ralph, It all depends on your application. Will you be peering with your clients, or planning to at some point. If that router has anything to do with transit, then you need the BGP tables, as you need to be able to hold path attributes. Every time you see one of us mention ISIS or OSPF, all it has to do with is carrying loopback/infrastructure routes. The rest of our prefixes are carried over BGP. If your application doesn't require you hold-on to attributes (Basically, you never plan on running BGP with a client off of that router or that router will never be doing transport for a router in your network that has clients running BGP.) Or, you could run MPLS, but we won't even go there. Derek the
corner for 5 minutes.
-Ralph
Every time you see one of us mention ISIS or OSPF, all it has to do with is carrying loopback/infrastructure routes.
I don't think anyone has said to Ralph why the above is done. Just in case it isn't obvious: you need to make sure the next-hops are known on each router by a means other than bgp. -mark
On Thu, 29 Aug 2002, Ralph Doncaster wrote:
Ralph, you have never failed to amaze me with your love for WCP (Worst Current Practices.)
OK, then hand me a clue and explain why ruing an iBGP mesh with 3-4 routers is so bad (seeing as Bassam Halabi didn't in his book).
I would definately not recommend route reflection or a confederation in a network with such a small number of BGP routers: you add lots of complexity, potential for less optimal routing (and possibly obscure implementation problems) and the only thing you gain is not having to reconfigure the existing BGP routers when you add a new one. If this is really your main concern, you can always make one router a route reflector but still keep the full IBGP mesh. Then you can add a new BGP router and you just have to configure a session with the reflector and save setting up IBGP sessions with the rest of the BGP routers for when you have some time on your hands. (Obviously you have a problem if the reflector fails before you get around to this.) I'm pretty sure this will earn me a place on someone's worst practices list, though. :-) Iljitsch van Beijnum
Practices.)
OK, then hand me a clue and explain why ruing an iBGP mesh with 3-4 routers is so bad (seeing as Bassam Halabi didn't in his book).
I would definately not recommend route reflection or a confederation in a network with such a small number of BGP routers: you add lots of complexity, potential for less optimal routing (and possibly obscure implementation problems) and the only thing you gain is not having to the existing BGP routers when you add a new one.
In spirit of the above, I suggest you also do not have maintenance window, forget about re-checking configs, enable to do the simpliest commands, and of course let everyone try debugging any problems by typing random commands into your routers. After all, your network consists of only 4 routers. Alex P.S. In case it not clear, the above paragraph is not what you should ever do.
On Fri, 30 Aug 2002 alex@yuriev.com wrote:
I would definately not recommend route reflection or a confederation in a network with such a small number of BGP routers:
In spirit of the above, I suggest you also do not have maintenance window, forget about re-checking configs, enable to do the simpliest commands, and of course let everyone try debugging any problems by typing random commands into your routers. After all, your network consists of only 4 routers.
Ok, but how is _not_ using route reflection or a confederation similar to the above??? Iljitsch
On Fri, 30 Aug 2002 alex@yuriev.com wrote:
I would definately not recommend route reflection or a confederation in a network with such a small number of BGP routers:
In spirit of the above, I suggest you also do not have maintenance window, forget about re-checking configs, enable to do the simpliest commands, and of course let everyone try debugging any problems by typing random commands into your routers. After all, your network consists of only 4 routers.
Ok, but how is _not_ using route reflection or a confederation similar to the above???
I fail to see how using confederations makes it complicated. In fact, should a network consist of two or above pops, and has more than one exit point, confederations are an excellent way to give one ability to fine tune traffic distribution. Alex
On Thu, 29 Aug 2002, Peter van Dijk wrote:
On Thu, Aug 29, 2002 at 01:09:54PM -0400, alex@yuriev.com wrote:
Has anybody mentioned the benefits of ISIS as an IGP to them. Link-state protocols are evil, and when they break, they *really* break. I still do not see a compeling argument for not using BGP as your IGP.
Slow convergence.
As well there is the issues of running a full iBGP mesh. I've actually been doing it, and now that I'm about to add my 5th router, OSPF is looking a lot better than configuring 4 more BGP sessions. I've heard some people recommend a route-reflector, but that would mean if the route-reflector goes down you're screwed.
Confederations are your friends. route-map set-igp-community is your friend. Alex
-Ralph
--
On Thu, Aug 29, 2002 at 01:09:54PM -0400, alex@yuriev.com wrote:
Has anybody mentioned the benefits of ISIS as an IGP to them. Link-state protocols are evil, and when they break, they *really* break. I still do not see a compeling argument for not using BGP as your IGP.
Slow convergence.
If you do not randomly MESS with your routers and circuits outside the maintenance window, your customers will apreciate stability over how fast it converge. As alternative, you get full blown outages a-la AboveNet and AT&T. Alex
Has anybody mentioned the benefits of ISIS as an IGP to them.
Link-state protocols are evil, and when they break, they *really* break. I still do not see a compeling argument for not using BGP as your IGP.
Alex
iBGP is only one half of an IGP. It is the "where to go" half. You still need some other igp (isis, ospf, rip, static routes, etc) for the "how to get there" half. Most large providers carry next-hops (loopbacks, border /30 (or /31), etc) around in either isis or ospf, and carry the remainder in iBGP. The reason? With link-state, one interface flap can mean doing SPF on every route. If "every route" is only a couple hundred, rather than 100K, you fare better when circuits are flapping. At that point, comparing the precomputed metric amongst 100k routes is (imho) rather trivial, especially when "igp metric" is a few steps down the decision tree. In all practicality, you need to haul, at least, eBGP routes around in iBGP, you need some kind of other igp to jumpstart iBGP, and is advised that this other igp have some concept of metric or cost to be able to give BGP some hints. Whether you choose to make your non-BGP igp lean and mean (disabling synchronization, with the attendant caveats) or fat and happy (and suffer the consequences during repeated link state changes), is up to the reader, but you still really need two igps.
bdragon@gweep.net wrote:
With link-state, one interface flap can mean doing SPF on every route.
Only if you learned every one of your routes from different neighbor. If you have two exits and 100000 routes, you calculate twice and apply the results to the prefixes. Note that this does not apply to a proprietary, "hybrid", semi-link state protocol marketed with name "EIGRP" where all routes need per-prefix calculation. (OSPF and IS-IS work fine) Pete
On Mon, 2 Sep 2002 bdragon@gweep.net wrote:
Has anybody mentioned the benefits of ISIS as an IGP to them.
Link-state protocols are evil, and when they break, they *really* break. I still do not see a compeling argument for not using BGP as your IGP.
Convergence time?
Alex
iBGP is only one half of an IGP. It is the "where to go" half. You still need some other igp (isis, ospf, rip, static routes, etc) for the "how to get there" half.
Most large providers carry next-hops (loopbacks, border /30 (or /31), etc) around in either isis or ospf, and carry the remainder in iBGP.
The reason?
With link-state, one interface flap can mean doing SPF on every route. If "every route" is only a couple hundred, rather than 100K, you fare
As you say disable synchronization and try and control the physical reach of your igp by some mechanism.. areas, summaries, ASes etc Steve
better when circuits are flapping. At that point, comparing the precomputed metric amongst 100k routes is (imho) rather trivial, especially when "igp metric" is a few steps down the decision tree.
In all practicality, you need to haul, at least, eBGP routes around in iBGP, you need some kind of other igp to jumpstart iBGP, and is advised that this other igp have some concept of metric or cost to be able to give BGP some hints. Whether you choose to make your non-BGP igp lean and mean (disabling synchronization, with the attendant caveats) or fat and happy (and suffer the consequences during repeated link state changes), is up to the reader, but you still really need two igps.
Has anybody mentioned the benefits of ISIS as an IGP to them.
Link-state protocols are evil, and when they break, they *really* break. I still do not see a compeling argument for not using BGP as your IGP.
Convergence time?
What is better - relatively long convergence time on affected routes or a problem on unaffected route? Ask your customers. They do not care if someone else is having a problem. They care that they dont.
With link-state, one interface flap can mean doing SPF on every route. If "every route" is only a couple hundred, rather than 100K, you fare
As you say disable synchronization and try and control the physical reach of your igp by some mechanism.. areas, summaries, ASes etc
Which is exactly what you are doing when you inject nailed routes into bgp. So, why do you need IGP such as OSPF again? Alex
Which is exactly what you are doing when you inject nailed routes into bgp.
So, why do you need IGP such as OSPF again?
Alex
To carry the bgp next-hops around the network? You could add in statics for every next-hop on every router, but this kind of configuration is complex and prone to errors such as loops in relatively minor cases. statically routing every next-hop "does not scale". Not to mention that many of us like the "compare igp metric" portion of the BGP decision tree. Having had the displeasure of having to deal with a network which had static routes as its sole igp, I'ld never want to see _that_ again. Thankfully, we managed to merely migrate the customers off that network, rather than even try to pull apart the twisty maze of static routes.
Which is exactly what you are doing when you inject nailed routes into bgp.
So, why do you need IGP such as OSPF again?
Alex
To carry the bgp next-hops around the network? You could add in statics for every next-hop on every router, but this kind of configuration is complex and prone to errors such as loops in relatively minor cases. statically routing every next-hop "does not scale". Not to mention that many of us like the "compare igp metric" portion of the BGP decision tree.
Rubbish again. *Every* interface that you bring up has a connected route. You redistribute those routes into IGP. You redistgribute statics from that router into IGP. Nail those routes into bgp and set internal community on it. network xxx.yyy.zzz.www mask ppp.hhh.ooo.lll route-map set-igp-community.
Having had the displeasure of having to deal with a network which had static routes as its sole igp, I'ld never want to see _that_ again. Thankfully, we managed to merely migrate the customers off that network, rather than even try to pull apart the twisty maze of static routes.
Alex
On Mon, Sep 02, 2002 at 11:16:51PM -0400, alex@yuriev.com wrote:
To carry the bgp next-hops around the network? You could add in statics for every next-hop on every router, but this kind of configuration is complex and prone to errors such as loops in relatively minor cases. statically routing every next-hop "does not scale". Not to mention that many of us like the "compare igp metric" portion of the BGP decision tree.
Rubbish again.
*Every* interface that you bring up has a connected route. You redistribute those routes into IGP. You redistgribute statics from that router into IGP. Nail those routes into bgp and set internal community on it.
network xxx.yyy.zzz.www mask ppp.hhh.ooo.lll route-map set-igp-community.
So how does this provide equivalent functionality to "compare igp metric"? I think there are a lot of folks out there who might like to do the whole nearest-exit thing. Even if you went to the trouble of setting up route-maps to your heart's content and managed to get each router to prefer paths from the nearest exit router, it wouldn't do you much good when a link failure turns that "nearest" into "furthest" but the iBGP session stays up. I think maybe the word "need" is being taken a little too seriously here. No, you don't NEED a separate IGP to make BGP work. But then again, you don't NEED a lot of things to make a network go in its most basic form. However, without some of those "unnecessary" things you might not find it to perform quite to your liking either. For my network, I'd much rather deal with some extra SPF calculations than slow convergence and playing route map games to get things like nearest-exit working. Links and loopbacks => IGP Everything else => BGP But then, nobody ever accused any two engineers of having the same personal preferences... -c
Rubbish again.
*Every* interface that you bring up has a connected route. You redistribute those routes into IGP. You redistgribute statics from that router into IGP. Nail those routes into bgp and set internal community on it.
network xxx.yyy.zzz.www mask ppp.hhh.ooo.lll route-map set-igp-community.
So how does this provide equivalent functionality to "compare igp metric"?
First of all, did we agree by now that there is no scalability issue or do we need to go over this again?
I think there are a lot of folks out there who might like to do the whole nearest-exit thing.
set metric +1 is your friend.
Even if you went to the trouble of setting up route-maps to your heart's content and managed to get each router to prefer paths from the nearest exit router, it wouldn't do you much good when a link failure turns that "nearest" into "furthest" but the iBGP session stays up.
You metric would be appropriately affected. next-hop-self and confederations are your friends.
I think maybe the word "need" is being taken a little too seriously here. No, you don't NEED a separate IGP to make BGP work. But then again, you don't NEED a lot of things to make a network go in its most basic form. However, without some of those "unnecessary" things you might not find it to perform quite to your liking either. For my network, I'd much rather deal with some extra SPF calculations than slow convergence and playing route map games to get things like nearest-exit working.
There are no route-map games. You can basically have the same route-map on all internal links. Of course it requires to be able to construct logical "if then" trees, as well as know some fundamental algebra.
Links and loopbacks => IGP Everything else => BGP
IGP trouble -> entire network down with hundreds of otherwise unaffected customers experiencing connectivity problems and getting hard earned $$ back, sending yet another network into ochapter 11.
But then, nobody ever accused any two engineers of having the same personal preferences...
-c
--
On Tue, Sep 03, 2002 at 12:55:03AM -0700, Clayton Fiske wrote:
Links and loopbacks => IGP
Why on earth does you want your link addresses in your IGP ? Sometimes it cannot be avoided, due to bad implementation, but why do you need it ? Allways use loopback addresses for BGP next-hop, then all you need is the loopback addresses in your IGP. /Jesper -- Jesper Skriver, jesper(at)skriver(dot)dk - CCIE #5456 Senior network engineer @ AS3292, TDC Tele Danmark One Unix to rule them all, One Resolver to find them, One IP to bring them all and in the zone to bind them.
On Tue, 3 Sep 2002, Jesper Skriver wrote:
Links and loopbacks => IGP
Why on earth does you want your link addresses in your IGP ?
Sometimes it cannot be avoided, due to bad implementation, but why do you need it ?
Routers that learn a route over IBGP need to know where the next hop address for route from other AS points to. Since this can't be a loopback address and you typically don't run an IGP on subnets between border routers in your AS and a remote AS, you need to either set next-hop-self on all IBGP sessions or redistribute connected in your IGP. Iljitsch van Beijnum
On Tue, Sep 03, 2002 at 05:26:54PM +0200, Iljitsch van Beijnum wrote:
On Tue, 3 Sep 2002, Jesper Skriver wrote:
Links and loopbacks => IGP
Why on earth does you want your link addresses in your IGP ?
Sometimes it cannot be avoided, due to bad implementation, but why do you need it ?
Routers that learn a route over IBGP need to know where the next hop address for route from other AS points to. Since this can't be a loopback address and you typically don't run an IGP on subnets between border routers in your AS and a remote AS, you need to either set next-hop-self on all IBGP sessions or redistribute connected in your IGP.
Yes, next-hop-self on iBGP sessions is a way to ensure that all BGP routes have a loopback address as next-hop. This also solve nasty issues with IXP's, and someone advertising a more specific of the peering LAN prefix. /Jesper -- Jesper Skriver, jesper(at)skriver(dot)dk - CCIE #5456 Senior network engineer @ AS3292, TDC Tele Danmark One Unix to rule them all, One Resolver to find them, One IP to bring them all and in the zone to bind them.
Which is exactly what you are doing when you inject nailed routes into bgp.
So, why do you need IGP such as OSPF again?
Alex
To carry the bgp next-hops around the network? You could add in statics for every next-hop on every router, but this kind of configuration is complex and prone to errors such as loops in relatively minor cases. statically routing every next-hop "does not scale". Not to mention that many of us like the "compare igp metric" portion of the BGP decision tree.
Rubbish again.
*Every* interface that you bring up has a connected route. You redistribute those routes into IGP. You redistgribute statics from that router into IGP. Nail those routes into bgp and set internal community on it.
network xxx.yyy.zzz.www mask ppp.hhh.ooo.lll route-map set-igp-community.
Having had the displeasure of having to deal with a network which had static routes as its sole igp, I'ld never want to see _that_ again. Thankfully, we managed to merely migrate the customers off that network, rather than even try to pull apart the twisty maze of static routes.
Alex
So, are you saying: 1) You should have a static route for every bgp next-hop, on every router? Including both router loopbacks and EBGP next-hops? 2) You should have a static route for every router loopback, on every router? 3) You have lots and lots and lots of iBGP sessions not only across loopbacks but between directly-connected interfaces in order to jumpstart the "real" ibgp sessions? 4) something else? And that: you don't use "closest-exit" at all, but haul traffic, wherever, around your network based upon steps below the igp-metric step in the bgp decision tree? The only thing that has been clear is that you redistribute statics into BGP, which I'm fairly certain most people already do. I'm sorry, but so far, I'm not buying how a static net is better. You seem to be trading off the complexity of automatically performing SPF, for the complexity of manually performing SPF. I'ld certainly hate to be in your Ops group when a particular path fails, requiring someone to sit with pad and paper and recompute SPF, by hand, for a hundred routers. On the up-side, the original failure might be fixed by the time the computation is about 50% complete.
On Tue, 3 Sep 2002 bdragon@gweep.net wrote:
So, are you saying:
4) something else?
I think what Alex is trying to say is pretty much make every router a confederation sub-AS of its own. That way, you never talk BGP with a router you're not directly connected to, so you don't need loopback routes to find BGP peers. If you then configure next-hop-self on every session, you don't need "redistribute connected" either so you've eliminated the need for an IGP.
And that: you don't use "closest-exit" at all, but haul traffic, wherever, around your network based upon steps below the igp-metric step in the bgp decision tree?
That's the part I can't figure out without some lab time either, but obviously you can tweak this setup to _statically_ do whatever it is you need, the only question is what happens when something goes down.
I'm sorry, but so far, I'm not buying how a static net is better.
"No IGP" doesn't mean "static", "no IGP" really means "no IGP". :-)
And that: you don't use "closest-exit" at all, but haul traffic, wherever, around your network based upon steps below the igp-metric step in the bgp decision tree?
That's the part I can't figure out without some lab time either, but obviously you can tweak this setup to _statically_ do whatever it is you need, the only question is what happens when something goes down.
route-map internal-link permit 10 match blah set metric +10 do blah Alex
On Tue, 3 Sep 2002 alex@yuriev.com wrote:
That's the part I can't figure out without some lab time either, but obviously you can tweak this setup to _statically_ do whatever it is you need, the only question is what happens when something goes down.
route-map internal-link permit 10 match blah set metric +10 do blah
Yes, now I see it. Under normal circumstances this doesn't make sense since IBGP doesn't "see" the infrastructure between to BGP routers, but in this setup each router adds to the MED metric so it works more or less the same as an OSPF or IS-IS metric. Iljitsch van Beijnum
1) You should have a static route for every bgp next-hop, on every router? Including both router loopbacks and EBGP next-hops?
Absolutely not.
2) You should have a static route for every router loopback, on every router?
Asolutely not.
3) You have lots and lots and lots of iBGP sessions not only across loopbacks but between directly-connected interfaces in order to jumpstart the "real" ibgp sessions?
Absolutely not.
4) something else?
Yes.
And that: you don't use "closest-exit" at all, but haul traffic, wherever, around your network based upon steps below the igp-metric step in the bgp decision tree?
Nope, we did.
The only thing that has been clear is that you redistribute statics into BGP, which I'm fairly certain most people already do.
Nope, we dont and never did.
I'm sorry, but so far, I'm not buying how a static net is better. You seem to be trading off the complexity of automatically performing SPF, for the complexity of manually performing SPF. I'ld certainly hate to be in your Ops group when a particular path fails, requiring someone to sit with pad and paper and recompute SPF, by hand, for a hundred routers. On the up-side, the original failure might be fixed by the time the computation is about 50% complete.
Again, there is no static net. Alex
On Mon, 2 Sep 2002 alex@yuriev.com wrote:
Has anybody mentioned the benefits of ISIS as an IGP to them.
Link-state protocols are evil, and when they break, they *really* break. I still do not see a compeling argument for not using BGP as your IGP.
Convergence time?
What is better - relatively long convergence time on affected routes or a problem on unaffected route?
Ask your customers. They do not care if someone else is having a problem. They care that they dont.
Do you run a decent sized network? Convergence time in the order of that taken by BGP is not acceptable, things go crazy when traffic pours in and theres no routes to carry it. Other example, what about static dialup users, they dial up and wait a few minutes whilst their route is installed throughout BGP??
With link-state, one interface flap can mean doing SPF on every route. If "every route" is only a couple hundred, rather than 100K, you fare
As you say disable synchronization and try and control the physical reach of your igp by some mechanism.. areas, summaries, ASes etc
Which is exactly what you are doing when you inject nailed routes into bgp.
No its not? I'm suggesting some level of order can help control the number of routers required to reconverge a network, I dont see the comparison with inserting routes in BGP which is how the routes get in not how they converge. Steve
So, why do you need IGP such as OSPF again?
Alex
Has anybody mentioned the benefits of ISIS as an IGP to them.
Link-state protocols are evil, and when they break, they *really* break. I still do not see a compeling argument for not using BGP as your IGP.
Convergence time?
What is better - relatively long convergence time on affected routes or a problem on unaffected route?
Ask your customers. They do not care if someone else is having a problem. They care that they dont.
Do you run a decent sized network?
No, I have never touched a router in my life.
Convergence time in the order of that taken by BGP is not acceptable, things go crazy when traffic pours in and theres no routes to carry it.
This is a great blanked statement. What is convergence time?
Other example, what about static dialup users, they dial up and wait a few minutes whilst their route is installed throughout BGP??
That is why their route is *nailed* via BGP to the router that *always* provide connectivity to them. If they have to move, BGP injectors are your friends. Takes seconds.
With link-state, one interface flap can mean doing SPF on every route. If "every route" is only a couple hundred, rather than 100K, you fare
As you say disable synchronization and try and control the physical reach of your igp by some mechanism.. areas, summaries, ASes etc
Which is exactly what you are doing when you inject nailed routes into bgp.
No its not? I'm suggesting some level of order can help control the number of routers required to reconverge a network, I dont see the comparison with inserting routes in BGP which is how the routes get in not how they converge.
If you dont have a network wide meltdown due to IGP failure you wont need to wait for entire network to come up. It is timing of discrete events. Isn't math grand. Alex
On Tue, 3 Sep 2002 alex@yuriev.com wrote:
> Has anybody mentioned the benefits of ISIS as an IGP to them.
Link-state protocols are evil, and when they break, they *really* break. I still do not see a compeling argument for not using BGP as your IGP.
Convergence time?
What is better - relatively long convergence time on affected routes or a problem on unaffected route?
Ask your customers. They do not care if someone else is having a problem. They care that they dont.
Do you run a decent sized network?
No, I have never touched a router in my life.
Possibly..
Convergence time in the order of that taken by BGP is not acceptable, things go crazy when traffic pours in and theres no routes to carry it.
This is a great blanked statement. What is convergence time?
The time from when traffic starts hitting down interfaces or null to when it starts going again. Preferably without the rest of the network needing to know about it and suffer meltdown.
Other example, what about static dialup users, they dial up and wait a few minutes whilst their route is installed throughout BGP??
That is why their route is *nailed* via BGP to the router that *always* provide connectivity to them. If they have to move, BGP injectors are your friends. Takes seconds.
See previous comment about network size - theres no such thing as always in dialup with multiple geographic PoPs.
With link-state, one interface flap can mean doing SPF on every route. If "every route" is only a couple hundred, rather than 100K, you fare
As you say disable synchronization and try and control the physical reach of your igp by some mechanism.. areas, summaries, ASes etc
Which is exactly what you are doing when you inject nailed routes into bgp.
No its not? I'm suggesting some level of order can help control the number of routers required to reconverge a network, I dont see the comparison with inserting routes in BGP which is how the routes get in not how they converge.
If you dont have a network wide meltdown due to IGP failure you wont need to wait for entire network to come up. It is timing of discrete events. Isn't math grand.
Reconvergence after a single link failing is hardly failure/meltdown? Steve - Bored of arguing - private responses only
Alex
problem on unaffected route?
Ask your customers. They do not care if someone else is having a problem. They care that they dont.
Do you run a decent sized network?
No, I have never touched a router in my life.
Possibly..
Convergence time in the order of that taken by BGP is not acceptable, things go crazy when traffic pours in and theres no routes to carry it.
This is a great blanked statement. What is convergence time?
The time from when traffic starts hitting down interfaces or null to when it starts going again. Preferably without the rest of the network needing to know about it and suffer meltdown.
How many seconds does it take your network to meltdown from traffic?
Other example, what about static dialup users, they dial up and wait a few minutes whilst their route is installed throughout BGP??
That is why their route is *nailed* via BGP to the router that *always* provide connectivity to them. If they have to move, BGP injectors are your friends. Takes seconds.
See previous comment about network size - theres no such thing as always in dialup with multiple geographic PoPs.
Route-injection is your friend. bgp-push and avi-bgp are your friends.
With link-state, one interface flap can mean doing SPF on every route. If "every route" is only a couple hundred, rather than 100K, you fare
As you say disable synchronization and try and control the physical reach of your igp by some mechanism.. areas, summaries, ASes etc
Which is exactly what you are doing when you inject nailed routes into bgp.
No its not? I'm suggesting some level of order can help control the number of routers required to reconverge a network, I dont see the comparison with inserting routes in BGP which is how the routes get in not how they converge.
If you dont have a network wide meltdown due to IGP failure you wont need to wait for entire network to come up. It is timing of discrete events. Isn't math grand.
Reconvergence after a single link failing is hardly failure/meltdown?
How many SECONDS does it take to for your network to meltdown from normal traffic level? Alex --
On Tue, 3 Sep 2002 alex@yuriev.com wrote:
That is why their route is *nailed* via BGP to the router that *always* provide connectivity to them. If they have to move, BGP injectors are your friends. Takes seconds.
Talking about things that take seconds: would you mind sharing your BGP hold time values with us? Iljitsch van Beijnum
On Tue, Sep 03, 2002 at 10:24:44AM +0100, Stephen J. Wilcox wrote:
Do you run a decent sized network? Convergence time in the order of that taken by BGP is not acceptable, things go crazy when traffic pours in and theres no routes to carry it.
Alex ran a decent sized network back when there were a lot of Crisco bugs which managed to wipe out entire networks running OSPF. Now admittidly I havn't seen one of these in a while (according to UU they've been shifting their focus to ISIS bugs which wipe out entire networks :P), but we are all a product of our past experiences. Experience is a good thing, but it can also be limiting. Things change, and sometimes being burned by experience can prevent us from trying things which may have a different outcome now. That is why humans get old and die, so new people can make fresh mistakes, and fresh discoveries. Otherwise we'd still be thinking the world was flat, the sun revolves around the earth, and man could never travel faster than 20mph. Just remember that computers change a lot faster than the human lifespan (and no I'm not encouraging anyone to bump off Alex :P), so even Cisco might manage to get one right. :) Trusting your IGP is a judgement call, but if you ever find yourself in a situation where you really can't trust it, you can look back on Alex's post and find a fairly reasonable way to run a network without it. I suggest we all take it as such and move on, 'cause otherwise this debate will go on until someone DOES die (and not necessarily old age :P). -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
Link-state protocols are evil, and when they break, they *really* break. I still do not see a compeling argument for not using BGP as your IGP.
Alex
iBGP is only one half of an IGP. It is the "where to go" half. You still need some other igp (isis, ospf, rip, static routes, etc) for the "how to get there" half.
Most large providers carry next-hops (loopbacks, border /30 (or /31), etc) around in either isis or ospf, and carry the remainder in iBGP.
The reason?
With link-state, one interface flap can mean doing SPF on every route. If "every route" is only a couple hundred, rather than 100K, you fare better when circuits are flapping. At that point, comparing the precomputed metric amongst 100k routes is (imho) rather trivial, especially when "igp metric" is a few steps down the decision tree.
In all practicality, you need to haul, at least, eBGP routes around in iBGP, you need some kind of other igp to jumpstart iBGP, and is advised that this other igp have some concept of metric or cost to be able to give BGP some hints. Whether you choose to make your non-BGP igp lean and mean (disabling synchronization, with the attendant caveats) or fat and happy (and suffer the consequences during repeated link state changes), is up to the reader, but you still really need two igps.
Rubbish. Every interface that you have has a connected route or static route. Those routes are distributed into your favourite BROKEN[1] IGP. There is no need to do that. Originate that route as a BGP route with set-igp-community route-map on itm if you want to do something only with routes that are internal to your network. Your customers wont care about convergence of BGP if your network does not break due to the funky IGP implementations. Alex [1] Any protocol that allows a non-affected path to be taken out of service by an affected path is BROKEN BY DESIGN. Any network that requires a protocol, which is broken by deisgn to function, is a badly designed network.
participants (26)
-
alex＠yuriev.com -
bdragon＠gweep.net -
Brian Wallingford -
Chris Woodfield -
Clayton Fiske -
Daniel Golding -
Derek Samford -
Dmitri Krioukov -
Frank Scalzo -
Iljitsch van Beijnum -
Jason Lixfeld -
Jesper Skriver -
Kurtis Lindqvist -
Mark Kent -
Me -
Michael Hallgren -
Mikael Abrahamsson -
NAIDOO Kesva FTLD/IAP -
Peter van Dijk -
Petri Helenius -
Ralph Doncaster -
Richard A Steenbergen -
Robert A. Hayden -
Ross Chandler -
Stephen J. Wilcox -
William Waites