I would have to disagree on a lot of these points. See below. Steven Naslund

-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Daniel Golding Sent: Thursday, April 04, 2002 5:25 PM To: Andy Dills; nanog@merit.edu Subject: RE: Qwest Support

I suppose. Except it's not even certain you were having a problem of any kind at all.

Qwest's presence or absence from public IX's really has nothing to do with your routes being announced. In fact, Qwest privately peers with all the other large networks. While there are many peering sessions at the public NAPs, most traffic is carried over private network interconnects, at least domestically. Certain peering points in Europe (Linx), tend to run the other way.

If the routes cannot be seen at the public IXs then a lot of people who are connected to the public IXs will not see it either. Depends if you are only talking to the "big networks".

In fact, if Qwest were publically peering with other networks, it might be a reason why your routes through UUNet were being prefered - private peer originated routes are almost always assigned higher local preferences in carrier networks, then public peer originated routes.

Local prefs are just that LOCAL. They will not matter to other networks, they merely show the routes I prefer in and out of my network. This should have no impact on AS path hop counts which is the primary method of selecting BGP routes.

I'm not sure your annoyance with Qwest has any basis in their lack of performance, as far as IP routing. BGP decision rules and other networks' routing policies will govern which paths are used for your routes. Here is an example...

- Network X peers with UUNet in 8 locations. Network X also peers with Qwest, lets say in 6 locations. For whatever reason, network X chooses UUNet's routes to you over, Qwest's. This could be due to local routing policy, dictating that 701 routes get a higher local pref. Or AS path lengths could be the same, and the decision could be falling to something like router ID. Whatever.

What I would wonder here is : If network X prefers UUnet over Quest then maybe UUnet offers better performance than Quest. I think that most networks will not set a local pref unless there is a reason to override the default BGP behavior which is to use AS path length. If service providers are avoiding Quest there is probably a good reason for it. I don't think many people would try to give UUnet more preference that they already get by default, more likely networks lower the UUnet pref in order to balance their traffic.

- In general, all the UUNet peering will get treated the same by Network X's routing policy. This won't always be the case, but let's say that none of the peering links are congested, etc. So, a certain number of paths are carried throughout Network X via iBGP. If UUNet's routes "won" at all those peering points, you will not see any paths through Qwest on a single carrier route server like Nitrous.

Not true. Nitrous shows all routes it knows about whether they are preferred or not.

- Route-views, and the like are different animals. They get ebgp multihop views from many providers, so you will tend to see paths from many different vantage points, and are more likely to see paths from both your upstreams.

ISPs get a heavy volume of calls every day. While Qwest may not have the greatest customer service, it's not like you were actually down or had a qwest originated routing issue. If that were the case, my sympathy would be greater.

How would Quest have known if he actually had a problem since they never really talked to him ? What if he had a real routing problem ?

- Daniel Golding

-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Andy Dills Sent: Thursday, April 04, 2002 5:43 PM To: nanog@merit.edu Subject: Qwest Support

Wow, Qwest support is indeed terrible.

Turned up the DS3 today...the connectivity seems fine. I decided to check a couple of routeservers (nitrous); all had my much-prepended UUnet announcement, but NONE had my Qwest announcement. Not a huge deal, but curious to me. Is Qwest just not at the public peering points? When I checked route-views.oregan-ix.net, I felt better, but yet annoyed. Even with the prepends, most networks were announcing UUnet's path.

So I decided to call them and ask...man what a mistake. The guy is like, "Ok, hold on, let me get somebody from our IP noc." 10 minutes goes by, and he comes back with "Couldn't get anybody in the IP noc, let me try to get somebody in your install group" (being that I turned up the DS3 today). Comes back another 10 minutes later with "Well, I left a message for them, but there isn't much I can do. Nobody seems to be answering their phone. If somebody doesn't call you back within 30 minutes, here's a number to call..."

So what if my routes were actually hosed? I'd just be screwed because they can't get anybody at the IP noc?

I wait. Nobody calls back within 30 minutes. I call the number he gave me. Busy. You gotta be kidding me.

So I call the main number again, talk to somebody different. She has me hold, and then brings some guy on the line "who can help me". I start to talk about route servers, and he's immediately like "Woah, this is a BGP problem...I can't help you. Let me try to get somebody from the IP noc."

So, I wait on hold for about 15 minutes, only to be given dial tone.

Please tell me it isn't always THIS bad?

Andy

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Andy Dills 301-682-9972 Xecunet, LLC www.xecu.net xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Dialup * Webhosting * E-Commerce * High-Speed Access

And here we go, down the rabbit hole... (see below)

Steve Naslund Said...

I would have to disagree on a lot of these points. See below.

Steven Naslund

...

Daniel Golding Said...

I suppose. Except it's not even certain you were having a problem of any kind at all.

Qwest's presence or absence from public IX's really has nothing to do with your routes being announced. In fact, Qwest privately peers with all the other large networks. While there are many peering sessions at the public NAPs, most traffic is carried over private network interconnects, at least domestically. Certain peering points in Europe (Linx), tend to run the other way.

If the routes cannot be seen at the public IXs then a lot of people who are connected to the public IXs will not see it either. Depends if you are only talking to the "big networks".

How is this so? There are plenty of routes that are not announced at public exchance points, that are floating around in the global routing table. That's because almost all of the folks who are at public peering points, are also buying transit from the "big networks", in one way or another. Needless to say, this is how the internet works.

...

In fact, if Qwest were publically peering with other networks, it might be a reason why your routes through UUNet were being prefered - private peer originated routes are almost always assigned higher local preferences in carrier networks, then public peer originated routes.

Local prefs are just that LOCAL. They will not matter to other networks, they merely show the routes I prefer in and out of my network. This should have no impact on AS path hop counts which is the primary method of selecting BGP routes.

I think you are missing the point. Almost all large networks have similar sorts of routing policies, where routes recieved through private interconnect, get pref-ed higher than routes through public exchanges. Because of this confluence of routing policies, private peer-originated routes tend to be prefered. Look at the NANOG presentation archives for good examples of how this works.

...

I'm not sure your annoyance with Qwest has any basis in their lack of performance, as far as IP routing. BGP decision rules and other networks' routing policies will govern which paths are used for your routes. Here is an example...

- Network X peers with UUNet in 8 locations. Network X also peers with Qwest, lets say in 6 locations. For whatever reason, network X chooses UUNet's routes to you over, Qwest's. This could be due to local routing policy, dictating that 701 routes get a higher local pref. Or AS path lengths could be the same, and the decision could be falling to something like router ID. Whatever.

What I would wonder here is : If network X prefers UUnet over Quest then maybe UUnet offers better performance than Quest. I think that most networks will not set a local pref unless there is a reason to override the default BGP behavior which is to use AS path length. If service providers are avoiding Quest there is probably a good reason for it. I don't think many people would try to give UUnet more preference that they already get by default, more likely networks lower the UUnet pref in order to balance their traffic.

This is simply not true. There is no incentive to balance outbound traffic amongst peers, EXCEPT to preserve inbound/outbound ratios. Almost all backbones set local prefs on the vast majority of routes, in order to establish a routing policy. Also, backbones do not tend to alter their local prefs or routing policies based on the perceived quality of their peers - only on things like congestion across a specific peering link. It is often tempting to think that because one manages traffic in a certain way, in a smaller or non-transit AS, that one would do it in the same way in a large transit AS, with significant peering. However, in reality, these are much different animals.

...

- In general, all the UUNet peering will get treated the same by Network X's routing policy. This won't always be the case, but let's say

that none of

...

the peering links are congested, etc. So, a certain number of paths are carried throughout Network X via iBGP. If UUNet's routes "won" at all those peering points, you will not see any paths through Qwest on a single carrier route server like Nitrous.

Not true. Nitrous shows all routes it knows about whether they are preferred or not.

Looking Glasses like Nitrous, show the routes on the routers that they look into. If paths from a specific provider are selected on all peering routers, for a given route, than no paths to alternative providers will appear. This is how BGP works. I suggest reading the appropriate RFCs, and perhaps, Routing TCP/IP volume II by Jeff Doyle. Remember: large carriers tend to have dedicated peering routers, where numerous private peers are terminated.

...

- Route-views, and the like are different animals. They get

ebgp multihop

...

views from many providers, so you will tend to see paths from many different vantage points, and are more likely to see paths from both your upstreams.

ISPs get a heavy volume of calls every day. While Qwest may not have the greatest customer service, it's not like you were actually down or had a qwest originated routing issue. If that were the case, my sympathy would be greater.

How would Quest have known if he actually had a problem since they never really talked to him ? What if he had a real routing problem ?

If this was a real routing problem, it would be a much more interesting conversation.

...

- Daniel Golding

-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Andy Dills Sent: Thursday, April 04, 2002 5:43 PM To: nanog@merit.edu Subject: Qwest Support

Wow, Qwest support is indeed terrible.

Turned up the DS3 today...the connectivity seems fine. I

decided to check

...

a couple of routeservers (nitrous); all had my much-prepended UUnet announcement, but NONE had my Qwest announcement. Not a huge deal, but curious to me. Is Qwest just not at the public peering points? When I checked route-views.oregan-ix.net, I felt better, but yet annoyed. Even with the prepends, most networks were announcing UUnet's path.

So I decided to call them and ask...man what a mistake. The guy is like, "Ok, hold on, let me get somebody from our IP noc." 10 minutes goes by, and he comes back with "Couldn't get anybody in the IP noc, let me try to get somebody in your install group" (being that I turned up the DS3 today). Comes back another 10 minutes later with "Well, I left a message for them, but there isn't much I can do. Nobody seems to be answering their phone. If somebody doesn't call you back within 30 minutes, here's a number to call..."

So what if my routes were actually hosed? I'd just be screwed because they can't get anybody at the IP noc?

I wait. Nobody calls back within 30 minutes. I call the number he gave me. Busy. You gotta be kidding me.

So I call the main number again, talk to somebody different. She has me hold, and then brings some guy on the line "who can help me". I start to talk about route servers, and he's immediately like "Woah, this is a BGP problem...I can't help you. Let me try to get somebody from the IP noc."

So, I wait on hold for about 15 minutes, only to be given dial tone.

Please tell me it isn't always THIS bad?

Andy

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Andy Dills 301-682-9972 Xecunet, LLC www.xecu.net xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Dialup * Webhosting * E-Commerce * High-Speed Access

One final post on this... I called today because I needed to announce a couple of new routes. I called their 877-37-LIGHT number...I get an amazing array of people when I call this line. The first lady is very nice, and even goes to check with somebody else ("BGP, I've heard of that"), and she tells me to email my filter change request to support@qwestip.com. Ok, sounds fair. I do so, at around 2:30. I wanted to get it in by 5, so I was in no rush. I also find out from the nice lady that Qwest has a webpage that will submit the request for me (qwestsource.net). I didn't have a username and password, so I filled out the form and waited for a response. At 3:30, I get the username and password for qwestsource. My luck, it doesn't work. Around 4:15, with not even an autoresponse from my email to support, I call the LIGHT number again. I get the most clued-sounding person I've talked to with Qwest yet. He's all crisp and Johnny TalkShow, and authoritively states "There's a special address for BGP requests. Email bgp@qwestip.net." I mention the bad username and password on qwestsource, and he says "Ah yes, that sometimes happens. Here's a number to call for the group that manages that..." So I hang up the phone, satisfied. I email bgp@qwestip.net, and pick up the phone to call about my qwestsource access. As I'm listening to the phone tell me that I've called the telco provisioning group, I see the bounce show up from bgp@qwestip.net. I'm had to laugh at that...some dude in this mass queue in a cubicle somewhere in Qwest is just sitting there talking like a tv character issuing bum information authoritively, with a strange hint of pleasure now that I think about it. So I call the LIGHT number one more time. Third time's the charm, apparently. I got a nice lady who put me right through to an engineer who had access to the filters. He added the filters, we cleared, and he was seeing all but one of the routes. We scratched our heads for a few minutes and cleared again, which of course fixed it. He then proceeds to log into the qwestsource backend to fix my account. All taken care of. So, I guess my bottom line is that Qwest has good people. But not a good setup. The problem isn't that the people I'm talking to aren't qualified to do their jobs, it's just that Qwest's setup makes me talk to the wrong people. I like UUnet's way. I call up, hit 2-1-1, enter a series of digits, #, another series of digits, #, and then I talk to the people who have enable. With Qwest, I still have to enter our account number, but I'm sure I end up in the same place the dsl customers go. At least, it seems like it. One thing I must give Qwest (this goes for UUnet as well) is that I have yet to wait on hold initially. I've spent a fair number of minutes on hold, but not until after talking to somebody first. If Qwest was setup like UUnet, I would think their IP customers would be much more content. What am I going to call Qwest about? Two things. Circuit down, routing change. Now that I have qwestsource working, I can submit requests. The circuit really shouldn't go down. Regardless, it would certainly be nice to be able to quickly talk to the people in charge of running the network without trying to con my way through mostly good intentioned front line phone jockeys. Network seems pretty solid though, and I can live with the occaisional support escapades when cutting my price in half and increasing my available bandwidth significantly...overall, I'm satisfied so far. It's not like I have a lot of time to give to this craziness, but if I have to choose two from "bandwidth, really inexpensive, and my time", I'd have to be spending somebody else's money to not pick the first two... Andy xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Andy Dills 301-682-9972 Xecunet, LLC www.xecu.net xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Dialup * Webhosting * E-Commerce * High-Speed Access

Yeah. I got that part of it, and I don't disagree that many large carriers have less than stellar support. Some things to consider when looking at the support you get from a large ISP... - Most problems you will have are going to be circuit problems - i.e. "my T-1 is down, AGAIN". Therefore, most carrier's support operations are built around fixing Layer 1 problems. You give them the circuit ID, and then, in theory, you get your circuit fixed. - Routing problems occur, but most are global, across a carrier's entire network, or regional, at a POP or POPs. There are many things that can cause this - bad router code, misconfiguration, etc. In a properly designed network, routing problems that impact a single user are rare. - Resources to assist customers in diagnosing routing problems are scarce, for a variety of reasons, some good, some very bad. This is compounded by the fact that the "hit rate" for customer routing problems is low. Most times when a customer calls and says that their T-3 is down, it really is. Most of the time when a customer calls and says they are having a BGP problem, it's rarely originated by the carrier, and is usually a customer misconfiguration or misunderstanding. - Daniel Golding

-----Original Message----- From: Gregory Urban [mailto:urban@cs.umbc.edu] Sent: Friday, April 05, 2002 11:14 AM To: Daniel Golding; nanog@merit.edu Subject: RE: Qwest Support

You totally missed the point. Had this been a real emergency, he would be unable to get resolution since Qwest was unable to dredge up a clue within their customer support machine.

Greg U

At 05:24 PM 4/4/2002, you wrote:

...

I suppose. Except it's not even certain you were having a problem of any kind at all.

Qwest's presence or absence from public IX's really has nothing to do with your routes being announced. In fact, Qwest privately peers with all the other large networks. While there are many peering sessions at the public NAPs, most traffic is carried over private network interconnects, at least domestically. Certain peering points in Europe (Linx), tend to run the other way.

In fact, if Qwest were publically peering with other networks, it might be a reason why your routes through UUNet were being prefered - private peer originated routes are almost always assigned higher local preferences in carrier networks, then public peer originated routes.

I'm not sure your annoyance with Qwest has any basis in their lack of performance, as far as IP routing. BGP decision rules and other networks' routing policies will govern which paths are used for your routes. Here is an example...

- Network X peers with UUNet in 8 locations. Network X also peers with Qwest, lets say in 6 locations. For whatever reason, network X chooses UUNet's routes to you over, Qwest's. This could be due to local routing policy, dictating that 701 routes get a higher local pref. Or AS path lengths could be the same, and the decision could be falling to something like router ID. Whatever.

- In general, all the UUNet peering will get treated the same by Network X's routing policy. This won't always be the case, but let's say that none of the peering links are congested, etc. So, a certain number of paths are carried throughout Network X via iBGP. If UUNet's routes "won" at all those peering points, you will not see any paths through Qwest on a single carrier route server like Nitrous.

- Route-views, and the like are different animals. They get ebgp multihop views from many providers, so you will tend to see paths from many different vantage points, and are more likely to see paths from both your upstreams.

ISPs get a heavy volume of calls every day. While Qwest may not have the greatest customer service, it's not like you were actually down or had a qwest originated routing issue. If that were the case, my sympathy would be greater.

- Daniel Golding

-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Andy Dills Sent: Thursday, April 04, 2002 5:43 PM To: nanog@merit.edu Subject: Qwest Support

Wow, Qwest support is indeed terrible.

Turned up the DS3 today...the connectivity seems fine. I decided to check a couple of routeservers (nitrous); all had my much-prepended UUnet announcement, but NONE had my Qwest announcement. Not a huge deal, but curious to me. Is Qwest just not at the public peering points? When I checked route-views.oregan-ix.net, I felt better, but yet annoyed. Even with the prepends, most networks were announcing UUnet's path.

So I decided to call them and ask...man what a mistake. The guy is like, "Ok, hold on, let me get somebody from our IP noc." 10 minutes goes by, and he comes back with "Couldn't get anybody in the IP noc, let me try to get somebody in your install group" (being that I turned up the DS3 today). Comes back another 10 minutes later with "Well, I left a message for them, but there isn't much I can do. Nobody seems to be answering their phone. If somebody doesn't call you back within 30 minutes, here's a number to call..."

So what if my routes were actually hosed? I'd just be screwed because they can't get anybody at the IP noc?

I wait. Nobody calls back within 30 minutes. I call the number he gave me. Busy. You gotta be kidding me.

So I call the main number again, talk to somebody different. She has me hold, and then brings some guy on the line "who can help me". I start to talk about route servers, and he's immediately like "Woah, this is a BGP problem...I can't help you. Let me try to get somebody from the IP noc."

So, I wait on hold for about 15 minutes, only to be given dial tone.

Please tell me it isn't always THIS bad?

Andy

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Andy Dills 301-682-9972 Xecunet, LLC www.xecu.net xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Dialup * Webhosting * E-Commerce * High-Speed Access

On Fri, 5 Apr 2002, Chris Woodfield wrote:

I think the main point here isn't the fact that the poster's routing was, in fact, not set up properly; it was the fact that he was unable to get a live body at Qwest to check it out.

Interestingly enough, there was indeed a problem. I'm not satisfied with the answer; it doesn't really make any sense. Not that I'm too concerned about knowing what actually happened, but hopefully somebody can suggest a possible explanation. I got a call this morning from my install manager (who is very nice...I like Christina) who told me that for some reason, when she came in this morning, and found my voice mail, she looked and their router wasn't getting our routes. Which seems really weird, because they were in plenty of views on the oregon route server...and our traffic graphs show plenty of ingress traffic since the turnup. So anyway, I checked nitrous. And yes, my Qwest routes are there now...so it looks like I did have a legit beef after all. Once qwest's router started seeing my routes (or whatever got fixed), my ingress UUnet traffic dropped to nil, just like I wanted... Strangeness, but probably related to global router configs that needed to be auto-updated I'm guessing... Andy xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Andy Dills 301-682-9972 Xecunet, LLC www.xecu.net xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Dialup * Webhosting * E-Commerce * High-Speed Access