October 2013 - Test - lists.nanog.org

Upstream / Handoff UPS?
by Kenny Kant 07 Nov '13

07 Nov '13

We have tons of circuits with various providers. Often times the demarc / handoff switch from the provider is not running on battery backup. Sometimes if the demarc device is located in the same room as our equipment we mitigate this and plug the device into our backup systems. Am I wrong to think that the demarc from the provider is a sacred thing that should only be touched by said provider. Thus they should provide their own battery system? Is it normal for this equipment not to be battery protected? We are not dealing with any crazy SLA's however I think it would be standard build practice to put UPS's on your gear. Even if its small handoff switch sitting right next to my switch. :) Kenny

5 4

latest Snowden docs show NSA intercepts all Google and Yahoo DC-to-DC traffic
by Jacque O'Lantern 07 Nov '13

07 Nov '13

http://www.washingtonpost.com/world/national-security/nsa-infiltrates-links…

19 28

Reverse DNS RFCs and Recommendations
by Nolan Rollo 06 Nov '13

06 Nov '13

I've been (probably needlessly) pouring over the Reverse DNS RFCs for long enough to actually have questions about a subject that should be relatively unimportant. I do want to make sure that we set up our reverse DNS correctly and most efficiently the first time so that we don't irritate other network operators with difficult regex based filtering ( http://www.gossamer-threads.com/lists/nanog/users/113633 ) and we don't have to change things as per a recommendation down the road. RFC draft-msullivan-dnsop-generic-naming-schemes-00.txt states: When using IP addresses in host names, their numbers SHOULD be separated by '.'s (dots) rather than any meta character such as a '-' (dash) and expressed in decimal. Host names SHOULD NOT use the '_' (underscore) character, host names for hosts with any form of SMTP mail service MUST NOT use the '_' (underscore) character. It is preferable to use the IP address in reverse format in the same way the the IN-ADDR.ARPA. domain is defined. Now since it is only a first revision draft I'm taking what it has to say with a grain of salt and it seems has taken quite a bit of criticism on forums. I'm also not singling out on Time Warner, WOW, Comcast or Charter for their naming conventions nor do I think they are bad, I'm just using them as examples because they are local, well-known ISPs. Actual Examples: cpe-67-XX-XX-XX.stny.res.rr.com - 67.XX.XX.XX d28-XX-XX-XX.dim.wideopenwest.com - 28.XX.XX.XX c-68-XX-XX-XX.hsd1.mi.comcast.net - 68.XX.XX.XX 24-XX-XX-XX.static.bycy.mi.charter.com - 24.XX.XX.XX *Most ISP Reverse DNS Hostnames (from what I've observed) seem to use the dash "-" character with the forward format, as opposed to the reverse IN-ADDR.ARPA. dotted scheme as recommended in the draft *Comcast and Charter all have geographic based furthest-right-hand tokens. *Charter, WideOpenWest, Time Warner, and Comcast all have some acronym that is not immediately clear, at least to me (HSD - High Speed Data?, BYCY - Bay City, MI?, DIM - Dynamic IP Mapping?, STNY - Southern Tier New York?) Which finally brings me to my questions: It seems like the unspoken de facto that mail admins appreciate given the IP 203.0.113.15 is "203-0-113-15.[type].[static/dynamic].yourdomain.tld". This seems perfectly acceptable, it's short, detailed and to the point. Is there really anything bad about this? What, if any would you name a network, gateway, broadcast address? Should the PTR be empty? <tinfoilhat> I've seen a lot about naming what type of technology it is (wireless, adsl, cable, etc.) in order to filter out the "high speed spammers". It seems to me that this would open up the likelihood of a targeted attack. We've all heard of security though obscurity and of course no one relies on it but we have to face the fact there are CVEs every day for various networking hardware/firmware. If an attacker can query DNS and find out that the IP is for wireless they could filter all wireless gear exploits. Is this still a good practice given the abundance of high speed data connections or is this just opening yourself up to reconnaissance?</tinfoilhat> There is a Merit Network mailing list discussion that outlines most of what I've read that can be found here ( http://www.merit.edu/mail.archives/nanog/msg06843.html ) Nolan Rollo VoIP Engineer Main: 517.223.3610x114 Fax: 517.223.4120 www.kw-corp.com<http://www.kw-corp.com/>

26 63

BGP failure analysis and recommendations
by JRC NOC 04 Nov '13

04 Nov '13

Hello Nanog - On Saturday, October 19th at about 13:00 UTC we experienced an IP failure at one of our sites in the New York area. It was apparently a widespread outage on the East coast, but I haven't seen it discussed here. We are multihomed, using EBGP to three (diverse) upstream providers. One provider experienced a hardware failure in a core component at one POP. Regrettably, during the outage our BGP session remained active and we continued receiving full routes from the affected AS. And our prefixes continued to be advertised at their border. However basically none of the traffic between those prefixes over that provider was delivered. The bogus routes stayed up for hours. We shutdown the BGP peering session when the nature of the problem became clear. This was effective. I believe that all customer BGP routes were similarly affected, including those belonging to some large regional networks and corporations. I have raised the questions below with the provider but haven't received any information or advice. My question is why did our BGP configuration fail? I'm guessing the basic answer is that the IGP and route reflectors within that provider were still connected, but the forwarding paths were unavailable. My BGP session basically acted like a bunch of static routes, with no awareness of the failure(s) and no dynamic reconfiguration of the RIB. Is this just an unavoidable issue with scaling large networks? Is it perhaps a known side effect of MPLS? Have we/they lost something important in the changeover to converged mutiprotocol networks? Is there a better way for us edge networks to achieve IP resiliency in the current environment? This is an operational issue. Thanks in advance for any hints about what happened or better practices to reduce the impact of a routine hardware fault in an upstream network. - Eric Jensen >Date: Wed, 23 Oct 2013 21:26:43 -0400 >To: cj(a)chrisjensen.org >From: JRC NetOps <noc(a)jensenresearch.com> >Subject: Fwd: BGP failure analysis and recommendations > > >>Date: Mon, 21 Oct 2013 23:19:28 -0400 >>To: christopher.smith(a)level3.com >>From: Eric Jensen <ejensen(a)jensenresearch.com> >>Subject: BGP failure analysis and recommendations >>Cc: "Joe Budelis Fast-E.com" <Joe(a)Fast-E.com> >>Bcc: noc(a)jensenresearch.com >> >>Hello Christopher Smith - >> >>I left you a voicemail message today. The Customer Service folks also >>gave me your email address. >> >>We have a small, but high-value multi-homed corporate network. >>We operate using our AS number 17103. >> >>We have BGP transit circuits with Level 3, Lightpath, and at our colo >>center (AS8001) >>The Level 3 circuit ID is BBPM9946 >> >>On Saturday, October 19 2013 we had a large IP outage. I tracked it back >>to our Level 3 circuit and opened a ticket (7126634). >>I have copied (below) an email I sent our channel salesman with more >>details about our BGP problems during your outage. >>Briefly, I am very concerned that Level 3 presented routes to us that >>were not actually reachable through your network, and even worse Level 3 >>kept advertising our prefixes even though your network couldn't deliver >>the traffic to us for those prefixes. >> >>I believe that the BGP NLRI data should follow the same IP path as the >>forwarded data itself. Apparently this isn't the case at Level 3. >>I also believe that your MPLS backbone should have recovered >>automatically from the forwarding failure, but this didn't happen either. >>My only fix was to manually shutdown the BGP peering session with Level 3. >> >>Can you explain to me how Level 3 black-holed my routes? >>Can you suggest some change to our or your BGP configuration to eliminate >>this BGP failure mode? >> >>Just to be clear, I don't expect our circuit, or your network, to be up >>all the time. But I do expect that the routes you advertise to us and to >>your BGP peers actually be reachable through your network. On Saturday >>this didn't happen. The routes stayed up while the data transport was down. >> >>Our IPv4 BGP peering session with Level 3 remains down in the interim. >>Please get back to me as soon as possible. >> >>- Eric Jensen >>AS17103 >>201-741-9509 >> >> >> >>>Date: Mon, 21 Oct 2013 22:55:35 -0400 >>>To: "Joe Budelis Fast-E.com" <Joe(a)Fast-E.com> >>>From: Eric Jensen <ejensen(a)jensenresearch.com> >>>Subject: Re: Fwd: Level3 Interim Response >>>Bcc: noc(a)jensenresearch.com >>> >>>Hi Joe- >>> >>>Thanks for making the new inquiry. >>>This was a big outage. Apparently Time Warner Cable and Cablevision >>>were affected greatly. Plus many large corporate networks. And of course >>>all the single-homed Level 3 customers worldwide. My little network was >>>just one more casualty. >>> >>>See: >>> >>>http://www.dslreports.com/forum/r28749556-Internet-Level3-Outage- >>> >>>http://online.wsj.com/news/articles/SB1000142405270230486450457914581369858… >>> >>>For our site, the massive outage started at about 9:00 am Saturday and >>>lasted until after 2:00PM. I opened a ticket about 9:30 am but only >>>realized the routing issues and took down our BGP session about 12:00 to >>>try to minimize the problems for our traffic caused by their >>>misconfigured BGP. >>> >>>There can always be equipment failures and fiber cuts. That's not the >>>problem. >>> From my point of view the problem was/is that Level 3 kept >>> "advertising" our prefix but couldn't deliver the packets to us. They >>> did this for all their customer's prefixes, thereby sucking in about >>> half the NYC area internet traffic and dumping into the Hudson River, >>> for a huge period of time. >>>They also kept advertising all their BGP routes to me, thereby fooling >>>my routers into sending outbound traffic to Level 3 where they again >>>dumped my traffic into the Hudson. >>> >>>I called Level 3 customer service today and have the name of a network >>>engineer to discuss options for fixing the BGP failure. >>>If you get any response with an engineering contact please let me know. >>> >>>I shouldn't have to manually intervene to route around problems. Even >>>sadder is the response from Level 3 explaining that they spent hours >>>trying to find the problem and had to manually reconfigure their >>>network, leading to saturated links and more problems. Their network >>>only healed when the faulty line card was replaced. >>> >>>I had reactivated the BGP session later that night, but after reviewing >>>the actual damage that we incurred, and the widespread nature of the >>>failure, I have decided to leave our Level 3 BGP session down, at least >>>until the engineering situation improves. >>>There may not be any good way to use a Level 3 BGP session without >>>risking the same "black hole" problem going forward. It's that type of >>>failure that BGP is specifically designed to deal with, but it was >>>developed in the days of point-to-point circuits carrying IP traffic. >>> >>>Nowadays some networks have a new layer between the wires and IP, namely >>>MPLS, and this allowed BGP to stay up but deprived the routers of >>>functioning IP next-hops, which they (both the Level 3 IP routers and >>>the Level 3 personnel) were unaware of. Apparently the Level 3 IP-based >>>BGP routers all believed they had working circuits edge-to-edge, but in >>>fact their network was partitioned. >>> >>>MPLS must have some redundancy features, but they obviously weren't >>>working on Saturday. This is a huge engineering failure. No large ISP >>>could function this way for long. >>> >>>I can wait the 72 hours for their response. I expect it will be full of >>>mealy-mouth platitudes about how no system is foolproof and it will all >>>be fine now. >>> >>>It would be more interesting to me to be in the meeting room where some >>>engineer has to explain how they could lose so much traffic and not be >>>able to operate a functioning, if degraded, network after a single line >>>card failure. It wouldn't be the head of network design, because that >>>person would already have been fired. >>> >>>Let me know if your hear anything. I will do the same. >>> >>>- Eric Jensen >>>AS17103 >>>201-741-9509 >>> >>> >>>

6 7

abha ahuja
by Randy Bush 04 Nov '13

04 Nov '13

abha ahuja, researcher and operator, died this day in 2001 at a tragically early age. if you did not know her, search a bit. she did a lot, and with an open mind and heart. randy

7 6

RE: If you're on LinkedIn, and you use a smart phone...
by Phil Bedard 03 Nov '13

03 Nov '13

I had to answer the question of "Why is LinkedIn asking for my GMail account information" to one of my parents recently. "Oh it is so they can access your information and use it...". It is how some random guys I play tennis with in a league keep popping up as people I should add, since they likely succumbed to that prompt. Another practice of theirs I do not like. Phil From: Laszlo Hanyecz Sent: 10/26/2013 1:44 To: Chris Hartley Cc: Phil Bedard; Nanog Subject: Re: If you're on LinkedIn, and you use a smart phone... When a user signs up for a social media account they generally do so by providing an email address like victim(a)freewebmailsite.com and selecting a password. The social media site can obviously probe freewebmailsite.com and attempt to authenticate using the same password that you just provided to them (for the purpose of logging into their social media site). I guess offering an email proxy or asking if it's ok to worm through your email for contacts is merely a formality. How many social media users do you guess would use the same password on the social media site as they would for freewebmailsite.com (and likely their employer's organization's email)? It's kind of like when google asks their users with android phones to provide their mobile phone number for SMS password recovery. Laszlo On Oct 25, 2013, at 11:43 PM, Chris Hartley <hartleyc(a)gmail.com> wrote: > Anyone who has access to logs for their email infrastructure ought > probably to check for authentications to user accounts from linkedin's > servers. Likely, people in your organization are entering their > credentials into linkedin to add to their contact list. Is it a > problem if a social media company has your users' credentials? I > guess it depends on your definition of "is." The same advice might > apply to this perversion of trust as well, but I'm not sure how > linkedin is achieving this "feat." > > On Fri, Oct 25, 2013 at 7:25 PM, Phil Bedard <bedard.phil(a)gmail.com> wrote: >> I saw some antectdotal stuff on this yesterday but reading their >> engineering blog entry makes me feel all warm and fuzzy inside. Oh >> nevermind, that's just the alcohol. This is perhaps one of the worst >> ideas I've seen concocted by a social media company yet. >> >> >> -Phil >> >> On 10/25/13, 6:56 PM, "George Bakos" <gbakos(a)alpinista.org> wrote: >> >>> next thing you know, Google is going to be offering free email so they >>> can do the same thing. >>> >>> On Fri, 25 Oct 2013 08:45:40 -0700 >>> Shrdlu <shrdlu(a)deaddrop.org> wrote: >>> >>>> I hate to do this, but it's something that anyone managing email >>>> servers (or just using a smart phone to update LI) needs to know >>>> about. I just saw this on another list I'm on, and I know that there >>>> are folks on NANOG that are on LinkedIn. >>>> >>>> ++++++++++ >>>> http://www.bishopfox.com/blog/2013/10/linkedin-intro/ >>>> >>>> LinkedIn released a new product today called Intro. They call it >>>> ___doing the impossible___, but some might call it ___hijacking >>>> email___. >>>> Why do we say this? Consider the following: >>>> >>>> Intro reconfigures your iOS device (e.g. iPhone, iPad) so that all of >>>> your emails go through LinkedIn___s servers. You read that right. Once >>>> you install the Intro app, all of your emails, both sent and received, >>>> are transmitted via LinkedIn___s servers. LinkedIn is forcing all your >>>> IMAP and SMTP data through their own servers and then analyzing and >>>> scraping your emails for data pertaining to___whatever they feel like. >>>> >>>> ++++++++++ >>>> >>>> Read the full article. If you're using LI via your smart phone, and >>>> you have already installed this app, you probably need to save off >>>> your contacts and data, and wipe the phone. I wouldn't trust >>>> uninstalling as enough, myself. In the long run, I'll be deleting my >>>> account. >>>> >>>> No, I don't use a smart phone to update any social media. No, I >>>> especially do not trust LI (never have, never will). BTW, they're >>>> currently adding back any contacts you've deleted. Thanks for >>>> reminding me that Joe Barr, Len Sassaman, and Jay D Dyson are gone >>>> from this world. >>>> >>>> -- >>>> Life may not be the party we hoped for, but while we are here, >>>> we might as well dance. >>>> >>>> >>> >>> >>> -- >>> >> >> >> >

2 1

Re: latest Snowden docs show NSA intercepts all Google and Yahoo DC-to-DC traffic
by Scott Weeks 01 Nov '13

01 Nov '13

On Wed, Oct 30, 2013 at 1:46 PM, Jacque O'Lantern < jacque.olantern(a)yandex.com> wrote: > http://www.washingtonpost.com/world/national-security/nsa-infiltrates-links… --- brandon.galbraith(a)gmail.com wrote: From: Brandon Galbraith <brandon.galbraith(a)gmail.com> Google is speeding up its initiative to encrypt all DC to DC traffic, as this was suspected a short time ago. http://www.informationweek.com/security/government/nsa-fallout-google-speed… --------------------------------------------------------- This goes back to our conversation last June: http://mailman.nanog.org/pipermail/nanog/2013-June/thread.html#59352 now $189K may not seem as 'big'! ;-) (http://mailman.nanog.org/pipermail/nanog/2013-June/059371.html) scott

6 6

Re: Happy Birthday, ARPANET!
by Scott Weeks 30 Oct '13

30 Oct '13

--- dougb(a)dougbarton.us wrote: From: Doug Barton <dougb(a)dougbarton.us> lo ---------------------------------------- Just in case some folks thought this was a typo... https://en.wikipedia.org/wiki/ARPANET#ARPANET_deployed :-) scott

1 0

Happy Birthday, ARPANET!
by Jay Ashworth 30 Oct '13

30 Oct '13

The Paley Center for Media reminds us that on this day in 1969 at 2230 PST, the first link was turned up between UCLAs Sigma 7 and SRIs 940. A photo of the laboratory logbook is included in the Wikipedia article: en.m.wikipedia.org/wiki/ARPANET Cheers, - jra -- Sent from my Android phone with K-9 Mail. Please excuse my brevity.

7 8

DDoS Prevention for a Transit Provider
by Jack Stonebraker 30 Oct '13

30 Oct '13

I'm looking to pick the brain of any Engineers out there who have deployed a DDoS Prevention strategy for an MSO that also runs their own transport network. Recently, we have been seeing increasingly large spikes of traffic traversing our core. We have determined the destination to be arbitrary, but often it is some host (A Customer CPE) south of one of our CMTS's. While we enforce ingress and egress rate limits facing the customers, the core facing network is pretty wide open, allowing the BGP mesh to steer traffic as needed. Initially, we've been trying to do root analysis of the traffic makeup via JFLOW data to see if simple ACL's might be a temporary stop gap, but I also want to explore a more elegant, long term solution. The introduction of IPS's feels cost prohibitive, especially since they would need to performing control at the core, as we provide wholesale transport services on top of our enterprise services and that makes for a huge amount of homogenized traffic to be inspected. Generally, the core can weather these spikes. Instead, it's the edge end corresponding L3 to L2 Trunks that becomes saturated. Any thoughts or comments would be greatly appreciated. Thanks. JJ Stonebraker IP Network Engineering Grande Communications

2 1