- NANOG - lists.nanog.org

Level3 and AT&T Latency
by Eric Williams 07 Nov '13

07 Nov '13

Is anybody else seeing or having major latency between Level 3 and AT&T today? We are multi-homed with Level 3 being one of our ISP's and had to divert traffic after seeing these issues. http://www.internetpulse.net/ Eric

8 9

M40e "AER" and MG9k's
by Shaw, Matthew 07 Nov '13

07 Nov '13

Good morning all, This is a bit of a longshot, but by any chance is there anyone watching this list who is currently or was formerly with VZ that's familiar with a configuration involving M40e's aggregating a number of customer facing MG9k's over ATM? We're having a heck of a time with them if you could reach out off list with any background or experiences with them from elsewhere I'd really appreciate it. Matt _______________________________________________________________________ This e-mail message and its attachments are for the sole use of the intended recipients. They may contain confidential information, legally privileged information or other information subject to legal restrictions. If you are not the intended recipient of this message, please do not read, copy, use or disclose this message or its attachments, notify the sender by replying to this message and delete or destroy all copies of this message and attachments in all media.

1 0

Re: Do you obfuscate email headers when reporting spam issues to clients?
by Anne P. Mitchell, Esq. 07 Nov '13

07 Nov '13

> On Wed, Nov 6, 2013 at 1:30 PM, Landon <landonstewart(a)gmail.com> wrote: >> How much trouble does your abuse department go to in order to obfuscate >> headers when providing evidence of spamming activity regardless of if it?s >> intentional/professional spammer activity or some kind of malware infection >> allowing a third party to spam. Especially for the pro spammers, we don?t >> want them list washing anything or worse yet becoming privy to spamtrap >> data if the reporting party wasn?t smart enough to obfuscate their own data >> before sending in the report. > > Howdy, > > It depends on the exact situation, but the general-purpose answer is: > none. zero. zip. > > The customer usually can't act on your information unless he can line > it up with an entry in his own logs. He needs lots of details in the > headers to figure out which computer or which of his users the message > came from. And he needs that information to determine whether the > message really came from his system -- headers get forged, you know. Because this is an issue inherent primarily with bulk mail, we remove all identifying information *except* the unsub link, which *should* have a unique identifying token embedded within, from which the sender *should* be able to determine the complainant's email address. And, if there is no such link, we use that as an opportunity to educate them as to *why* they need to include such a link (mind you, in order to be accredited with us the sender has to have already demonstrated that they comply with including an unsub link, but because many of our accreditation customers are ESPs, their customers may sometimes not be modelling 100% of best practices). Regardless of unsub link, or anything else, if we get a spam complaint against one of our customers, we hold their feet to the fire, and require them to explain exactly how the particular list was built, how the address was acquired, etc.. Failure to do so can (and usually does) result in termination of their accreditation - in the case of an ESP, they have to take corrective measures against their spamming customer or the ESP will lose their accreditation. Anne Anne P. Mitchell, Esq. Author: Section 6 of the CAN-SPAM Act of 2003 CEO/President Institute for Social Internet Public Policy http://www.ISIPP.com Member, Cal. Bar Cyberspace Law Committee How do you get to the inbox instead of the spam filter? SuretyMail! Helping businesses keep their email out of the junk folder since 1998 http://www.isipp.com/SuretyMail

4 6

latest Snowden docs show NSA intercepts all Google and Yahoo DC-to-DC traffic
by Jacque O'Lantern 07 Nov '13

07 Nov '13

http://www.washingtonpost.com/world/national-security/nsa-infiltrates-links…

19 28

Reverse DNS RFCs and Recommendations
by Nolan Rollo 06 Nov '13

06 Nov '13

I've been (probably needlessly) pouring over the Reverse DNS RFCs for long enough to actually have questions about a subject that should be relatively unimportant. I do want to make sure that we set up our reverse DNS correctly and most efficiently the first time so that we don't irritate other network operators with difficult regex based filtering ( http://www.gossamer-threads.com/lists/nanog/users/113633 ) and we don't have to change things as per a recommendation down the road. RFC draft-msullivan-dnsop-generic-naming-schemes-00.txt states: When using IP addresses in host names, their numbers SHOULD be separated by '.'s (dots) rather than any meta character such as a '-' (dash) and expressed in decimal. Host names SHOULD NOT use the '_' (underscore) character, host names for hosts with any form of SMTP mail service MUST NOT use the '_' (underscore) character. It is preferable to use the IP address in reverse format in the same way the the IN-ADDR.ARPA. domain is defined. Now since it is only a first revision draft I'm taking what it has to say with a grain of salt and it seems has taken quite a bit of criticism on forums. I'm also not singling out on Time Warner, WOW, Comcast or Charter for their naming conventions nor do I think they are bad, I'm just using them as examples because they are local, well-known ISPs. Actual Examples: cpe-67-XX-XX-XX.stny.res.rr.com - 67.XX.XX.XX d28-XX-XX-XX.dim.wideopenwest.com - 28.XX.XX.XX c-68-XX-XX-XX.hsd1.mi.comcast.net - 68.XX.XX.XX 24-XX-XX-XX.static.bycy.mi.charter.com - 24.XX.XX.XX *Most ISP Reverse DNS Hostnames (from what I've observed) seem to use the dash "-" character with the forward format, as opposed to the reverse IN-ADDR.ARPA. dotted scheme as recommended in the draft *Comcast and Charter all have geographic based furthest-right-hand tokens. *Charter, WideOpenWest, Time Warner, and Comcast all have some acronym that is not immediately clear, at least to me (HSD - High Speed Data?, BYCY - Bay City, MI?, DIM - Dynamic IP Mapping?, STNY - Southern Tier New York?) Which finally brings me to my questions: It seems like the unspoken de facto that mail admins appreciate given the IP 203.0.113.15 is "203-0-113-15.[type].[static/dynamic].yourdomain.tld". This seems perfectly acceptable, it's short, detailed and to the point. Is there really anything bad about this? What, if any would you name a network, gateway, broadcast address? Should the PTR be empty? <tinfoilhat> I've seen a lot about naming what type of technology it is (wireless, adsl, cable, etc.) in order to filter out the "high speed spammers". It seems to me that this would open up the likelihood of a targeted attack. We've all heard of security though obscurity and of course no one relies on it but we have to face the fact there are CVEs every day for various networking hardware/firmware. If an attacker can query DNS and find out that the IP is for wireless they could filter all wireless gear exploits. Is this still a good practice given the abundance of high speed data connections or is this just opening yourself up to reconnaissance?</tinfoilhat> There is a Merit Network mailing list discussion that outlines most of what I've read that can be found here ( http://www.merit.edu/mail.archives/nanog/msg06843.html ) Nolan Rollo VoIP Engineer Main: 517.223.3610x114 Fax: 517.223.4120 www.kw-corp.com<http://www.kw-corp.com/>

26 63

How anti-NSA backlash could fracture the Internet along national borders - The Washington Post
by Jay Ashworth 06 Nov '13

06 Nov '13

The balkanizing of the Net? http://www.washingtonpost.com/blogs/worldviews/wp/2013/11/01/how-anti-nsa-b… Cheers, - jra -- Sent from my Android phone with K-9 Mail. Please excuse my brevity.

11 19

[pfsinoz@gmail.com: [APRICOT-INFO] APRICOT 2014 call for papers]
by bmanning＠vacation.karoshi.com 05 Nov '13

05 Nov '13

of possible interest. /bill ----- Forwarded message from Philip Smith <pfsinoz(a)gmail.com> ----- X-Mailman-Approved-At: Tue, 05 Nov 2013 19:37:41 +1000 Subject: [APRICOT-INFO] APRICOT 2014 call for papers Hi everyone, We have just released the call for presentations for APRICOT 2014. Please consider presenting at APRICOT, or encourage a colleague or friend to do so. Also we'd really appreciate it if you would help inform members of your local operations community about this CfP. Many thanks! Philip, Mark, Dean APRICOT PC Chairs -- Asia Pacific Regional Internet Conference on Operational Technologies (APRICOT) 18 - 28 February 2014, Bangkok, Thailand https://2014.apricot.net CALL FOR PAPERS =============== The APRICOT 2014 Programme Committee is now seeking contributions for Presentations and Tutorials for APRICOT 2014. We are looking for presenters who would: - Offer a technical tutorial on an appropriate topic; - Participate in the technical conference sessions as a speaker; - Convene and chair panel sessions of relevant topics. Please submit on-line at: http://papers.apricot.net/user/login.php?event=6 CONFERENCE MILESTONES --------------------- Call for Papers Opens: 1 November 2013 First Draft Programme Published: 6 December 2013 Final Deadline for Submissions: 7 February 2014 Final Programme Published: 14 February 2014 Final Slides Received: 21 February 2014 PROGRAMME MATERIAL ------------------ The APRICOT Programme is organised in three parts, including workshops, tutorials and the conference. Topics for tutorials and the conference must be relevant to Internet Operations and Technologies: - IPv4 / IPv6 Routing and operations - IPv6 deployment and transition technologies - Internet backbone operations - ISP and Carrier services - IXPs and Peering - Research on Internet Operations and Deployment - Thai Internet - Network security issues (NSP-SEC, DDoS, Anti-Spam, Anti-Malware) - DNS / DNSSEC - Internet policy (Security, Regulation, Content Management, Addressing, etc) - Access and Transport Technologies, including Cable/DSL, 3G/LTE, wireless, metro ethernet, fibre, MPLS - Content & Service Delivery (Multicast, Voice, Video, "telepresence", Gaming) and Cloud Computing CfP SUBMISSION -------------- Draft slides for both tutorials and conference sessions MUST be provided with CfP submissions otherwise the Programme Committee will be unable to review the submission. For work in progress, the most current information available at time of submission is acceptable. All draft and complete slides must be submitted in PDF format only. Final slides are to be provided by the specified deadline for publication on the APRICOT website. Prospective presenters should note that the majority of speaking slots will be filled well before the final submission deadline. The PC will retain a limited number of slots up to the final submission deadline for presentations that are exceptionally timely, important, or of critical operational importance. Please submit on-line at: http://papers.apricot.net/user/login.php?event=6 Any questions or concerns should be addressed to the Programme Committee by e-mail at: pc-chairs at apricot.net We look forward to receiving your presentation proposals. Dean Pemberton, Mark Tinka & Philip Smith Co-Chairs, APRICOT Programme Committee pc-chairs(a)apricot.net -- _______________________________________________ Apricot-info mailing list Apricot-info(a)apricot.net http://mailman.apnic.net/mailman/listinfo/apricot-info ----- End forwarded message -----

1 0

Fwd: [apops] APRICOT 2014 call for papers is now open
by Mark Tinka 05 Nov '13

05 Nov '13

FYI. Cheers, Mark.

1 0

Fwd: Re: Some stats on IPv6 fragments and EH filtering on the Internet
by Fernando Gont 05 Nov '13

05 Nov '13

Folks, FYI. Thought this might be of interest. P.S.: Input/comments welcome Thanks! Cheers, Fernando -------- Original Message -------- Subject: Some stats on IPv6 fragments and EH filtering on the Internet Date: Mon, 04 Nov 2013 15:01:48 -0800 From: Fernando Gont <fernando(a)gont.com.ar> To: 6man(a)ietf.org <6man(a)ietf.org>, IPv6 Operations <v6ops(a)ietf.org> Folks, I did a presentation on the topic at the IEPG meeting earlier this week. It provides some concrete data regarding IPv6 fragmentation and Extension Header filtering on the Internet. The slideware is available at: <http://www.iepg.org/2013-11-ietf88/fgont-iepg-ietf88-ipv6-frag-and-eh.pdf> Certainly there's *much* more work to be done in this area, but I thought that this could be good food sfor some of the discussions that we were having on the topic. Thanks, -- Fernando Gont e-mail: fernando(a)gont.com.ar || fgont(a)si6networks.com PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1 -------- Original Message -------- Subject: Re: Some stats on IPv6 fragments and EH filtering on the Internet Date: Mon, 4 Nov 2013 23:27:12 +0000 From: Tim Chown <tjc(a)ecs.soton.ac.uk> To: 6man(a)ietf.org <6man(a)ietf.org>, IPv6 Operations <v6ops(a)ietf.org> Hi, Also as per the IEPG discussion, the results I had in conjunction with a summer MSc project student can be summarised as follows. The headline is that he saw a 37.7% failure rate for the Fragmentation Header (alone), a bit better than Fernandos results, but still not good. He tested the top 1,000 IPv6-enabled Alexa sites. He used the scapy toolkit which supports the four main extension header types (routing, fragmentation, destination and hop-by-hop) He tested a) valid combinations of those 4 extension headers as per RFC 2460 b) other non-valid combinations c) duplicated extension headers d) fragmentation header Primarily TCP tests, doing HTTP GET requests. For single extension headers, acceptance was Routing header 63.9% Frag header 62.3% Hop by hop header 60% Destination option header 15.8% When using no extension headers, success rate was 100% When using multiple headers, the rates fall markedly, not dissimilar with Fernandos numbers for longer headers. About 120 sites accept all four types of extension headers. A small number of sites accepted illegal combinations/ordering of extension headers. A more detailed set of results is being pushed to a conference paper. I now have another student taking this further, and validating the above results, so feel free to contact me off-list if youre interested. Tim On 4 Nov 2013, at 23:01, Fernando Gont <fernando(a)gont.com.ar> wrote: > Folks, > > I did a presentation on the topic at the IEPG meeting earlier this week. > It provides some concrete data regarding IPv6 fragmentation and > Extension Header filtering on the Internet. > > The slideware is available at: > <http://www.iepg.org/2013-11-ietf88/fgont-iepg-ietf88-ipv6-frag-and-eh.pdf> > > Certainly there's *much* more work to be done in this area, but I > thought that this could be good food sfor some of the discussions that > we were having on the topic. > > Thanks, > -- > Fernando Gont > e-mail: fernando(a)gont.com.ar || fgont(a)si6networks.com > PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1 > > > > -------------------------------------------------------------------- > IETF IPv6 working group mailing list > ipv6(a)ietf.org > Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6 > -------------------------------------------------------------------- -------------------------------------------------------------------- IETF IPv6 working group mailing list ipv6(a)ietf.org Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6 -------------------------------------------------------------------- -- Fernando Gont e-mail: fernando(a)gont.com.ar || fgont(a)si6networks.com PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

1 0

BGP failure analysis and recommendations
by JRC NOC 04 Nov '13

04 Nov '13

Hello Nanog - On Saturday, October 19th at about 13:00 UTC we experienced an IP failure at one of our sites in the New York area. It was apparently a widespread outage on the East coast, but I haven't seen it discussed here. We are multihomed, using EBGP to three (diverse) upstream providers. One provider experienced a hardware failure in a core component at one POP. Regrettably, during the outage our BGP session remained active and we continued receiving full routes from the affected AS. And our prefixes continued to be advertised at their border. However basically none of the traffic between those prefixes over that provider was delivered. The bogus routes stayed up for hours. We shutdown the BGP peering session when the nature of the problem became clear. This was effective. I believe that all customer BGP routes were similarly affected, including those belonging to some large regional networks and corporations. I have raised the questions below with the provider but haven't received any information or advice. My question is why did our BGP configuration fail? I'm guessing the basic answer is that the IGP and route reflectors within that provider were still connected, but the forwarding paths were unavailable. My BGP session basically acted like a bunch of static routes, with no awareness of the failure(s) and no dynamic reconfiguration of the RIB. Is this just an unavoidable issue with scaling large networks? Is it perhaps a known side effect of MPLS? Have we/they lost something important in the changeover to converged mutiprotocol networks? Is there a better way for us edge networks to achieve IP resiliency in the current environment? This is an operational issue. Thanks in advance for any hints about what happened or better practices to reduce the impact of a routine hardware fault in an upstream network. - Eric Jensen >Date: Wed, 23 Oct 2013 21:26:43 -0400 >To: cj(a)chrisjensen.org >From: JRC NetOps <noc(a)jensenresearch.com> >Subject: Fwd: BGP failure analysis and recommendations > > >>Date: Mon, 21 Oct 2013 23:19:28 -0400 >>To: christopher.smith(a)level3.com >>From: Eric Jensen <ejensen(a)jensenresearch.com> >>Subject: BGP failure analysis and recommendations >>Cc: "Joe Budelis Fast-E.com" <Joe(a)Fast-E.com> >>Bcc: noc(a)jensenresearch.com >> >>Hello Christopher Smith - >> >>I left you a voicemail message today. The Customer Service folks also >>gave me your email address. >> >>We have a small, but high-value multi-homed corporate network. >>We operate using our AS number 17103. >> >>We have BGP transit circuits with Level 3, Lightpath, and at our colo >>center (AS8001) >>The Level 3 circuit ID is BBPM9946 >> >>On Saturday, October 19 2013 we had a large IP outage. I tracked it back >>to our Level 3 circuit and opened a ticket (7126634). >>I have copied (below) an email I sent our channel salesman with more >>details about our BGP problems during your outage. >>Briefly, I am very concerned that Level 3 presented routes to us that >>were not actually reachable through your network, and even worse Level 3 >>kept advertising our prefixes even though your network couldn't deliver >>the traffic to us for those prefixes. >> >>I believe that the BGP NLRI data should follow the same IP path as the >>forwarded data itself. Apparently this isn't the case at Level 3. >>I also believe that your MPLS backbone should have recovered >>automatically from the forwarding failure, but this didn't happen either. >>My only fix was to manually shutdown the BGP peering session with Level 3. >> >>Can you explain to me how Level 3 black-holed my routes? >>Can you suggest some change to our or your BGP configuration to eliminate >>this BGP failure mode? >> >>Just to be clear, I don't expect our circuit, or your network, to be up >>all the time. But I do expect that the routes you advertise to us and to >>your BGP peers actually be reachable through your network. On Saturday >>this didn't happen. The routes stayed up while the data transport was down. >> >>Our IPv4 BGP peering session with Level 3 remains down in the interim. >>Please get back to me as soon as possible. >> >>- Eric Jensen >>AS17103 >>201-741-9509 >> >> >> >>>Date: Mon, 21 Oct 2013 22:55:35 -0400 >>>To: "Joe Budelis Fast-E.com" <Joe(a)Fast-E.com> >>>From: Eric Jensen <ejensen(a)jensenresearch.com> >>>Subject: Re: Fwd: Level3 Interim Response >>>Bcc: noc(a)jensenresearch.com >>> >>>Hi Joe- >>> >>>Thanks for making the new inquiry. >>>This was a big outage. Apparently Time Warner Cable and Cablevision >>>were affected greatly. Plus many large corporate networks. And of course >>>all the single-homed Level 3 customers worldwide. My little network was >>>just one more casualty. >>> >>>See: >>> >>>http://www.dslreports.com/forum/r28749556-Internet-Level3-Outage- >>> >>>http://online.wsj.com/news/articles/SB1000142405270230486450457914581369858… >>> >>>For our site, the massive outage started at about 9:00 am Saturday and >>>lasted until after 2:00PM. I opened a ticket about 9:30 am but only >>>realized the routing issues and took down our BGP session about 12:00 to >>>try to minimize the problems for our traffic caused by their >>>misconfigured BGP. >>> >>>There can always be equipment failures and fiber cuts. That's not the >>>problem. >>> From my point of view the problem was/is that Level 3 kept >>> "advertising" our prefix but couldn't deliver the packets to us. They >>> did this for all their customer's prefixes, thereby sucking in about >>> half the NYC area internet traffic and dumping into the Hudson River, >>> for a huge period of time. >>>They also kept advertising all their BGP routes to me, thereby fooling >>>my routers into sending outbound traffic to Level 3 where they again >>>dumped my traffic into the Hudson. >>> >>>I called Level 3 customer service today and have the name of a network >>>engineer to discuss options for fixing the BGP failure. >>>If you get any response with an engineering contact please let me know. >>> >>>I shouldn't have to manually intervene to route around problems. Even >>>sadder is the response from Level 3 explaining that they spent hours >>>trying to find the problem and had to manually reconfigure their >>>network, leading to saturated links and more problems. Their network >>>only healed when the faulty line card was replaced. >>> >>>I had reactivated the BGP session later that night, but after reviewing >>>the actual damage that we incurred, and the widespread nature of the >>>failure, I have decided to leave our Level 3 BGP session down, at least >>>until the engineering situation improves. >>>There may not be any good way to use a Level 3 BGP session without >>>risking the same "black hole" problem going forward. It's that type of >>>failure that BGP is specifically designed to deal with, but it was >>>developed in the days of point-to-point circuits carrying IP traffic. >>> >>>Nowadays some networks have a new layer between the wires and IP, namely >>>MPLS, and this allowed BGP to stay up but deprived the routers of >>>functioning IP next-hops, which they (both the Level 3 IP routers and >>>the Level 3 personnel) were unaware of. Apparently the Level 3 IP-based >>>BGP routers all believed they had working circuits edge-to-edge, but in >>>fact their network was partitioned. >>> >>>MPLS must have some redundancy features, but they obviously weren't >>>working on Saturday. This is a huge engineering failure. No large ISP >>>could function this way for long. >>> >>>I can wait the 72 hours for their response. I expect it will be full of >>>mealy-mouth platitudes about how no system is foolproof and it will all >>>be fine now. >>> >>>It would be more interesting to me to be in the meeting room where some >>>engineer has to explain how they could lose so much traffic and not be >>>able to operate a functioning, if degraded, network after a single line >>>card failure. It wouldn't be the head of network design, because that >>>person would already have been fired. >>> >>>Let me know if your hear anything. I will do the same. >>> >>>- Eric Jensen >>>AS17103 >>>201-741-9509 >>> >>> >>>

6 7