February 2019 - Test - lists.nanog.org

Analysing traffic in context of rejecting RPKI invalids using pmacct
by Job Snijders 14 Mar '19

14 Mar '19

Dear all, Whether to deploy RPKI Origin Validation with an "invalid == reject" policy really is a business decision. One has to weigh the pros and cons: what are the direct and indirect costs of accepting misconfigurations or hijacks for my company? what is the cost of deploying RPKI? What is the cost of honoring misconfigured RPKI ROAs? There are a few thousand misconfigured ROAs, what does this mean for me? To answer these questions, Paolo Lucente and myself worked to extend pmacct traffic analysis engine (http://pmacct.net/) in such a way that it can do perform the RFC 6811 Origin Validation procedure and present the outcome as a property in the flow aggregation process. Pmacct has the ability to ingest BGP feeds and correlate the BGP data to the sflow/netflow/ipfix data. This allows for fantastic business intelligence, you can see exactly how much traffic is flowing from what customers to what endpoints for what reason! Pmacct implemented Origin Validation in a cute way: it separates out RPKI invalid BGP announcements into two categories: a) "invalid with no overlapping or alternative route" (aka will be blackholed if 'invalid == reject') b) "invalid but an overlapping unknown/valid announcement also exists" (end-to-end connectivity can still work). Because pmacct separates out the various types kinds of (invalid) BGP announcements, operators don't have to do deploy *anything* in their network to get a good grasp on how their connectivity to the rest of the Internet would look like after deploying a "invalid == reject" policy. No changes to your network configurations are required to make use of this feature, you don't need to tag routes with communities or do other tricks. All the analysis happens inside pmacct. Of course we tested this first in the NTT global backbone AS 2914! At the moment of writing, we're seeing less than a handful of gigabits per second being send towards BGP announcements that are RPKI Invalid and for which no alternative route exists. In context of NTT's backbone that amount of traffic is just statistical noise. This is a very encouraging sign, it may help us move towards the goal of deploying RPKI Origin Validation in AS 2914. Nusenu wrote a great blog post on where these RPKI ROA misconfigurations are located, i recommend reading their posts to develop a better understanding of the problem space: https://medium.com/@nusenu/where-are-rpki-unreachable-networks-located-65c7… Even if you don't intend to deploy RPKI Origin Validation (or are single-homed), pmacct's RPKI capabilities can be useful in forensic investigations. It'll be easier to analyse how much and what kind of traffic for what period of time was sent to a possible hijack. This will help you when writing RFOs! If you want to testdrive this feature, fetch pmacct version 1.7.3-rc1 from https://github.com/pmacct/pmacct/releases/tag/1.7.3-rc1 Documentation on how to configure the feature: https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L1783-#L1833 https://github.com/pmacct/pmacct/blob/master/CONFIG-KEYS#L2626-#L2647 Let us know what you think! Or if you'd like to chat telemetry with Paolo or me about analysing the effects of BGP hijacks and RPKI, we'll both be at the San Francisco NANOG meeting next week! Kind regards, Job ps. Dear Kentik & Deepfield, please copy+paste this feature! We'll happily share development notes with you, you can even look at pmacct's source code for inspiration. :-)

4 4

ICMPv6 "too-big" packets ignored (filtered ?) by Cloudflare farms
by Jean-Daniel Pauget 13 Mar '19

13 Mar '19

hello, I confess using IPv6 behind a 6in4 tunnel because the "Business-Class" service of the concerned operator doesn't handle IPv6 yet. as such, I realised that, as far as I can figure, ICMPv6 packet "too-big" (rfc 4443) seem to be ignored or filtered at ~60% of ClouFlare's http farms as a result, random sites such as http://nanog.org/ or https://www.ansible.com/ are badly reachable whenever small mtu are involved ... support@cloudflare answered me that because I'm not the owner of concerned site, and because of security reasons, they wouldn't investigate further. are there security concerns with ICMP-too-big ? regards, -- Jean-Daniel Pauget http://rezopole.net/ Rezopole/LyonIX +33 (0)4 27 46 00 50

19 38

Arista Layer3
by Romeo Czumbil 09 Mar '19

09 Mar '19

So I've been using Arista as layer2 for quite some time, and I'm pretty happy with them. Kicking the idea around to turn on some Layer3 features but I've been hearing some negative feedback. The people that I did hear negative feedback don't use Arista themselves. (they just heard....) So do we have any Arista L3 people out here that can share some negatives or positives? Use case: Just some MPLS IPv4/IPv6 routing, l2vpn OSPF/BGP Maybe 20k routes (no full internet routes) 7050 Series 7280 Series -Romeo

20 33

WIndows Updates Fail Via IPv6
by Mark Tinka 07 Mar '19

07 Mar '19

Hi all. Anyone ever figured out why Windows updates fail when the computer has an IPv6 connection? Google has tickets and tickets of this to and outside of Microsoft since 2013, with no real solution or answer as to what the problem actually is. In essence, many of the solutions out there point toward making sure the updates do not occur over IPv6, which, in effect, is the same as disabling it. I have a family PC at home running Windows 10 Pro, and noticed updates would fail in recent months. It took me a moment to realize that this started happening only after I enabled IPv6 in the TCP/IP stack. Disabling it immediately solves the issue. Quite odd that this is happening in 2018... Mark.

20 49

a quick survey about LLDP and similar
by Pierfrancesco Caci 05 Mar '19

05 Mar '19

Hello, having a bit of a debate in my team about turning on LLDP and/or CDP. I would appreciate if you could spend a minute answering this survey so I have some numbers to back up my reasoning, or to accept defeat. https://www.surveymonkey.com/r/TH3WCWP Feel free to cross-post to other relevant lists. Thank you Pf -- Pierfrancesco Caci, ik5pvx

8 8

Re: How to choose a transit provider?
by nanog-isp＠mail.com 05 Mar '19

05 Mar '19

Mike Hammett wrote: > Usually, DIA (as transit delivered to a customer) is more expensive than transport + transit + small colo > (1U\2U stuff) + IX... at least as observed by many of my brethren. Is this really true in the general case? Adding colo and IX to transport and transit involves at least one additional cross connect and an IX port fee. This is likely to push the total above the pure DIA price. However, regardless of how the numbers pencil out, this isn't really a fair comparison. For small ISPs, the yardstick against which adding an IX to the mix is usually measured against is the marginal cost of IP transit. Given that the cost of transport is fixed, is it more economical to buy more IP transit or to join an IX? Transit being so cheap means that joining an IX isn't always so enticing from a financial perspective, although there are other non-monetary benefits. I certainly subscribe to the notion that transport + transit is usually less expensive than DIA, but this does depend on the market and location. Jared

4 4

HULU NOC
by John Alcock 04 Mar '19

04 Mar '19

Afternoon, I have searched the forums and have had no luck. We have just received a new block of ip's. None of my subscribers can get to Hulu. I have started updating all the major GeoIP Databases. I figure I need to get Hulu to update their database. Of course calling regular support is useless Anyone have a contact? John Alcock john(a)alcock.org Network Engineer Highland Communications

2 1

Re: A Deep Dive on the Recent Widespread DNS Hijacking
by James Renken 04 Mar '19

04 Mar '19

On Feb 25, 2019, at 5:20 AM, Bill Woodcock <woody(a)pch.net> wrote: > We know that neither Comodo nor Let's Encrypt were DNSSEC validating before issuing certs. I’d like to clarify that Let’s Encrypt has always validated DNSSEC, dating to before we issued our first publicly trusted certificate in September 2015. -- James Renken (pronouns: he/him) Internet Security Research Group Let's Encrypt: A Free, Automated, and Open CA

2 1

Question about ISP billing procedures
by Daniel Rohan 02 Mar '19

02 Mar '19

Can anyone shed light on how ISPs handle missing samples when calculating p95s for monthly billing cycles? Do they fill null samples with zeros or leave them as null? I’m working on a billing sanity tool and want to make sure to cover my corner cases well. Thanks! Dan -- Thanks, Dan

9 8

RE: A Deep Dive on the Recent Widespread DNS Hijacking
by Montgomery, Douglas (Fed) 27 Feb '19

27 Feb '19

You might have missed reading the very article you cite. "Woodcock said PCH’s reliance on DNSSEC almost completely blocked that attack, but that it managed to snare email credentials for two employees who were traveling at the time. .... Aside from that, DNSSEC saved us from being really, thoroughly owned.” Or maybe ACME .. https://tools.ietf.org/html/draft-ietf-acme-acme-12#section-11.2 "It is therefore RECOMMENDED that ACME-based CAs make all DNS queries via DNSSEC-validating stub or recursive resolvers. This provides additional protection to domains which choose to make use of DNSSEC.” I am not sure how many of the domains listed as being hijacked are DNSSEC signed, but it seems if they were, and had a reasonable long TTL on a DS record at their parent, many if not most of these could have been prevented/detected. ICANN seems to think that is the case: ICANN Calls for Full DNSSEC Deployment https://www.icann.org/news/announcement-2019-02-22-en Of course, DNSSEC is often blamed for not protecting those who did not deploy/use it. Not sure what can be said about that line of reasoning. Dougm -- Doug Montgomery, Manager Internet & Scalable Systems Research @ NIST ============ Date: Sat, 23 Feb 2019 12:13:41 -0700 From: "Keith Medcalf" <kmedcalf(a)dessus.com> To: "nanog(a)nanog.org" <nanog(a)nanog.org> Subject: RE: A Deep Dive on the Recent Widespread DNS Hijacking Attacks Message-ID: <6e31d305aee69c4d85116e6a81d0c91d(a)mail.dessus.com> Content-Type: text/plain; charset="us-ascii" On Saturday, 23 February, 2019 10:03, Stephane Bortzmeyer wrote: >Very good article, very detailed, with a lot of technical precisions, >about the recent domain name hijackings (not using the DNS, just good >old hijackings at registrar or hoster). >https://krebsonsecurity.com/2019/02/a-deep-dive-on-the-recent-widespread-dn… So in other words this was just an old school script kiddie taking advantage of DNS registrars, the only difference being this was a whole whack of script kiddies acting in concert directed by a not-quite-so-stupid script kiddie, with some "modernz" thrown in for good measure. (Sounds like an NSA operation to me -- and the targets perfectly match those that the NSA would choose -- plus some good old misdirection just for the jollies of it) The second takeaway being that DNSSEC is useless in preventing such an occurrence because the script kiddies can merely turn it off at the same time as they redirect DNS. However, having DNSSEC can protect you from incompetent script-kiddies. It can also give you a false sense of security. Did I miss anything? --- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume.

35 79