On Tue, 31 Jan 2012, Grant Ridder wrote:

What is keeping you from advertising a more specific route (i.e /25's)?

Many providers filter out anything longer (smaller) than /24. jms

On Tue, Jan 31, 2012 at 12:00 PM, Kelvin Williams <kwilliams@altuscgi.com>wrote:

...

Greetings all.

We've been in a 12+ hour ordeal requesting that AS19181 (Cavecreek Internet Exchange) immediately filter out network blocks that are being advertised by ASAS33611 (SBJ Media, LLC) who provided to them a forged LOA.

The routes for networks: 208.110.48.0/20, 63.246.112.0/20, and 68.66.112.0/20 are registered in various IRRs all as having an origin AS 11325 (ours), and are directly allocated to us.

The malicious hijacking is being announced as /24s therefore making route selection pick them.

Our customers and services have been impaired. Does anyone have any contacts for anyone at Cavecreek that would actually take a look at ARINs WHOIS, and IRRs so the networks can be restored and our services back in operation?

Additionally, does anyone have any suggestion for mitigating in the interim? Since we can't announce as /25s and IRRs are apparently a pipe dream.

-- Kelvin Williams Sr. Service Delivery Engineer Broadband & Carrier Services Altus Communications Group, Inc.

"If you only have a hammer, you tend to see every problem as a nail." -- Abraham Maslow

Many/most transit providers filter prefixes longer than /24, so the effectiveness may be minimal. At the very least I'd advertise /24s yourself because if the forger is geographically further away, some local sites may still work. Better than nothing. On Tue, Jan 31, 2012 at 11:19 AM, Grant Ridder <shortdudey123@gmail.com>wrote:

Hi,

What is keeping you from advertising a more specific route (i.e /25's)?

-Grant

On Tue, Jan 31, 2012 at 12:00 PM, Kelvin Williams <kwilliams@altuscgi.com

...

wrote:

...

Greetings all.

We've been in a 12+ hour ordeal requesting that AS19181 (Cavecreek Internet Exchange) immediately filter out network blocks that are being advertised by ASAS33611 (SBJ Media, LLC) who provided to them a forged LOA.

The routes for networks: 208.110.48.0/20, 63.246.112.0/20, and 68.66.112.0/20 are registered in various IRRs all as having an origin AS 11325 (ours), and are directly allocated to us.

The malicious hijacking is being announced as /24s therefore making route selection pick them.

Our customers and services have been impaired. Does anyone have any contacts for anyone at Cavecreek that would actually take a look at ARINs WHOIS, and IRRs so the networks can be restored and our services back in operation?

Additionally, does anyone have any suggestion for mitigating in the interim? Since we can't announce as /25s and IRRs are apparently a pipe dream.

-- Kelvin Williams Sr. Service Delivery Engineer Broadband & Carrier Services Altus Communications Group, Inc.

"If you only have a hammer, you tend to see every problem as a nail." -- Abraham Maslow

Surely something is better than nothing. Advertise the /24's and the /25's, see what happens. At the least it's a step forwards until you get their routes filtered. Tony On 31 January 2012 18:22, Kelvin Williams <kwilliams@altuscgi.com> wrote:

Upstream requirements. Additionally, I don't believe it would do us any good. If they're announcing /24 now, why would they not announce a /25. On Jan 31, 2012 1:19 PM, "Grant Ridder" <shortdudey123@gmail.com> wrote:

...

Hi,

What is keeping you from advertising a more specific route (i.e /25's)?

-Grant

On Tue, Jan 31, 2012 at 12:00 PM, Kelvin Williams < kwilliams@altuscgi.com>wrote:

...

Greetings all.

We've been in a 12+ hour ordeal requesting that AS19181 (Cavecreek Internet Exchange) immediately filter out network blocks that are being advertised by ASAS33611 (SBJ Media, LLC) who provided to them a forged LOA.

The routes for networks: 208.110.48.0/20, 63.246.112.0/20, and 68.66.112.0/20 are registered in various IRRs all as having an origin AS 11325 (ours), and are directly allocated to us.

The malicious hijacking is being announced as /24s therefore making route selection pick them.

Our customers and services have been impaired. Does anyone have any contacts for anyone at Cavecreek that would actually take a look at ARINs WHOIS, and IRRs so the networks can be restored and our services back in operation?

Additionally, does anyone have any suggestion for mitigating in the interim? Since we can't announce as /25s and IRRs are apparently a pipe dream.

-- Kelvin Williams Sr. Service Delivery Engineer Broadband & Carrier Services Altus Communications Group, Inc.

"If you only have a hammer, you tend to see every problem as a nail." -- Abraham Maslow

On Wednesday, February 01, 2012 02:57:46 AM Tony McCrory wrote:

Surely something is better than nothing. Advertise the /24's and the /25's, see what happens.

The fact that the hijacking ISP's upstreams accepted routes through their network that didn't belong to that ISP is bad enough. That we should still be able to advertise anything without an appropriate filter being in place and expecting it to work (even if it's with good intention, as in this case) is equally as bad. A big fail to our community, for up to this day, not implementing basic routing and forwarding filters that would do away with all this cruft in the first place. Clearly the Youtube/Pakistan/PCCW incident has long been forgotten. But that's life, I guess... Mark.

On Monday, February 06, 2012 12:26:51 PM Suresh Ramasubramanian wrote:

I had this happen to me in 2008 - http://www.gossamer-threads.com/lists/nanog/users/110097 Total pain in the ass when it does happen. Funnily enough in that case it was another downstream of the same ISP who was pulling this stunt ..

Clearly the joy of running a clean network is not shared by all :-). Yes, it is a little bit of extra hassle having to update filters when your downstreams make change requests (including verification, e.t.c.). But when some of our upstreams make us go through this, I'm much happier they do than they if they didn't. Some are even asking us to "fax" documents of such requests; a little extreme, but okay, I'll bite :-). We do have some upstreams who are pretty lax about this. But we certainly are not, and as a result, are yet to put one of our customers or the Internet in jeopardy because we let one through the cracks. It's 2012, we really shouldn't be seeing this type of thing anymore, particularly after what happened in Pakistan. Mark.

On Monday, February 06, 2012 01:14:20 PM Christopher Morrow wrote:

o not having filters at all (pccw/pktel)

Well, we know what this leads to (part of the reasons you find some eBGP sessions carrying /25's or longer + RFC 1918 space is because of this).

o filtering using old/stale data (ntt/l3)

Well, we think that this problem resolves itself rather well: o If a customer is not getting traffic hitting a prefix, they'll check with their upstream on if their prefix is being received and propagated. o If a customer is not seeing their prefix on the Internet, they'll check with their upstream on if their prefix is being received and propagated. o If a customer starts announcing a new prefix without updating their upstream, one e-mail or phone call to the upstream will get this fixed. In all cases above, it's better not to have an unauthorized prefix in the yonder than have open filters, because one is more likely to quickly get resolved without any colleteral than the other.

why aren't filters applied at all? ("its hard, people keep asking me to update them, bah! work!")

Well, so was running the Internet without BGP communities or prefix lists or AS_PATH filters, until they appeared. And while some folk still use so-called "distribute lists" today, it's fine with me as long as they maintain secure operations with whatever solution they choose, hard or easy. Some ISP's have automated this process either by using internal IRR software, or scripting web GUI's that their NOC can use to simply stick in the prefix, select which router to update, and bam! Compared to the alternative, this is a small price to pay.

why aren't filter data bases kept up to date? ("its hard, i have to email something to radb/altdb/etc... bah, work!")...

That's why we use the RIPE RR. They have a cool web GUI that you can use to edit on object, including IRR data (and they're respected by all other major RR's and operators): https://apps.db.ripe.net/webupdates/ It certainly beats the old "MAIL FROM" system, although I believe that is still operational.

why aren't checks of the filter data simple and mechanical? (and accurate?) ("Bah! work! plus, have you looked at the ouptput? bah! work!")

See above. We manually check the RIR WHOIS database. I'm sure some networks might automate this with an internal system that checks the RIR WHOIS database, and probably integrates with their provisioning system that can automatically create and update filters on routers. I don't know... But it's certainly better than the alternative.

resource certification would at least get us to the point where checking the data in the IRR is 'easy', it's not going to get people to PUT FILTERS ON CUSTOMER SESSIONS, and it's not going to get people to update their IRR objects (add AND DELETE!!!)

I support RPKI, but also realize that operator support will take a very long time for various reasons, e.g., education, delayed software upgrades, persistence with older methods, fear of centralization, e.t.c. In such a case, operators will need to support "Invalid" and "NotFound" states of origin information for a long time. As adoption and deployment increases, operators can begin dropping "Invalid" results, "NotFound" results, or both. Or even mark them down with poor LOCAL_PREF values so as not to use those routes for forwarding unless it is really necessary. At some point, when diffusion of RPKI is sufficiently prolific, anything that does not return a "Valid" result will be dropped. This should force every operator around the world to support it, much like the large carriers forced us all to use IRR's just so they won't ignore our routes, wherever we are in the world. But before all this happens, we have to prevent more hijacks. And we have to use the tools we have today. Mark.

On Monday, February 06, 2012 03:06:24 PM Christopher Morrow wrote:

do you have customers with 10k long prefix lists? it gets hard when the lists get long, or the data is for downstream folks of your customer. Good that someone's checking though, I'd love to see this part automated.

No, we don't have customers with 10,000-long prefix lists, but we have planned to implement automation (either using the IRR Toolset or home-grown solutions) to make this possible as a means to scaling, should that time arise. As it is now, throwing NOC staff at the problem for the volumes we have works well enough. But this is, by no means, a panacea as we scale up. Heck, one must even worry whether a some router configurations can take that many lines. But there are other ways around that.

RPKI doesn't necessarily mean that the router knows anything about certificates in the short-term. I think there's a time when 'the resource certification system' (which is really, today, the rpki) holds cert/roa data that you could use to filter what the IRR tells you for a customer. You could even do this in any automated manner!

Well, given validation information will be available within a network, one may use it in non-obvious ways to implement policy.

The time between the previous and next paragraphs though is when all isp's will need to beat the drums with their customers saying: "Hey, you REALLY need to get that shit into the 'resource certification system' (rpki), NOW." (because shortly we'll stop accepting your "invalid" routes... and then the interwebs won't be able to find you, and we'll all be sad.)

Well, we have all seen how doing this with DNSSEC, IPv6 and 4-byte ASN's worked out. We need to understand that different operators and different customers will have varying reasons as to why they can't yet "do the right thing". There is abundant evidence of this with other similar initiatives. Adoption will have to be incremental. During that time, the Internet must not break.

sure... it's not working so well though :(

Not for lack of having some kind of solution. What we have today may not be the best, most scalable option, but it is one nonetheless. Automating it (like what RPSL does) is how you can make it scale to some extent, but it's still the same solution. We can't just sit around waiting for RPKI. There will be some reason why some of us can't deploy it, while someone else is thinking up its successor. Rinse, repeat. Mark.

With regards to RPKI, I'd like to point out what is possible now, and what the maturity is of the implementations. All RIRs have a system up an running. As John Curran pointed out in an earlier message, ARIN will have a production system up this year, but right now you can already gain experience with their testbed: https://www.arin.net/resources/rpki.html If you create your ROAs there, there are actually quite some parties who will already validate this information. Of course, basing an actual routing decision on this is a second step; for that we'll need more and better quality data. As it stands there are about 1400 IPv4 and 300 IPv6 announcements that have a ROA associated with them. There are some public test beds up that will give a valid route a higher pref, and an invalid one a lower pref. So create a ROA for your announcements, and then watch it show up on actual RPKI-capable Cisco and Juniper routers: EuroTransit have made their instance of the RIPE NCC RPKI Validator public, so you can easily verify the validity of your announcement here by searching for it: http://rpki01.fra2.de.euro-transit.net:8080/bgp-preview Then you can log in on a public Juniper here: 193.34.50.25, 193.34.50.26 telnet username: rpki password: testbed Try commands such as: - show validation session detail - show validation statistics - show validation database - show route protocol bgp 204.127.128.0/17 - show route protocol bgp validation-state valid You can also log into a public Cisco here: rpki-rtr.ripe.net telnet username: ripe no password Try commands such as: - sh ip bgp rpki table - sh ip bgp rpki server - sh ip bgp 93.175.146.0/24 - sh ip bgp ipv6 unicast rpki table - sh ip bgp ipv6 unicast 2001:630::/32 Both routers are configured along these lines: https://www.ripe.net/certification/router-configuration and talk to the RIPE NCC RPKI Validator service available here: https://www.ripe.net/certification/tools-and-resources Try it out, and give feedback! Cheers, Alex P.S. RFCs 6480-6493 have been published. A big thank you goes out to all the people who made that possible. On 6 Feb 2012, at 08:59, Mark Tinka wrote:

On Monday, February 06, 2012 03:06:24 PM Christopher Morrow wrote:

...

do you have customers with 10k long prefix lists? it gets hard when the lists get long, or the data is for downstream folks of your customer. Good that someone's checking though, I'd love to see this part automated.

No, we don't have customers with 10,000-long prefix lists, but we have planned to implement automation (either using the IRR Toolset or home-grown solutions) to make this possible as a means to scaling, should that time arise.

As it is now, throwing NOC staff at the problem for the volumes we have works well enough. But this is, by no means, a panacea as we scale up.

Heck, one must even worry whether a some router configurations can take that many lines. But there are other ways around that.

...

RPKI doesn't necessarily mean that the router knows anything about certificates in the short-term. I think there's a time when 'the resource certification system' (which is really, today, the rpki) holds cert/roa data that you could use to filter what the IRR tells you for a customer. You could even do this in any automated manner!

Well, given validation information will be available within a network, one may use it in non-obvious ways to implement policy.

...

The time between the previous and next paragraphs though is when all isp's will need to beat the drums with their customers saying: "Hey, you REALLY need to get that shit into the 'resource certification system' (rpki), NOW." (because shortly we'll stop accepting your "invalid" routes... and then the interwebs won't be able to find you, and we'll all be sad.)

Well, we have all seen how doing this with DNSSEC, IPv6 and 4-byte ASN's worked out.

We need to understand that different operators and different customers will have varying reasons as to why they can't yet "do the right thing". There is abundant evidence of this with other similar initiatives.

Adoption will have to be incremental. During that time, the Internet must not break.

...

sure... it's not working so well though :(

Not for lack of having some kind of solution.

What we have today may not be the best, most scalable option, but it is one nonetheless. Automating it (like what RPSL does) is how you can make it scale to some extent, but it's still the same solution.

We can't just sit around waiting for RPKI. There will be some reason why some of us can't deploy it, while someone else is thinking up its successor. Rinse, repeat.

Mark.

On Mon, 6 Feb 2012, Christopher Morrow wrote:

why aren't filters applied at all?

filters don't generate revenue. -Dan

On Monday, February 06, 2012 02:41:53 PM goemon@anime.net wrote:

filters don't generate revenue.

Neither does traffic - that does generate revenue - not reaching your customer. Mark.

On Tue, Jan 31, 2012 at 10:19 AM, Grant Ridder <shortdudey123@gmail.com> wrote:

Hi,

What is keeping you from advertising a more specific route (i.e /25's)?

Most large transits and NSPs filter out prefixes more specific than a /24. Conventionally, at least in my experience, /24's are the most-specific prefix you can use and expect that it will end up in most places. Some shops with limited router processing or table storage capacity will filter even more restrictively, so a bigger aggregate is worth announcing as well. Cheers, jof

We are. On Tue, Jan 31, 2012 at 1:32 PM, Chuck Church <chuckchurch@gmail.com> wrote:

Shouldn't a forged LOA be justification to contact law enforcement?

Chuck

-----Original Message----- From: Kelvin Williams [mailto:kwilliams@altuscgi.com] Sent: Tuesday, January 31, 2012 1:01 PM To: nanog@nanog.org Subject: Hijacked Network Ranges

Greetings all.

We've been in a 12+ hour ordeal requesting that AS19181 (Cavecreek Internet Exchange) immediately filter out network blocks that are being advertised by ASAS33611 (SBJ Media, LLC) who provided to them a forged LOA.

The routes for networks: 208.110.48.0/20, 63.246.112.0/20, and 68.66.112.0/20 are registered in various IRRs all as having an origin AS 11325 (ours), and are directly allocated to us.

The malicious hijacking is being announced as /24s therefore making route selection pick them.

Our customers and services have been impaired. Does anyone have any contacts for anyone at Cavecreek that would actually take a look at ARINs WHOIS, and IRRs so the networks can be restored and our services back in operation?

Additionally, does anyone have any suggestion for mitigating in the interim? Since we can't announce as /25s and IRRs are apparently a pipe dream.

-- Kelvin Williams Sr. Service Delivery Engineer Broadband & Carrier Services Altus Communications Group, Inc.

"If you only have a hammer, you tend to see every problem as a nail." -- Abraham Maslow

-- Kelvin Williams Sr. Service Delivery Engineer Broadband & Carrier Services Altus Communications Group, Inc. Office - Direct: 404.682.2151 Office - Main: 404.682.2150 Mobile: 404.931.4888 Fax: 866.895.8557 "If you only have a hammer, you tend to see every problem as a nail." -- Abraham Maslow

Looks fixed now.. --heather -----Original Message----- From: Keegan Holley [mailto:keegan.holley@sungard.com] Sent: Tuesday, January 31, 2012 2:50 PM To: Schiller, Heather A Cc: Kelvin Williams; nanog@nanog.org Subject: Re: Hijacked Network Ranges - paging Cogent and GBLX/L3 To be honest I haven't had much success it convincing a tier 1 to modify someone else's routes on my behalf for whatever reason. I also have had limited success in getting them to do anything quickly. I'd first look to modify your advertisements as much as possible to mitigate the issue and then work with the other guys upstreams second. 2012/1/31 Schiller, Heather A <heather.schiller@verizon.com>:

Or roll it up hill:

33611 looks like they get transit from 19181, who's only upstream appears to be 12189. 12189 gets connectivity from 174 and 3549.

174 = Cogent 3549 = GBLX/L3

 --Heather

-----Original Message----- From: Kelvin Williams [mailto:kwilliams@altuscgi.com] Sent: Tuesday, January 31, 2012 1:01 PM To: nanog@nanog.org Subject: Hijacked Network Ranges

Greetings all.

We've been in a 12+ hour ordeal requesting that AS19181 (Cavecreek Internet Exchange) immediately filter out network blocks that are being advertised by ASAS33611 (SBJ Media, LLC) who provided to them a forged LOA.

The routes for networks: 208.110.48.0/20, 63.246.112.0/20, and 68.66.112.0/20 are registered in various IRRs all as having an origin AS 11325 (ours), and are directly allocated to us.

The malicious hijacking is being announced as /24s therefore making route selection pick them.

Our customers and services have been impaired.  Does anyone have any contacts for anyone at Cavecreek that would actually take a look at ARINs WHOIS, and IRRs so the networks can be restored and our services back in operation?

Additionally, does anyone have any suggestion for mitigating in the interim?  Since we can't announce as /25s and IRRs are apparently a pipe dream.

-- Kelvin Williams Sr. Service Delivery Engineer Broadband & Carrier Services Altus Communications Group, Inc.

"If you only have a hammer, you tend to see every problem as a nail." -- Abraham Maslow

I would go at first by advertising your prefixes as a /24 as well, just randomly checked 2 different locations and the as-path to 11325 is shorter than to 33611 This seems to be the case for customers of Tiscali and L3, so this will probably get most of your traffic back to you... Regards, Ido -----Original Message----- From: Kelvin Williams [mailto:kwilliams@altuscgi.com] Sent: Tuesday, January 31, 2012 1:01 PM To: nanog@nanog.org Subject: Hijacked Network Ranges Greetings all. We've been in a 12+ hour ordeal requesting that AS19181 (Cavecreek Internet Exchange) immediately filter out network blocks that are being advertised by ASAS33611 (SBJ Media, LLC) who provided to them a forged LOA. The routes for networks: 208.110.48.0/20, 63.246.112.0/20, and 68.66.112.0/20 are registered in various IRRs all as having an origin AS 11325 (ours), and are directly allocated to us. The malicious hijacking is being announced as /24s therefore making route selection pick them. Our customers and services have been impaired. Does anyone have any contacts for anyone at Cavecreek that would actually take a look at ARINs WHOIS, and IRRs so the networks can be restored and our services back in operation? Additionally, does anyone have any suggestion for mitigating in the interim? Since we can't announce as /25s and IRRs are apparently a pipe dream. -- Kelvin Williams Sr. Service Delivery Engineer Broadband & Carrier Services Altus Communications Group, Inc. "If you only have a hammer, you tend to see every problem as a nail." -- Abraham Maslow