Thanks & Let's Prevent this in the Future.
First off, I'd like to thank everyone on this list who have reached out today and offered us help with our hijacked network space. It's so refreshing to see that there are still so many who refuse to leave a man/woman down. I'm not going to place any blame, its useless. There were lies, there were incompetencies, and there was negligence but that is now water under the bridge. However, I think that we as network operators have a duty to each other to make sure we don't allow a downstream customer wreck the operations of another entity who has been rightfully allocated resources. A few months ago, when establishing a new peering relationship I was encouraged (actually required) to utilize one of the IRRs. I took the time to register all of my routes, ASNs, etc. However, as I learned today, this was probably done in vain. Too many people won't spend the extra 30-seconds to verify the information listed there or in ARINs WHOIS. I don't care what a customer tells me, too many times I've found they aren't 100% honest either for malicious/fraudulent reasons or they are unknowing. So, for our networks or the networks we manage, we want to verify what a customer is saying to prevent what happened to us today. I'd like to get a conversation going and possibly some support of an initiative to spend that extra 30-seconds to verify ownership and authorization of network space to be advertised. Additionally, if someone rings your NOC's line an industry-standard process of verifying "ownership" and immediately responding by filtering out announcements. There's no sense in allowing a service provider to be impaired because a spammer doesn't want to give up clean IP space. Do you protect a bad customer or the Internet as a whole? I pick the Internet as a whole. How can we prevent anyone else from ever enduring this again? While we may never stop it from ever happening, spammers (that's what we got hit by today) are a dime a dozen and will do everything possible to hit an Inbox, so how can we establish a protocol to immediate mitigate the effects of an traffic-stopping advertisement? I thought registering with IRRs and up-to-date information in ARINs WHOIS was sufficient, apparently I was wrong. Not everyone respects them, but then again, they aren't very well managed (I've got several networks with antiquated information I've been unable to remove, it doesn't impair us normally, but its still there). What can we do? Better yet, how do we as a whole respond when we encounter upstream providers who refuse to look at the facts and allow another to stay down? kw -- Kelvin Williams Sr. Service Delivery Engineer Broadband & Carrier Services Altus Communications Group, Inc. "If you only have a hammer, you tend to see every problem as a nail." -- Abraham Maslow
On 1 Feb 2012, at 09:01, "Kelvin Williams" <kwilliams@altuscgi.com> wrote:
A few months ago, when establishing a new peering relationship I was encouraged (actually required) to utilize one of the IRRs. I took the time to register all of my routes, ASNs, etc. However, as I learned today, this was probably done in vain. Too many people won't spend the extra 30-seconds to verify the information listed there or in ARINs WHOIS.
It's amazing isn't it. It isn't only fraud and maliciousness that this prevents. A number of times I have been asked to advertise space or open filters for space based on typos and people not understanding address notation and CIDR. So it's with doing if only to prevent this. -- Leigh ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________
I'd like to get a conversation going and possibly some support of an initiative to spend that extra 30-seconds to verify ownership and authorization of network space to be advertised. Additionally, if someone rings your NOC's line an industry-standard process of verifying "ownership" and immediately responding by filtering out announcements. There's no sense in allowing a service provider to be impaired because a spammer doesn't want to give up clean IP space. Do you protect a bad customer or the Internet as a whole? I pick the Internet as a whole.
How can we prevent anyone else from ever enduring this again? While we may never stop it from ever happening, spammers (that's what we got hit by today) are a dime a dozen and will do everything possible to hit an Inbox, so how can we establish a protocol to immediate mitigate the effects of an traffic-stopping advertisement?
One problem is the number of routing registries and the requirements differ for them. The nefarious operator can enter routes in an IRR just as easily as a legitimate operator. There was a time when some significant networks used the IRRs for their filtration policy. I'm not sure how many still do. But generally speaking, if someone calls me and I can verify that they really are a POC for the entity that is assigned an address allocation (generally some verification method beyond email if the subnet their MX record points to is part of the hijacking!) then I am going to do whatever I can to help them out provided what they are asking for is reasonable and within my capabilities. It shouldn't be too hard to verify. If someone claims to be with a commercial entity of Foo.COM then the entity is likely listed in the phone book and a phone call can take care of my personal verification requirement. Back in the days of Cyberpromo and Sanford Wallace, what I did was used TCP wrappers on my mail server so that when I received a connection from a Cyberpromo net block, I hairpinned the connection back to his MX server using netcat so when he connected to me, the HELO he received was from his own mail server, which gladly accepted mail from Cyberpromo. He could pump mail to me all day long if he wanted to, but his mailq wasn't going to get any smaller. The problem of email spam is an interesting one that has been battled for a very long time and is probably better discussed on a list dedicated to that topic.
On Wed, 1 Feb 2012, George Bonser wrote:
One problem is the number of routing registries and the requirements differ for them. The nefarious operator can enter routes in an IRR just as easily as a legitimate operator. There was a time when some significant networks used the IRRs for their filtration policy. I'm not sure how many still do.
A few do, but IRR data really can't be trusted as a means of authenticating authority to advertise IP space. It's a nice system for automating route filter updates (between the customer and provider...i.e. I add data to an IRR, and Level3 auto-updates my prefix filter), but when anyone can put anything into the IRR dbs, obviously everyone using the IRR dbs isn't going to stop someone from hijacking someone else's space. As a regional ISP / hosting provider, my policy for accepting BGP routes from customers has been to check RIR whois data, make sure the space appears to be owned by the customer, and if there's any question, ask for clarification and/or an LOA from the customer showing that they are authorized to use the space. Every customer gets a prefix filter that allows them to announce only what we've verified they're authorized to announce. Level3, I suppose, just trusts customers won't announce space they shouldn't. The alternative, which I've dealt with with some other "Tier 1's" is that they require customers to provide an LOA every time they want to add a CIDR to their prefix filter. That's more paper work for me and for them, and tends to be slower, as someone has to manually approve (maybe manually apply) the change request. Given what we have available today, there's security or convenience...pick one. It seems like the IRR thing could reasonably easily be solved if the RIRs got more deeply involved. Suppose, as an ARIN member, I used ARIN's IRR. I'd start out by registering my maintainer object, my ASN, my routes, and an as-set. Then, when I wanted to add customer ASNs to my as-set, the system wouldn't allow me to do so unless the customer had already registered their ASN with my ASN as an approved provider/path. The customer, if they had no ASN, would be responsible for updating their route (PI or PA) in the IRR db, stating my ASN is a valid origin for their route. This would solve the problem of larger providers like Level3 blindly accepting my as-set because all the authentication/authorization would already have been done before the data went into the IRR db. Things get a little more complicated when I pick up a customer who has "out of region" objects, i.e. RIPE IPs/ASN, but if all the RIRs worked together and standardized how the information is maintained, it could work. i.e. When I try to add the RIPE ASN to my as-set, ARIN's system would check RIPE's system to see if that ASN's owner had set my ASN as a valid provider/path for their ASN. This seems so simple, either it should have been implemented a decade or more ago, or perhaps I'm overlooking something. It wouldn't stop route advertisements from providers who don't care / don't filter, but it would make systems like Level3's more secure...and if all the "Tier 1's" adopted similar automated filter generators based on the RIR IRR dbs, it would stop unauthorized announcements from getting far.
The problem of email spam is an interesting one that has been battled for a very long time and is probably better discussed on a list dedicated to that topic.
Definitely...but the issue here is ownership/authorization to use IP space, and how stop unauthorized BGP announcements when providers don't or won't filter their BGP customers. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On Thursday, February 02, 2012 01:00:43 AM George Bonser wrote:
One problem is the number of routing registries and the requirements differ for them. The nefarious operator can enter routes in an IRR just as easily as a legitimate operator. There was a time when some significant networks used the IRRs for their filtration policy. I'm not sure how many still do.
I've dealt with AfriNIC and APNIC WHOIS databases, and they normally control the 'inetnum' and inet6num' entries that go into the WHOIS databases. So there is some degree of certainty that what is in there is generally true. You're right, anyone can create an IRR record, and it's quite terrible how easy it is to create false information that could break another person's network. This is why we don't generally trust IRR or PeeringDB data when verifying downstream prefixes which we should permit through our filters. We rely on the RIR 'inetnum' and 'inet6num' records for that. My memory fails me on what ARIN do, but before AfriNIC was established and the majority of Africa's prefixes were allocated by RIPE and ARIN, I recall the ARIN policy (SWIP templates, et al) being a hassle-rich experience that anything else is long forgotten :-). Mark.
One option is to use RPKI and origin validation. But it won't help much unless prefix holders create their certificates and ROAs and networks operators use those to validate origins. It won't solve all the issues but at least some fat fingers/un-expierience errors. We are running an experiment to detect route-hijacks/missconf using RPKI. So far, not many routes are "signed" but at least we can periodically check our own prefix (or any other with ROAs) to detect some inconsistencies: http://www.labs.lacnic.net/rpkitools/looking_glass/rest/all/pfx/200.7.84.0/ http://www.labs.lacnic.net/rpkitools/looking_glass/ Regards, -as On 1 Feb 2012, at 06:58, Kelvin Williams wrote:
First off, I'd like to thank everyone on this list who have reached out today and offered us help with our hijacked network space. It's so refreshing to see that there are still so many who refuse to leave a man/woman down.
I'm not going to place any blame, its useless. There were lies, there were incompetencies, and there was negligence but that is now water under the bridge.
However, I think that we as network operators have a duty to each other to make sure we don't allow a downstream customer wreck the operations of another entity who has been rightfully allocated resources.
A few months ago, when establishing a new peering relationship I was encouraged (actually required) to utilize one of the IRRs. I took the time to register all of my routes, ASNs, etc. However, as I learned today, this was probably done in vain. Too many people won't spend the extra 30-seconds to verify the information listed there or in ARINs WHOIS.
I don't care what a customer tells me, too many times I've found they aren't 100% honest either for malicious/fraudulent reasons or they are unknowing. So, for our networks or the networks we manage, we want to verify what a customer is saying to prevent what happened to us today.
I'd like to get a conversation going and possibly some support of an initiative to spend that extra 30-seconds to verify ownership and authorization of network space to be advertised. Additionally, if someone rings your NOC's line an industry-standard process of verifying "ownership" and immediately responding by filtering out announcements. There's no sense in allowing a service provider to be impaired because a spammer doesn't want to give up clean IP space. Do you protect a bad customer or the Internet as a whole? I pick the Internet as a whole.
How can we prevent anyone else from ever enduring this again? While we may never stop it from ever happening, spammers (that's what we got hit by today) are a dime a dozen and will do everything possible to hit an Inbox, so how can we establish a protocol to immediate mitigate the effects of an traffic-stopping advertisement?
I thought registering with IRRs and up-to-date information in ARINs WHOIS was sufficient, apparently I was wrong. Not everyone respects them, but then again, they aren't very well managed (I've got several networks with antiquated information I've been unable to remove, it doesn't impair us normally, but its still there).
What can we do? Better yet, how do we as a whole respond when we encounter upstream providers who refuse to look at the facts and allow another to stay down?
kw
-- Kelvin Williams Sr. Service Delivery Engineer Broadband & Carrier Services Altus Communications Group, Inc.
"If you only have a hammer, you tend to see every problem as a nail." -- Abraham Maslow
In related news, the IETF working group that is writing standards for the RPKI is having an interim meeting in San Diego just after NANOG. They deliberately chose that place/time to make it easy for NANOG attendees to contribute, so comments from this community are definitely welcome. <http://www.ietf.org/mail-archive/web/sidr/current/msg03923.html> <http://trac.tools.ietf.org/wg/sidr/trac/wiki/InterimMeeting20120209> On Fri, Feb 3, 2012 at 7:16 AM, Arturo Servin <aservin@lacnic.net> wrote:
One option is to use RPKI and origin validation. But it won't help much unless prefix holders create their certificates and ROAs and networks operators use those to validate origins. It won't solve all the issues but at least some fat fingers/un-expierience errors.
We are running an experiment to detect route-hijacks/missconf using RPKI. So far, not many routes are "signed" but at least we can periodically check our own prefix (or any other with ROAs) to detect some inconsistencies:
http://www.labs.lacnic.net/rpkitools/looking_glass/rest/all/pfx/200.7.84.0/
http://www.labs.lacnic.net/rpkitools/looking_glass/
Regards, -as
On 1 Feb 2012, at 06:58, Kelvin Williams wrote:
First off, I'd like to thank everyone on this list who have reached out today and offered us help with our hijacked network space. It's so refreshing to see that there are still so many who refuse to leave a man/woman down.
I'm not going to place any blame, its useless. There were lies, there were incompetencies, and there was negligence but that is now water under the bridge.
However, I think that we as network operators have a duty to each other to make sure we don't allow a downstream customer wreck the operations of another entity who has been rightfully allocated resources.
A few months ago, when establishing a new peering relationship I was encouraged (actually required) to utilize one of the IRRs. I took the time to register all of my routes, ASNs, etc. However, as I learned today, this was probably done in vain. Too many people won't spend the extra 30-seconds to verify the information listed there or in ARINs WHOIS.
I don't care what a customer tells me, too many times I've found they aren't 100% honest either for malicious/fraudulent reasons or they are unknowing. So, for our networks or the networks we manage, we want to verify what a customer is saying to prevent what happened to us today.
I'd like to get a conversation going and possibly some support of an initiative to spend that extra 30-seconds to verify ownership and authorization of network space to be advertised. Additionally, if someone rings your NOC's line an industry-standard process of verifying "ownership" and immediately responding by filtering out announcements. There's no sense in allowing a service provider to be impaired because a spammer doesn't want to give up clean IP space. Do you protect a bad customer or the Internet as a whole? I pick the Internet as a whole.
How can we prevent anyone else from ever enduring this again? While we may never stop it from ever happening, spammers (that's what we got hit by today) are a dime a dozen and will do everything possible to hit an Inbox, so how can we establish a protocol to immediate mitigate the effects of an traffic-stopping advertisement?
I thought registering with IRRs and up-to-date information in ARINs WHOIS was sufficient, apparently I was wrong. Not everyone respects them, but then again, they aren't very well managed (I've got several networks with antiquated information I've been unable to remove, it doesn't impair us normally, but its still there).
What can we do? Better yet, how do we as a whole respond when we encounter upstream providers who refuse to look at the facts and allow another to stay down?
kw
-- Kelvin Williams Sr. Service Delivery Engineer Broadband & Carrier Services Altus Communications Group, Inc.
"If you only have a hammer, you tend to see every problem as a nail." -- Abraham Maslow
Thanks for the reminder, Richard. Yes, as I announced earlier (see http://mailman.nanog.org/pipermail/nanog/2012-January/044493.html - the message with the corrected date), there is an interim sidr meeting on Thu *9* Feb in San Diego. Registration is free. Registration is easy (email). Registration is open to all. Registration is open ended. Registration is encouraged (so we know how many to expect). As the message says, see the sidr wiki http://trac.tools.ietf.org/wg/sidr/trac/wiki/InterimMeeting20120209 for agenda and a list of attendees. --Sandy Murphy ________________________________________ From: Richard Barnes [richard.barnes@gmail.com] Sent: Friday, February 03, 2012 8:09 AM To: Arturo Servin Cc: nanog Subject: Re: Thanks & Let's Prevent this in the Future. In related news, the IETF working group that is writing standards for the RPKI is having an interim meeting in San Diego just after NANOG. They deliberately chose that place/time to make it easy for NANOG attendees to contribute, so comments from this community are definitely welcome. <http://www.ietf.org/mail-archive/web/sidr/current/msg03923.html> <http://trac.tools.ietf.org/wg/sidr/trac/wiki/InterimMeeting20120209> <snip>
participants (8)
-
Arturo Servin -
George Bonser -
Jon Lewis -
Kelvin Williams -
Leigh Porter -
Mark Tinka -
Murphy, Sandra -
Richard Barnes