I've got a problem where AS20115 continues to announce prefixes after BGP neighbors were shutdown. They claim it's a wedged BGP process but aren't in any hurry to fix it outside of a maintenance window. I'm at a loss of what else I can do. They admit the problem but won't take action saying it needs to wait for a maintenance window. Am I out of line insisting that's an unacceptable response to a problem that results in prefix/traffic hijacking? ~Seth
On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us> wrote:
I've got a problem where AS20115 continues to announce prefixes after BGP neighbors were shutdown. They claim it's a wedged BGP process but aren't in any hurry to fix it outside of a maintenance window.
If they weren't lying to you, they'd fix it now. That's not the kind of problem that waits. Thing is: they lied to you. Long ago they "helpfully" programmed their router to announce your route regardless of whether you sent a route to them. They want to wait for a maintenance window to remove that configuration.
I'm at a loss of what else I can do. They admit the problem but won't take action saying it needs to wait for a maintenance window. Am I out of line insisting that's an unacceptable response to a problem that results in prefix/traffic hijacking?
Try dropping the link entirely. If they still announce your addresses, bring it back up but report it as emergency down, escalate, and call back every 10 minutes until the junior tech understands that it's time to call and wake up the guy who makes the decision to fix it now. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
On 9/28/15 18:30, William Herrin wrote:
On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us> wrote:
I've got a problem where AS20115 continues to announce prefixes after BGP neighbors were shutdown. They claim it's a wedged BGP process but aren't in any hurry to fix it outside of a maintenance window.
If they weren't lying to you, they'd fix it now. That's not the kind of problem that waits.
Thing is: they lied to you. Long ago they "helpfully" programmed their router to announce your route regardless of whether you sent a route to them. They want to wait for a maintenance window to remove that configuration.
I'm at a loss of what else I can do. They admit the problem but won't take action saying it needs to wait for a maintenance window. Am I out of line insisting that's an unacceptable response to a problem that results in prefix/traffic hijacking?
Try dropping the link entirely. If they still announce your addresses, bring it back up but report it as emergency down, escalate, and call back every 10 minutes until the junior tech understands that it's time to call and wake up the guy who makes the decision to fix it now.
I'm at the tail end here almost 8 hours later since the hijacking started. Their NOC is just blowing me off now and they're happy to continue the hijacking until it's convenient for them to have a maintenance window. And that's apparently the final decision. ~Seth
Start announcing their prefixes? Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Sep 28, 2015 11:09 PM, "Seth Mattinen" <sethm@rollernet.us> wrote:
On 9/28/15 18:30, William Herrin wrote:
On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us> wrote:
I've got a problem where AS20115 continues to announce prefixes after BGP neighbors were shutdown. They claim it's a wedged BGP process but aren't in any hurry to fix it outside of a maintenance window.
If they weren't lying to you, they'd fix it now. That's not the kind of problem that waits.
Thing is: they lied to you. Long ago they "helpfully" programmed their router to announce your route regardless of whether you sent a route to them. They want to wait for a maintenance window to remove that configuration.
I'm at a loss of what else I can do. They admit the problem but won't take
action saying it needs to wait for a maintenance window. Am I out of line insisting that's an unacceptable response to a problem that results in prefix/traffic hijacking?
Try dropping the link entirely. If they still announce your addresses, bring it back up but report it as emergency down, escalate, and call back every 10 minutes until the junior tech understands that it's time to call and wake up the guy who makes the decision to fix it now.
I'm at the tail end here almost 8 hours later since the hijacking started. Their NOC is just blowing me off now and they're happy to continue the hijacking until it's convenient for them to have a maintenance window. And that's apparently the final decision.
~Seth
That's something I would do. Announce announce and keep adding ports until I hit a 10 Gig port worth of traffic or saw it fixed. Be sure to put in a blackhole route for the prefixes. Try to pick blocks that are as geographically located to your peering routers as possible ...IE in Reno pick the blocks that seem to be near by - like Reno, Tahoe, Sacramento ..... when that batch of customers makes their phones ring all night someone will listen. Would be nice if our membership organization ARIN ( that we all pay to keep us somewhat organized) had an ability to do something for you.... I never looked into it...i don't know....maybe it does ? But, in the mean time I am pretty sure you can document this well and prove your announcements of theirs was due to the fact you couldn't get proper technical attention and needed to desperately before your customers cancel after 8 hours of this. Tomorrow call your lawyers and begin to sue that cable company (did I recognize that ASN as cable TV ? ) for damages this must be causing you in ill-will amongst your customer base. I wonder just how you prove the damage...some equation based on customer calls and complaints together with how many years you have been in business as well as the number of contracts that are coming up for renewal. etc etc. Now that would be interesting to see a formula for that if anyone has been through it. Thank You Bob Evans CTO
Start announcing their prefixes?
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Sep 28, 2015 11:09 PM, "Seth Mattinen" <sethm@rollernet.us> wrote:
On 9/28/15 18:30, William Herrin wrote:
On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us> wrote:
I've got a problem where AS20115 continues to announce prefixes after BGP neighbors were shutdown. They claim it's a wedged BGP process but aren't in any hurry to fix it outside of a maintenance window.
If they weren't lying to you, they'd fix it now. That's not the kind of problem that waits.
Thing is: they lied to you. Long ago they "helpfully" programmed their router to announce your route regardless of whether you sent a route to them. They want to wait for a maintenance window to remove that configuration.
I'm at a loss of what else I can do. They admit the problem but won't take
action saying it needs to wait for a maintenance window. Am I out of line insisting that's an unacceptable response to a problem that results in prefix/traffic hijacking?
Try dropping the link entirely. If they still announce your addresses, bring it back up but report it as emergency down, escalate, and call back every 10 minutes until the junior tech understands that it's time to call and wake up the guy who makes the decision to fix it now.
I'm at the tail end here almost 8 hours later since the hijacking started. Their NOC is just blowing me off now and they're happy to continue the hijacking until it's convenient for them to have a maintenance window. And that's apparently the final decision.
~Seth
On Mon, Sep 28, 2015 at 11:59 PM, Bob Evans <bob@fiberinternetcenter.com> wrote:
That's something I would do. Announce announce and keep adding ports until I hit a 10 Gig port worth of traffic or saw it fixed. Be sure to put in a blackhole route for the prefixes. Try to pick blocks that are as geographically located to your peering routers as possible ...IE in Reno pick the blocks that seem to be near by - like Reno, Tahoe, Sacramento ..... when that batch of customers makes their phones ring all night someone will listen.
that seems like a pretty poor strategy... guaranteed to get you into some hot water, I suspect. Keep in mind that the 'noc' at 20115 isn't the same thing as the customer-service-center. There's likely little to link the 2 things together there :(
Would be nice if our membership organization ARIN ( that we all pay to keep us somewhat organized) had an ability to do something for you.... I never looked into it...i don't know....maybe it does ?
arin does not guarantee 'routability' of netblocks assigned to your org.
But, in the mean time I am pretty sure you can document this well and prove your announcements of theirs was due to the fact you couldn't get proper technical attention and needed to desperately before your customers cancel after 8 hours of this. Tomorrow call your lawyers and begin to sue that cable company (did I recognize that ASN as cable TV ? ) for damages this must be causing you in ill-will amongst your customer base.
I wonder just how you prove the damage...some equation based on customer calls and complaints together with how many years you have been in business as well as the number of contracts that are coming up for renewal. etc etc. Now that would be interesting to see a formula for that if anyone has been through it.
you COULD find a charter person on-list...there are nine names on the attendees list for the upcoming meeting... I imagine peeringdb likely has folk listed... gosh it sure does: <https://www.peeringdb.com/private/participant_view.php?id=2144> what with their emails and everything.
Thank You Bob Evans CTO
Start announcing their prefixes?
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Sep 28, 2015 11:09 PM, "Seth Mattinen" <sethm@rollernet.us> wrote:
On 9/28/15 18:30, William Herrin wrote:
On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us> wrote:
I've got a problem where AS20115 continues to announce prefixes after BGP neighbors were shutdown. They claim it's a wedged BGP process but aren't in any hurry to fix it outside of a maintenance window.
If they weren't lying to you, they'd fix it now. That's not the kind of problem that waits.
Thing is: they lied to you. Long ago they "helpfully" programmed their router to announce your route regardless of whether you sent a route to them. They want to wait for a maintenance window to remove that configuration.
I'm at a loss of what else I can do. They admit the problem but won't take
action saying it needs to wait for a maintenance window. Am I out of line insisting that's an unacceptable response to a problem that results in prefix/traffic hijacking?
Try dropping the link entirely. If they still announce your addresses, bring it back up but report it as emergency down, escalate, and call back every 10 minutes until the junior tech understands that it's time to call and wake up the guy who makes the decision to fix it now.
I'm at the tail end here almost 8 hours later since the hijacking started. Their NOC is just blowing me off now and they're happy to continue the hijacking until it's convenient for them to have a maintenance window. And that's apparently the final decision.
~Seth
On Mon, Sep 28, 2015 at 11:59 PM, Bob Evans <bob@fiberinternetcenter.com> wrote:
That's something I would do. Announce announce and keep adding ports until I hit a 10 Gig port worth of traffic or saw it fixed. Be sure to put in a blackhole route for the prefixes. Try to pick blocks that are as geographically located to your peering routers as possible ...IE in Reno pick the blocks that seem to be near by - like Reno, Tahoe, Sacramento ..... when that batch of customers makes their phones ring all night someone will listen.
that seems like a pretty poor strategy... guaranteed to get you into some hot water, I suspect. Keep in mind that the 'noc' at 20115 isn't the same thing as the customer-service-center. There's likely little to link the 2 things together there :(
You are right - probably creates more problems than good.
Would be nice if our membership organization ARIN ( that we all pay to keep us somewhat organized) had an ability to do something for you.... I never looked into it...i don't know....maybe it does ?
arin does not guarantee 'routability' of netblocks assigned to your org.
Yep, I was pretty sure of that - but wouldn't it be nice if arin could have some communication line or at least try. Yes, never any guarantees really. bob
But, in the mean time I am pretty sure you can document this well and prove your announcements of theirs was due to the fact you couldn't get proper technical attention and needed to desperately before your customers cancel after 8 hours of this. Tomorrow call your lawyers and begin to sue that cable company (did I recognize that ASN as cable TV ? ) for damages this must be causing you in ill-will amongst your customer base.
I wonder just how you prove the damage...some equation based on customer calls and complaints together with how many years you have been in business as well as the number of contracts that are coming up for renewal. etc etc. Now that would be interesting to see a formula for that if anyone has been through it.
you COULD find a charter person on-list...there are nine names on the attendees list for the upcoming meeting... I imagine peeringdb likely has folk listed... gosh it sure does:
<https://www.peeringdb.com/private/participant_view.php?id=2144>
what with their emails and everything.
Thank You Bob Evans CTO
Start announcing their prefixes?
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Sep 28, 2015 11:09 PM, "Seth Mattinen" <sethm@rollernet.us> wrote:
On 9/28/15 18:30, William Herrin wrote:
On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us> wrote:
I've got a problem where AS20115 continues to announce prefixes after BGP neighbors were shutdown. They claim it's a wedged BGP process but aren't in any hurry to fix it outside of a maintenance window.
If they weren't lying to you, they'd fix it now. That's not the kind of problem that waits.
Thing is: they lied to you. Long ago they "helpfully" programmed their router to announce your route regardless of whether you sent a route to them. They want to wait for a maintenance window to remove that configuration.
I'm at a loss of what else I can do. They admit the problem but won't take
action saying it needs to wait for a maintenance window. Am I out of line insisting that's an unacceptable response to a problem that results in prefix/traffic hijacking?
Try dropping the link entirely. If they still announce your addresses, bring it back up but report it as emergency down, escalate, and call back every 10 minutes until the junior tech understands that it's time to call and wake up the guy who makes the decision to fix it now.
I'm at the tail end here almost 8 hours later since the hijacking started. Their NOC is just blowing me off now and they're happy to continue the hijacking until it's convenient for them to have a maintenance window. And that's apparently the final decision.
~Seth
On Tue, Sep 29, 2015 at 2:04 AM, Bob Evans <bob@fiberinternetcenter.com> wrote:
On Mon, Sep 28, 2015 at 11:59 PM, Bob Evans <bob@fiberinternetcenter.com> wrote:
That's something I would do. Announce announce and keep adding ports until I hit a 10 Gig port worth of traffic or saw it fixed. Be sure to put in a blackhole route for the prefixes. Try to pick blocks that are as geographically located to your peering routers as possible ...IE in Reno pick the blocks that seem to be near by - like Reno, Tahoe, Sacramento ..... when that batch of customers makes their phones ring all night someone will listen.
that seems like a pretty poor strategy... guaranteed to get you into some hot water, I suspect. Keep in mind that the 'noc' at 20115 isn't the same thing as the customer-service-center. There's likely little to link the 2 things together there :(
You are right - probably creates more problems than good.
Would be nice if our membership organization ARIN ( that we all pay to keep us somewhat organized) had an ability to do something for you.... I never looked into it...i don't know....maybe it does ?
arin does not guarantee 'routability' of netblocks assigned to your org.
Yep, I was pretty sure of that - but wouldn't it be nice if arin could have some communication line or at least try. Yes, never any guarantees really.
I'm fairly sure that the arin (or ripe or apnic or...) answer to your question is: "read the contact info in whois... call the stated numbers." pretty sure that's also not going to be super helpful, email the poc's in the peering-db.
bob
But, in the mean time I am pretty sure you can document this well and prove your announcements of theirs was due to the fact you couldn't get proper technical attention and needed to desperately before your customers cancel after 8 hours of this. Tomorrow call your lawyers and begin to sue that cable company (did I recognize that ASN as cable TV ? ) for damages this must be causing you in ill-will amongst your customer base.
I wonder just how you prove the damage...some equation based on customer calls and complaints together with how many years you have been in business as well as the number of contracts that are coming up for renewal. etc etc. Now that would be interesting to see a formula for that if anyone has been through it.
you COULD find a charter person on-list...there are nine names on the attendees list for the upcoming meeting... I imagine peeringdb likely has folk listed... gosh it sure does:
<https://www.peeringdb.com/private/participant_view.php?id=2144>
what with their emails and everything.
Thank You Bob Evans CTO
Start announcing their prefixes?
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Sep 28, 2015 11:09 PM, "Seth Mattinen" <sethm@rollernet.us> wrote:
On 9/28/15 18:30, William Herrin wrote:
On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us> wrote:
> I've got a problem where AS20115 continues to announce prefixes > after > BGP > neighbors were shutdown. They claim it's a wedged BGP process but > aren't > in > any hurry to fix it outside of a maintenance window. >
If they weren't lying to you, they'd fix it now. That's not the kind of problem that waits.
Thing is: they lied to you. Long ago they "helpfully" programmed their router to announce your route regardless of whether you sent a route to them. They want to wait for a maintenance window to remove that configuration.
I'm at a loss of what else I can do. They admit the problem but won't take > action saying it needs to wait for a maintenance window. Am I out of > line > insisting that's an unacceptable response to a problem that results > in > prefix/traffic hijacking? >
Try dropping the link entirely. If they still announce your addresses, bring it back up but report it as emergency down, escalate, and call back every 10 minutes until the junior tech understands that it's time to call and wake up the guy who makes the decision to fix it now.
I'm at the tail end here almost 8 hours later since the hijacking started. Their NOC is just blowing me off now and they're happy to continue the hijacking until it's convenient for them to have a maintenance window. And that's apparently the final decision.
~Seth
On Sep 28, 2015, at 11:59 PM, Bob Evans <bob@FiberInternetCenter.com> wrote:
Would be nice if our membership organization ARIN ( that we all pay to keep us somewhat organized) had an ability to do something for you.... I never looked into it...i don't know....maybe it does ?
No one else has said this, so… RPKI. Which ARIN does do. —Sandy P.S. The following has numerous points of weirdness. about 104.73.161.0/24, RADB says: route: 104.73.161.0/24 descr: Proxy for Akamai (AS20940) and Roller Networks (AS11170) origin: AS20115 mnt-by: MAINT-CHTR-WD changed: tim.weber@charter.com 20150312 #20:32:27Z source: RADB route: 104.73.161.0/24 descr: Akamai Technologies origin: AS20940 mnt-by: AKAM1-RIPE-MNT changed: unread@ripe.net 20000101 source: RIPE remarks: **************************** remarks: * THIS OBJECT IS MODIFIED remarks: * Please note that all data that is generally regarded as personal remarks: * data has been removed from this object. remarks: * To view the original object, please query the RIPE Database at: remarks: * http://www.ripe.net/whois remarks: **************************** route: 104.64.0.0/10 descr: Akamai origin: AS35994 mnt-by: AKAM1-ALTDB-MNT changed: ablock@akamai.com 20140518 source: ALTDB
Is this related to 104.73.161.0/24? That's ours. :-) We'll take a look and get back to you. Thanks for caring! Best, Marty
On Sep 28, 2015, at 23:08, Seth Mattinen <sethm@rollernet.us> wrote:
On 9/28/15 18:30, William Herrin wrote:
On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen <sethm@rollernet.us> wrote: I've got a problem where AS20115 continues to announce prefixes after BGP neighbors were shutdown. They claim it's a wedged BGP process but aren't in any hurry to fix it outside of a maintenance window.
If they weren't lying to you, they'd fix it now. That's not the kind of problem that waits.
Thing is: they lied to you. Long ago they "helpfully" programmed their router to announce your route regardless of whether you sent a route to them. They want to wait for a maintenance window to remove that configuration.
I'm at a loss of what else I can do. They admit the problem but won't take action saying it needs to wait for a maintenance window. Am I out of line insisting that's an unacceptable response to a problem that results in prefix/traffic hijacking?
Try dropping the link entirely. If they still announce your addresses, bring it back up but report it as emergency down, escalate, and call back every 10 minutes until the junior tech understands that it's time to call and wake up the guy who makes the decision to fix it now.
I'm at the tail end here almost 8 hours later since the hijacking started. Their NOC is just blowing me off now and they're happy to continue the hijacking until it's convenient for them to have a maintenance window. And that's apparently the final decision.
~Seth
On 9/28/15, 10:24 PM, "NANOG on behalf of Seth Mattinen" <nanog-bounces@nanog.org on behalf of sethm@rollernet.us> wrote:
On 9/28/15 20:19, Martin Hannigan wrote:
Is this related to 104.73.161.0/24? That's ours. :-)
We'll take a look and get back to you. Thanks for caring!
Yep, that's one of the affected prefixes.
~Seth Hi Seth, which market was this occurring? Was this already removed? I'm not seeing it this morning. I would like to figure out what went wrong here. We shouldn't be nailing up any static configuration to have caused a situation like this.
On 29/Sep/15 16:26, Rampley Jr, Jim F wrote:
Hi Seth, which market was this occurring? Was this already removed? I'm not seeing it this morning. I would like to figure out what went wrong here. We shouldn't be nailing up any static configuration to have caused a situation like this.
You'd be surprised how often this happens, especially on the back of a conference rocking into a city/country and the local provider having minimal BGP experience. Once the conference is done, folk leave, and the provider forgets about things - which is not a problem since the conference would have come with its own IP address space. The issue goes unnoticed for 12x months when the conference is trying to route their usual block in some other city/country, and things just seem "strange". Someone remembers the previous year's event, calls up the previous provider, and finds out that the tech. who worked the activation has since left. It's not easy... Many other situations closer to home (i.e., paying customers) where things like this happen, especially if the customer has IP address space but does not do BGP (until they want to or leave to the competition). Blackholing operations that go wrong that folk forget about as well, not to mention other networks that cut themselves off by using public IP address space for their enterprise network. It's not easy at all... Mark.
On 9/29/15 7:26 AM, Rampley Jr, Jim F wrote:
On 9/28/15, 10:24 PM, "NANOG on behalf of Seth Mattinen" <nanog-bounces@nanog.org on behalf of sethm@rollernet.us> wrote:
On 9/28/15 20:19, Martin Hannigan wrote:
Is this related to 104.73.161.0/24? That's ours. :-)
We'll take a look and get back to you. Thanks for caring!
Yep, that's one of the affected prefixes.
~Seth Hi Seth, which market was this occurring? Was this already removed? I'm not seeing it this morning. I would like to figure out what went wrong here. We shouldn't be nailing up any static configuration to have caused a situation like this.
Reno, NV. I do believe they've finally withdrawn this morning (I just woke up, it was a long night). ~Seth
On 9/29/15 7:26 AM, Rampley Jr, Jim F wrote:
On 9/28/15, 10:24 PM, "NANOG on behalf of Seth Mattinen" <nanog-bounces@nanog.org on behalf of sethm@rollernet.us> wrote:
On 9/28/15 20:19, Martin Hannigan wrote:
Is this related to 104.73.161.0/24? That's ours. :-)
We'll take a look and get back to you. Thanks for caring!
Yep, that's one of the affected prefixes.
~Seth Hi Seth, which market was this occurring? Was this already removed? I'm not seeing it this morning. I would like to figure out what went wrong here. We shouldn't be nailing up any static configuration to have caused a situation like this.
Reno, NV. I do believe they've finally withdrawn this morning (I just woke up, it was a long night).
~Seth This issue was caused by a hung BGP process which was resolved last night. Nothing nefarious. No static configuration nailed up, no BGP highjacking
On 9/29/15, 9:49 AM, "Seth Mattinen" <sethm@rollernet.us> wrote: purposely done. ;)
If this is anything like what I deal with the aging timer for the bgp session is set to 180s by default. After 2 years I've been unable to get the charter noc to enable bfd on my links to address this issue On Sep 29, 2015 10:59 AM, "Seth Mattinen" <sethm@rollernet.us> wrote:
On 9/29/15 8:18 AM, Rampley Jr, Jim F wrote:
This issue was caused by a hung BGP process which was resolved last night. Nothing nefarious. No static configuration nailed up, no BGP highjacking purposely done. ;)
Is there a Cisco bug ID?
~Seth
On Tue, Sep 29, 2015 at 1:29 PM, N M <digitallystoned@gmail.com> wrote:
If this is anything like what I deal with the aging timer for the bgp session is set to 180s by default. After 2 years I've been unable to get the charter noc to enable bfd on my links to address this issue
because bfd brings it's own special sort of pain...
Nice of you to check Jim. This brings up the old idea - A long time ago I had an INOC phone by PCH.NET - It never rang, as we filter our outbound with detail everywhere we announce. ISPs need to provide us their address list. And the few times I needed to use it , no one ever answered. ( It was a decade ago before NANOG membership.) So after a while I too ignored it. Maybe this was an idea ahead of it's time ? From this painful mishap, it could have been a great solution for NOC Engineers to help each. I find peeringdb often outdated as companies change around and sluggish return call if at all. Most are like a sales line number post. I see now a long list of registered networks in the PCH directory. Are networks actually paying attention and using it. Is it time to take another look ? At midnight in your organization could you get a NOC person with " proper BGP skills and access " to answer and care about a bad announcement ? https://inoc-dba-web.pch.net/inoc-dba/console.cgi?op=show_pubdir&list=org Link above shows lots more networks listed on the INOC-DBA Public Directory: Organizations But have you used it? Did it work for you when you needed it ? Any further comments are appreciated. This seems like a very good proper civil approach - maybe this or something like it ARIN might help promote and endorse as a benefit to the community ? Be nice if with the cash they did something simple like this and got all of us to use it? Special line forwarding ? A Emergency Only NOC App for our phones for just this kind of situation - one that registers a specific ASN and pin code we set on the registration page ? Thank You Bob Evans CTO
On 9/28/15, 10:24 PM, "NANOG on behalf of Seth Mattinen" <nanog-bounces@nanog.org on behalf of sethm@rollernet.us> wrote:
On 9/28/15 20:19, Martin Hannigan wrote:
Is this related to 104.73.161.0/24? That's ours. :-)
We'll take a look and get back to you. Thanks for caring!
Yep, that's one of the affected prefixes.
~Seth Hi Seth, which market was this occurring? Was this already removed? I'm not seeing it this morning. I would like to figure out what went wrong here. We shouldn't be nailing up any static configuration to have caused a situation like this.
Hi Bob, On Tue, Sep 29, 2015 at 08:05:45AM -0700, Bob Evans wrote:
This seems like a very good proper civil approach - maybe this or something like it ARIN might help promote and endorse as a benefit to the community ? Be nice if with the cash they did something simple like this and got all of us to use it? Special line forwarding ? A Emergency Only NOC App for our phones for just this kind of situation - one that registers a specific ASN and pin code we set on the registration page ?
In this day and age people use IRC or Facebook to quickly get to a friend of a friend of a friend to get to a good contact. Get on with the times :-) Kind regards, Job
A friend is not someone that allows their company to hijack your prefixes. A friend is one that can get it to stop. Dude - wake up and drink some coffee. Thank You Bob Evans CTO
Hi Bob,
On Tue, Sep 29, 2015 at 08:05:45AM -0700, Bob Evans wrote:
This seems like a very good proper civil approach - maybe this or something like it ARIN might help promote and endorse as a benefit to the community ? Be nice if with the cash they did something simple like this and got all of us to use it? Special line forwarding ? A Emergency Only NOC App for our phones for just this kind of situation - one that registers a specific ASN and pin code we set on the registration page ?
In this day and age people use IRC or Facebook to quickly get to a friend of a friend of a friend to get to a good contact. Get on with the times :-)
Kind regards,
Job
I have actually found this NANOG email to be more effective than a chat or mombook public service. We need something more private like that. Thank You Bob Evans CTO
A friend is not someone that allows their company to hijack your prefixes. A friend is one that can get it to stop. Dude - wake up and drink some coffee.
Thank You Bob Evans CTO
Hi Bob,
On Tue, Sep 29, 2015 at 08:05:45AM -0700, Bob Evans wrote:
This seems like a very good proper civil approach - maybe this or something like it ARIN might help promote and endorse as a benefit to the community ? Be nice if with the cash they did something simple like this and got all of us to use it? Special line forwarding ? A Emergency Only NOC App for our phones for just this kind of situation - one that registers a specific ASN and pin code we set on the registration page ?
In this day and age people use IRC or Facebook to quickly get to a friend of a friend of a friend to get to a good contact. Get on with the times :-)
Kind regards,
Job
I entirely disagree, Job. The idea of a private tieline network that is connected, by SIP, to a line appearance in the NOC of each AS, and no one else is on it, seems like a fine idea to me. And that was INOC-DBA's original goal, as I understand it: You're having a problem? It's coming from some specific AS? Pick up the phone, mash the red INOC line button, dial the AS number, and you're talking to their NOC. And that's *authenticated*: since it's low enough churn to set up by hand, it's authenticated by humans. Show of hands: who has it set up, correctly, right now? ----- Original Message -----
From: "Job Snijders" <job@instituut.net> To: "Bob Evans" <bob@FiberInternetCenter.com> Cc: nanog@nanog.org Sent: Tuesday, September 29, 2015 11:12:43 AM Subject: Re: PCH.net questions and thoughts - Re: Prefix hijacking by AS20115 Hi Bob,
On Tue, Sep 29, 2015 at 08:05:45AM -0700, Bob Evans wrote:
This seems like a very good proper civil approach - maybe this or something like it ARIN might help promote and endorse as a benefit to the community ? Be nice if with the cash they did something simple like this and got all of us to use it? Special line forwarding ? A Emergency Only NOC App for our phones for just this kind of situation - one that registers a specific ASN and pin code we set on the registration page ?
In this day and age people use IRC or Facebook to quickly get to a friend of a friend of a friend to get to a good contact. Get on with the times :-)
Kind regards,
Job
-- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
On 29/09/2015 16:19, Jay Ashworth wrote:
The idea of a private tieline network that is connected, by SIP, to a line appearance in the NOC of each AS, and no one else is on it, seems like a fine idea to me.
it's a great idea: I had my inoc-dba phone connected and live for 15 years. I used it exactly once. Nick
On Tue 2015-Sep-29 11:19:57 -0400, Jay Ashworth <jra@baylink.com> wrote: <some stuff, then>:
Show of hands: who has it set up, correctly, right now?
I had this in my to-do, and this thread poked me again to get on with it. Sadly, https://inoc-dba-web.pch.net/inoc-dba/console.cgi?op=new_account gives me: Account sign up is disabled. Please wait for the new system! :'( -- Hugo
* jra@baylink.com (Jay Ashworth) [Tue 29 Sep 2015, 17:31 CEST]:
The idea of a private tieline network that is connected, by SIP, to a line appearance in the NOC of each AS, and no one else is on it, seems like a fine idea to me.
Until you take into account that SIP doesn't work through many firewalls, that people generally don't give a second thought to timezones, that network engineers generally dislike having to mess with voice systems, etc. etc. 2 out of 3 INOC-DBA calls I ever received were silent on their end (presumably) due to firewalls; the third call was a test.
And that was INOC-DBA's original goal, as I understand it:
You're having a problem? It's coming from some specific AS?
Pick up the phone, mash the red INOC line button, dial the AS number, and you're talking to their NOC.
And that's *authenticated*: since it's low enough churn to set up by hand, it's authenticated by humans.
In other words, it wasn't secure, it wouldn't scale and churn killed it.
Show of hands: who has it set up, correctly, right now?
No. There is nothing I'd do after receiving a phone call that I wouldn't do via email anyway. -- Niels.
Neils, do you actually work at in a NOC operation with BGP operations and policies you can change - a backbone with customers? If not - I would understand why email is fast enough for you. Maybe SIP iNOC phone isn't the right answer - but it seems to work fine everywhere I go. There just has to be a better way of communicating other than posting an email to a board - which isn't focused on a live network emergency. Something that's self filtered by all of us for a specific use. Say....An email/ text might work well or even better than SIP - if we had an APP that noticed a specific key or coded line plus your ASN to then ring my phone with an urgent ring tone.....hence, the idea of an NOC APP for that. Something other than "No I won't do anything different" - an idea or concept something you would embrace for such a moment. The iNOC phone wasn't embraced. Maybe a APP is a better idea than a phone. Thank You Bob Evans CTO
* jra@baylink.com (Jay Ashworth) [Tue 29 Sep 2015, 17:31 CEST]:
The idea of a private tieline network that is connected, by SIP, to a line appearance in the NOC of each AS, and no one else is on it, seems like a fine idea to me.
Until you take into account that SIP doesn't work through many firewalls, that people generally don't give a second thought to timezones, that network engineers generally dislike having to mess with voice systems, etc. etc.
2 out of 3 INOC-DBA calls I ever received were silent on their end (presumably) due to firewalls; the third call was a test.
And that was INOC-DBA's original goal, as I understand it:
You're having a problem? It's coming from some specific AS?
Pick up the phone, mash the red INOC line button, dial the AS number, and you're talking to their NOC.
And that's *authenticated*: since it's low enough churn to set up by hand, it's authenticated by humans.
In other words, it wasn't secure, it wouldn't scale and churn killed it.
Show of hands: who has it set up, correctly, right now?
No. There is nothing I'd do after receiving a phone call that I wouldn't do via email anyway.
-- Niels.
On 29 September 2015 at 17:13, Bob Evans <bob@fiberinternetcenter.com> wrote:
Neils, do you actually work at in a NOC operation with BGP operations and policies you can change - a backbone with customers?
"lolz" as the kids say.
Say....An email/ text might work well or even better than SIP - if we had an APP that noticed a specific key or coded line plus your ASN to then ring my phone with an urgent ring tone.....hence, the idea of an NOC APP for that.
This isn't an iPhone developers conference, the answer is very rarely "there's an app for that". The chance of that being integrated with ISP phone systems is slim to none. Email works. When it doesn't IRC works. It has done for a decade, it will for the next decade. Yes, even when the 200 people post to Outages saying "XYZ is down for me, anyone else" or the far more annoying "can someone from XYZ contact me offlist" posts to NANOG. M
On 30/09/2015, at 6:19 AM, Matthew Walster <matthew@walster.org> wrote:
"lolz" as the kids say.
Current stats indicate it's actually only the old-timers that say lolz now days! ;) http://www.huffingtonpost.com/entry/facebook-study-laughter_55c8b148e4b0f1cb... Pete
On Tue, Sep 29, 2015 at 7:12 AM, Job Snijders <job@instituut.net> wrote:
Hi Bob,
On Tue, Sep 29, 2015 at 08:05:45AM -0700, Bob Evans wrote:
This seems like a very good proper civil approach - maybe this or something like it ARIN might help promote and endorse as a benefit to the community ? Be nice if with the cash they did something simple like this and got all of us to use it? Special line forwarding ? A Emergency Only NOC App for our phones for just this kind of situation - one that registers a specific ASN and pin code we set on the registration page ?
In this day and age people use IRC or Facebook to quickly get to a friend of a friend of a friend to get to a good contact. Get on with the times :-)
This seems lossy and unscriptable to me. There are maxint different flavors of $social, so it's not suitable for escalation, IMO. Also, many people opt out of half of them when they're not on the clock. And, many of them have "I don't know you so I'll bury your message" options, which makes being tickled by a stranger for emergency purposes hard. And their "APIs", so to speak, are constantly shifting. But we already have a reliable, widespread, high-SNR channel: this list. It's the place that people go when they can't get an answer any other way. Email works when many other things are broken. What if all NOCs used their NOC email distro/alias to subscribe, filter for posts containing their own ASes/admin-domains/prefixes, plus the string "problem|issue|etc", and flag them as higher priority. A junior NOCling could check it manually every couple of hours, and maybe a public web archive of the list, in case of filter failures. I would expect most NOCs worth their salt to be monitoring nanog anyway. Why not leverage it? A sibling list could be spun off -- nanog-panic-button? ;) -- if that would be preferable. Royce
Well, there *is* outages@outages.org... :-) ----- Original Message -----
From: "Royce Williams" <royce@techsolvency.com> To: nanog@nanog.org Sent: Tuesday, September 29, 2015 11:31:54 AM Subject: Re: PCH.net questions and thoughts - Re: Prefix hijacking by AS20115 On Tue, Sep 29, 2015 at 7:12 AM, Job Snijders <job@instituut.net> wrote:
Hi Bob,
On Tue, Sep 29, 2015 at 08:05:45AM -0700, Bob Evans wrote:
This seems like a very good proper civil approach - maybe this or something like it ARIN might help promote and endorse as a benefit to the community ? Be nice if with the cash they did something simple like this and got all of us to use it? Special line forwarding ? A Emergency Only NOC App for our phones for just this kind of situation - one that registers a specific ASN and pin code we set on the registration page ?
In this day and age people use IRC or Facebook to quickly get to a friend of a friend of a friend to get to a good contact. Get on with the times :-)
This seems lossy and unscriptable to me. There are maxint different flavors of $social, so it's not suitable for escalation, IMO. Also, many people opt out of half of them when they're not on the clock. And, many of them have "I don't know you so I'll bury your message" options, which makes being tickled by a stranger for emergency purposes hard. And their "APIs", so to speak, are constantly shifting.
But we already have a reliable, widespread, high-SNR channel: this list. It's the place that people go when they can't get an answer any other way. Email works when many other things are broken.
What if all NOCs used their NOC email distro/alias to subscribe, filter for posts containing their own ASes/admin-domains/prefixes, plus the string "problem|issue|etc", and flag them as higher priority. A junior NOCling could check it manually every couple of hours, and maybe a public web archive of the list, in case of filter failures.
I would expect most NOCs worth their salt to be monitoring nanog anyway. Why not leverage it?
A sibling list could be spun off -- nanog-panic-button? ;) -- if that would be preferable.
Royce
-- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
We have a big, red rotary phone that sits in our NOC that we have attached to a VoIP box just to use for that. :) On 9/29/2015 10:05 AM, Bob Evans wrote:
Nice of you to check Jim. This brings up the old idea - A long time ago I had an INOC phone by PCH.NET - It never rang, as we filter our outbound with detail everywhere we announce. ISPs need to provide us their address list.
And the few times I needed to use it , no one ever answered. ( It was a decade ago before NANOG membership.) So after a while I too ignored it. Maybe this was an idea ahead of it's time ? From this painful mishap, it could have been a great solution for NOC Engineers to help each. I find peeringdb often outdated as companies change around and sluggish return call if at all. Most are like a sales line number post.
I see now a long list of registered networks in the PCH directory. Are networks actually paying attention and using it. Is it time to take another look ? At midnight in your organization could you get a NOC person with " proper BGP skills and access " to answer and care about a bad announcement ?
https://inoc-dba-web.pch.net/inoc-dba/console.cgi?op=show_pubdir&list=org Link above shows lots more networks listed on the INOC-DBA Public Directory: Organizations
But have you used it? Did it work for you when you needed it ? Any further comments are appreciated.
This seems like a very good proper civil approach - maybe this or something like it ARIN might help promote and endorse as a benefit to the community ? Be nice if with the cash they did something simple like this and got all of us to use it? Special line forwarding ? A Emergency Only NOC App for our phones for just this kind of situation - one that registers a specific ASN and pin code we set on the registration page ?
Thank You Bob Evans CTO
On 9/28/15, 10:24 PM, "NANOG on behalf of Seth Mattinen" <nanog-bounces@nanog.org on behalf of sethm@rollernet.us> wrote:
On 9/28/15 20:19, Martin Hannigan wrote:
Is this related to 104.73.161.0/24? That's ours. :-)
We'll take a look and get back to you. Thanks for caring!
Yep, that's one of the affected prefixes.
~Seth Hi Seth, which market was this occurring? Was this already removed? I'm not seeing it this morning. I would like to figure out what went wrong here. We shouldn't be nailing up any static configuration to have caused a situation like this.
-- ================================================================ Aaron Wendel Chief Technical Officer Wholesale Internet, Inc. (AS 32097) (816)550-9030 http://www.wholesaleinternet.com ================================================================
Since it’s come up on the list and we haven’t given a public update recently, I thought I’d just write a quick note on the state of INOC-DBA. For those who aren’t familiar with it, INOC-DBA is a SIP-based hotline communications system between NOCs and CERTs: https://www.pch.net/services/INOC_DBA https://en.wikipedia.org/wiki/INOC-DBA PCH has been the secretariat for INOC-DBA for the past thirteen years as a function of our not-for-profit purpose, serving network operators. During that time, the INOC-DBA back-end and self-provisioning systems have been completely replaced three times, and we’re currently at work on moving from the SER-driven 3.0 series of releases to a more modern BE7k-driven 4.0 system. Because INOC-DBA has only been intermittently directly grant-funded, sometimes, like now, it is funded entirely out of our overhead budget, so progress can be slow. The consequence is that, in order to make headway on the 4.0 transition, we’ve had to move people off of active support of the old 3.0 self-provisioning system. So, it’s fine for people who are already using it, but there’s not currently a way to create a new user within the 3.0 system, nor for existing users to make significant changes to call routing. ASNs have proven to be a good identifier, allowing network operators to communicate with each other in a way that’s vetted, while avoiding putting PCH in the position of judging who qualifies to join and who doesn't. Whether you know the name of a network, or where it’s located, or even what timezone they’re in, you know them by their ASN. And a hotline system that bypasses directories and receptionists and escalation chains is a quick and low-friction way of reaching someone who has the authority and access to resolve a problem. While email is the most venerable and well-known communication method it is often filtered, missed, or funneled through helpdesks that don’t have sufficient clue, or are stymied by dealing with someone who isn’t one of their own customers. Facebook and general-purpose chat systems are less than ideal as well, as they’re un-vetted and quickly suffer the same fate as email: if they’re paid attention to at all, filters or automated systems are put in place to block the noise. So, a closed network for voice, video, presence and chat has proven to be an immediate, low-noise way for those network operators who choose to use it, to communicate with each other. In the 4.0 system, XMPP chat using the same identifiers in the same closed network is a natural extension and the new feature that, though hardly revolutionary, we’re most looking forward to releasing. The technical issues that were discussed in this thread about NAT/PAT problems are certainly valid, but can be circumvented in a number of different ways, some of which are addressed in our documentation. SIP and RTP can work through NAT if correctly configured in simple circumstances, or in the presence of a NAT-traversal server, such as is included in INOC-DBA. An organization may have multiple INOC-DBA users and opt to have a SIP-capable system at the border of their network with one side facing the public Internet, and one side facing their private network, and which manages call flow and media handling (Asterisk, Freeswitch, or any one of a number of free or commercial SIP PBX-like systems will do this fairly easily; again, there are tutorials in our documentation). This also allows after-hours routing to PSTN lines or to call groups as needed, controlled by a local administrator. We also have considered keeping the media path through our servers, which aids the NAT traversal issue while not precluding local SIP enclaves as described above. One of the things that we struggle with is maintaining an appropriate balance between, on the one hand, keeping the network operations community informed of the status of the system, so they don’t feel compelled to ask on NANOG, versus not pro-actively over-sharing on lists and making a nuisance of ourselves. Admittedly, if the 4.0 transition were going faster, this would be less of an issue. So, we’re glad of the continued interest (particularly in the NANOG community, where INOC-DBA is not as widely used as in, for instance, the LACNIC community), and we apologize for the slow transition to the new 4.0 back-end and self-provisioning system. As always, you can contact us directly about INOC-DBA related stuff on operator@pch.net JT --- John Todd - jtodd@pch.net - +1-415-831-3123 On 29 Sep 2015, at 8:05, Bob Evans wrote:
Nice of you to check Jim. This brings up the old idea - A long time ago I had an INOC phone by PCH.NET - It never rang, as we filter our outbound with detail everywhere we announce. ISPs need to provide us their address list.
And the few times I needed to use it , no one ever answered. ( It was a decade ago before NANOG membership.) So after a while I too ignored it. Maybe this was an idea ahead of it's time ? From this painful mishap, it could have been a great solution for NOC Engineers to help each. I find peeringdb often outdated as companies change around and sluggish return call if at all. Most are like a sales line number post.
I see now a long list of registered networks in the PCH directory. Are networks actually paying attention and using it. Is it time to take another look ? At midnight in your organization could you get a NOC person with " proper BGP skills and access " to answer and care about a bad announcement ?
https://inoc-dba-web.pch.net/inoc-dba/console.cgi?op=show_pubdir&list=org Link above shows lots more networks listed on the INOC-DBA Public Directory: Organizations
But have you used it? Did it work for you when you needed it ? Any further comments are appreciated.
This seems like a very good proper civil approach - maybe this or something like it ARIN might help promote and endorse as a benefit to the community ? Be nice if with the cash they did something simple like this and got all of us to use it? Special line forwarding ? A Emergency Only NOC App for our phones for just this kind of situation - one that registers a specific ASN and pin code we set on the registration page ?
Thank You Bob Evans CTO
[snip]
On Mon, 28 Sep 2015, Seth Mattinen wrote:
I'm at the tail end here almost 8 hours later since the hijacking started. Their NOC is just blowing me off now and they're happy to continue the hijacking until it's convenient for them to have a maintenance window. And that's apparently the final decision.
Willful negligence. Will only be in your favor when it comes to collect damages. -Dan
participants (21)
-
Aaron
-
Bob Evans
-
Christopher Morrow
-
goemon@anime.net
-
Hugo Slabbert
-
Jay Ashworth
-
Job Snijders
-
John Todd
-
Josh Luthman
-
Mark Tinka
-
Martin Hannigan
-
Matthew Walster
-
N M
-
Nick Hilliard
-
Niels Bakker
-
Pete Mundy
-
Rampley Jr, Jim F
-
Royce Williams
-
Sandra Murphy
-
Seth Mattinen
-
William Herrin