- Test - lists.nanog.org

Questions for Level3 & Choopa about their Enabling of IP Hijacking (RPKI Invalid)
by lists＠geeksolutions.ca 11 Dec '20

11 Dec '20

Is there anyone on this list at Level3 or Choopa who can respond to why: 1. Level3/Centurylink/Lumen: How come RPKI invalid prefixes were allowed to be announced and considered valid? 2. Level3/Centurylink/Lumen: Is there any actual NOC hidden behind the numbers where someone can reasonably be of assistance? Why are your SOC, Support and DDoS team saying you can't blackhole a hijacked prefix because of "GOVERNMENT regulations"? 3. CHOOPA: Why does your Vultr brand allow anyone who can successfully insert an IRR record (eg, in RaDB) to be imported (even when RPKI is invalid) and permit hijacking? 4. CHOOPA: Why does your Network Team need 3 days to check an RPKI valid or invalid? It is not rocket science. Please feel free to contact me off the list with these answers. If you are interested in the long story of the whole ordeal of being hijacked for 3 days, it is below. IP hijacking ordeal we went through: We are extremely disappointed with Choopa/Vultr and Level3. On 2020-12-07 at exactly 23:48:10, we notified Choopa and Vultr via email, ticket and contact form that one of our IPv4 prefixes was being hijacked by one of their customers. Our prefixes are RPKI signed, and the ASN announcing the prefix was not in the RPKI sign. We continued to follow up on the request, and sent more requests in to Vultr/Choopa's system. When we phoned Choopa, we were told by the individual on the phone that they see the ticket, and they'll bump it up on the network engineering ticket list. Frustrated after three days of an ongoing hijack, and consistently no assistance from Vultr or it's parent Choopa, we reached out to the only Tier-1 Choopa has in London that was NOT filtering according to RPKI, Level3. We sent an email to the noc(a)level3carrier.com, which was listed as a point of contact on the ASN & PeeringDB pages. The email had no response. We then called the number (1-833-453-8353), and spoke to Technical Support, who transferred us the first time to the SOC, the second time to their DDoS department. The person who responded in the SOC said that they can filter it "very fast" if we validate ownership of the prefix (despite being RPKI signed). So we complied, we were told to email "abuse(a)centurylinkservices.net" with a message saying the prefix is being hijacked, the ASN of the hijacker and the direct upstream (Vultr/Choopa). We sent that email, he said we "should get" an automatic reply (none arrived, confirmed with mx it was delivered). We waited half an hour, which they agreed was a "reasonable time" to wait for it to be filtered. After half an hour, we followed up, and this time landed on the DDoS department (I have no clue how they thought this through). At the DDoS department, they said that they can't help, and I should "keep emailing" abuse(a)centurylinkservices.net. He offered a phone number for me to call as a "direct line to abuse", upon hanging up and dialing, I got to the generic prompts for *customer services*. They were no help either. Fast forward four hours, we have no point of contact at Level3 or Choopa, we have had no communication from either. We finally get a message from Choopa, reading: "Greetings, This ticket has been forwarded to our networking team so they can examine your situation, check the infrastructure configuration, and apply any relevant changes. Please allow for significant additional time while we review this ticket." "Significant additional time" to check an RPKI valid? That seems incredibly odd. Fast forward to another hour, we receive the next of Choopa messages, this time saying: "Thanks for the update on this. We have validated the removal request and have removed the prefix from our network. Please allow additional time for this to update to the providers.". We finally thought we were in the clear. Almost two hours go by, and the prefix is still not filtered what-so-ever. We follow up once more, and are told "We have removed the prefix and is not announced by Vultr anymore. Please allow 24 to 48 hours for the internet providers to update their routing database." Finally, an hour later, nlnog & other lg's are seeing Level3 no longer announcing the prefix. What we found out in the process is Vultr ignores RPKI invalid (despite having a table on their system which shows RPKI Invalid/Invalid ASN), as long as at one point in time an IRR record existed (or, is created). Once Vultr gets their hands on it, they make the IRR records at RaDB, and keep updating them even when they're not valid. With the run around of Level3, and Choopa/Vultr, they're practically inviting IP hijackers to play. Insanity!

1 0

Fwd: Weather Service faces Internet bandwidth shortage, proposes limiting key data
by Rich Kulawiec 11 Dec '20

11 Dec '20

----- Forwarded message from Dave Farber <farber(a)gmail.com> ----- > From: Dave Farber <farber(a)gmail.com> > Date: Thu, 10 Dec 2020 15:47:44 +0900 > Subject: [IP] Weather Service faces Internet bandwidth shortage, proposes limiting key data > > Weather Service faces Internet bandwidth shortage, proposes limiting key data > The National Weather Service is proposing to place limits on accessing its > life-saving weather data in a bid to fix Internet outages. > By Jason Samenow and Andrew Freedman > > https://www.washingtonpost.com/weather/2020/12/09/nws-data-limits-internet-… [snip] This seems like a problem that this group could solve rather rapidly with minimal incremental expense. It also seems like one that's very much worth solving. ---rsk

10 10

Woefully outdated Verizon & Lumen ARIN whois/SWIP data
by nanog＠jima.us 10 Dec '20

10 Dec '20

Hi all, I apologize for dredging up the topic on this forum, but Lumen/CenturyLink/Qwest/US West has been continually unable to help me despite multiple attempts over years (yes, years), and Verizon's response to my colleague was to blame a downstream consumer of the whois data (?), and to use the opportunity to try to sell value-added services as a prerequisite to escalating the matter (!?). Does anyone have a lead on someone within those organizations who understands how to delete IP space assignments? Thanks in advance! - Jima

1 1

95th billing and automation
by Mehmet Akcin 10 Dec '20

10 Dec '20

hi there, i have asked about this in the past. What is the best tool out there to do 95th percentile billing. I have decided to use observium and librenms as result of responses but there seems to be some kind of billing module issue with these tools (thy are basically the same code). What are other systems besides observium and librenms (and old fashion cacti) people are using these days with 95th billing and integration with a CRM like salesforce/zoho, etc. I appreciate the responses. Mehmet

4 3

OpenNMS, openstreetmap, geocoding APIs and SNMP
by Eric Kuhnke 10 Dec '20

10 Dec '20

Anyone that has used a recent version of OpenNMS has probably noticed that the default home page view now includes an openstreetmap based view of node/device status, by geographical location. Section 18.3 here: https://docs.opennms.org/opennms/releases/latest/guide-admin/guide-admin.ht… https://wiki.opennms.org/wiki/Geographical_Maps example image here: https://www.tecmint.com/wp-content/uploads/2019/05/opennms-default-admin-da… While customizing a newly setup opennms VM for a unique purpose, I began thinking that there's two possible ways to go about accurately placing nodes on such a map, on a very large scale, with scripting/automation in the provisioning and monitoring process. *Method 1 *(as opennms does now): As defined in RFC3418, put the standard street address of the node in the SNMP sysLocation field. Typically in a human readable text format that would be understood when posted to a geolocation API SaaS. Example being: 2200 6th Avenue, Seattle WA 98121 sysLocation OBJECT-TYPE SYNTAX DisplayString (SIZE (0..255)) MAX-ACCESS read-write STATUS current DESCRIPTION "The physical location of this node (e.g., 'telephone closet, 3rd floor'). If the location is unknown, the value is the zero-length string." ::= { system 6 } The above is dependent on a few things. First, every node needs to have a standard format street address entered into its sysLocation field. But what if you have a telecom tower site on a mountain or hillside? What if it's a node in a developing nation environment where standard third-party geolocation API providers (S.18.3 of the opennms documentation) can't resolve anything? What if it's a device that moves around, like a router on an ocean going vessel or aircraft? *Method 2 *(backwards compatible, possibly an improvement): Treat the 255 character sysLocation field as a rudimentary three column CSV file with pipe delimiters. Put the standard human readable description of the node location in the first column. Populate the second column with the latitude and longitude in WGS84 decimal degree format, such as: Suite 402 2200 6th Ave, Seattle WA 98121|47.616380|-122.341673 Where both columns have six decimal places of precision Column 2 is positive integer if north of the equator Column 3 is negative integer if west of Greenwich. Optional 4th column: Elevation in metres, MSL In the method 2 example described above, opennms can be customized to retrieve the lat/long from sysLocation and place nodes on the dashboard, rendering the underlying map from a self-hosted openstreetmap intranet server without needing to call an external geolocation API. Provided that the data provisioned into the CPE/node in the first place is accurate (this is absolutely a GIGO problem), node locations for difficult-to-geocode install locations should become much more accurate. *Some issues that may be encountered:* Very old or improperly implemented snmpd on some devices does not support a 255 character field length, sometimes not even 100 characters. Varies greatly by vendor implementation. Provisioning and configuration system for nodes need an automated way to enter those lat/long coordinates into columns 2 and 3 in a scripted, automated way. In an example of small 1/10GbE capable business-class ISP CPE demarc devices, *perhaps* as the result of a provisioning system's call to a geolocation API. But with some sort of optional step for configuring a node in a location that won't geocode - such as having a provisioning tech manually scroll to and click a location on a map in a browser. For ISPs with large numbers of CPE endpoint/stub nodes that don't presently have valid sysLocation, it will be a manually laborious one-time process to hunt down all extant CPEs and verify their street address vs customer records, google maps/bing maps/openstreetmap, and manually edit the sysLocation field. Environments that would have many nodes expected to fail third-party geolocation APIs, would probably want to customize some sort of click-on-map GUI for the persons doing the provisioning and hand-off of CPEs to field installation techs. Or possibly using GPS data acquired from an installation technician's mobile phone (standard iOS and Android browser APIs for requesting GPS data), as a custom data field in a ticketing system. Data from that field in the ticketing system could then be fed to another piece of software and pushed to a script to write the sysLocation. *Possibly outside scope*: It's possible to self host your own openstreetmap tile server. The vector data for the entire world is freely available to download. The tile rendering engine and all software needed to set it up is all open source. This can be implemented for ISP management/monitoring networks (NOC map dashboard functionality, etc) that for security reasons have no connection whatsoever to the global routing table. Nodes that move around because they're mounted on some sort of vehicle may be able to update and rewrite their own SNMP sysLocation field from GPS data. Would require custom scripting or being accompanied by a very small embedded Linux system with serial interface to a low-cost GPS chipset. That's a data acquisition system that already exists for some category of equipment such as two-way satellite terminals on ocean going vessels.

2 1

anyone from cisco.com DNS Team around?
by Thomas Mieslinger 09 Dec '20

09 Dec '20

Hi, I have trouble to activate my Cisco NCS5xxx Devices. Turns out that tools.cisco.com. resolves to either 173.37.145.8 (this works) or 72.163.4.38 (which was decommissioned earlier this year). By running dig A tools.cisco.com @alln01-ucs-dcz03n-gslb1-snip.cisco.com four times I can reproduce this. Cisco, please fix your DNS. Thanks Thomas

1 0

OARC 34 Workshop, February 4th & 5th, Registration and Call for Contributions now open
by Jake Zack 09 Dec '20

09 Dec '20

OARC 34 will be an online meeting on February 4th & 5th starting at 16:00 UTC. The Programme Committee is seeking contributions from the community. All DNS-related subjects and suggestions for discussion topics are welcome. Based on the feedback from the previous workshop, the DNS-OARC audience is interested to see more content related to DNS operations. Therefore we are particularly interested in submissions from DNS Operators about attack mitigation and any major (public or non-public) DNS outages that your organization might have faced over the last year or so, the steps taken to resolve/mitigate the immediate issue, and to prevent future such events and any lessons learned. As it is an online workshop, we'd like to encourage brevity; presentations should not be longer than 20 minutes (with additional time for questions). **Workshop Milestones:** * 9 Dec 2020 - Submissions open via Indico * 4 Jan 2021 - Deadline for submission (23:59 UTC) * 14 Jan 2021 - Initial Contribution list published * 21 Jan 2021 - Full agenda published * 28 Jan 2021 - Deadline for slideset submission and Rehearsal * 4 Feb 2021 - OARC 34 Workshop The Registration page and details for presentation submission are published at: <https://www.dns-oarc.net/oarc34> To allow the Programme Committee to make objective assessments of submissions, so as to ensure the quality of the workshop, submissions SHOULD include slides. Draft slides are acceptable on submission. Additional information for speakers of OARC 34: * your talk will be broadcast live and recorded for future reference * your presentation slides will be available for delegates and others to download and refer to, before, during and after the meeting * you will be expected to attend a rehearsal on January 28th. It would be very useful to have your slides (even if draft) ready for this. If you have questions or concerns you can contact the Programme Committee: https://www.dns-oarc.net/oarc/programme via <submissions(a)dns-oarc.net> Jacob Zack, for the DNS-OARC Programme Committee OARC depends on sponsorship to fund its workshops and associated social events. Please contact <sponsor(a)dns-oarc.net> if your organization is interested in becoming a sponsor. (Please note that OARC is run on a non-profit basis, and is not in a position to reimburse expenses or time for speakers at its meetings.)

1 0

Submit a talk for NANOG 81 Virtual — Deadline is Mon, 12/14!
by NANOG News 09 Dec '20

09 Dec '20

*Bring your research + ideas to life* *With a presentation at NANOG 81 Virtual, February 8-10, 2021* The NANOG Program Committee is accepting proposals for our 81st community-wide conference through Monday, December 14, and for all other future conferences on a rolling basis. We encourage anyone interested in sharing their research, ideas, and best practices with the greater NANOG community to submit a talk, tutorial, or panel for consideration. Learn More <https://www.nanog.org/meetings/nanog-81/nanog-81-call-presentations/> *Topics of interest for NANOG 81 Virtual* In addition to proposals focused on current or soon-to-be deployed technologies and industry innovation, the following topics are of particular interest for NANOG 81: - Automation - SDN - Traffic analysis - ISP Issues: sharing lessons learned / what to watch out for - Cloud-related topics: scale concerns, etc. - Routing - Best current practices - Career development skills - Tutorials: 101, deep dives, etc. Submit Now <https://www.nanog.org/meetings/submit-presentation/>

1 0

Spoofer Report for NANOG for Nov 2020
by CAIDA Spoofer Project 08 Dec '20

08 Dec '20

In response to feedback from operational security communities, CAIDA's source address validation measurement project (https://spoofer.caida.org) is automatically generating monthly reports of ASes originating prefixes in BGP for systems from which we received packets with a spoofed source address. We are publishing these reports to network and security operations lists in order to ensure this information reaches operational contacts in these ASes. This report summarises tests conducted within usa, can. Inferred improvements during Nov 2020: none inferred Source Address Validation issues inferred during Nov 2020: ASN Name First-Spoofed Last-Spoofed 209 CENTURYLINK-US-LEGACY-QWEST 2016-08-16 2020-11-28 6128 CABLE-NET-1 2016-09-03 2020-11-29 20412 CLARITY-TELECOM 2016-09-30 2020-11-30 6181 FUSE-NET 2016-10-10 2020-11-30 14971 PINETEL 2016-10-21 2020-11-28 32440 LONI 2016-11-03 2020-11-30 12083 WOW-INTERNET 2016-11-09 2020-11-28 30036 MEDIACOM-ENTERPRISE-BUSINESS 2016-11-16 2020-11-29 20473 AS-CHOOPA 2017-01-08 2020-11-24 25623 CISN2 2017-03-13 2020-11-17 63296 AWBROADBAND 2017-09-01 2020-11-24 393564 SPOKANE 2018-06-05 2020-11-17 33452 RW 2018-09-19 2020-11-27 20448 VPNTRANET-LLC 2018-09-20 2020-11-30 8047 GCI 2019-04-11 2020-11-27 7859 PAIR-NETWORKS 2020-04-03 2020-11-29 3491 BTN-ASN 2020-10-18 2020-11-11 13837 POINT5-MS 2020-11-05 2020-11-05 11814 DISTRIBUTEL-AS11814 2020-11-22 2020-11-22 Further information for these tests where we received spoofed packets is available at: https://spoofer.caida.org/recent_tests.php?country_include=usa,can&no_block… Source Address Validation issues inferred during Nov 2020 using open resolver tests: ASN Name Last-Detected 16524 METTEL 2020-11-30 15081 ASN-CROS-1 2020-11-30 19257 SUBRIGO 2020-11-30 395262 CENTRE-OF-EXCELLENCE-IN-NEXT-G 2020-11-30 11260 EASTLINK-HSI 2020-11-30 53274 3DB-WIRELESS 2020-11-27 397081 AUWIRELESS-01 2020-11-27 53338 ITCI 2020-11-27 47023 NOR-NET-COMM 2020-11-27 40651 MONSTERGOVSOL 2020-11-27 32233 PERSONA 2020-11-26 394232 TELKEL-MTL 2020-11-26 46375 AS-SONICTELECOM 2020-11-26 9009 M247 2020-11-26 46328 FOUR-U 2020-11-26 6539 GT-BELL 2020-11-26 16922 OUHSC-EDU 2020-11-25 29953 MORNCOMM 2020-11-25 396236 MIMOKC 2020-11-25 53456 WESTRUX 2020-11-24 11287 AS-MSEA 2020-11-24 4565 MEGAPATH2-US 2020-11-24 55175 CLOUDBURST 2020-11-24 395261 AS-HCCL-7 2020-11-24 25636 ONTL-2002 2020-11-24 30600 AS-CMN 2020-11-24 53850 GORILLASERVERS 2020-11-23 14403 JJI-DEFAULT-00 2020-11-14 577 BACOM 2020-11-11 Further information for these tests is available at: https://spoofer.caida.org/ornog.php?country_include=usa,can Please send any feedback or suggestions to spoofer-info(a)caida.org

1 0

CWC advises of a possible fibre fault between Antigua and Guadeloupe on the Xtera-Optima, ECFS.
by John Curran 07 Dec '20

07 Dec '20

FYI - https://www.facebook.com/CNC3Television/posts/10159065715522996 /John === From CNC3 Television, Trinidad and Tobago ***Broadband outage limits internet access across the region*** A major broadband outage has forced the suspension of online classes, limiting internet access and other mobile services across the country and further up the Caribbean region. Two of the country’s main service providers—FLOW and Digicel—have been severely impacted by a major ‘fibre fault” on an underground cable. Because of this, customers will experience challenges browsing the internet, loading emails, sending and receiving WhatsApp messages and making landline calls. People have reported difficulty using internet services from all corners of the country. There are also reports that some schools have been forced to suspend online classes because of a lack of internet access. A broken underwater fibre cable between Antigua and Guadeloupe is suspected to be the reason for the outage. A statement by Cable & Wireless Communications (CWC) stated: “CWC advises of a possible fibre fault between Antigua and Guadeloupe on the Xtera-Optima, ECFS.” Digicel customers in Barbados and St. Vincent are also affected by the outage. Sources at FLOW stated it could take some time before services are fully restored. Information from the company’s network operations centre estimate full repair will be completed by Thursday, because damage to the underwater cables appears to be “major”. They indicated that a temporary solution is currently being put in place. While internet will return, customers are likely expected to encounter unstable connections with dropouts until the situation is rectified. Digicel posted an advisory to its Facebook page a short while ago. “Valued customer, We are aware that some of you may be experiencing an outage with your internet services. Preliminary investigations indicate that the cause is a technical issue with our international upstream internet suppliers which is affecting multiple internet providers across T&T and the Caribbean.” ===

1 0