- Test - lists.nanog.org

Tiered operations support
by harbor235 23 Aug '19

23 Aug '19

Hi noggers, First some background; I have inherited a real-time services delivery infrastructure that while technically functional is absent of a wworking tiered operational support structure. In addition, the infrastructure was not implemented with technology best practices and has some remaining single points of failure. T3/engineering handles engineering, operational support, testing, and development. T1 and T2 perform some tasks but not what is expected of a traditional support structure, As we are attempting to scale our program our limitations are obvious. We are trying to scale the program and need to remedy all single points of failure and implement technical and management best practices. While we try to remedy all of the above, our customer expects routine capability enhancements. Question(s) 1) how to pivot to a tiered operational support structure and set expectations for each tier level. How do I do that without having my entire staff leave? Current staff is not a professional organization and are used to a purely reactive state. 2) ITIL process are great and we have started to implement what makes sense but I need a operational support stucture/model to support and manage this effort. 3) How do you manage your engineering and operational projects? Currently I am using Excel for all projects short and long term and also use a seperate spreadsheet for my short term project sprints? Basic project tracking with no mature processes involve. How are you pulling this all together? I would like to hire key positions that can bring essential capabilities to my project but I am limited hiring new staff. thoughts, suggestions Mike

1 0

Fwd: [safnog] SAFNOG-5 Is Ready For You
by Mark Tinka 23 Aug '19

23 Aug '19

FYI - SAFNOG-5 in Johannesburg, next week. Mark. -------- Forwarded Message -------- Subject: [safnog] SAFNOG-5 Is Ready For You Date: Thu, 22 Aug 2019 13:30:12 +0000 From: Portia Rabonda <portia.rabonda(a)safnog.net> To: safnog(a)safnog.org <safnog(a)safnog.org> Hello All, With just 4 days to go to SAFNOG-5 in Johannesburg, South Africa, we are excited to have you join us for what is going to be a very informative, educational and fun agenda. Highlights from the program include: * 3 keynotes: o Sunil Tagare (OpenCables), talking about the impact of the submarine cables - on Africa - that Facebook and Google have announced, and what it could mean for the continent. o Byron Clatterbuck (SEACOM), also speaking about submarine cable realities in Africa, how to make them sustainable and help effect change in the region. o Jerome Fleury (Cloudflare), who will be going into detail about the outage his organization faced on June 24th of this year, due to a massive BGP leak. * The first carrier-neutral data centre that will go live in Kampala, Uganda, at the end of 2019. * A panel on the relevance (or lack of thereof) of generalized speed tests in 2019 as minimum bandwidth increases for consumers, enterprises and service providers. * South Africa's first FTTH service provider that is delivering native IPv6 to home users. * Numerous tutorials/workshops that will provide training on: o IPv6 deployment. o IRR, RPKI and ROV for a secure, global BGP. o Critical infrastructure vis a vis what the Internet should be, and what it actually is. There will also be a number of social events taking place, which will connect all participants to further network with one another and create opportunities on helping further grow the Internet within the region, and the rest of the world. To register for the event, and for more details on the agenda, please visit: http://www.safnog.org/ We look forward to seeing you in Johannesburg. Regards, SAFNOG-5 Programme Committee

1 0

FCC Takes Action Against WISPs That Interfered with FCC Weather Radar
by Sean Donelan 23 Aug '19

23 Aug '19

I haven't been paying attention to the WISP market, so I'm not up to speed on these issues. https://www.fcc.gov/document/fcc-proposes-fines-against-wisps-and-issues-wa… The Federal Communications Commission’s Enforcement Bureau today announced proposed fines and issued a formal industry warning related to devices that apparently caused interference to the Federal Aviation Administration’s terminal doppler weather radar station in San Juan, Puerto Rico. The Enforcement Bureau proposed three separate $25,000 fines against wireless Internet service providers Boom Solutions, Integra Wireless, and WinPR. [...] In addition to the proposed fines, the Bureau’s Enforcement Advisory warned operators, manufacturers, and marketers of Unlicensed National Information Infrastructure devices that these devices must be certified under FCC rules. Such devices that operate in the 5.25 GHz to 5.35 GHz and 5.47 GHz to 5.725 GHz bands risk interfering with radar systems if not properly configured to share the spectrum [...]

5 8

Re: syn flood attacks from NL-based netblocks
by Jakob Heitz (jheitz) 20 Aug '19

20 Aug '19

The source address in the SYN is spoofed. What if the real owner of the source address wanted to connect to you? Then your penaltybox would block him. An attacker could now use your penaltybox to cause a DoS to the real owner of the IP address. > Date: Sun, 18 Aug 2019 08:48:08 -0700 > From: Mike <mike-nanog(a)tiedyenetworks.com> > > My idea is to maintain a penaltybox for any client IP that initiated a > connection but did not complete, while also maintaining a whitelist of > 'frequent fliers' who have previously completed their connections > successful. The penalty could simply be to drop traffic sourced from > those client ips that do not complete the handshake, for some > configurable timeout period. The whitelisting feature could give a pass > to good clients and allow these to bypass the penalty filtering, for a > longer timeout period (but of course, passing it along so other ACL's > can take effect). I'd say, perhaps, a 5 minute timeout would be > sufficient for a penalty, while 1 day or longer would be sufficient for > whitelisting. It would depend on your traffic of course, and definitely > you would want something efficient such as linux ipset as opposed to > individual iptables rules. > > While looking around, I came across the SYNPROXY netfilter module.. it > appears to be very complete but missing the above functionality to avoid > responding to spoofed clients. I'm going to see about hacking up a proof > of concept. I'll post here if I come up with something to play with. > > Mike-

1 0

FCC: December 27, 2018 CenturyLink Network Outage Report
by Sean Donelan 20 Aug '19

20 Aug '19

The FCC announced the release of a report detailing the cause and impact of a nationwide CenturyLink network outage that occurred last December, along with recommendations to help prevent similar outages from occurring in the future https://www.fcc.gov/document/fcc-issues-report-centurylink-network-outage December 27, 2018 CenturyLink Network Outage Report In the early morning of December 27, 2018, CenturyLink experienced a nationwide outage on its fiber network that lasted for almost 37 hours. This outage was caused by an equipment failure catastrophically exacerbated by a network configuration error. It affected communications service providers, business customers, and consumers who directly or indirectly relied upon CenturyLink’s transport services, which route communications traffic from various providers to locations across the country, resulting in extensive disruptions to phone service, including 911 calling. The effects included dropped calls, disconnected 911 call centers (known as “Public Safety Answering Points”), and fast-busy signals for people who called 911. As many as 22 million customers across 39 states were affected by the outage, including approximately 17 million customers across 29 states who lacked reliable access to 911. Indeed, at least 886 calls to 911 were not delivered.1 Fortunately, based on discussions with affected service providers and public safety officials from affected states, as well as a review of media reports, the Public Safety and Homeland Security Bureau (Bureau) is not aware of any harm to life or property resulting from the outage. [...]

1 0

SMF Tie Cable Standards for Data Center Applications
by Chris Costa 19 Aug '19

19 Aug '19

In our new data center builds we're transitioning from MMF to SMF for the tie cabling between networking gear in the MDF/IDF racks to the server racks. Today those interconnects are short (under 100 meters) 10GE and 40GE-LX4 over MMF. We’re transitioning to SMF to support services beyond that, such as 100GBASE-LR4. Would G.652.D be the right specification for the infrastructure tie cabling? Are there other considerations around this? Thank you

1 0

Re: syn flood attacks from NL-based netblocks
by Töma Gavrichenkov 19 Aug '19

19 Aug '19

On Mon, Aug 19, 2019, 9:24 PM Florian Brandstetter <florianb(a)squareflow.net> wrote: > Load balancing is done on Layer 4 or Layer 3 when routing, so your > ingress connection will have the same hash as the outgoing connection > (unless the source port of the connection changes on the ACK - which it > really should not). > If the hash is symmetric, yes. No wonder topology issues would have more impact, my point was that there are also other things to look at here. -- Töma >

1 0

syn flood attacks from NL-based netblocks
by Jim Shankland 19 Aug '19

19 Aug '19

Greetings, I'm seeing slow-motion (a few per second, per IP/port pair) syn flood attacks ostensibly originating from 3 NL-based IP blocks: 88.208.0.0/18 , 5.11.80.0/21, and 78.140.128.0/18 ("ostensibly" because ... syn flood, and BCP 38 not yet fully adopted). Why is this syn flood different from all other syn floods? Well ... 1. Rate seems too slow to do any actual damage (is anybody really bothered by a few bad SYN packets per second per service, at this point?); but 2. IPs/port combinations with actual open services are being targeted (I'm seeing ports 22, 443, and 53, just at a glance, to specific IPs with those services running), implying somebody checked for open services first; 3. I'm seeing this in at least 2 locations, to addresses in different, completely unrelated ASes, implying it may be pretty widespread. Is anybody else seeing the same thing? Any thoughts on what's going on? Or should I just be ignoring this and getting on with the weekend? Jim

12 25

Centurylink Looking Glass fail
by Ca By 19 Aug '19

19 Aug '19

Paging someone at Centurylink to fix your looking glass. https://lookingglass.centurylink.com None of the functions in any of the cities work Your network is kind of a big deal, so please try to provide visibility to your routing state so the rest of us can do our day-job, on Sunday

2 1

new BGP hijack & visibility tool “BGPalerter”
by Job Snijders 16 Aug '19

16 Aug '19

Dear NANOG, Recently NTT investigated how to best monitor the visibility of our own and our subsidiaries’ IP resources in the BGP Default-Free Zone. We were specifically looking how to get near real-time alerts funneled into an actionable pipeline for our NOC & Operations department when BGP hijacks happen. Previously we relied on a commercial “BGP Monitoring as a Service” offering, but with the advent of RIPE NCC’s “RIS Live” streaming API [1] we saw greater potential for a self-hosted approach designed specifically for custom integrations with various business processes. We decided to write our own tool “BGPalerter” and share the source code with the Internet community. BGPalerter allows operators to specify in great detail how to distribute meaningful information from the firehose from various BGP data sources (we call them “connectors”), through data processors (called “monitors”), finally outputted through “reports” into whatever mechanism is appropriate (Slack, IRC, email, or a call to your ticketing system’s API). The source code is available on Github, under a liberal open source license to foster community collaboration: https://github.com/nttgin/BGPalerter If you wish to contribute to the project, please use Github’s “issues” or “pull request” features. Any help is welcome! We’d love suggestions for new features, updates to the documentation, help with setting up a CI regression testing pipeline, or packaging for common platforms. Kind regards, Job & Massimo NTT Ltd [1]: https://ris-live.ripe.net/

8 10