- NANOG - lists.nanog.org

Reporter's questions
by Dave Burstein 29 Apr '16

29 Apr '16

Folks I've long lurked here and thought to ask about the story below and related. The news here is that Hurricane Electric is opening Africa pops and likely bringing down prices significantly. I made some (semi-informed) guesses about the relation of backhaul/transit to ISP costs. Anyone with date, I'd welcome it on or off list. Also, pointers to other sources of attractive pricing in the Global South welcome. I've been to several international meetings where I hear that's a huge problem. Improvements welcome. I'm always looking for news about broadband and related policy. I've just been to a major 5G conference and have been learing a great deal. Quick takeaways: Field trials coming next year at both Verizon & AT&T. Test data so far is very positive. MU MIMO should be ready for market this decade, with 10x improvement in the right places. Millimeter wave is also looking good but few expect volume before 2020-2023. Very expensive because short range of high frequencies means many, many cell sites. Short range means that those with fiber in place for backhaul have an enormous advantage. That's mostly the incumbents. I believe this will be a huge problem for competition but DC hasn't caught on yet. If any of this isn't appropriate for the list, let me know. Dave Burstein http://fastnet.news/index.php/97-fnn/368-hurricane-s-incredible-backhaul-pr… Hurricane's Expected Incredible Backhaul Prices to Joburg & Nairobi could kickstart Africa's Internet <http://fastnet.news/index.php/97-fnn/368-hurricane-s-incredible-backhaul-pr…> [image: Hurricane Africa 250]Not quite as low as the $0.27/megabit price in Riga. Hurricane Electric, a major Internet backbone, is coming to Africa. Although they aren't announcing prices, I predict they will offer much less expensive backhaul/transit. Most of the continent suffers from brutally high, cartel-like prices for wholesale connections. If HE prices as I expect they will, that could easily reduce consumer Internet prices by 15% to 35%. In turn, that will allow tens of millions more to connect. In 2012 at the WCIT, a dozen Africans told me the high the cost of transit/backhaul was by far the most important international obstacle to bringing the Internet to more Africans. A friend of mine was quoted $70/megabit for a Gig-E in Lagos last year. The cost in most European or American cities would be between $0.50 and perhaps $4.00. The current high price of backhaul raises the cost of a robust broadband connection probably $10-20 over what it could be. Even a second-rate service with a low cap is probably $5 more expensive than necessary. There is a real cost involved carrying data 6,000 miles undersea but that explains only a small part of the 50-1 difference in the price between London and most of Africa. Most coastal countries are now served by more than one undersea cable. South Africa has five and Kenya three, Take a look at Steve Song's remarkable map. <https://manypossibilities.net/african-undersea-cables/>The Africans have been chomping at the bit to reduce the impact of the cartel, but the U.S. has been blocking that at the ITU. Hurricane offers a $2,700/month price for a 10 gigabit pipe in several dozen cities in Europe and North America, including Riga, Latvia and Rochester, Minnesota. That's on a five year contract. Three years would be somewhat higher. The Africa price will be higher but I believe will create a new market level. The cable middleman must be paid. A Gig-E can serve a thousand robust connections and easily five thousand with low caps. 10 gigs is enough for ten thousand good connections. HE keeps prices down by doing little advertising and less pr and run lean in all ways. They;ve been around since 1994, were early to IPv6 and other technical advances, and service many world-class clients. The ISP needs to carry the data from the Hurricane POP to their local networks. Fortunately, land-based fiber nets are growing rapidly. With 700M mobile subs in Africa, the market is booming. Afterfibre has a map <https://afterfibre.nsrc.org/> worth looking at. Africa will have more than 315M broadband users, the population of the United States, within the next 18 months. Ericsson expects a total of 500M new broadband users in Africa between 2015 & 2020. A bit further - All cost figures here other than actual price quotes are my somewhat informed estimates based on scattered details of actual ISP internal costs I've accumulated over the years. They aren't precise. Cost accounting in a high fixed cost/low marginal cost industry is highly subjective. That's especially true here because the decision how to allocate backhaul costs between voice and data varies. Also, expect a lag between the wholesale and retail price changes. In developed countries with typically robust regional fiber, a large carrier's network cost from the peering point to the local exchange was similar to transit costs. Both seem to typically equate to 3-5 hops worth of switches/routers. So if I double transit costs, which have a competitive market, I get a usable estimate of the marginal cost of the bandwidth delivered to the local exchange. If transit costs $0.27/megabit in the local market, I believe the marginal cost of bandwidth will be well under $1. That's confirmed off the record numerous times by the carriers but very rarely discussed in public. (Because bandwidth is such a small part of broadband costs, any policy discussion that focuses on how much bandwidth a consumer takes is unsound.) However, as I testified to the U.S. Broadband Commission, rural areas and small companies often have much higher costs. ISPs and telcos under ~ 20,000 customers cannot efficiently buy backhaul in many locations. (That's when I developed some rules of thumb between transit costs and the delivered cost of broadband,) I suspect the effect is even more extreme in African rural areas or where customers are relatively few. I'd welcome any data people might have. -- Editor, Fast Net News, Net Policy News and DSL Prime Author with Jennie Bourne DSL (Wiley) and Web Video: Making It Great, Getting It Noticed (Peachpit)

1 0

Re: Arista Routing Solutions
by Ryan Woolley 29 Apr '16

29 Apr '16

Colton Conor wrote: > I know Arista is typically a switch manufacturer, but with their recently > announced Arista 7500R Series and soon to be announced but already shipping > 7280R Series Arista is officially getting into the routing game. The fixed > 1U 7280R Series looks quite impressive. The 7500R series is your > traditional chassis and line card based solution. > > Both of these products have the ability to hold the full internet routing > table, and Arista is working on MPLS features. Both of these new products > use the latest Broadcom Jerico chipsets. We (Netflix) have been deploying the previous gen (7500E) as edge routers for about two years in high traffic, low route count applications in our CDN, and have been working with Arista for almost as long to improve route scale so that we could turn off all our traditional routers. The features that enable full routes on Jericho are running in our production network today and we also have the 7500R and 7280R working with full tables. I can't speak to MPLS, but for our use case (all L3, very high-density 10/40/100G, BGP, IS-IS and light QoS), it's working well. So, yes, I'd say those two products are quite viable and competitive options in the edge router space.

7 8

carrier grade fax boards?
by Ryan Finnesey 28 Apr '16

28 Apr '16

I was wondering if anyone had any recommendations on carrier grade fax boards that are SIP based? Cheers Ryan

8 8

Network Weathermap
by James Bensley 28 Apr '16

28 Apr '16

Hi All, Those of you that have a network weathermap similar to [1], what software are you using to edit the maps, the built in editor or something custom, 3rd party/external editor? Paid or free editor? I'm look for a better editing tool I can preferably use with that plugin (we are feeding it from Cacti). I'm open to both open source editors and paid commercial ones. Maybe you use a different weathermap tool, again I'd be interested to hear about that too. Doesn't have to be fed from Cacti/RRDs. Maybe I'm just pants at being visual [2] but I'm finding that making large maps isn't that well supported (the 'move' option on nodes and using the 'via' option on links becomes less accurate) and it's difficult to keep "busy" images clean (no grid, or snap-to-grid, scaling for higher res maps for larger networks etc). The main problem is keeping everything neat though. An example would be that we acquire a network, want to extend an existing map to incorporate the new network, I can’t select a bunch of objects and move them along all together I have to painstakingly move each node by hand, align each link and node by hand etc. Cheers, James. [1] http://network-weathermap.com/manual/0.97b/ [2] That is most likely the main problem here

3 5

contact with mail support for domain cisco.com
by Piotr 28 Apr '16

28 Apr '16

Hi, There is a problem with sending emails from employees in @cisco.com domain to some certain domain. Emails in opposite direction pass without problem. No errors, warnings or any other logs at cisco's employees desktop.. We checked popular rbls, spamhauses, senderbase etc. Other domains on the same MX cluster receive emails from @cisco.com.. I try to get help in many ways ( cisco tac, a few account managers, channel partners) but without success.. Big thanks for contact via email: peter.handke.1966 at gmail.com or any other advice best regards, Peter

2 1

NYC & Philly Metro rack stack and config
by Daniel Corbe 28 Apr '16

28 Apr '16

If anyone in the NYC and Philly metro areas want to make a few extra bucks, contact me off list. I need someone to drop an initial config on some brocade routers and run some cabling. Best, Daniel

1 0

IPv6 prefix from T-Mobile USA used but not announced in BGP (2607:7700::/32)
by Baptiste Jonglez 27 Apr '16

27 Apr '16

Hi, While doing statistics on the participants of a public DHT, I was surprised to see some IP addresses that are not present in the DFZ: 2607:7700:0:25::4e00:605b 2607:7700:0:25::4e25:8ce8 2607:7700:0:25::c808:db2c 2607:7700:0:4::3294:6683 2607:7700:0:4::4c09:4d39 2607:7700:0:4::5985:87d1 2607:7700:0:4::5d7d:3df8 All those IP are in 2607:7700::/32, allocated to T-Mobile USA but never announced: https://stat.ripe.net/2607%3A7700%3A%3A%2F32#tabId=at-a-glance What could explain their usage in an application? Some form of address translation? Maybe these addresses have been used as source address in outgoing packets, which would indicate IPv6 tests in T-Mobile? (but obviously, nobody is able to answer such packets). Baptiste

3 5

Re: IPv6 prefix from T-Mobile USA used but not announced in BGP (2607:7700::/32)
by Baptiste Jonglez 27 Apr '16

27 Apr '16

On Wed, Apr 27, 2016 at 02:38:41PM -0700, Ca By wrote: > What behavior do you expect when an ipv6only node connects to an ipv4only > node , which is the tmobile case? How is that address of the address > report? As far as I know, IPv4-only DHT nodes do not directly communicate with IPv6-only DHT nodes, I don't see how this would be possible in general. The code of the DHT client is here: https://github.com/savoirfairelinux/opendht > On Wednesday, April 27, 2016, Baptiste Jonglez <baptiste(a)bitsofnetworks.org> > wrote: > > > On Wed, Apr 27, 2016 at 01:16:28PM -0700, Aaron Hopkins wrote: > > > On Wed, 27 Apr 2016, Baptiste Jonglez wrote: > > > > > > >While doing statistics on the participants of a public DHT, I was > > > >surprised to see some IP addresses that are not present in the DFZ: > > > > > > I believe those are used by T-mobile's 464XLAT (RFC 6877) implementation. > > > > > > Recent Android on T-mobile is IPv6-only and has no ability to connect to > > > raw IPv4 addresses. T-mobile's DNS servers are only asked by these > > devices > > > to translate hostnames to IPv6 addresses. If they can't find an IPv6 > > > address, they will look up the IPv4 address for a hostname, and pack it > > into > > > the bottom 32 bits of an IPv6 address that routes to a IPv6-to-IPv4 NAT > > > device. > > > > Thanks, I had forgotten that DNS64 is possible without using the > > well-known prefix. The encoded IPv4 addresses seem to belong to other > > peers of the DHT. > > > > So, if this is basically DNS64/NAT64, these IP addresses should not be > > seen as source or destination address outside of T-Mobile's network, and > > are not attached to the interface of any device. > > > > I can see two possible explanations: > > > > 1/ packets with src or dest IP in 2607:7700::/32 somehow escaped > > T-Mobile's network, without being translated back to IPv4. They caused > > other DHT nodes to believe they have incoming peers in 2607:7700::/32. > > > > 2/ there is an interesting bug in the DHT software when run behind 464XLAT > > (btw, the DHT is dual-stack and supports IPv6 just fine) > > > > I still wonder how this can happen, because the DHT does not use DNS at > > all... > > > > Baptiste > >

1 0

GeoIP database issues and the real world consequences
by Chris Boyd 27 Apr '16

27 Apr '16

Interesting article. http://fusion.net/story/287592/internet-mapping-glitch-kansas-farm/ An hour’s drive from Wichita, Kansas, in a little town called Potwin, there is a 360-acre piece of land with a very big problem. The plot has been owned by the Vogelman family for more than a hundred years, though the current owner, Joyce Taylor née Vogelman, 82, now rents it out. The acreage is quiet and remote: a farm, a pasture, an old orchard, two barns, some hog shacks and a two-story house. It’s the kind of place you move to if you want to get away from it all. The nearest neighbor is a mile away, and the closest big town has just 13,000 people. It is real, rural America; in fact, it’s a two-hour drive from the exact geographical center of the United States. But instead of being a place of respite, the people who live on Joyce Taylor’s land find themselves in a technological horror story. For the last decade, Taylor and her renters have been visited by all kinds of mysterious trouble. They’ve been accused of being identity thieves, spammers, scammers and fraudsters. They’ve gotten visited by FBI agents, federal marshals, IRS collectors, ambulances searching for suicidal veterans, and police officers searching for runaway children. They’ve found people scrounging around in their barn. The renters have been doxxed, their names and addresses posted on the internet by vigilantes. Once, someone left a broken toilet in the driveway as a strange, indefinite threat. --Chris

52 115

trout views
by Randy Bush 27 Apr '16

27 Apr '16

nfs0.dfw.rg.net:/root# ping 128.223.51.20 PING 128.223.51.20 (128.223.51.20) 56(84) bytes of data. From 4.69.145.11 icmp_seq=1 Time to live exceeded From 4.69.145.11 icmp_seq=2 Time to live exceeded ^C --- 128.223.51.20 ping statistics --- 2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1001ms nfs0.dfw.rg.net:/root# traceroute 128.223.51.20 traceroute to 128.223.51.20 (128.223.51.20), 30 hops max, 60 byte packets 1 r0.dfw.rg.net (198.180.152.3) 1.571 ms 1.559 ms 1.597 ms 2 r1.dfw.rg.net (198.180.152.2) 0.558 ms 0.551 ms 0.640 ms 3 ge-102-0-0-29-53.r08.dllstx09.us.bb.gin.ntt.net (157.238.224.205) 1.562 ms 1.598 ms 1.716 ms 4 ae-12.r07.dllstx09.us.bb.gin.ntt.net (129.250.3.107) 0.488 ms 0.538 ms 0.582 ms 5 * * * 6 * * * 7 ae-65-65.csw1.Dallas1.Level3.net (4.69.206.133) 0.514 ms 0.555 ms ae-95-95.csw4.Dallas1.Level3.net (4.69.206.145) 0.525 ms 8 ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 0.627 ms 0.769 ms ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 0.633 ms 9 vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 0.517 ms vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 0.589 ms 0.676 ms 10 ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 0.519 ms ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 0.632 ms ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 13.412 ms 11 vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 13.264 ms 0.604 ms vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 0.530 ms 12 ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 0.633 ms 0.519 ms 0.513 ms 13 vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 0.540 ms vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 13.318 ms vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 0.554 ms 14 ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 0.723 ms ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 12.793 ms ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 0.494 ms 15 vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 0.554 ms vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 0.541 ms vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 2.040 ms 16 ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 12.294 ms ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 0.896 ms 0.790 ms 17 vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 0.590 ms vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 0.610 ms vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 0.932 ms 18 ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 0.522 ms ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 11.423 ms ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 0.747 ms 19 vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 0.673 ms vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 0.803 ms vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 0.638 ms 20 ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 0.525 ms ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 0.826 ms 0.876 ms 21 vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 0.635 ms 9.840 ms 9.404 ms 22 ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 0.622 ms 0.604 ms ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 8.751 ms 23 vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 0.671 ms 0.817 ms 0.687 ms 24 ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 0.785 ms 0.754 ms ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 4.774 ms 25 vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 0.633 ms 0.683 ms 0.795 ms 26 ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 0.659 ms ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 0.758 ms 0.907 ms 27 vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 0.776 ms vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 0.721 ms vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 0.705 ms 28 ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 15.057 ms 14.660 ms ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 0.871 ms 29 vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 0.864 ms vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 0.752 ms 0.747 ms 30 ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 0.877 ms ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 0.721 ms ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 14.302 ms and, of course, email does not work

3 4