On Tue, Mar 13, 2012 at 7:09 PM, chris <tknchris@gmail.com> wrote:

On Mar 13, 2012 7:04 PM, "William Herrin" <bill@herrin.us> wrote:

...

On Tue, Mar 13, 2012 at 6:26 PM, Justin M. Streiner <streiner@cluebyfour.org> wrote:

...

I realize this might be a bit of a fool's errand, but I'm trying to determine if Verizon will speak BGP with FiOS business customers.

No. If you want to do BGP with Verizon, you have to buy a T1 at 10 times the cost and 1/10th of the speed.

Haha true that. How else would.they.push their atm and.Ethernet products.

A cost I could live with. It's the fact that they won't sell me BGP service in the FiOS product line *at all* that makes me pine for the days of FCC mandated unbundling. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com  bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

Comcast same deal ethernet only chris On Mar 13, 2012 7:42 PM, "Nathan Stratton" <nathan@robotics.net> wrote:

On Tue, 13 Mar 2012, William Herrin wrote:

A cost I could live with. It's the fact that they won't sell me BGP

...

service in the FiOS product line *at all* that makes me pine for the days of FCC mandated unbundling.

Having the same problem with Comcast, even on there business Cable service they wont do BGP with me.

-Nathan

Regards,

...

Bill Herrin

-- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On Tue, Mar 13, 2012 at 8:20 PM, Faisal Imtiaz <faisal@snappydsl.net> wrote:

So I have to ask you the big question...

Why do you want to do BGP with Comcast or Verizon ? (Over FIOS or Cable ?)

Is the intent to Peer with their network ? (which they will rightfully only allow on bigger fatter connections)..

'peer' has many connotations, I think most of the cases of it over FIOS are just: "I want bgp so I can announce my prefixes, and see yours/default/etc" (which leads to 'multihoming' and other normal (for businesses) activities on the Internet.

or Are you trying to delivery your IP's to a End Customer behind that FIOS / Cable Connection ? ... (there a ways to accomplish this without needing their cooperation..)

or you are multihomed or you want some semblence of 'the internet is down' so other bits of your infrastructure can take over or you want ... a thousand other things.

On Tue, Mar 13, 2012 at 8:35 PM, Mark Gauvin <MGauvin@dryden.ca> wrote:

Peering is generally for a comercial endevor to my understandind fios is a residential service so which are you trying to accomplish

'peering' really is a loaded term... 'settlement free peering' ? 'bgp peering' ? there are other meanings as well, but I think in the case the person I responded (Faisal?) was asking about he meant 'settlement free peering', which I don't think is what Justin meant, Justin just wants the same as most of the bgp speakers want: "multihoming". -chris

On Tue, 13 Mar 2012, Faisal Imtiaz wrote:

Why do you want to do BGP with Comcast or Verizon ? (Over FIOS or Cable ?)

To gain redundancy for a consulting client.

Is the intent to Peer with their network ? (which they will rightfully only allow on bigger fatter connections)..

I think you mean "higher margin connections" ;) As far as I know, most major carriers will still sell you a T1 for Internet access (and even BGP!) if you want it.

Are you trying to delivery your IP's to a End Customer behind that FIOS / Cable Connection ? ... (there a ways to accomplish this without needing their cooperation..)

Running BGP over a tunnel is one (albeit sub-optimal) option, but I don't know of any providers that sell such a service. All of the other options have varying degrees of downside, i.e. how much of an outage are you willing to put up with when provider A fails, transferring DNS records, etc. jms

4 of the 6 downstreams are multihomed. Only 40321 (Emigrant Bank) and 18762 (Dominick & Dominick LLC) are single homed to 19262 (Verizon Online LLC). -Grant On Tue, Mar 13, 2012 at 7:43 PM, Justin M. Streiner <streiner@cluebyfour.org

wrote:

On Tue, 13 Mar 2012, Christopher Morrow wrote:

A) DHCP only, single address, dynamic

...

B) Single Static address (uplift of 25$/month I believe?)

I think that might be $40/mo now, but I could be mistaken.

Also, I know that on 701 the rate of BGP to non-BGP customers was

...

increasing and was at ~30% or so as of ~2007... You'd think that 19262 would see that, see the business opportunity and offer it? Though, I suppose they DO see the business opportunity: "You want bgp? you want to bring your own ips? you want more than a DHCP address? Pay up, a lot."

I wonder if something is cooking there. When I look at a full BGP view, I see quite a few ASNs downstream of 19262, beyond some that appear to be internal VZ ASNs:

* 12.195.9.0/24 x.x.x.x 701 19262 30079 * 65.198.73.0/24 x.x.x.x 701 19262 40321 * 68.236.226.0/24 x.x.x.x 701 19262 18762 * 137.71.229.0/24 x.x.x.x 701 19262 20258 * 141.155.220.0/24 x.x.x.x 701 19262 36512 * 143.165.216.0/21 x.x.x.x 701 19262 2923 .....

jms

C) 5 ips STATICALLY ROUTED AS /32's!! (WTF??) for 25$ above the option-B above/month.

And people wonder why Verizon is the first to whine about routing table growth from deaggregation? ;-) In all seriousness, though, I don't think they are routed as /32s. I think that's one for the Verizon CPE, 5 for your devices all routed as a single /29. Owen

On Mar 13, 2012, at 8:57 PM, Christopher Morrow wrote:

On Tue, Mar 13, 2012 at 11:20 PM, Owen DeLong <owen@delong.com> wrote:

...

...

C) 5 ips STATICALLY ROUTED AS /32's!! (WTF??) for 25$ above the option-B above/month.

And people wonder why Verizon is the first to whine about routing table growth from deaggregation? ;-)

eh, these end up (I think) aggregated on the edge router, so you get 5 /32's from a /23 (or the like) routed to the edge layer3 device. not as bloaty for the rest of their network as it at first seems.

...

In all seriousness, though, I don't think they are routed as /32s. I think that's one for the Verizon CPE, 5 for your devices all routed as a single /29.

owen, seen the config on a live router, yes they are routed as /32's to the VC you are connected to. I probably have the config for my old link in IM/email somewhere. apparently their automation either doesn't understand CIDR, or it was 'too expensive' to make the automation do CIDR once they started to offer extra ips to the business customers.

-chris

Interesting. I guess to each their own. Many other providers I know are selling "5 IP" packages done the other way. Owen

in the DSL world, when we were providing service using Bridge PVC's, it was easier to allocate (as many needed) /32 to a customer CPE, than to route a subnet. This changed when the AT&T/BellSouth infrastructure changed from being able to get ATM PVC's to PPPoE only network. Faisal Imtiaz Snappy Internet& Telecom On 3/13/2012 11:57 PM, Christopher Morrow wrote:

On Tue, Mar 13, 2012 at 11:20 PM, Owen DeLong<owen@delong.com> wrote:

...

...

C) 5 ips STATICALLY ROUTED AS /32's!! (WTF??) for 25$ above the option-B above/month.

And people wonder why Verizon is the first to whine about routing table growth from deaggregation? ;-)

eh, these end up (I think) aggregated on the edge router, so you get 5 /32's from a /23 (or the like) routed to the edge layer3 device. not as bloaty for the rest of their network as it at first seems.

...

In all seriousness, though, I don't think they are routed as /32s. I think that's one for the Verizon CPE, 5 for your devices all routed as a single /29.

owen, seen the config on a live router, yes they are routed as /32's to the VC you are connected to. I probably have the config for my old link in IM/email somewhere. apparently their automation either doesn't understand CIDR, or it was 'too expensive' to make the automation do CIDR once they started to offer extra ips to the business customers.

-chris

On Wed, 14 Mar 2012 00:19:16 -0400, William Herrin <bill@herrin.us> wrote:

Nope. I have FiOS and the 5 IPs. They are 5 IPs, in sequence, at a completely arbitrary location in a /24 subnet. ...

Time Warner (TWTC, not TWC) does the same thing... we have 8 addresses from them... 131 - 138; it's a /24 and we get to use those 8 addresses. [Yes, that causes problems trying to access anything else in that /24] I have no clue what's on the other end of that, and really don't care. (it's more or less bridged ethernet over a T1, that's also carrying voice.)

We have the same problem in our FTTH access network (due to L2 isolation CPE can't directly ARP those in the same subnet), hence the vendor's move towards MAC force forwarding (MACFF). Frank -----Original Message----- From: Ricky Beam [mailto:jfbeam@gmail.com] Sent: Wednesday, March 14, 2012 2:18 PM To: William Herrin Cc: nanog@nanog.org Subject: Re: Verizon FiOS - is BGP an option? On Wed, 14 Mar 2012 00:19:16 -0400, William Herrin <bill@herrin.us> wrote:

Nope. I have FiOS and the 5 IPs. They are 5 IPs, in sequence, at a completely arbitrary location in a /24 subnet. ...

Time Warner (TWTC, not TWC) does the same thing... we have 8 addresses from them... 131 - 138; it's a /24 and we get to use those 8 addresses. [Yes, that causes problems trying to access anything else in that /24] I have no clue what's on the other end of that, and really don't care. (it's more or less bridged ethernet over a T1, that's also carrying voice.)

Is that is needed, what is wrong with that ? Isn't MPLS a form of encapsulation ? Don't the enterprise folks run routing protocols on it ? With carriers today it is very common to deliver L2 connectivity over L3 networks. One does not have to like it...and just because someone else (upstream) does it for one, it does not mean it is wrong......just the nature of networks and growth, solving problems with the available set of tools... Faisal On Mar 14, 2012, at 12:40 AM, chris <tknchris@gmail.com> wrote:

next lets encapsulate bgp over http next so we can run bgp at wifi hotspots :)

On Wed, Mar 14, 2012 at 12:34 AM, Frank Bulk <frnkblk@iname.com> wrote:

...

One possible avenue is put a router/computer in a colo and build a GRE tunnel over your FiOS connection to the data center, and then "peer" with folk there.

Frank

-----Original Message----- From: Justin M. Streiner [mailto:streiner@cluebyfour.org] Sent: Tuesday, March 13, 2012 5:27 PM To: nanog@nanog.org Subject: Verizon FiOS - is BGP an option?

All:

I realize this might be a bit of a fool's errand, but I'm trying to determine if Verizon will speak BGP with FiOS business customers. Their website is relatively lean on details. Everything that mentions BGP points to VZB services, which does not appear to include FiOS. Looking at the routing table, I do see several non-VZ ASNs downstream of AS19262, so it looks like it might be possible.

If that is the case, could anyone lend any insight to get past the "what is BGP?" response that likely awaits from their salescritters?

jms

I am not familiar with VZ's FIOS network... however I suspect that if they are using a Redback at the Headend, it would allow you to have a 'bridge' network with secure arp settings. (it's a feature that we have seen on Redback's...) Allows you to have a 'flat network' for all your subs, and there is a mechanism built in to allow for assign static ip's and also not allowing for someone to 'fake' / 'steal' someone else assigned IP's. This is nice, because one can be very efficient in their use of IPv4 addresses.... This is why I had said earlier that some of these networks are built with infrastructure that is not designed / meant to run advance routing protocols for an End User customers... too much overhead in running bgp sessions to hand off a single IP or even a /29 ........ Faisal Imtiaz Snappy Internet& Telecom On 3/14/2012 9:13 AM, Jay Ashworth wrote:

----- Original Message -----

...

From: "Faisal Imtiaz"<faisal@snappydsl.net> Is that is needed, what is wrong with that ? Well, we just had FiOS Business 150/65 dropped this week, and my /27 isn't even a /27; we're sharing a /24 with, presumably, a bunch of other customers.

Not sure how BGP would handle that...

Cheers, -- jra

Yes, Dane is not only very smart but also a very sharp and savvy business operator.. But I am also sure they are not doing this as a 'no charge' offering for a 'resi' circuit. Most competitive ISP's (such as Sonic and ourselves) a very flexible to customer's needs and are willing to support custom configurations but .. it has to make business sense...and the underlying infrastructure be able to support that configuration. Faisal Imtiaz Snappy Internet& Telecom 7266 SW 48 Street Miami, Fl 33155 Tel: 305 663 5518 x 232 Helpdesk: 305 663 5518 option 2 Email: Support@Snappydsl.net On 3/14/2012 2:41 PM, Mehmet Akcin wrote:

As far as I know only ISP that will let you do BGP with them on DSL is Sonic and their Fusion service is awesome but very limited to bay area.

they rock though.

mehmet

On Mar 14, 2012, at 6:36 AM, Faisal Imtiaz wrote:

...

I am not familiar with VZ's FIOS network... however I suspect that if they are using a Redback at the Headend, it would allow you to have a 'bridge' network with secure arp settings. (it's a feature that we have seen on Redback's...)

Allows you to have a 'flat network' for all your subs, and there is a mechanism built in to allow for assign static ip's and also not allowing for someone to 'fake' / 'steal' someone else assigned IP's. This is nice, because one can be very efficient in their use of IPv4 addresses....

This is why I had said earlier that some of these networks are built with infrastructure that is not designed / meant to run advance routing protocols for an End User customers... too much overhead in running bgp sessions to hand off a single IP or even a /29 ........

Faisal Imtiaz Snappy Internet& Telecom

On 3/14/2012 9:13 AM, Jay Ashworth wrote:

...

----- Original Message -----

...

From: "Faisal Imtiaz"<faisal@snappydsl.net> Is that is needed, what is wrong with that ? Well, we just had FiOS Business 150/65 dropped this week, and my /27 isn't even a /27; we're sharing a /24 with, presumably, a bunch of other customers.

Not sure how BGP would handle that...

Cheers, -- jra

On Wed, Mar 14, 2012 at 3:32 PM, Justin M. Streiner <streiner@cluebyfour.org> wrote:

The point (and this goes back to my original post) was that VZ is missing out on revenue (and customer service, but let's not get ahead of ourselves...) opportunities by not offering such a thing as an add-on for their business- class FiOS services.  If they brand it and bill at as a business-class service, then allowing someone to multihome using FiOS and something else does not seem like such an unreasonable request.

Well... they brand it as a SOHO service and AFAICT they refuse to install "business fios" anywhere zoned commercial. If they would actually connect it to a commercial facility, there'd be no shortage of third parties willing to do a fios line pair plus a tunnel and then drive whatever data over it that you wanted. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com  bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On 3/14/2012 3:32 PM, Justin M. Streiner wrote:

On Wed, 14 Mar 2012, Faisal Imtiaz wrote:

...

Most competitive ISP's (such as Sonic and ourselves) a very flexible to customer's needs and are willing to support custom configurations but .. it has to make business sense...and the underlying infrastructure be able to support that configuration.

I don't think anyone is disagreeing with you, or implying otherwise here. That is understood.. The point (and this goes back to my original post) was that VZ is missing out on revenue (and customer service, but let's not get ahead of ourselves...) Our concept of 'revenue' and large provider's concept of 'revenue' is very different .... opportunities by not offering such a thing as an add-on for their business- class FiOS services. It may sound simple to you and I, but the bigger challenge is on the support side how to deal with a more complex issue, and how to justify having more expensive support engineers ... etc.. If they brand it and bill at as a business-class service, then allowing someone to multihome using FiOS and something else does not seem like such an unreasonable request.

On the surface this does not sound unreasonable, until you take into consideration that Support issues now would requires someone (or more like a team of folks) who is running cost is about $100 to $250 / hr (engineers, support structure, etc etc etc) ...... for a service which is approx in the same $ figure for MRR...

As others have mentioned, if 19262 would toss in a few route-reflectors and let their customers EBGP-multihop to them, that would be a step in the right direction. In the scenario I'm working on at the moment, default, or default+customer routes would be perfectly fine. I neither want nor need a full view for this application.

while this is reasonable, we all have to keep in mind, that you can I can 'toss' in route-reflectors for a few hundred to a few thousand dollars each... Folks like VZ and AT&T pay top dollars for top capacity equipment to handle stuff.. so you are talking about a few 'route-reflectors' for $50k or $150k each ? .... (Remember these are the folks who are paying full prices on the Cisco / Juniper boxes.....) if you ever looked at the Cisco Top of the line router.. ( I don't remember what it called, but do remember the starting price for a base config is $250K, going up to $750k... was designed to meet the demands of larger network operators such as VZ / AT&T etc...) As for the Cable Companies, most of them out-source network management / upgrade and upkeep to folks like Scientific Atlanta (a division of Cisco).. that is why when you call in for network support issues, you get someone who is not technically proficient with all aspects of networking... because they don't have them....

...

...

...

In the scenario I'm working on at the moment, default, or default+customer routes would be perfectly fine. I neither want nor need a full view for this >>>>application.

There are existing solutions which are very easy to implement, which will allow you to do this, without having to deal too much with the underlying carrier .... so my question becomes.... if you need this, and you can solve this easily from your side... why do you want a behemoth to change and deliver ?.... (Even if they did you are not going to be happy with how they are performing ...).

jms

Faisal Imtiaz <faisal@snappydsl.net> writes:

I am not familiar with VZ's FIOS network... however I suspect that if they are using a Redback at the Headend, it would allow you to have a 'bridge' network with secure arp settings. (it's a feature that we have seen on Redback's...)

AFAIK Verizon does not use Redback/Ericsson stuff for FIOS and never has. A cursory survey of two (older, BPON, Tellabs) builds found ethernet OUI 00:90:1a, i.e. Juniper ERX. -r

Christopher Morrow <morrowc.lists@gmail.com> writes:

On Wed, Mar 14, 2012 at 8:14 PM, Robert E. Seastrom <rs@seastrom.com> wrote:

...

Faisal Imtiaz <faisal@snappydsl.net> writes:

...

I am not familiar with VZ's FIOS network... however I suspect that if they are using a Redback at the Headend, it would allow you to have a 'bridge' network with secure arp settings. (it's a feature that we have seen on Redback's...)

AFAIK Verizon does not use Redback/Ericsson stuff for FIOS and never has.

A cursory survey of two (older, BPON, Tellabs) builds found ethernet OUI 00:90:1a, i.e. Juniper ERX.

yes, all edge boxes for FIOS are ERX... better support for CALEA there was one of the major drivers.

So it was _one_ of the drivers, but was it a more major driver than "for the love of God, not Redback!"? :) -r

On 3/14/2012 9:00 PM, Robert E. Seastrom wrote:

Christopher Morrow<morrowc.lists@gmail.com> writes:

...

On Wed, Mar 14, 2012 at 8:14 PM, Robert E. Seastrom<rs@seastrom.com> wrote:

...

Faisal Imtiaz<faisal@snappydsl.net> writes:

...

I am not familiar with VZ's FIOS network... however I suspect that if they are using a Redback at the Headend, it would allow you to have a 'bridge' network with secure arp settings. (it's a feature that we have seen on Redback's...) AFAIK Verizon does not use Redback/Ericsson stuff for FIOS and never has.

A cursory survey of two (older, BPON, Tellabs) builds found ethernet OUI 00:90:1a, i.e. Juniper ERX. yes, all edge boxes for FIOS are ERX... better support for CALEA there was one of the major drivers. So it was _one_ of the drivers, but was it a more major driver than "for the love of God, not Redback!"? :)

the last I knew, Verizon was an Alcatel house for switching and Alcatel managed to get tcp/ip into their switching gear. so I'm left to wonder. --C

This is a fascinating thread! I have had multiple class C address blocks assigned to us for many years (since the 80's) I have 2 T1 connections and one of them is up for contract renewal. I have wanted to replace one of the expensive T1s for a long time. DSL and Cable are available here at reasonable prices (no FIOS yet) However, even after they tell me they will do it, no provider will route even a single /24 (/30) for me. Mostly it's Verizon and/or Time Warner. I would love to have another solution. All I really need is to maintain the IPs on my servers so they are public/world accessible. (Email/Web/FTP/telnet(!)) Perhaps I can route to a co-located server then a tunnel back to the server farm over a static IP DSL or Cable link??? I am stumped. Any ideas? Rich

On Fri, 3 Aug 2012, Christopher Morrow wrote:

On Fri, Aug 3, 2012 at 10:31 AM, Richard Miller <rmiller@millerad.com> wrote:

...

I am stumped.

Any ideas?

time to migrate to carriers that care about you and your business?

The tough part there is that Verizon is not required (as I understand it) to open access to the plant they've built out for FiOS, unlike their copper plant. That's one of the reasons they've been keen to push people away from DSL and onto FiOS. Short of pulling dark fiber (not cost-effective for hope use yet ;) ) from one of the providers that serves this area, there were no other viable options for getting >T1/DSL speeds to the house :( I ended up having FiOS business service installed about two weeks ago, but it's definitely pricey. jms

Hi, Yes the easier way to do it is have your subnet routed to someone that is willing to colo your router, or provide your with something like NHRP, and use a 87x on your brand new unnamed Cable/DSL provider to create a NHRP tunnel for it. We have many customers which required that kind of tunnel to bypass some belligerent TelCo. But if you're going to drop your T1 for Cable/DSL get 2 of them using different technology and from different provider (aka 1 Cable and 1 DSL =D). Have fun. ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 08/03/12 10:31, Richard Miller wrote:

This is a fascinating thread!

I have had multiple class C address blocks assigned to us for many years (since the 80's) I have 2 T1 connections and one of them is up for contract renewal. I have wanted to replace one of the expensive T1s for a long time. DSL and Cable are available here at reasonable prices (no FIOS yet) However, even after they tell me they will do it, no provider will route even a single /24 (/30) for me.

Mostly it's Verizon and/or Time Warner.

I would love to have another solution. All I really need is to maintain the IPs on my servers so they are public/world accessible. (Email/Web/FTP/telnet(!))

Perhaps I can route to a co-located server then a tunnel back to the server farm over a static IP DSL or Cable link???

I am stumped.

Any ideas?

Rich

On 8/3/12 8:56 AM, William Herrin wrote:

On Fri, Aug 3, 2012 at 11:26 AM, Alain Hebert <ahebert@pubnix.net> wrote:

...

Yes the easier way to do it is have your subnet routed to someone that is willing to colo your router, or provide your with something like NHRP, and use a 87x on your brand new unnamed Cable/DSL provider to create a NHRP tunnel for it.

We have many customers which required that kind of tunnel to bypass some belligerent TelCo.

But if you're going to drop your T1 for Cable/DSL get 2 of them using different technology and from different provider (aka 1 Cable and 1 DSL =D).

I'm doing this. Works well most of the time. A couple months ago we had major storm related outages in the area that persisted a couple of days. Internet service on both lines dropped out after 12 hours. It seems the telcos and cable companies don't consider the commodity Internet part of their equipment to be something which needs electricity during an extended grid outage. Cox. Verizon. I'm looking at you.

Most don't, and for the price being paid on commodity connections I feel indifferent about it. The central plant days are mostly gone; there's fiber huts everywhere and not enough trucks/manpower (in my area a lineman sits in his truck and reads a book while tethered to the power kiosk) to run them all if the outage is too widespread for too long. ~Seth

On Aug 3, 2012, at 12:31 , William Herrin <bill@herrin.us> wrote:

On Fri, Aug 3, 2012 at 8:51 AM, Seth Mattinen <sethm@rollernet.us> wrote:

...

On 8/3/12 8:56 AM, William Herrin wrote:

...

It seems the telcos and cable companies don't consider the commodity Internet part of their equipment to be something which needs electricity during an extended grid outage. Cox. Verizon. I'm looking at you.

Most don't, and for the price being paid on commodity connections I feel indifferent about it.

Back in the day they kept my land line phone on during extended power outages. And that was when they had to power the phone. Now all they have to do is power the equipment on their end of the line. My phone's out because hey, voip. My Sprint cell phone's out because the fools can't power their towers. It's 105 degrees out and I'm screwed if someone has a heat stroke because we can't even call 911.

48vDC battery to power your phone up to 3 ringer equivalences was a pretty light load overall, compared to PON aggregators for all those neighborhoods. Further, as noted above the PON equipment is much more widely distributed than powering your phone. Powering your phone was straight DC down the same copper wire as your service. Powering the PON aggregators, well, unless you've got some magic new technology for powering them via fiber is a bit more involved and quite a bit more amperage per conductor than POTS.

...

The central plant days are mostly gone; there's fiber huts everywhere and not enough trucks/manpower (in my area a lineman sits in his truck and reads a book while tethered to the power kiosk) to run them all if the outage is too widespread for too long.

They put a quarter million dollars into the fiber hut. They can't put a $500 gasoline generator in a warehouse 50 miles away and go pick it up when there's an extended outage?

That's a lot of generators and a lot of people to go pull them out and make sure they don't walk off during said extended outage.

I'll give Verizon a little credit. They restored service after about 12 hours of outage. Cox didn't restore service until 12 hours *after* my power came back on.

Seems pretty reasonable to me given the scale of the outage.

Could be worse. I could have Pepco instead of Dominion. But it could be better. And 20 years ago the reliability was.

20 years ago you didn't have a megabit to your home let alone many megabits. 20 years ago, POTS was much simpler than the converged networks we have today. There is something to be said for the simplicity of POTS. If you're that concerned about calling 911 for a heat stroke, why don't you maintain a POTS line? Owen

On Fri, Aug 3, 2012 at 5:17 PM, William Herrin <bill@herrin.us> wrote:

On Fri, Aug 3, 2012 at 10:01 AM, Owen DeLong <owen@delong.com> wrote:

...

On Aug 3, 2012, at 12:31 , William Herrin <bill@herrin.us> wrote:

...

Could be worse. I could have Pepco instead of Dominion. But it could be better. And 20 years ago the reliability was.

20 years ago you didn't have a megabit to your home let alone many megabits. 20 years ago, POTS was much simpler than the converged networks we have today. There is something to be said for the simplicity of POTS.

If you're that concerned about calling 911 for a heat stroke, why don't you maintain a POTS line?

When Verizon installed FIOS in the neighborhood they removed the copper lines to each house. It was understood and accepted that if the

ACTUALLY... no. they are NOT supposed to do this, in fact they said to congress that they were NOT removing copper, not clipping it outside the prem (despite what I've seen with my own eyes...). I think it's actually a violation for them to clip the copper, and to not support it, since it was put in with public funds... but ianal and all that patrick stuff.

household fiber adapters did not receive power the battery would fail in a few hours. That the upstream would fail, even for folks who took measures to continue to power the fiber adapter, was unexpected and very unfortunate. If they can run a copper pair back to a powerable location then it escapes me why they can't do the same with a single strand of fiber.

they do not want to be beholden to the PUC if they can avoid it...

Regards, Bill Herrin

-- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

Actually, it's a choice. You just tell them you want to keep your POTS when you sign up for service. They can definitely bundle Fios TV & POTS. The VOIP package might be cheaper. I suspect that's where most people wind up, not realizing the difference in service until there is a power outage. --Heather -----Original Message----- From: William Herrin [mailto:bill@herrin.us] Sent: Friday, August 03, 2012 5:18 PM To: Owen DeLong Cc: nanog@nanog.org Subject: Re: Verizon FiOS - is BGP an option? On Fri, Aug 3, 2012 at 10:01 AM, Owen DeLong <owen@delong.com> wrote:

On Aug 3, 2012, at 12:31 , William Herrin <bill@herrin.us> wrote:

...

Could be worse. I could have Pepco instead of Dominion. But it could be better. And 20 years ago the reliability was.

20 years ago you didn't have a megabit to your home let alone many megabits. 20 years ago, POTS was much simpler than the converged networks we have today. There is something to be said for the simplicity of POTS.

If you're that concerned about calling 911 for a heat stroke, why don't you maintain a POTS line?

When Verizon installed FIOS in the neighborhood they removed the copper lines to each house. It was understood and accepted that if the household fiber adapters did not receive power the battery would fail in a few hours. That the upstream would fail, even for folks who took measures to continue to power the fiber adapter, was unexpected and very unfortunate. If they can run a copper pair back to a powerable location then it escapes me why they can't do the same with a single strand of fiber. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

A good portable generator is more than $500, and if it's a wide-spread outage there's not enough portable generators to go around, and if there were, not enough people to set them and give them their fluids. And it doesn't pay to put a natural gas (or similar) generator at every node for those rare instances where the battery does not suffice. Frank -----Original Message----- From: William Herrin [mailto:bill@herrin.us] Sent: Friday, August 03, 2012 2:31 PM To: Seth Mattinen Cc: nanog@nanog.org Subject: Re: Verizon FiOS - is BGP an option? On Fri, Aug 3, 2012 at 8:51 AM, Seth Mattinen <sethm@rollernet.us> wrote: <snip>

The central plant days are mostly gone; there's fiber huts everywhere and not enough trucks/manpower (in my area a lineman sits in his truck and reads a book while tethered to the power kiosk) to run them all if the outage is too widespread for too long.

They put a quarter million dollars into the fiber hut. They can't put a $500 gasoline generator in a warehouse 50 miles away and go pick it up when there's an extended outage? I'll give Verizon a little credit. They restored service after about 12 hours of outage. Cox didn't restore service until 12 hours *after* my power came back on. Could be worse. I could have Pepco instead of Dominion. But it could be better. And 20 years ago the reliability was. -Bill -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

As someone else posted, many FTTH installations are centralized as much as possible to avoid having non-passive equipment in the plant, allowing for the practicality of onsite generators. That's what we do. But for those who have powered nodes in the field (distributed/tiered BPON or GPON configurations and cable plants), it's not realistic to keep them all powered. Despite what the DOT may be able to do. Frank -----Original Message----- From: wherrin@gmail.com [mailto:wherrin@gmail.com] On Behalf Of William Herrin Sent: Friday, August 03, 2012 4:53 PM To: frnkblk@iname.com Cc: nanog@nanog.org; Seth Mattinen Subject: Re: Verizon FiOS - is BGP an option? On Fri, Aug 3, 2012 at 11:26 AM, Frank Bulk <frnkblk@iname.com> wrote:

A good portable generator is more than $500, and if it's a wide-spread outage there's not enough portable generators to go around, and if there were, not enough people to set them and give them their fluids.

Doesn't take a "good" generator to maintain a -48V battery string. Drop it off. Plug it in. Start it up. Task some folks on an 8 hour loop to keep the tanks topped off. If the DOT, not noted for its efficiency, can get the major traffic lights up and running on generators the next day, why can't Sprint, Cox and Verizon get their towers and fiber concentrators powered up? That's a condemnation worthy of the word: that your company performed worse in the storm recovery than the local department of transportation. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On 4 August 2012 04:07, Frank Bulk <frnkblk@iname.com> wrote:

...

As someone else posted, many FTTH installations are centralized as much as possible to avoid having non-passive equipment in the plant, allowing for the practicality of onsite generators. That's what we do. But for those who have powered nodes in the field (distributed/tiered BPON or GPON configurations and cable plants), it's not realistic to keep them all powered. Despite what the DOT may be able to do.

If only they had some kind of copper cabling running from some kind of central location (like perhaps the same place the fiber runs to, I imagine the same buildings that the old POTS lines ran to) that went all the way out to the huts full of powered equipment (that would likely be next to the old POTS junction boxes) that as a result of their new fiber installs would have a few pairs unused, then they could possibly have hooked those up as backup power when grid power becomes unavailable for a large area (poor power distribution efficiency would probably stop you wanting to power it that way all the time).

On 8/4/12 8:44 AM, Mike Jones wrote: providing line voltage has a bit different current requirements than a remote ip dslam sitting in a hut. you're not powering something like: http://us.zyxel.com/Products/details.aspx?PC1IndexFlag=20040812100619&CategoryGroupNo=109D87CE-A152-4245-BE66-D455B07FE7A6 over 5000' of 24awg twisted pair.

It's a shame that there isn't any such copper infrastructure owned by those same companies already in place, but perhaps they could have thrown an extra copper cable in to the middle of that fiber bundle at the same time they were running it at negligible additional cost.

- Mike

On Fri, Aug 03, 2012 at 11:52:53AM -1000, William Herrin wrote:

On Fri, Aug 3, 2012 at 11:26 AM, Frank Bulk <frnkblk@iname.com> wrote:

...

A good portable generator is more than $500, and if it's a wide-spread outage there's not enough portable generators to go around, and if there were, not enough people to set them and give them their fluids.

Doesn't take a "good" generator to maintain a -48V battery string. Drop it off. Plug it in. Start it up. Task some folks on an 8 hour loop to keep the tanks topped off.

Even battery-buffered overnight, solar PV works great if grid is down or even completely absent.

If the DOT, not noted for its efficiency, can get the major traffic lights up and running on generators the next day, why can't Sprint, Cox and Verizon get their towers and fiber concentrators powered up? That's a condemnation worthy of the word: that your company performed worse in the storm recovery than the local department of transportation.

-----Original Message----- From: Frank Bulk [mailto:frnkblk@iname.com] Sent: Friday, August 03, 2012 5:27 PM To: 'William Herrin' Cc: nanog@nanog.org Subject: RE: Verizon FiOS - is BGP an option? A good portable generator is more than $500, and if it's a wide-spread outage there's not enough portable generators to go around, and if there were, not enough people to set them and give them their fluids. And it doesn't pay to put a natural gas (or similar) generator at every node for those rare instances where the battery does not suffice. Frank -----Original Message----- From: William Herrin [mailto:bill@herrin.us] Sent: Friday, August 03, 2012 2:31 PM To: Seth Mattinen Cc: nanog@nanog.org Subject: Re: Verizon FiOS - is BGP an option? On Fri, Aug 3, 2012 at 8:51 AM, Seth Mattinen <sethm@rollernet.us> wrote: <snip>

The central plant days are mostly gone; there's fiber huts everywhere and not enough trucks/manpower (in my area a lineman sits in his truck and reads a book while tethered to the power kiosk) to run them all if the outage is too widespread for too long.

They put a quarter million dollars into the fiber hut. They can't put a $500 gasoline generator in a warehouse 50 miles away and go pick it up when there's an extended outage? I'll give Verizon a little credit. They restored service after about 12 hours of outage. Cox didn't restore service until 12 hours *after* my power came back on. Could be worse. I could have Pepco instead of Dominion. But it could be better. And 20 years ago the reliability was. -Bill -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004 One would think the head end of the fiber would have batteries and a generator. I have TDS fiber at home and I believe it goes all the way back to the CO with no active items between. I do have UPS's and a genset to keep the ONT and servers running. Here in Fairpoint (former Verizon) land, most of the SLC huts I've seen, have either a genset or a plug for a mobile generator on the side of the bldg. The generators in the service vehicles can plug into these. The cable (HFC) infrastructure, on the other hand, has pole mounted power supplies that apparently (to me) go dead within an hour of a power failure. No way to back them up easily that I can see. Running BGP and hosting over a residential service such as cable or DSL, has it's limitations as I have learned. I doubt your LEC has an SLA for DSL service. I would look at hosting somewhere closer to your eyeball networks and let them worry about power, cooling and network availability. -Keith

On Sat, Aug 4, 2012 at 11:02 AM, Chris Adams <cmadams@hiwaay.net> wrote:

Besides, where do you think you're going to get gasoline in a wide-spread extended power failure? Few gas stations have generators, and even if they do, they'll sell out of gas quickly. That distribution system also needs power. The diesel for our generator had to be trucked in from outside the affected area (Birmingham IIRC).

I managed to get gasoline for my generator. I had to drive upwards of 5 miles and pass as many as 7 closed stations to get it. But it was available and if I'd planned better with respect to containers to carry it in I'd have had zero difficulty. Some stations did have generators. And some were in locations that didn't lose power in the first place. The kind of event which ends access to fuel tends to destroy the communications infrastructure anyway so that loss of power is not the main barrier to operations. Extended loss of power is a regular, high-probability threat. I think it reasonable to expect the local communications companies to be ready for it and capable of keeping the key infrastructure online. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

long-haul fiber all stayed up. Most of the other problems were because of the power failure (some local fiber rings dropped, especially one CLEC's that puts their nodes in customer premises and were broken by customers' power failures). [snip] I would go as far as to say most electric utilities I know of specialize in safe efficient long-distance transmission for a mass consumption audience (massive number of users, very few users with specific high reliability requirements who can't accept a 5 day outage at least), and favor safety and efficiency over reliability, using many components that are not buried or heavily shielded, and are highly susceptible to weather events, trucks knocking poles over,

On 8/4/12, Chris Adams <cmadams@hiwaay.net> wrote: lightning, tornados, solar flares, etc, and that their answer to outage prevention is to let it fail, or shut it down in case of damage, and then repair later. If a telco provider's answer to powering remote comms facilities is to just let the electric company bring in AC, to charge a battery which will last a short time, then disaster survivability was not the driving design goal for that selection. Possibly because their customers or their local government weren't demanding it, or weren't willing to pay enough for them to install suitably designed power distribution and backup. If you really care about building a reliable communications infrastructure, then you have at least two independent paths and sources for communications, and at least two independent paths and sources for power, to each major component of the system, so you provide your own power distribution, favoring reliability. That increases the price of the service, and if the consumer doesn't want it so badly that they'll pay significantly more, then it could be a waste, financially; most of the time, people won't notice extra fault tolerance measures versus a competitor's cheaper service that isn't so resilient in disasters. -- -JH

Residences aren't critical infrastructure, no matter how angry the owners get. I can assure you that infrastructure that really *is* critical do have their own generators and receive priority attention from the service providers. Frank -----Original Message----- From: William Herrin [mailto:bill@herrin.us] Sent: Saturday, August 04, 2012 11:56 AM To: Chris Adams; nanog@nanog.org Subject: Re: Verizon FiOS - is BGP an option? <snip> Got news for these folks: if you have cable on the poles spidering in to lots of homes and businesses you are a critical infrastructure provider and you need to act like it. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004 -----Original Message----- From: William Herrin [mailto:bill@herrin.us] Sent: Saturday, August 04, 2012 10:15 AM To: Chris Adams; nanog@nanog.org Subject: Re: Verizon FiOS - is BGP an option? <snip> Extended loss of power is a regular, high-probability threat. I think it reasonable to expect the local communications companies to be ready for it and capable of keeping the key infrastructure online. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

My apologies, I think we crossed wires here. This thread included a discussion of continual residential Internet access in an extended power outage. That's the topic I was responding to, not E911. Frank -----Original Message----- From: Nathan Eisenberg [mailto:nathan@atlasnetworks.us] Sent: Saturday, August 04, 2012 9:27 PM To: nanog@nanog.org Subject: RE: Verizon FiOS - is BGP an option?

Residences aren't critical infrastructure, no matter how angry the owners get.

911 access isn't a critical service? Fire and security panels aren't critical services? If basic life safety and property protection aren't critical services, I'm not sure what is. These are peoples' lives and families and homes. There is nothing - repeat, nothing - more important than that. It is absolutely a critical service. Nathan Eisenberg