I've seen it with "static" public IP pppoe assignment. No NAT. Reliability? Best effort at best. Coverage area is ok. Speed and reliability is completely dependant on your location. Test first. Always. And then do not set a decent expectation. IPSec tunnels dropping? Could be. Again, depends on your locations. My overall impression? Get a T1. My overall recommendation? Complete site surveys and then have a back out plan. tv ----- Original Message ----- From: "Crooks, Sam" <Sam.Crooks@experian.com> To: <nanog@nanog.org> Sent: Tuesday, April 14, 2009 11:25 PM Subject: Looking for AT&T / Verizon / Sprint WWAN service impressions - on oroff-list replies welcome I'm considering use of AT&T / Verizon / Sprint WWAN services and the Cisco 3G router interface cards/integrated module in C880 routers for primary or backup WAN network connectivity for routers. I'm looking for information from users of these services on the following: - addressing - Do these WWAN services use dynamic, PPPoE or static IP assignment typically? Any of the 3? All? - is static IP assignment available? - do these service providers use NAT within their network? - How is the service reliability? In most cases, is the service available for use when you need to use it? - How is the service coverage area? Do you have problems getting sufficient coverage in the deplouyment location to support desired speeds (say 512kbps up/down as a minimum)? - is ESP / IKE / IPsec permitted through un-rate-limited and un-molested by the providers? - If you build a IPsec/GRE tunnel over these services, do you have frequent issues with the tunnel dropping, or a dynamic routing protocol running through the tunnel going down frequently? Also interested in similar information on impressions of similar EMEA WWAN service providers, particularly Vodaphone and T-Mobile, if anyone has experiences with these. Replies on-list or off-list are welcome.... Your choice. Cisco 3G interface and provider information: http://www.cisco.com/en/US/products/ps7272/index.html http://www.cisco.com/en/US/prod/routers/networking_solutions_products_ge nericcontent0900aecd80601f7e.html#~north-america Regards, Sam Crooks

Crooks, Sam wrote:

I'm considering use of AT&T / Verizon / Sprint WWAN services and the Cisco 3G router interface cards/integrated module in C880 routers for primary or backup WAN network connectivity for routers.

My comments are only for Sprint EVDO/1xRTT since that's what I use.

I'm looking for information from users of these services on the following:

- addressing - Do these WWAN services use dynamic, PPPoE or static IP assignment typically? Any of the 3? All?

My IP changes every time the session establishes.

- is static IP assignment available?

I've never asked about static because there was no benefit to me when other workarounds were available, i.e. DMVPN.

- do these service providers use NAT within their network?

Sprint doesn't, you get a public IP and I can establish inbound connections. They seem to filter incoming port 80 though. I regularly SSH to the wireless IP without any problems, although if the radio is sleeping sometimes it takes two attempts.

- How is the service reliability? In most cases, is the service available for use when you need to use it?

I've been using it for years with no complaints.

- How is the service coverage area? Do you have problems getting sufficient coverage in the deplouyment location to support desired speeds (say 512kbps up/down as a minimum)?

I get full EVDO rates. It's as reliable as any other CDMA phone I've used in my area. Standard bad and good coverage areas apply. They will do site surveys for you though, plus you can get fancy antennas for the cards. I picked EVDO because it has a better upstream rate.

- is ESP / IKE / IPsec permitted through un-rate-limited and un-molested by the providers?

As far as I can tell.

- If you build a IPsec/GRE tunnel over these services, do you have frequent issues with the tunnel dropping, or a dynamic routing protocol running through the tunnel going down frequently?

Sometimes latency sucks and timers will expire. It always recovers on its own though.

Also interested in similar information on impressions of similar EMEA WWAN service providers, particularly Vodaphone and T-Mobile, if anyone has experiences with these.

Replies on-list or off-list are welcome.... Your choice.

Cisco 3G interface and provider information:

http://www.cisco.com/en/US/products/ps7272/index.html

http://www.cisco.com/en/US/prod/routers/networking_solutions_products_ge nericcontent0900aecd80601f7e.html#~north-america

If uplink rates matter, for AT&T, you'll have to wait for the HWIC-3G-HSPA-A to come out. If you want better than 384 up right now, go EVDO Rev. A and make sure they do a site survey for you first. In the end, it's just a fancy cell phone in your router. ~Seth

My understanding is that AT&T uses an MPLS/VRF CE router facing the user such that the resulting network connectivity is a private MPLS VPN. VZW apparently requires the user to implement a GRE/IPSec configuration just to reach their MPLS/VRF layer. The resulting user router config is thus much simpler with AT&T. Haven't heard about Sprint though. -----Original Message----- From: Crooks, Sam [mailto:Sam.Crooks@experian.com] Sent: Tuesday, April 14, 2009 9:26 PM To: nanog@nanog.org Subject: Looking for AT&T / Verizon / Sprint WWAN service impressions - on oroff-list replies welcome I'm considering use of AT&T / Verizon / Sprint WWAN services and the Cisco 3G router interface cards/integrated module in C880 routers for primary or backup WAN network connectivity for routers. I'm looking for information from users of these services on the following: - addressing - Do these WWAN services use dynamic, PPPoE or static IP assignment typically? Any of the 3? All? - is static IP assignment available? - do these service providers use NAT within their network? - How is the service reliability? In most cases, is the service available for use when you need to use it? - How is the service coverage area? Do you have problems getting sufficient coverage in the deplouyment location to support desired speeds (say 512kbps up/down as a minimum)? - is ESP / IKE / IPsec permitted through un-rate-limited and un-molested by the providers? - If you build a IPsec/GRE tunnel over these services, do you have frequent issues with the tunnel dropping, or a dynamic routing protocol running through the tunnel going down frequently? Also interested in similar information on impressions of similar EMEA WWAN service providers, particularly Vodaphone and T-Mobile, if anyone has experiences with these. Replies on-list or off-list are welcome.... Your choice. Cisco 3G interface and provider information: http://www.cisco.com/en/US/products/ps7272/index.html http://www.cisco.com/en/US/prod/routers/networking_solutions_products_ge nericcontent0900aecd80601f7e.html#~north-america Regards, Sam Crooks

Crooks, Sam wrote:

I'm considering use of AT&T / Verizon / Sprint WWAN services and the Cisco 3G router interface cards/integrated module in C880 routers for primary or backup WAN network connectivity for routers.

I'm looking for information from users of these services on the following:

I have only Verizon at the remote locations. But I have messed with the others.

- addressing - Do these WWAN services use dynamic, PPPoE or static IP assignment typically? Any of the 3? All? - is static IP assignment available?

Verizon is private NAT by default. They will provide a static public IP address for a one time fee of $500. If I recall right, it's still PPP, but you get the same IP address every time. As of 6 months ago, Sprint did not have a static IP option. I now hear that they do.

- do these service providers use NAT within their network?

- How is the service reliability? In most cases, is the service available for use when you need to use it?

Depends on your location and the type of radio (USB or pcmcia) you use. The USB ones tend to flake out on me more, but it might be the router or other issues.

- How is the service coverage area? Do you have problems getting sufficient coverage in the deplouyment location to support desired speeds (say 512kbps up/down as a minimum)?

Once again, this depends on your area. I use it as a last ditch effort when DSL is not available.

- is ESP / IKE / IPsec permitted through un-rate-limited and un-molested by the providers?

No problems using IPSEC tunnels (Cisco PIX) over Verizon EVDO.

- If you build a IPsec/GRE tunnel over these services, do you have frequent issues with the tunnel dropping, or a dynamic routing protocol running through the tunnel going down frequently?

The Cisco Pix series tend to rebuild a dropped tunnel, so I can't say I have looked into it that deeply.

Also interested in similar information on impressions of similar EMEA WWAN service providers, particularly Vodaphone and T-Mobile, if anyone has experiences with these.

I have had issues running IPSEC tunnels over a local WISP. Whenever they would drop out do to maint (normally once a year), I could no longer establish an VPN tunnel. It would often take days to weeks before the tunnel would work again. Something in their system would drop the packets. Never found the cause and switched over to EVDO. No problems bringing up a VPN connection on my laptop, tethered to my G1 phone on T-Mobile.

Replies on-list or off-list are welcome.... Your choice.

Cisco 3G interface and provider information:

http://www.cisco.com/en/US/products/ps7272/index.html

http://www.cisco.com/en/US/prod/routers/networking_solutions_products_ge nericcontent0900aecd80601f7e.html#~north-america

Regards,

Sam Crooks

That's funny, because our company is a (very small) LEC and a member of a (small) regional network, and we've been asked by a larger consortium to give them protected 10-Gig waves between two cities. It's not been a problem to find DWDM vendors that can do that. Frank -----Original Message----- From: Rod Beck [mailto:Rod.Beck@hiberniaatlantic.com] Sent: Wednesday, April 15, 2009 7:39 AM To: joel.mercado@verizon.net; Wallace Keith; nanog@nanog.org Subject: Re: Diversity - was: Fiber cut in SF area That service is probably very expensive. There is no known way to provide cheap 10 wave protection. Not carrier grade. Protected 10 GigE service (LAN PHY 10 GigE) will tolerate a very high BER before switching. And the cost of switching STM64 is very high as well. Bottom line is that it will cost more than two diversely routed 10 gig waves. There is no real market for protected 10 gig waves. Occasionally a bank will request the service, but backoff as soon as they see the price tag. "Hopefully none of these customers had service and protect ckts that went down... I would be pissed as a ceo if that happen to my company. Hopefully level3's new service offering is 100@percent redundant as stated The new service offerings include: - Protected Wavelengths: Level 3 now provides automatic protection-switching to a dedicated diversely routed wavelength in the event of a network failure. The protection switch, fully automated and managed by Level 3, happens at switching speeds approaching SONET restoration times. The single interface to the customer requires no additional capital cost for customer optical ports, and the diverse restoration path is fixed and fully known to the customer. These features allow customers to achieve fast restoration with predictable performance in their network without adding significant cost and routing complexity. -" Roderick S. Beck Director of European Sales Hibernia Atlantic 13-15, rue Sedaine, 75011 Paris http://www.hiberniaatlantic.com

Rod Beck wrote:

That service is probably very expensive.

There is no known way to provide cheap 10 wave protection. Not carrier grade. Protected 10 GigE service (LAN PHY 10 GigE) will tolerate a very high BER before switching. And the cost of switching STM64 is very high as well.

Bottom line is that it will cost more than two diversely routed 10 gig waves.

There is no real market for protected 10 gig waves. Occasionally a bank will request the service, but backoff as soon as they see the price tag.

"Hopefully none of these customers had service and protect ckts that went down... I would be pissed as a ceo if that happen to my company. Hopefully level3's new service offering is 100@percent redundant as stated

The new service offerings include: - Protected Wavelengths: Level 3 now provides automatic protection-switching to a dedicated diversely routed wavelength in the event of a network failure. The protection switch, fully automated and managed by Level 3, happens at switching speeds approaching SONET restoration times. The single interface to the customer requires no additional capital cost for customer optical ports, and the diverse restoration path is fixed and fully known to the customer. These features allow customers to achieve fast restoration with predictable performance in their network without adding significant cost and routing complexity. -"

Surely a simple wideband optomechanical switch, actuated by detected signal degradation on a pilot wavelength or wavelengths, would do the job with high reliability and relatively low cost, without any extra need for switching the STM64 signal at the bitstream level? -- Neil

Rod Beck wrote:

And if the 10 gig wave is from 1 Wilshire to 60 Hudson with hundreds of regen huts and 30 POPs in between?

How that affect the capex cost?

Sure, the capex cost of offering full diversity is substantial; my point was just that the cost of switching STM64 signals at the endpiints need not be a significant issue, since you only have to switch the optical path, which is cheap to do and highly reliable, and the kit to do that will only make up a tiny fraction of the rest of the capital and operations cost. -- Neil

On Wed, Apr 15, 2009 at 01:38:43PM +0100, Rod Beck wrote:

There is no known way to provide cheap 10 wave protection. Not carrier grade. Protected 10 GigE service (LAN PHY 10 GigE) will tolerate a very high BER before switching. And the cost of switching STM64 is very high as well.

Bottom line is that it will cost more than two diversely routed 10 gig waves. ... Every span has to be protected.

Hi Rod, I don't think thats true. Most "carrier grade" DWDM platforms deployed over the last few years have been capable of doing protected 10GE LAN PHY service without a SONET/STM layer and without costing more than two diversely routed waves. Also, many of the modern systems in use by modern competetive carriers are capable of providing > 2 degree (ring) protection. They essentially act like an optical "switch", and can automatically seek out (and signal via GMPLS) an available channel to restore or protect the overall path on a dynamic basis, and in more than 2 directions.

There is no real market for protected 10 gig waves. Occasionally a bank will request the service, but backoff as soon as they see the price tag.

I think the pricing is the result of trying to charge what the market will bear rather than an underlying technical cost to deliver service. Think "if the customer wants a want stop solution where we're managing everything for them they should be willing to pay more for the convenience".

"Hopefully none of these customers had service and protect ckts that went down... I would be pissed as a ceo if that happen to my company. Hopefully level3's new service offering is 100@percent redundant as stated

Protected vs 2x diverse unprotected circuits each have their advantages and disadvantages. One thing a protected circuit is not good at is providing higher availability than 2x diverse unprotected circuits. That's because you're trading diversity at the endpoints for simplicity, so you've still done nothing to protect yourself against endpoint failures. Protected circuits may provide other advantages though, such as > 2 degree protection, or better latency than may be reasonably available to purchase independently. It depends on the carrier, the network, and even the customer to figure out which is the better solution.

The new service offerings include: - Protected Wavelengths: Level 3 now provides automatic protection-switching to a dedicated diversely routed wavelength in the event of a network failure. The protection switch, fully automated and managed by Level 3, happens at switching speeds approaching SONET restoration times. The single interface to the customer requires no additional capital cost for customer optical ports, and the diverse restoration path is fixed and fully known to the customer. These features allow customers to achieve fast restoration with predictable performance in their network without adding significant cost and routing complexity. -"

I believe this is what I was talking about above, on their Infinera platform. This is much more powerful than traditional ring designs.

But bear in mind that DWDM infrastructure that does 80 to 120 waves per fiber pair is very expensive.

I suppose expensive is in the eye of the beholder. Every modern long-haul "carrier grade" DWDM platform I know of has done at least 80 channel 50GHz spacing at the same cost as a 40ch solution for quite a few years now. Only in the metro space does the statement above hold true. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

On Wed, Apr 15, 2009 at 1:37 PM, Rod Beck <Rod.Beck@hiberniaatlantic.com>wrote:

Hi Richard,

I never said that protected LAN PHY 10 GigE was more expensive than two diversely routed waves. However, Hibernia's engineers have advised that route protected LAN PHY 10 GigE will tolerate a relatively high BER before switching. I stand by that statement.

I said that protected STM64 service was more expensive and that is true. Not only do you need two diversely STM64 waves, but you need protection as well.

Finally, you're wrong about "trying to charge what the market will bear".

Rod, Unless you are lucky enough to be doing large,cost +, IRU deals all day, supply and demand economics should prevail, right? The minimum a market would bear is based on costs and then supply vs. demand. Best, Marty -- Martin Hannigan martin@theicelandguy.com p: +16178216079 Power, Network, and Costs Consulting for Iceland Datacenters and Occupants

On Wed, Apr 15, 2009 at 06:37:36PM +0100, Rod Beck wrote:

Hi Richard,

I never said that protected LAN PHY 10 GigE was more expensive than two diversely routed waves.

Strange, the e-mail from you that I quoted specifically said:

Bottom line is that it will cost more than two diversely routed 10 gig waves.

But at any rate...

However, Hibernia's engineers have advised that route protected LAN PHY 10 GigE will tolerate a relatively high BER before switching. I stand by that statement.

I said that protected STM64 service was more expensive and that is true. Not only do you need two diversely STM64 waves, but you need protection as well.

Modern DWDM systems don't care about they content of the payload, they use a system called OTN (optical transport networks) as a generic digital wrapper around the payload, and then they deal entirely with the OTN frame. This makes features like optical protection protocol agnostic, and remove any kind of cost difference based on the type of service. I think you're confusing the old style system of implementing a SONET/SDH based ring as a method of delivering protected services, with the modern techniques of delivering 10G or other subrate services as LAN PHY or SONET/SDH or some other protocol. These are completely different things.

I have sold almost 30 ten gig waves (leases) and I have only received one request (global bank) for protected service. When I priced at the twice the price of an unprotected service plus a 10% premium, that request was downsized to a protected STM16.

Well, I DID point out some compelling reasons why one might want to do 2x (or more) diversely routed unprotected wavelengths rather than a protected service. There are many other reasons, such as statistically multiplexed oversubscribtion on multiple unprotected circuits during the normal non-failure state. At any rate, I'm not in a position to explain the logic or motivations of the people who buy waves from you, all I can tell you is how the technology works and what it costs to deploy it. As such, my previous explanation was correct. :)

Customers in general are simply not willing to pay for protection. Indeed, most of them prefer to load balance among diversely routed 10 gig waves or buy waves on several network or cable systems.

All perfectly legitimate reasons why one might want to do multiple diverse unprotected wavelengths, but this is still orthogonal to the assertions that protected wavelengths are not possible, not reliable, or cost more to implement than 2x unprotected waves. Also, keep in mind that the availability of > 2-degree protection on modern DWDM platforms could *easily* result in optically protected wavelengths which are much cheaper to deliver than diversely routed unprotected paths. For example, lets consider the scenerio you previously gave of a 10G wavelength from Chicago to Frankfurt. Using an optical switching protection system, a provider could survive a fiber cut between Chicago and Cleveland in Detroit by wrapping the wavelengths via Chicago Indianapolis Cincinatti Cleveland, before continuing on its way to Frankfurt. This eliminates the need to provision capacity on two completely diverse paths, which may not even exist or which may have extremely poor latency choices, and reduces the cost to deliver the service. As always, the benefits of such a system depend on both the carrier's and the customers' footprints. I suspect you'll start to see more of this in the future, as Level3 seems to be adopting it. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)