On Fri, 3 Aug 2007, Robert Boyle wrote:

At 02:17 AM 8/3/2007, you wrote:

...

Hi,, group

I need some help.

Which equipment is better ( perfomance, availability, scalability, features, Support, and Price ($$$) ) ???

Some experience in the real life ????

Dependent on your interface needs, if GigE, 10G, (40G & 100G in the future) and POS are all you need, include the Foundry XMR in your eval too. Very solid software and excellent support at a price point which is significantly lower than C & J. I don't know the pricing for H.

Any experiences of Foundry routing w/ more complex protocols (PIM, MSDP, various IPv6 stuff)? The last time we tried running non-C/J as a router was a very Extreme experience and we swore never again to touch similar router underdogs in the future. -- Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

Can anyone give a breakdown of the different kinds of content deliver networks? For example, we have Akamai, which appears to be a pure Layer 3 network that is tailored to pushing relatively small files like web pages and we have Lime Light Networks, which is a mix of Layer 1 and Layer 3, that focuses on bigger files like video streams. Any insights out there? And what are the major challenges in making scalable content delivery networks? Roderick S. Beck Director of EMEA Sales Hibernia Atlantic 1, Passage du Chantier, 75012 Paris http://www.hiberniaatlantic.com Wireless: 1-212-444-8829. Landline: 33-1-4346-3209 AOL Messenger: GlobalBandwidth rod.beck@hiberniaatlantic.com rodbeck@erols.com ``Unthinking respect for authority is the greatest enemy of truth.'' Albert Einstein. -----Original Message----- From: owner-nanog@merit.edu on behalf of Jason J. W. Williams Sent: Fri 8/3/2007 10:32 PM To: Pekka Savola; Robert Boyle Cc: ALEJANDRO ESQUIVEL RODRIGUEZ; nanog@merit.edu Subject: RE: Cisco CRS-1 vs Juniper 1600 vs Huawei NE5000E We're Juniper right now, but we're looking at the Foundry MLX line for possible future sites due to cost/performance. So I'd be interested in folks' experience with Foundry's Terathon gear and associated IronWare revs. Its supposed to be a lot better than the JetCore stuff (cam-trashing problems etc.) but it'd be nice to hear what folks are seeing in real life. Best Regards, Jason -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Pekka Savola Sent: Friday, August 03, 2007 3:07 PM To: Robert Boyle Cc: ALEJANDRO ESQUIVEL RODRIGUEZ; nanog@merit.edu Subject: Re: Cisco CRS-1 vs Juniper 1600 vs Huawei NE5000E On Fri, 3 Aug 2007, Robert Boyle wrote:

At 02:17 AM 8/3/2007, you wrote:

...

Hi,, group

I need some help.

Which equipment is better ( perfomance, availability, scalability, features, Support, and Price ($$$) ) ???

Some experience in the real life ????

Dependent on your interface needs, if GigE, 10G, (40G & 100G in the future) and POS are all you need, include the Foundry XMR in your eval too. Very solid software and excellent support at a price point which is significantly lower than C & J. I don't know the pricing for H.

Any experiences of Foundry routing w/ more complex protocols (PIM, MSDP, various IPv6 stuff)? The last time we tried running non-C/J as a router was a very Extreme experience and we swore never again to touch similar router underdogs in the future. -- Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings !SIG:46b39bc6156532946815078! This e-mail and any attachments thereto is intended only for use by the addressee(s) named herein and may be proprietary and/or legally privileged. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this email, and any attachments thereto, without the prior written permission of the sender is strictly prohibited. If you receive this e-mail in error, please immediately telephone or e-mail the sender and permanently delete the original copy and any copy of this e-mail, and any printout thereof. All documents, contracts or agreements referred or attached to this e-mail are SUBJECT TO CONTRACT. The contents of an attachment to this e-mail may contain software viruses that could damage your own computer system. While Hibernia Atlantic has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage that you sustain as a result of software viruses. You should carry out your own virus checks before opening any attachment

Content Delivery NetworksRod, I run a small CDN oriented to audio/video distribution in Central Europe region. You mentioned challenges that CDN are facing. There are several of these: 1) Geographic load distribution - in example, you have to have enough capacity in each distribution area that is being potentionally served by your customers (media companies). 2) Computing power - you have to have in each POP adequate computing power that allows you to use full bandwidth available in that POP 3) Network capacity in each POP - self explaining 4) Content moving - you have to optimise internal data flows between POPs 5) User redirection - You have to implement a scalable mechanisms that redirects users to the closes POP. You can use application redirect (fast, but not so much scalable), DNS redirect (scalable, but not so fast) or anycasting (this needs cooperation with ISP). 6) Costs of the system - Most CDNs are designed for peaking traffic, but load is more dynamic that in traditional networks. 7) Play alone There is almost no way for handover traffic to other CDN. No standards, no kinf of interconnection agreement. These points are only a rough overview, this ecosystem is developping and challenges also depend on the starting conditions and region where you are running you CDN. Regards Michal ----- Original Message ----- From: Rod Beck To: Jason J. W. Williams ; Pekka Savola ; Robert Boyle Cc: ALEJANDRO ESQUIVEL RODRIGUEZ ; nanog@merit.edu Sent: Monday, August 06, 2007 23:10 Subject: Content Delivery Networks Can anyone give a breakdown of the different kinds of content deliver networks? For example, we have Akamai, which appears to be a pure Layer 3 network that is tailored to pushing relatively small files like web pages and we have Lime Light Networks, which is a mix of Layer 1 and Layer 3, that focuses on bigger files like video streams. Any insights out there? And what are the major challenges in making scalable content delivery networks? Roderick S. Beck Director of EMEA Sales Hibernia Atlantic 1, Passage du Chantier, 75012 Paris http://www.hiberniaatlantic.com Wireless: 1-212-444-8829. Landline: 33-1-4346-3209 AOL Messenger: GlobalBandwidth rod.beck@hiberniaatlantic.com rodbeck@erols.com ``Unthinking respect for authority is the greatest enemy of truth.'' Albert Einstein.

Hi Patrick,

...

5) User redirection - You have to implement a scalable mechanisms that redirects users to the closes POP. You can use application redirect (fast, but not so much scalable), DNS redirect (scalable, but not so fast) or anycasting (this needs cooperation with ISP).

What is slow about handing back different answers to the same query via DNS, especially when they are pre-calculated? Seems very fast to me.

Yes DNS-based redirection scales very pretty. But there are two problems: 1) Client may not be in same network as DNS server (I'm using my home DNS server even if I'm at IETF or I2 meeting on other side of globe) 2) DNS TTL makes realtime traffic management inpossible. Remember you may not distribute network traffic, but sometimes also server load. If one server/POP fails or is overloaded, you need to redirect users to another one in realtime.

Application redirection is far, far slower. (I am assuming you are talking about something like HTTP level redirects. Did you mean something else?)

Yes, it is slower, but in some scenarios (streaming hours of content for thousands of spectators) works nicely. Currently I'm using both mechanisms. Typically I'm using application level redirets and when I know there will be significant peak (e.g. ten times more than average), I'll provide DNS FQDN :-)

As for anycast, with your own backbone, you don't need any cooperation. Even if you don't, the cooperation you need from your providers & peers is minimal at worst. (At least relative to writing the code for, say, DNS redirection.) But anycast assumes "BGP knows best", and we all know that is not even close to the truth.

I'm definitelly not talking about situation when you are running your own backbone. Anycast in general works for small transactions (root DNS fits perfectly), but Internet is "living beast" and if you provide live stream that lasts two hours, one flipping BGP session makes the content almost unusable for part of your users. Regards Michal

...

...

...

5) User redirection - You have to implement a scalable mechanisms that redirects users to the closes POP. You can use application redirect (fast, but not so much scalable), DNS redirect (scalable, but not so fast) or anycasting (this needs cooperation with ISP).

What is slow about handing back different answers to the same query via DNS, especially when they are pre-calculated? Seems very fast to me.

Yes DNS-based redirection scales very pretty.

But there are two problems: 1) Client may not be in same network as DNS server (I'm using my home DNS server even if I'm at IETF or I2 meeting on other side of globe)

This has been discussed. Operational experience posted here by Owen shows < 10% of users are "far" from their recursive NS.

Sure, but 10% of 5 Gb/s is 500 Mb/s. In my streaming scenario. I respect CDN for HTTP delivery has probably other experience. Also I'm using housing contracts for "deliver only to ISP users" and use no transit connectivity of housing ISPs (frankly - this is much cheaper).

You are the tiny minority. (Don't feel bad, so am I. :) Most "users" either use the NS handed out by their local DHCP server, or they are VPN'ing anyway.

10% is tiny minority, but in real world with real costs, this minority can squeeze my profit :-)

...

2) DNS TTL makes realtime traffic management inpossible. Remember you may not distribute network traffic, but sometimes also server load. If one server/POP fails or is overloaded, you need to redirect users to another one in realtime.

Define "real time"? To do it in 1 second or less is nigh impossible. But I challenge you to fail anything over in 1 second when IP communication with end users not on your LAN is involved.

I've seen TTLs as low as 20s, giving you a mean fail-over time of 10 seconds. That's more than fast enough for most applications these days.

I've tested (year ago) real scenario and got very disappointing feedback. It seemed that some corporate gateways here don't respect zone TTL. I'm so far to recommend my solutions to the community. I think that every CND provider has to choose its own solution that fits it's own services. Regards MK

On Aug 9, 2007, at 10:55 PM, Paul Reubens wrote:

How do you engineer around enterprise and ISP recursors that don't honor TTL, instead caching DNS records for a week or more?

In my "little" bit of research and experience over the last 10 years in this field, I have often pursued this "urban myth". It remains largely just that. The most common supposed violator of this was AOL. I found myself in a position at one stage to get to the "root" of this, and was rather impressed to find that it was indeed a myth. We've just finished a small research project where we looked at approximately 16 million recursive servers. The only ones violating this were some CPE devices that ran local recursive services, and they were generally along the lines of returning the appropriate TTL the first time they were queried, and if the TTL was zero, they returned a higher TTL (10000 seconds) to subsequent queries for a short period (5 minutes). It may have been a code bug, or a designed behavior given that these were CPE devices. Do you have any real examples of significant recursive servers doing this?

Very interesting. We've all heard and probably all passed along that little bromide at one time or another. Is it possible that at one time it was true (even possibly for AOL) but with the rise of CDNs, policies of not honoring TTL's have fallen by the wayside?

I think you'll still see it in spam zombies, some of which have the DNS info pre-loaded into them in order to avoid split-horizon anti-spam techniques. Not much we can do about that until we get sufficient backbone to deal with the zombie problem and its software enablers. Regards, John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies", Information Superhighwayman wanna-be, http://www.johnlevine.com, ex-Mayor "More Wiener schnitzel, please", said Tom, revealingly.

...

...

On 8/10/2007 at 11:55 AM, "Patrick W. Gilmore" <patrick@ianai.net> wrote:

On Aug 10, 2007, at 12:46 PM, John Levine wrote:

...

...

Very interesting. We've all heard and probably all passed along that little bromide at one time or another. Is it possible that at one time it was true (even possibly for AOL) but with the rise of CDNs, policies of not

...

...

honoring TTL's have fallen by the wayside?

I think you'll still see it in spam zombies, some of which have the

...

DNS info pre-loaded into them in order to avoid split-horizon anti-spam techniques.

Not much we can do about that until we get sufficient backbone to deal with the zombie problem and its software enablers.

Actually, I think the fact Zombies do not honor TTLs is a feature. :)

Fast-flux my MX records to avoid spam? Throw the spammers' technology back at 'em! I changed some MX records in mid-July for a domain. Spam was still flowing into the old MX hosts until I closed the firewall 25/tcp holes just today. Now just logging those zombies still banging on the gates. -- Crist J. Clark crist.clark@globalstar.com Globalstar Communications (408) 933-4387 B¼information contained in this e-mail message is confidential, intended only for the use of the individual or entity named above. If the reader of this e-mail is not the intended recipient, or the employee or agent responsible to deliver it to the intended recipient, you are hereby notified that any review, dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please contact postmaster@globalstar.com

On Aug 13, 2007, at 2:25 AM, Florian Weimer wrote:

* Rodney Joffe:

...

Do you have any real examples of significant recursive servers doing this?

nscd in GNU libc has issues related to cache expiry. I'm not sure if it is general brokenness, or some TTL-related issue. It's use is not terribly widespread, and it's a host-specific cache only, but there's a certain installation base.

Thanks Florian. So this looks like a code "feature", not stupid behavior by deployers. I'll keep a note when we fingerprint misbehaving systems in the future. /rlj

-- Florian Weimer <fweimer@bfk.de> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99

"Chris L. Morrow" <christopher.morrow@verizonbusiness.com> writes:

that sets a lower-bar on TTL in the nscd cache -

(from the manpage for nscd.con)

positive-time-to-live cachename value Sets the time-to-live for positive entries (successful queries) in the specified cache. value is in integer seconds. Larger values increase cache hit rates and reduce mean response times, but increase problems with cache coherence. Note that sites that push (update) NIS maps nightly can set the value to be the equivalent of 12 hours or more with very good perfor- mance implications.

This is still a client issue as, hopefully, the cache-resolvers don't funnel their business through nscd save when applications on them need lookups... (things like ping/telnet/traceroute/blah)

nscd may represent a problem if the application in question is a http-proxy without it's own resolver. There's also a number of more-or-less broken http-proxies doing their own resolver caching regardless of actual TTL. Such applications represent a problem wrt any DNS-based load balancing, including CDNs, since they can serve a large number of end-users, redirecting them to the "wrong" address long after the TTL should have expired. Bjørn

On Aug 10, 2007, at 1:55 AM, Paul Reubens wrote:

How do you engineer around enterprise and ISP recursors that don't honor TTL, instead caching DNS records for a week or more?

A friend of mine was working for a place that performed some service on data (not important what, you send them some data (through this really ugly client app that they wrote in-house) and they sent you back something...). Anyway, for various reasons they needed to move out of their current data-center to a new provider. They had this truly monumental plan for doing this that they had been working on for months --- MS Project printouts that covered entire walls in this huge rainbow of colors, 400 or so pages of plans, etc etc etc -- it all boiled down to: Decrease the TTL, then swap in the new A record at midnight on Friday. As soon as the TTL expired everything would start working in the new place and it will all be transparent to the end users... Anyway, my friend calls me at like 3 in the morning on Saturday -- they have updated DNS and none of their clients are connecting to the new place... It seems that they have burnt some bridges with the old provider and will be shut off on Saturday evening -- he's really desperate, so I agree to wander over and take a look... I arrive to find utter confusion -- the CEO is screaming at the CTO, who appears to have decided that the best way to fix things is by getting drunk, random other people are screaming (apparently just for fun), etc.... I manage to get someone to calm down for long enough to explain the summary of the plan to me and run nslookup.. Sure enough the TTL is really low and the new IP is being handed out, etc. I ask how long it took for the client to fail over during their tests -- "Oh, no, we didn't test like that, we didn't want to impact the current service, so we tested with a different domain and checked how long it took for a IE to pick up the change... It was less than 10 minutes..." We track down one of the developers and talk to him. He explains this long and involved system with the client performing heath-checks on the server and reconnecting wit exponential back-off, etc etc etc. Its all great -- apart from the fact that he calls gethostbyname() during startup, and then never again.... This is a *really* common issue.... W

On 8/7/07, Patrick W.Gilmore <patrick@ianai.net> wrote: On Aug 7, 2007, at 10:05 AM, Michal Krsek wrote:

...

...

...

5) User redirection - You have to implement a scalable mechanisms that redirects users to the closes POP. You can use application redirect (fast, but not so much scalable), DNS redirect (scalable, but not so fast) or anycasting (this needs cooperation with ISP).

What is slow about handing back different answers to the same query via DNS, especially when they are pre-calculated? Seems very fast to me.

Yes DNS-based redirection scales very pretty.

But there are two problems: 1) Client may not be in same network as DNS server (I'm using my home DNS server even if I'm at IETF or I2 meeting on other side of globe)

This has been discussed. Operational experience posted here by Owen shows < 10% of users are "far" from their recursive NS.

You are the tiny minority. (Don't feel bad, so am I. :) Most "users" either use the NS handed out by their local DHCP server, or they are VPN'ing anyway.

...

2) DNS TTL makes realtime traffic management inpossible. Remember you may not distribute network traffic, but sometimes also server load. If one server/POP fails or is overloaded, you need to redirect users to another one in realtime.

Define "real time"? To do it in 1 second or less is nigh impossible. But I challenge you to fail anything over in 1 second when IP communication with end users not on your LAN is involved.

I've seen TTLs as low as 20s, giving you a mean fail-over time of 10 seconds. That's more than fast enough for most applications these days.

-- TTFN, patrick

On Fri, Aug 03, 2007 at 07:47:44PM -0300, Giuliano (UOL) wrote:

It has excellent performance under MPLS, BGP and Multicast Networks.

But a CLI/config as modern as a grammophone. If only they would copy JunOS instead of IOS... sigh.

But ... I never saw it under extreme conditions with IPv6 ...

They already fail at light conditions, given that there is no multitopology IS-IS. This equals to "showstopper" if your network uses multitopo IS-IS for v4+v6 (and perhaps even for unicast and multicast). Best regards, Daniel -- CLUE-RIPE -- Jabber: dr@cluenet.de -- dr@IRCnet -- PGP: 0xA85C8AA0

Date: Fri, 03 Aug 2007 19:47:44 -0300 From: "Giuliano (UOL)" <giulianocm@uol.com.br> Subject: Re: Cisco CRS-1 vs Juniper 1600 vs Huawei NE5000E

You can use Foundry XMR box. It has excellent performance under MPLS, BGP and Multicast Networks. But ... I never saw it under extreme conditions with IPv6 ...

If you need sane MTU controls on both L2 and L3 stay *very* *far* away from Foundry gear... Despite years and years of telling them they need to allow different MTU settings both at the VLAN as on the VE level they still Don't Get It (TM)... :( And I definitely know I'm not the only one who's repeatedly asked them about it. :( Apart from that, if you need basic IPv4 stuff, aka not too fancy terribly new things, they have a very decent platform, with far lower port costs then C or J. And performance is also very good, especially since you get L2 and L3, whereas with J you'd need to go with the (very new) MX960, whose L2 featureset still eludes me, or the proven 6509's (with beefy sup's) from C... Kind regards, JP Velders