- NANOG - lists.nanog.org

RE: What is the limit? (was RE: multi-homing fixes)
by Roeland Meyer 30 Aug '01

30 Aug '01

|> From: Sam Thomas [mailto:sthomas@lart.net] |> Sent: Thursday, August 30, 2001 2:57 AM |> |> On Wed, Aug 29, 2001 at 11:36:03PM -0700, Roeland Meyer wrote: |> > |> > |> From: Randy Bush [mailto:randy@psg.com] |> > |> Sent: Wednesday, August 29, 2001 9:18 PM |> > |> |> > |> > there would appear to be people who assert its not |> > |> > unafordable to compute the routes for the current, |> > |> > and the forseeable network. |> > |> |> > |> and there are those of us who think we can probably do |> > |> so iff we stop the explosion of /24s. |> > |> > Okay, let's get it back out of the realm of opinion and |> > into requirements. |> > I'll even go down one step more, to /25. Take the entire |> > IP addr space, |> > break it ALL into /25's, times the BGP table entry size, |> > how big is that? |> > Also, I just got a price quote for 512MB ECC PC133 DIMMS |> > at $49US, retail. |> using Jonthan Disher's calculation of 155B/route (a check |> of my local neighborhood gsr gave ~185B/route, but this is |> all academic anyway so we'll use his number since it was |> first and so folk playing along at home can have some |> continuity), i get ~5GB for the whole address space as /25s. |> (using 185B/route, it came to just over 8GB). Thank you, it was the route-entry size that I didn't have defensible numbers on. I knew roughly what they were, but was already out on the limb.<g> |> i don't know of any off the shelf consumer hardware that |> supports >2GB (i looked, really!), so the price to stick |> that $800 ($16 is extremely reasonable shipping cost for |> 16 memory sticks, see if you can find it ;) of ram on |> something just went up an order of magnitude. Okay, well I wasn't talking about COTS boards anyway. I was talking about commodity tech, at the chip level. What us old microP silicon mechanics called the "component" level. Circuit boards all cost the same to build and design, regardless of the content of the design, with minor variations. It's ALL automated these days. Feed a schematic in one end and, about a week later, get a circuit board design out the other, with none-to-little human intervention. Manufacturing is about the same. It is the components that attach to them that contains the variable cost. It is those components whose cost is dropping like lemmings off a cliff. You see, chip design and manufacture is almost getting to the same place as circuit boards. The only human-introduced variable is in the original design and architecture. That cost is amortised over the volume of units sold. This is where commodity tech has a huge cost/performance advantage. |> > Considering what router vendors charge, with very fat |> > margins, for their low-tech routers, affordable tech |> > arguments don't hold water. Routers are expensive |> > because someone is stupid enough to pay the prices. |> i guess we're in the wrong business. we should go develop |> discount routers using near-current off-the-shelf |> technology. thinking this through, i'm already amazed at |> the lack of glut for discount high-capacity routers. You haven't been paying attention to the stock market then. Why do you think router vendor stock prices are dropping like flies? Maybe because there *is* a glut of cheap used equipment? This is forcing more sane prices on them and, as a direct result, their earnings reports are down. |> oops. some folk (like me) actually want the core router |> to be able to handle a chassis chock-full-o-oc48s. |> that'll take a bit of specialized hardware. oh, and |> it would be really nice-ah if the router could forward |> ip packets amongst those oc48s at something approaching |> line-rate. that'll take an industrial strength backplane. |> not finding too many multi-Gb/s backplanes on pricewatch.com, |> so guess that will have to be custom built, and we'll |> probably have to develop some custom circuitry to handle the |> custom backplane as well. I'll bet that COGm, for a Cisco Catalyst 6509, with all GigE blades, is under $5KUS. Further, that the major cost is the sheet-metal and paint work, not the electronix. In the last three years, I emplaced 6 of them at ~$200K each. Please tell me that isn't a very fat margin and yes, I do know about keystone pricing and how the distro channel works. Even R&D isn't a major cost, given enough volume. Manufacturing has been commoditized for over 20 years. Are you telling me that Cisco can't give us better tech, for less money, than they've been doing thus far? |> shoot, forgot software. we've run up the cost so much, we'll |> just drop in a peecee with a linux kernel and zebra. we |> won't be able to route packets through it, though, so i |> guess we'll have to custom manufacture some other |> device to handle the actual packing shuffling. or, we |> could custom develop a single unit and some software that |> runs on it. i'm sure there are plenty of |> bgp/mpls/multicast/etc clued programmers willing to work |> in our software sweatshop for minimum wage to keep costs low. see, Richard Stallman and the FSF. I'll save my thoughts on open-source "free" software for another day. Please know that MHSC is a Caldera VAR (still dealing with Caldera assilimating the "enemy", SCO), and we can sell you a copy of SCO OpenUNIX these days (but not cheap <sigh>). Yes, I've had GateD running for some time now. Software cost is a part of R&D costs. |> so, after all this, how big of a bargain is using that $800 |> in ram so far? All the additional conditions should make ZERO difference to the COGs of the product. Are you telling me that 8GB worth of table entries is a small market? |> ok, it's still probably at least a few bucks cheaper than |> vendor X. let's go sell it. hah! silly customers want a |> number to call when it breaks. well, it uses cheap |> off-the-shelf memory and processor, how much can it really |> cost to support? You know ... in the past 10 years, with many hundreds of servers, I have had ZERO CPU failures and ZERO RAM failures, past the infant mortality period. Even in workstations, all such failures were caused by brilliant actions, like restricting the airflow (wrapping the darned thing in a blanket, once), even then, the HDDs were the first to die. With zero moving parts, if you can keep them cool, they run forever. They obsolece long before they die. I have one old server, an amdK6-233, whose RAM (128MB of SIMMs) has been in 24x7 operation for over 6 years. It's also running a AHA2940 that has seen 24x7 operation since 1996. |> since we used commodity hardware (at least ram and |> processor), we won't have to stock spares. i'm sure |> we can trust our customers to buy the right parts |> off the shelf at "Bubba's Fine House of PeeCee Hardware". You can kid around if you want to. But, you're only showing irrational bias. An Intel part is an Intel part, regardless of whether Bubba or Einstein sells it to you. Likewise with RAM. If the board don't take standard PC133 RAM then the design is defective. Bubba doesn't make the parts. He only sells them. Unless, Bubba has a silicon foundry in his basement. In that case, his name probably isn't Bubba. |> now, would everyone kindly get off the $EXPENSIVE ram from |> vendor X whine? Those who discount, discounts, wind up broke, before those that don't. |> > royally ripped off there. Cisco takes special RAM because |> > Cisco designed incompatible RAM circuitry so they could |> > charge you more for the Cisco label. It also bolsters |> > the FUD-storm and the after-market support vendors. |> |> or, maybe it's because they had the foresight to know that |> if they used commodity ram, a bunch of cheapskate morons |> would be plugging in the cheapest ram they could find at |> the saturday morning swapmeet, and making dozens of |> "warranty" support calls because the router crashes all the |> time or half the packets come through looking like |> swiss cheese. Maybe the alternative was to hire competent circuit design engineers that could design circuits that don't have those vulnerabilities? In my book, the moron was the fool that didn't design with those cheapskates in mind.

1 0

Re: What is the limit? (was RE: multi-homing fixes)
by Vadim Antonov 30 Aug '01

30 Aug '01

Actually, talking about water/electricity interaction in refrigerators, lab equipment, etc, misses the very simple point - a probability of a leak is proportional (as a good approximation :) to the number of moveable components (PCBs, connectors, etc) in the system. In a typical CO it's tens or hundreds of thousands. --vadim PS. Water is not a good coolant, and even distilled and deionized water tends to pick ions from metal parts rather quickly. Spirits are flammable; CFCs are bad for environment. There's also an issue of toxicity. Of course, everything is doable, but what is the cost? Off-the-shelf components are all designed for air cooling. Switching to liquid cooling means a lot of custom stuff; which is expensive and which takes a long time to design and manufacture. PPS It is not voltage which matters, it's current :) Even if leaks do not cause shorts the moisture accumulation may corrode parts leading to mechanical shorts, it also may cause excessive cross-talk. Sporadically malfunctioning equipment is much worse than flat-out burned out. PPS Finally, getting your hands wet makes your changes to get killed by electricity _much_ higher. 48V won't do you any harm if your hands are dry, it may kill if they're wet.

1 0

Re: FIBER CUT: Dallas to West Coast
by Sean Donelan 30 Aug '01

30 Aug '01

On Wed, 29 August 2001, Patrick Greenwell wrote: > > The difference is the carrier's response, in particular how well they > > keep their customers informed. > > That is certainly *a* point of differentiation, however if the goal of > these "basement dual-homers" is to not suffer downtime due to the > outage of a single provider(much like the organizations that "matter"), > all the responsiveness in the world from a provider whose circuit to > one of the "basement dual-homers" which has failed isn't going to prevent > them from being down, is it? I don't know what the goal of "basement dual-homers" is. I think that is the other thread, which I haven't been keeping up with. It is possible to reduce the impact of most provider problems. I had a seven year streak without a service affecting outage at my NOC longer than the BGP route convergence interval. Even "hot-standby, fail-over" systems require a finite time to recover. Even with all the network insanity between 1993 and 2000, I was always able to reach the working portion of the Internet. If my own connection didn't work, how could I post about those problems. It was a lot of hard work, and I'll admit I had a very steep learning curve. And I'll be the first to admit, there were force maejure events my network wouldn't be able to handle. But those were my choices, and I could make those decisions with accurate information. The first step of operating a reliable network is choosing carriers willing to keep you, their customer, fully informed. Only after you have accurate information from the carriers can you make informed decisions concerning your network. Most of my biggest problems occurred due to inaccurate or lack of information from a carrier. If I know about the risk, I could choose to mitigate it or choose to do nothing. If the carrier doesn't keep me informed about jeopardy conditions in their network, managing that risk is more difficult. A side-effect of the lack of information is the extrodinary measures customers feel they must take to protect their network connectivity. Do you really need to multi-home, have your own AS number, have your own CIDR block for reliable network connectivity? With DHCP, you could just unplug from one provider and plug into the new one, and auto-magically have connectivity. With Dynamic DNS update you could automagically redirect your DNS name to your new address. IP addresses could be assigned geographically, instead of by provider, and aggregation could take place anywhere from the census tract level to the hemisphere level.

2 1

ot - new edge
by Steven J. Sobol 30 Aug '01

30 Aug '01

Looking for people who have experience reselling bandwidth from New Edge Networks.... replies offlist please. THanks -- JustThe.net LLC - Steve "Web Dude" Sobol, CTO - sjsobol(a)JustThe.net Donate a portion of your monthly ISP bill to your favorite charity or non-profit organization! E-mail me for details.

1 0

BGP Instability
by Mark Borchers 30 Aug '01

30 Aug '01

Currently seeing unusually high numbers of routes being dampened (~11K). The common denominator seems to be AS 4637 in the as-path of most of the dampened prefixes. Is any further information known? Thanks, -- Mark Borchers McLeodUSA Data Services IP Engineering 9012 New Trails Dr. mborchers(a)splitrock.net The Woodlands, TX 77381 (281) 465-1931 http://www.mcleod.net/

2 1

Liquid cooling... (was RE: What is the limit? (was RE: multi-homi ng fixes))
by James Smith 30 Aug '01

30 Aug '01

Actually, the fluid referenced in the URL (Fluorinert) is used to cool some Crays. It is only manufactured by one maker, and only a few systems use it. There are actual liquid cooled Cray parts in the NSA museum near Ft Meade. Non-operational, of course, just like this post... James H. Smith II NNCDS NNCSE Systems Engineer The Presidio Corporation -----Original Message----- From: Charles Sprickman [mailto:spork@inch.com] Sent: Thursday, August 30, 2001 3:45 PM To: Alex Rubenstein Cc: Daniel Senie; nanog(a)merit.edu Subject: Re: What is the limit? (was RE: multi-homing fixes) On Wed, 29 Aug 2001, Alex Rubenstein wrote: > I recall something about liquid (something, could be nitrogen, or perhaps > even mercury) that was chilled; PC Boards were then submersed in the > liquid to keep cool. > > Maybe on Crays? Maybe in someone's garage? :) http://www.octools.com/articles/submersion/submersion12.html C > -- Alex Rubenstein, AR97, K2AHR, alex(a)nac.net, latency, Al Reuben -- > -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net -- > >

1 0

Re: What is the limit? (was RE: multi-homing fixes)
by Daniel Senie 30 Aug '01

30 Aug '01

At 11:10 PM 8/29/01, Vadim Antonov wrote: >On Wed, 29 Aug 2001, Andrew Partan wrote: > > > I have proposed to various router vendors the possibility of giving > > them a chilled water feed instead of lots of cool air. At the > > moment they seem to not need it, but I would not be surprized to > > find something like this needed at some point. > >Err. Water and electricvity make a dangerous mix. And this was not a problem in IBM Mainframe computers because? I'm not registering an opinion one way or the other at this point on whether routers should consider other forms of cooling, but using water or other liquids to cool electronics is not a new concept. Properly engineered, there is no particular danger. ----------------------------------------------------------------- Daniel Senie dts(a)senie.com Amaranth Networks Inc. http://www.amaranth.com

7 6

Re: FIBER CUT: Dallas to West Coast
by Sean Donelan 30 Aug '01

30 Aug '01

On Wed, 29 August 2001, Patrick Greenwell wrote: > On Wed, 29 Aug 2001, Brian Whalen wrote: > > Routing around the problem, what a concept. More backup for the 2xt1 shop > > to contimue doing it.. > > I'm still awaiting the list of providers that never have a circuit go > down. :-) Every carrier has had a circuit go down. The difference is the carrier's response, in particular how well they keep their customers informed. On the other hand, there is a significant statistical difference between major carriers of service affecting cable cuts and the mean time to repair. Which might be somewhat of a surprise because the physical cuts are more or less evenly distributed among the carriers, on a pro-rata basis. Among "major" carriers (i.e. with more than 20,000 route miles active) Sprint has the fewest service affecting cable cuts AT&T is the best at keeping customer's informed

5 7

Re: What is the limit? (was RE: multi-homing fixes)
by Joe Loiacono 30 Aug '01

30 Aug '01

APNICs approach (FYI): Hi James, Thanks for your mail and questions. Answers in-line below. At 07:08 PM 29/08/2001 +1000, James Spenceley wrote: >http://www.apnic.net/meetings/12/docs/proposal-multihome-assign.html > >Are companies with existing historically assigned address space eligible for >additional space via the small multi-homing assignment policy ? Anyone is eligible. >Should there be a limit of 1 of these assignments made per 'entity', with >renumbering occurring if further address space is required ? For clarification, the current proposal did not specify any limitation on the number of such assignments. What would be the motivation for this? >If so should we consider reserving the next larger block for a period of >time, to account for possible growth ? No reservations will be made. It is generally assumed that applicants will be paying a one-off service fee, rather than an ongoing membership based fee. This is in fact identical to the way that PI assignments can currently be obtained. The proposal restricts the assignments to (about to be) multihomed orgs. >If entity has a 'small multi-homing' assignment and later joins as a member >and qualifies for a /20 PA allocation, are they required to renumber from >the multi-homing assignment ? Yes. The assumption would be that the organisation whose network has grown, would then take the shorter portable prefix would then announce that prefix instead of the longer one. Maybe this should be made more explicit in the policy? >Should 'small multi-homing' assignments be made from a specific (defined) >netblock ? I think there is a choice here. We can use "swamp" space, found mostly in 202/8 which by definition does not contain large ranges of address space, or we can take a range from the less "swampy" space ie. 218/8 and use that. My feeling is that it would be better to use the 202/8 range. >There is significant risk in assigning greater than /20 space from existing >APNIC allocated blocks given that operators often use RIR minium allocation >guides for filtering, if the minimum assignment/allocation for all current >APNIC blocks becomes /23 or /24 this may encourage the deaggregation of >existing PA blocks. I'm not sure I follow the logic. As Geoff pointed out, there is already significant de-aggregation. Despite this, in de-aggregating you still run the increased risk of exposure to more severe route flap dampening parameters as well as filters. The objective in filtering is to limit the size of the routing table, so I dont see this policy encouraging ISPs to adjusting their filters to allow longer prefixes, if anything, rather the other way round. regards Anne APNIC --- * APNIC-TALK: General APNIC Discussion List * * To unsubscribe: send "unsubscribe" to apnic-talk-request(a)apnic.net *

2 1

Re: What is the limit? (was RE: multi-homing fixes)
by Larry Sheldon 30 Aug '01

30 Aug '01

> > I have proposed to various router vendors the possibility of giving > > them a chilled water feed instead of lots of cool air. At the > > moment they seem to not need it, but I would not be surprized to > > find something like this needed at some point. > > Err. Water and electricvity make a dangerous mix. That sounds a bit FUDish--lots of electricity is generated using water (most of it is, actually, given that steam is part of the cycle even in most nukes). And we have used water-cooled CPU's for a lot of years. (And as a point of interest, distilled water at neutral ph is a reasonably good insulator, if I recall correctly.)

3 2