March 2003 - Test - lists.nanog.org

Port 80 Issues
by John Murphy 10 Mar '03

10 Mar '03

Anyone seeing any port 80 issues tonight? We're having some issues with our port 80 off-net traffic tonight.

8 10

RE: Question concerning authoritative bodies.
by McBurnett, Jim 09 Mar '03

09 Mar '03

See Comments In-line below.. > So I'm curious what people think. We have semi centralized > various things in > the past such as IP assignments and our beloved DNS root > servers. Would it > not also make sense to handle common security checks in a > similar manner? In > creating an authority to handle this, we cut back on the I would question the validity of this scan.. How easy would it be to put an ACL entry to block the Scan source? > amount of noise > issued. I bring this up because the noise is getting louder. This is almost the cost of being a business... > More and more > networks are issuing their own relay and proxy checks. At > this rate, in a > few years, we'll see more damage done to server resources by > scanners than > we do from spam and those who would exploit such vulnerabilities. Why not establish a system like dshield.org, where companies could reference the database and submit their data. Maybe get the backbones to sponsor, or Dept of Homeland Security. It needs to be global, and probrably should be an IETF / RIR / IANA thought process... Thoughts?? Jim

1 0

Re: 923Mbits/s across the ocean
by Joe St Sauver 09 Mar '03

09 Mar '03

Hi, #High speed at reasonable costs are the end-goal. However, it is important #to be able to plan for when one will need such links, to know what one #will be able to achieve, and for regular users to be ready to use them #when the commonly available. This takes some effort up front to achieve #and demonstrate. And that's the key point that I think folks have been missing so far about all this. Internet2 provides excellent connectivity to folks who generally have a minimum of switched 10Mbps ethernet connectivity, and routinely switched 100Mbps connectivity. However, if you look at the weekly Abilene netflow summary reports (see http://netflow.internet2.edu/ , or jump directly to a particular report such as http://netflow.internet2.edu/weekly/20030224/ ) you will see that for bulk TCP flows, the median throughput is still only 2.3Mbps. 95th%-ile is only ~9Mbps. That's really not all that great, throughput wise, IMHO. Add one further element to that: user expectations. Users hear, "Wow, we now have an OC12 to Abilene [or gigabit ethernet, or an OC48, or an OC192], I'll going to be able to *smoke* my fast ethernet connection ftping files from <insert far away place>!" ... but then they find out that no, in fact, if they are seeing 100Mbps for bulk TCP transfers, then they are in true throughput elite, the upper 1/10th of 1% of all I2 traffic. SO! The I2 Land Speed Record is not necessarily about making everyone be able to do gigabit-class traffic across the pond, it is about making LOTS of faculty be able to do 100Mbps at least across the US. Emprirically, it is clear to me that this "trivial" accomplishment, e.g., getting 100Mbps across the wide area, is actually quite hard, and it is only by folks pushing really hard (as Cotrell and his colleagues have) that the more mundane throughput targets (say, 100Mbps) will routinely be accomplished. Regards, Joe St Sauver (joe(a)oregon.uoregon.edu) University of Oregon Computing Center

3 2

RE: 923Mbits/s across the ocean
by Antony Antony 09 Mar '03

09 Mar '03

Les > The jumbo frames effectively increase the congestion avoidance additive Les > increase of the congestion avoidance phase of TCP by a factor of 6. Les > Thus after a congestion event, that reduces the window by a factor of 2, Jumbo frames helps a lot. Also NAPI in Linux 2.5.x will help a lot. The way most Linux (till 2.4.20) NIC drivers handle TX and RX interrupt moderation and SMP with multiple NICs are not very efficient. That really kill the performance of TCP over Long Fat Networks. Jumbo frames alleviate the problem very much. Poor x86 CPU is swamped with large number of interrupts when TCP send/receive burst of data about the size of congestion window, ~32Mbytes in our case. -antony

1 0

Re: Port 80 Issues
by John Murphy 09 Mar '03

09 Mar '03

Thanks Todd, It looks like AT&T is having an issue with one of their core routers in St. Louis and it is affecting a large number of their optical customers. About 44% of our packets > 516 bytes (that varies from 500 up to 1200) are being dropped. No ETR from the AT&T NOC. ----- Original Message ----- From: "Todd A. Blank" <todd.blank(a)ipoutlet.com> To: <john.murphy(a)charter.net>; <nanog(a)nanog.org> Sent: Saturday, March 08, 2003 10:31 PM Subject: RE: Port 80 Issues > I am seeing it on roadrunner - if that is any help. I am not seeing it > on our network. > > Todd A. Blank > IPOutlet LLC > > > -----Original Message----- > From: John Murphy [mailto:john.murphy@charter.net] > Sent: Saturday, March 08, 2003 11:13 PM > To: nanog(a)nanog.org > Subject: Port 80 Issues > > > Anyone seeing any port 80 issues tonight? We're having some issues with > our > port 80 off-net traffic tonight. > >

2 2

Re: Who uses RADB? [was BGP to doom us all]
by Danny McPherson 08 Mar '03

08 Mar '03

> as you say for customers only. Inter-provider we have basic bogon checking plus > maximum prefix. Its too unwieldy to build when you have peers exchanging > thousands of routes... theres a belief that the peer should be behaving > responsibly tho and this is a condition of most bilateral peering contracts. Unfortunately, contracts don't fix mis-(or malicious-) configurations on compromised routers or from a peers disgruntled worker. > Going back to the original topic on this thread I would expect a deliberate > attack on BGP routing to come from a customer not a provider such as Level3, if > they are filtering in turn to their customers we have a reasonable amount of > sanity checking going on A large provider I worked for in the past had a router maliciously configured to inject a more-specific prefix for a very "popular network". Even the "popular networks" provider sent the traffic to us. Had explicit prefix-based inter- provider filtering been in place it would not have occurred, or at least "the whole Internet" wouldn't have been affected. With the IRRs and inter-provider filtering it's the whole chicken and egg thing. Inter-provider filters aren't in place because no one cares about IRRs (even though they have other operational value as well). Vendors don't support the amount of prefix filters required because they say no one uses them. Heck, lots of folks still don't ingress filter routes (or packets) from their customers. When ANS used to employ inter-provider filters the biggest problem was getting them updated and bouncing routes or sessions. That's no excuse anymore because pretty much everyone supports the ability to incrementally update filters, and BGP Route Refresh fixes the bounce the session/route thing. So, let's recap why no one uses them (as many have said already in the related thread): Laziness. The same laziness that results in the slew of other things many folks have pointed out not being addressed. -danny

11 14

RE: 923Mbits/s across the ocean
by Cottrell, Les 08 Mar '03

08 Mar '03

The jumbo frames effectively increase the congestion avoidance additive increase of the congestion avoidance phase of TCP by a factor of 6. Thus after a congestion event, that reduces the window by a factor of 2, one can recover 6 times as fast. This is very important on large RTT fast links where the recovery rate(for TCP/Reno) goes as the MTU/RTT^2. This can be seen in some of the graphs at: http://www-iepm.slac.stanford.edu/monitoring/bulk/fast/stacks.png or more fully at: http://www-iepm.slac.stanford.edu/monitoring/bulk/fast/ We saw little congestion related packet loss on the testbed. With big windows SACK becomes increasingly important so one does not have recover a large fraction of the window for a single packet. Once one gets onto networks where one is really sharing the bandwidth with others performance drops off rapidly (see for example the measuremsnts at http://www-iepm.slac.stanford.edu/monitoring/bulk/fast/#Measurements%20from… and compare them with those at http://www-iepm.slac.stanford.edu/monitoring/bulk/fast/#TCP%20Stack%20Compa… One of the next things we want to look at next is how the various new TCP stacks work on production Academic & Research Networks (e.g. from Internet2, ESnet, GEANT, ...) with lots of other competing traffic. -----Original Message----- From: Iljitsch van Beijnum [mailto:iljitsch@muada.com] Sent: Saturday, March 08, 2003 1:49 PM To: Cottrell, Les Cc: 'nanog(a)nanog.org' Subject: Re: 923Mbits/s across the ocean On Sat, 8 Mar 2003, Cottrell, Les wrote: > We used a stock TCP (Linux kernel TCP). We did however, use jumbo > frames (9000Byte MTUs). What kind of difference did you see as opposed to standard 1500 byte packets? I did some testing once and things actually ran slightly faster with 1500 byte packets, completely contrary to my expectations... (This was UDP and just 0.003 km rather than 10,000, though.) > The remarks about window size and buffer are interesting also. It is > true large windows are needed. To approach 1Gbits/s we require 40MByte > windows. If this is going to be a problem, then we need to raise > question like this soon and figure out how to address (add more > memory, use other protocols etc.). In practice to approcah 2.5Gbits/s > requires 120MByte windows. So how much packet loss did you see? Even with a few packets in a million lost this would bring your transfer way down and/or you'd need even bigger windows. However, bigger windows mean more congestion. When two of those boxes start pushing traffic at 1 Gbps with a 40 MB window, you'll see 20 MB worth of lost packets due to congestion in a single RTT. A test where the high-bandwidth session or several high-bandwidth sessions have to live side by side with other traffic would be very interesting. If this works well it opens up possibilities of doing this type of application over real networks rather than (virtual) point-to-point links where congestion management isn't an issue.

1 0

923 Mbps across the Ocean ...
by Eric Germann 08 Mar '03

08 Mar '03

http://www.cnn.com/2003/TECH/internet/03/07/speed.record/index.html Comments folks? ========================================================================== Eric Germann CCTec ekgermann(a)cctec.com Van Wert OH 45801 http://www.cctec.com Ph: 419 968 2640 Fax: 603 825 5893 "The fact that there are actually ways of knowing and characterizing the extent of one’s ignorance, while still remaining ignorant, may ultimately be more interesting and useful to people than Yarkovsky" -- Jon Giorgini of NASA’s Jet Propulsion Laboratory

12 11

RE: 923Mbits/s across the ocean
by Cottrell, Les 08 Mar '03

08 Mar '03

>> High speed at reasonable costs are the end-goal. However, it is >> important to be able to plan for when one will need such links, to >> know what one will be able to achieve, and for regular users to be >> ready to use them when the commonly available. This takes some effort >> up front to achieve and demonstrate. >True, however as it was mentioned before, why not do the same type of testing in a lab environment >between a couple >of boxes having the TCP stack insert appropriate delays? When in 1995 we were >getting simplex IP links over >satellites up that is how we did the testing before bringing them up on the birds. Following up on and driven by the work leading up to and following the Land Speed Record, some of the Caltech people collaborating on this record together with collaborators from SLAC and elsewhere, are proposing a "WAN in the Lab" that can be used for just such testing. This saves on leasing fibers but there are still considerable expenses to run at 10Gbit/s rates (cpus, NICs, optical multiplexing equipment etc.). It is also a much more controlled environment that simplifies things. On the other hand it misses out on the real world experience, and so eventually has to be tested first on real world lightly used testbeds, and then on advanced research networks and finally on production networks, to understand how issues such as fairness, congestion avoidance, robustness to poor implemementations or configurations etc. really work.

1 0

RE: 923Mbits/s across the ocean
by Cottrell, Les 08 Mar '03

08 Mar '03

With the glossing over of details that goes with press releases there appears to be a misunderstanding here. I never said we paid list prices. I am well aware that one can get large discounts from vendors. However, I think it is important to quote a well known price (in this case list), which people can relate to how well they think they can negotiate (otherwise it just becomes a bragging point of who can get the largest discount), and gets away from the point of giving people an idea of what it might cost. In our case we got 100% (free) discounts from Level(3) and Cisco for the Sunnyvale to Chicago link and the GSR. The link from StarLight to Amsterdam was put in place for a European funded demonstration (since turned into a production link), the equipment was mainly funded by another European research project. At the same time, getting it for free has its costs, one has much less leverage with the vendors as to delivery (and retrieval) dates, reliability etc. as well as the headaches of getting everything (PCs, loaned NIC cards, Routers, links) to come together, to keep the vendors interest, extend the loan etc. High speed at reasonable costs are the end-goal. However, it is important to be able to plan for when one will need such links, to know what one will be able to achieve, and for regular users to be ready to use them when the commonly available. This takes some effort up front to achieve and demonstrate. -----Original Message----- From: alex(a)yuriev.com [mailto:alex@yuriev.com] Sent: Saturday, March 08, 2003 12:30 PM To: Cottrell, Les Cc: 'nanog(a)nanog.org' Subject: Re: 923Mbits/s across the ocean > You are modest in your budgetary request. Just the Cisco router (GSR > 12406) we had on free loan listed at close to a million dollars, and > the OC192 links just from Sunnyvale to Chicago would have cost what > was left of the million/per month. No, your budget folks have no clue, which they clearly demonstrate. Anyone here who buys Cisco at the list prices works for companies that for some reason want to waste money. We pay about 10c on a dollar. Anyone leasing OC-192 at that price as opposite to lighting it up is smoking. > "What am I missing here, theres OC48=2.4Gb, OC192=10Gb ..." > > We were running host to host (end-to-end) with a single stream with > common off the shelf equipment, there are not too many (I think none) > > 1GE host NICs available today that are in production (e.g. without > signing a non-disclosure agreement). Again, if this is all available today, what is so new that you guys have done, apart from blowing tons of money? > The remarks about window size and buffer are interesting also. It is > true large windows are needed. To approach 1Gbits/s we require 40MByte > windows. If this is going to be a problem, then we need to raise > question like this soon and figure out how to address (add more > memory, use other protocols etc.). In practice to approcah 2.5Gbits/s > requires 120MByte windows. > > I am quite happy to concede that this does not need to be about some > jocks beating a record. I do think it is important to catch the > public's attention to why high speeds are important, that they are > achievable today application to application (it would also be useful > to estimate when such speeds are available to universities, large > companies, small companies, the home etc.), and for techies it is > important to start to understand the challenges the high speeds raise, > e.g. cpu and router memories, bugs in TCP, OS, application etc., new > TCP stacks, new (possibly UDP based) protocols such as tsunami, need > for 64 bit counters in monitoring, effects of the NIC card, jumbo > requirements etc., and what is needed to address them. Also to try and > put it in meaningful terms (such as 2 full length DVD movies in a > minute, that could also increase the "cease and desist" legal messages > shipped ;-)) is important. High speeds are not important. High speeds at a *reasonable* cost are important. What you are describing is a high speed at an *unreasonable* cost. Alex

3 2