This is the exact issue. I can only get between 5-7 Mbps. So the question is really what incremental performance gain do WAN accelerators/optimizers offer? Can registry/OS tweaks really make a significant difference because so far with all the "speed enhancements" I have deployed to the registry based on the some of the aforementioned sites I have seen no improvement. I guess I am looking for a product that as a wrapper can multiplex a single socket connection. Philip ----- Original Message ---- From: Robert Boyle <robert@tellurian.com> To: Philip Lavine <source_route@yahoo.com>; nanog <nanog@merit.edu> Sent: Tuesday, March 27, 2007 2:04:20 PM Subject: Re: TCP and WAN issue At 04:26 PM 3/27/2007, Philip Lavine wrote:
I have an east coast and west coast data center connected with a DS3. I am running into issues with streaming data via TCP and was wondering besides hardware acceleration, is there any options at increasing throughput and maximizing the bandwidth? How can I overcome the TCP stack limitations inherent in Windows (registry tweaks seem to not functions too well)?
You will have problems obtaining anything more than 5-7Mbit/s based on 1500 byte Ethernet packets and a RTT latency of 70-90ms. You can increase your window size or use Jumbo Ethernet frames. Almost all GigE gear supports jumbo frames. I'm not sure of your application, but without OS tweaks, each stream is limited to 5-7Mbit/s. You can open multiple streams between the same two hosts or you can use multiple hosts to transfer your data. You can utilize the entire DS3, but not without OS TCP stack tweaks or a move to jumbo frames. You can also use UDP or another connectionless packet method to move the data between sites. Good luck. -Robert Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 "Well done is better than well said." - Benjamin Franklin ____________________________________________________________________________________ Finding fabulous fares is fun. Let Yahoo! FareChase search your favorite travel sites to find flight and hotel bargains. http://farechase.yahoo.com/promo-generic-14795097
On Mar 27, 2007, at 2:41 PM, Philip Lavine wrote:
This is the exact issue. I can only get between 5-7 Mbps. So the question is really what incremental performance gain do WAN accelerators/optimizers offer?
I don't know if you'd get much of a performance benefit from this approach. Bandwidth savings, possibly, depending upon your application. We have a box called a WAAS which is a WAN optimizer, so do several other vendors (search online for 'wan optimizer' or 'wan optimization', you should get a lot of hits), but I have no experience with these types of boxes.
Can registry/OS tweaks really make a significant difference because so far with all the "speed enhancements" I have deployed to the registry based on the some of the aforementioned sites I have seen no improvement.
I'm not a Windows person, so I don't know the answer to this; I know you can do a fair amount of optimization with other OSes, depending upon the OS and your NICs. The MTU, MSS, window-size stuff mentioned previously all applies, as do jumbo frames, if your end-stations and network infrastructure support them. What you want to see is large packets, as large as your end-to-end infrastructure can support.
I guess I am looking for a product that as a wrapper can multiplex a single socket connection.
Your application should be able to do that, potentially, and as other folks mentioned, your app can potentially be tweaked to open up multiple connections. I think there are also NICs which do something of this sort, but it's not something I've personally used (maybe others have experiences they can relate?). My general advice would be to look at all the things mentioned previously you can potentially do with your existing OSes, network infrastructure, and apps, and do them prior to looking at specialized boxes. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // 408.527.6376 voice Words that come from a machine have no soul. -- Duong Van Ngo
What you want to see is large packets, as large as your end-to-end infrastructure can support.
Personally, I would prefer to see more people fixing the infrastructure rather than accepting it as a limit. Install some Linux servers even if all they do is run an application layer proxy to turn the TCP stream from the LAN into something better suited for the WAN. They could use UDP, compress the stream, consolidate small MTU packets into big MTU packets, etc. I remember many years ago when a dentist asked me to help with an urgent problem on his billing system. The system worked fine but he split from his wife and had changed the corporate name of his dental practice. He needed to produce bills under the new name, but the software provider had gone out of business a couple of years earlier and there was no source code. Fortunately, it was a UNIX based application that used the print spooler for billing reports so I just inserted a shell script to change "Pleasant Valley Dental Inc." to "Dr. Smith Dental (1987) Inc." or some such. Fixing the infrastructure may also mean switching IP network providers to get someone who supports bulk data transfers between data centers with jumbo frames end-to-end.
Your application should be able to do that, potentially, and as other folks mentioned, your app can potentially be tweaked to open up multiple connections. I think there are also NICs which do something of this sort, but it's not something I've personally used (maybe others have experiences they can relate?).
Tweaking apps generally turns out to be heavy-duty stuff with lots of release control and testing. Also, the applications programmers generally have a poor understanding of network issues. If you can separate the applications stuff from the data transfer stuff, and tackle the network issues first, then you will have an easier time of it. --Michael Dillon
On Mar 27, 2007, at 3:24 PM, <michael.dillon@bt.com> wrote:
Personally, I would prefer to see more people fixing the infrastructure rather than accepting it as a limit.
Concur - what I meant is, 'can support when fully optimized'. ;>
Tweaking apps generally turns out to be heavy-duty stuff with lots of release control and testing. Also, the applications programmers generally have a poor understanding of network issues. If you can separate the applications stuff from the data transfer stuff, and tackle the network issues first, then you will have an easier time of it.
Concur - app-tweaking should be the penultimate approach (and then, maybe, look at boxes, if there's an issue which can't be resolved the other ways; but my guess is that doing the BCPs should yield good results). ----------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // 408.527.6376 voice Words that come from a machine have no soul. -- Duong Van Ngo
On Tue, Mar 27, 2007, Philip Lavine wrote:
This is the exact issue. I can only get between 5-7 Mbps. So the question is really what incremental performance gain do WAN accelerators/optimizers offer? Can registry/OS tweaks really make a significant difference because so far with all the "speed enhancements" I have deployed to the registry based on the some of the aforementioned sites I have seen no improvement.
I guess I am looking for a product that as a wrapper can multiplex a single socket connection.
The "WAN optimisers" (yuk, I do have to deal with them in my current work) offer some kind of TCP acceleration and flow control - the most advanced I've seen here is Packeteer. What they (claim to) do is a few things, notably: * spoof the TCP connection at both ends over high-latency links so the TCP stacks at either end think they're talking to something with low RTT/loss and thus open up their window sizes right up w/ less touchy expoential backoff; * then massage the TCP delay and window sizes to make things play happy inside the given pipe size. * Then insert all kinds of protocol spoofing to deal with WAN link speed/RTT to fix performance of "LAN only" protocols (such as pretending to be SMB end-to-end I believe) - this isn't a new thing btw, I seem to remember seeing routers in the 90's spoof various bits of IPX/SPX over ISDN to get a speedup and prevent spurious re-dial just to see if the other end is there.. Now, this stuff seems to work fine in the situations I'm using them for (low bandwidth links, lots of users end-to-end, TCP normally chokes rather easily) but they do allow things like windows boxes to get full speed out of satellite links. This, of course, makes 0% difference to the throughput of my well-tuned Linux Squid web proxies at either end; but it does mean the proxies don't end up using -all- of the bandwidth of the link as they want to do. So! * if you're after "fill this pipe please" and you don't require the super-fine application throttling/shaping/acceleration, then do whatever QoS you require on your DS3 (if any!) to ensure you're not causing your high-priority traffic to get tail-dropped in the "dumb" fashion and causing backoff, and * you don't need jumbo frames to get full DS3 speed over 70ms; just bump up tx/rx buffer sizes and maximum window sizes and re-test. * realise that the WAN accelerators make their money doing TCP optimisation because its easier to buy a box and put it in then push out group policies to windows desktops to fix their default TCP settings. :) * and finally, make sure the rest of your network isn't introducing issues - stuff like bung cables/drivers/duplex settings can cause packetloss which becomes much nastier over higher latency links (backoff/retry over LAN happens much faster than your 70ms cross-country link.) Adrian
In a message written on Tue, Mar 27, 2007 at 02:41:13PM -0700, Philip Lavine wrote:
This is the exact issue. I can only get between 5-7 Mbps. So the question is really what incremental performance gain do WAN accelerators/optimizers offer? Can registry/OS tweaks really make a significant difference because so far with all the "speed enhancements" I have deployed to the registry based on the some of the aforementioned sites I have seen no improvement.
In short, yes. Working for an ISP with colos on both coasts I helped customers on numerous occasions "tune" their operating system. A plain jane desktop from choose your favorite vendor today will do ~300-~500Mbps/sec of FTP with only minor tuning assuming you have good disks (e.g. not a laptop). Most servers can be easily tuned to chew a full gigabit. TCP Window Scale, Selective Acknowledgement, and a "TCP Send" and "TCP Receive" buffers that are big enough to handle your bandwidth*delay product (*2 for a good margin) are all that's required. Jumbo frames are not, and in fact make little difference. All of the TCP limits are 100% the same with Jumbo frames, and most NIC's generate roughly the same number of Interrupts with Jumbo frames enabled. You're saving a little bit of IP processing overhead on the end hosts, but I doubt you'll measure it. Wan optimizers are extremely complex points of failure that generally mess with the protocol in ways that are extremely dangerous, they should only be used if you don't have direct access to the end boxes to fix them. Google can turn up 50 write ups on how to tune your settings better than I could ever write here. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
On 3/27/2007 8:44 PM, Leo Bicknell wrote:
Jumbo frames are not [required], and in fact make little difference.
They are actually worse in some applications, since they have such a high propagation delay. Instead of streaming 6 segments spread across a 300 usec time window, you send a single segment that still requires 300 usec... not always with happy results. -- Eric A. Hall http://www.ehsco.com/ Internet Core Protocols http://www.oreilly.com/catalog/coreprot/
participants (6)
-
Adrian Chadd -
Eric A. Hall -
Leo Bicknell -
michael.dillon＠bt.com -
Philip Lavine -
Roland Dobbins