Troubleshooting TCP performance tutorial
Greetings, This past week I have been trying to find the root cause of tcp performance problems of a few clients that are using a third party metro Ethernet for transport. RFC2544 tests (Layer 2) and iperf using UDP give good symmetric performance almost 100% the speed of the circuit. However all kind of TCP tests result in some kind of asymmetrical deficiency, either the upstream or downstream of the client is hugely different. The latency is not a huge factor since all the metro Ethernet connections have less than 2 ms. So the question basically if is there a good tutorial or white paper for troubleshooting tcp with emphasis of using tools like Wireshark to debug and track this kind of problems. Regards, Abel.
In a situation like yours I found Internet Core Protocols: The Definitive Guide by Eric Hall an easy to read guide to insuring that what you are seeing via wireshark. I was able to find an issue with the DF bit in a load balancer that was causing confounding headaches in a network using wireshark and this book. Walk it through the syn-ack dance and don't trust that the devices are handling it correctly. Start at one end and work your way through and insure to YOUR satisfaction that every device proscribes to the protocol. Don't rush, don't jump to conclusions. Just follow the packet. That's the best advice I can give you. http://oreilly.com/catalog/9781565925724/ -- Joe Hamelin, W7COM, Tulalip, WA, 360-474-7474 On Fri, Sep 17, 2010 at 5:06 PM, Abel Alejandro <aalejandro@worldnetpr.com> wrote:
Greetings,
This past week I have been trying to find the root cause of tcp performance problems of a few clients that are using a third party metro Ethernet for transport. RFC2544 tests (Layer 2) and iperf using UDP give good symmetric performance almost 100% the speed of the circuit. However all kind of TCP tests result in some kind of asymmetrical deficiency, either the upstream or downstream of the client is hugely different. The latency is not a huge factor since all the metro Ethernet connections have less than 2 ms.
So the question basically if is there a good tutorial or white paper for troubleshooting tcp with emphasis of using tools like Wireshark to debug and track this kind of problems.
Regards, Abel.
To add on to that. Recently Wireshark Network Analysis was released. It's an excellent book covering wireshark and reading packet captures in general by Laura Chappell. I just finished reading it and I have to say it's an excellent book. Highly recommended. Between those two books I think you'll be very close to being a wireshark/packet capture guru. I hope this helps, -Tim Eberhard On Fri, Sep 17, 2010 at 7:33 PM, Joe Hamelin <joe@nethead.com> wrote:
In a situation like yours I found Internet Core Protocols: The Definitive Guide by Eric Hall an easy to read guide to insuring that what you are seeing via wireshark. I was able to find an issue with the DF bit in a load balancer that was causing confounding headaches in a network using wireshark and this book.
Walk it through the syn-ack dance and don't trust that the devices are handling it correctly. Start at one end and work your way through and insure to YOUR satisfaction that every device proscribes to the protocol. Don't rush, don't jump to conclusions. Just follow the packet. That's the best advice I can give you.
http://oreilly.com/catalog/9781565925724/ -- Joe Hamelin, W7COM, Tulalip, WA, 360-474-7474
On Fri, Sep 17, 2010 at 5:06 PM, Abel Alejandro <aalejandro@worldnetpr.com> wrote:
Greetings,
This past week I have been trying to find the root cause of tcp performance problems of a few clients that are using a third party metro Ethernet for transport. RFC2544 tests (Layer 2) and iperf using UDP give good symmetric performance almost 100% the speed of the circuit. However all kind of TCP tests result in some kind of asymmetrical deficiency, either the upstream or downstream of the client is hugely different. The latency is not a huge factor since all the metro Ethernet connections have less than 2 ms.
So the question basically if is there a good tutorial or white paper for troubleshooting tcp with emphasis of using tools like Wireshark to debug and track this kind of problems.
Regards, Abel.
http://www.amazon.com/Wireshark-Network-Analysis-Official-Certified/dp/18939... Spendy but looks good. I'll have to pick it up when the next consulting check comes in. Thanks! I was sad to see that Eric Hall's book was out of print. At least cheap used copies are available. I forgot my copy a few jobs ago... I'm sure someone is getting help from it. -- Joe Hamelin, W7COM, Tulalip, WA, 360-474-7474 On Fri, Sep 17, 2010 at 6:00 PM, Tim Eberhard <xmin0s@gmail.com> wrote:
To add on to that. Recently Wireshark Network Analysis was released. It's an excellent book covering wireshark and reading packet captures in general by Laura Chappell. I just finished reading it and I have to say it's an excellent book. Highly recommended.
Between those two books I think you'll be very close to being a wireshark/packet capture guru.
I hope this helps, -Tim Eberhard
On Fri, Sep 17, 2010 at 7:33 PM, Joe Hamelin <joe@nethead.com> wrote:
In a situation like yours I found Internet Core Protocols: The Definitive Guide by Eric Hall an easy to read guide to insuring that what you are seeing via wireshark. I was able to find an issue with the DF bit in a load balancer that was causing confounding headaches in a network using wireshark and this book.
Walk it through the syn-ack dance and don't trust that the devices are handling it correctly. Start at one end and work your way through and insure to YOUR satisfaction that every device proscribes to the protocol. Don't rush, don't jump to conclusions. Just follow the packet. That's the best advice I can give you.
http://oreilly.com/catalog/9781565925724/ -- Joe Hamelin, W7COM, Tulalip, WA, 360-474-7474
On Fri, Sep 17, 2010 at 5:06 PM, Abel Alejandro <aalejandro@worldnetpr.com> wrote:
Greetings,
This past week I have been trying to find the root cause of tcp performance problems of a few clients that are using a third party metro Ethernet for transport. RFC2544 tests (Layer 2) and iperf using UDP give good symmetric performance almost 100% the speed of the circuit. However all kind of TCP tests result in some kind of asymmetrical deficiency, either the upstream or downstream of the client is hugely different. The latency is not a huge factor since all the metro Ethernet connections have less than 2 ms.
So the question basically if is there a good tutorial or white paper for troubleshooting tcp with emphasis of using tools like Wireshark to debug and track this kind of problems.
Regards, Abel.
Date: Fri, 17 Sep 2010 20:06:09 -0400 From: "Abel Alejandro" <aalejandro@worldnetpr.com>
Greetings,
This past week I have been trying to find the root cause of tcp performance problems of a few clients that are using a third party metro Ethernet for transport. RFC2544 tests (Layer 2) and iperf using UDP give good symmetric performance almost 100% the speed of the circuit. However all kind of TCP tests result in some kind of asymmetrical deficiency, either the upstream or downstream of the client is hugely different. The latency is not a huge factor since all the metro Ethernet connections have less than 2 ms.
So the question basically if is there a good tutorial or white paper for troubleshooting tcp with emphasis of using tools like Wireshark to debug and track this kind of problems.
Regards, Abel.
You might look at http://fasterdata.es.net. A lot of it is aimed at very large volume data transfers, but quite a bit is relevant to all TCP issues. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751
On Saturday, September 18, 2010, Kevin Oberman <oberman@es.net> wrote:
You might look at http://fasterdata.es.net. A lot of it is aimed at very large volume data transfers, but quite a bit is relevant to all TCP issues. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751
+1 fasterdata.es.net. Excellent resource. -brandon -- Brandon Galbraith US Voice: 630.492.0464
Abel Alejandro wrote:
Greetings,
This past week I have been trying to find the root cause of tcp performance problems of a few clients that are using a third party metro Ethernet for transport. RFC2544 tests (Layer 2) and iperf using UDP give good symmetric performance almost 100% the speed of the circuit. However all kind of TCP tests result in some kind of asymmetrical deficiency, either the upstream or downstream of the client is hugely different. The latency is not a huge factor since all the metro Ethernet connections have less than 2 ms.
So the question basically if is there a good tutorial or white paper for troubleshooting tcp with emphasis of using tools like Wireshark to debug and track this kind of problems.
Regards, Abel.
It might be worth your while to run the analysis found here: http://netalyzr.icsi.berkeley.edu/index.html
participants (6)
-
Abel Alejandro -
Brandon Galbraith -
Joe Hamelin -
Kevin Oberman -
Michael Painter -
Tim Eberhard