Analysing traces for performance bottlenecks
Hi, Are there any packages (or Wireshark options that I've missed) that can follow a TCP stream and determine the limiting factor on throughput. E.g Latency, packet loss, out of sequence packets, window size, or even just the senders rate onto the wire. I know how to analyse a trace by hand for performance issues, but it's relatively time consuming. Googling for variations on "Analyse TCP stream limit throughput" didn't find anything. Sam
One potentially useful piece of software that is a work in progress is called Pcapdiff. (http://www.eff.org/testyourisp/pcapdiff/) Written by Seth Schoen and Steven Lucy it's a pretty useful piece of software. While still in a relative infant stage I think it could mature into a very useful tool to troubleshoot network connectivity. Pcapdiff was originally written to find out if your ISP was toying with you P2P packets (comcast) and injecting resets. I have worked with the authors a bit and found it highly useful to take two packet captures on my network and use it to verify A) All packets sent are recieved B) They are in their original state. Again the features of pcapdiff are pretty limited but I love the idea of the program and I really think it could grow into an excellent tool to analyze packet captures with just a few additional features (a few listed below) -Tim Eberhard On Tue, Jul 15, 2008 at 5:05 AM, Sam Stickland < sam_mailinglists@spacething.org> wrote:
Hi,
Are there any packages (or Wireshark options that I've missed) that can follow a TCP stream and determine the limiting factor on throughput. E.g Latency, packet loss, out of sequence packets, window size, or even just the senders rate onto the wire. I know how to analyse a trace by hand for performance issues, but it's relatively time consuming.
Googling for variations on "Analyse TCP stream limit throughput" didn't find anything.
Sam
A bit more googling has found the Web100 projects NDT (http://e2epi.internet2.edu/ndt/). I'm currently making a Linux VM that can run it. It's useful, but I'm still really after something that can do it's type of analysis from a packet capture. Sam Sam Stickland wrote:
Hi,
Are there any packages (or Wireshark options that I've missed) that can follow a TCP stream and determine the limiting factor on throughput. E.g Latency, packet loss, out of sequence packets, window size, or even just the senders rate onto the wire. I know how to analyse a trace by hand for performance issues, but it's relatively time consuming.
Googling for variations on "Analyse TCP stream limit throughput" didn't find anything.
Sam
Date: Tue, 15 Jul 2008 11:05:34 +0100 From: Sam Stickland <sam_mailinglists@spacething.org>
Hi,
Are there any packages (or Wireshark options that I've missed) that can follow a TCP stream and determine the limiting factor on throughput. E.g Latency, packet loss, out of sequence packets, window size, or even just the senders rate onto the wire. I know how to analyse a trace by hand for performance issues, but it's relatively time consuming.
Googling for variations on "Analyse TCP stream limit throughput" didn't find anything.
tcptrace is old and pretty basic, but it can provide a LOT if information. Combined with xplot, the graphs often point to the exact nature of a TCP problem, but you need a really good understanding of TCP to figure anything out. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751
Kevin Oberman <oberman <at> es.net> writes:
tcptrace is old and pretty basic, but it can provide a LOT if information. Combined with xplot, the graphs often point to the exact nature of a TCP problem, but you need a really good understanding of TCP to figure anything out.
Wireshark also provides tcptrace-like graphs ("Statistics -> TCP Stream Graph -> Time Sequence Graph (tcptrace)"). They're not quite as pretty, but are just as effective at tracking down all sorts of TCP problems, provided, as Kevin said, you have a really good understanding of how TCP behaves.
Matt Cable wrote:
Kevin Oberman <oberman <at> es.net> writes
tcptrace is old and pretty basic, but it can provide a LOT if information. Combined with xplot, the graphs often point to the exact nature of a TCP problem, but you need a really good understanding of TCP to figure anything out.
Wireshark also provides tcptrace-like graphs ("Statistics -> TCP Stream Graph -> Time Sequence Graph (tcptrace)"). They're not quite as pretty, but are just as effective at tracking down all sorts of TCP problems, provided, as Kevin said, you have a really good understanding of how TCP behaves
Thanks for all the replies so far. While the TCP graphs are useful they are very difficult to read in Wireshark - they really need to be displayed in xplot, but this requires an X11 setup? I've found NDT: http://e2epi.internet2.edu/ndt/ This uses a java applet hosted on a web100 patched linux server to record network diagnostics from connecting clients. A typical report might look like this: Web100 reports the Round trip time = 122.15 msec; the Packet size = 1260 Bytes; and No packet loss was observed. C2S throughput test: Packet queuing detected: 1.09% S2C throughput test: Packet queuing detected: 1.32% This connection is receiver limited 84.33% of the time. Increasing the the client's receive buffer (63.0 KB) will improve performance This connection is sender limited 1.70% of the time. Increasing the NDT server's send buffer (127.0 KB) will improve performance This connection is network limited 13.96% of the time. The theoretical network limit is 7869.69 Mbps The NDT server has a 127.0 KByte buffer which limits the throughput to 16.37 Mbps Your PC/Workstation has a 63.0 KByte buffer which limits the throughput to 4.09 Mbps The network based flow control limits the throughput to 8.73 Mbps Client Data reports link is 'OC-48', Client Acks report link is 'OC-12' Server Data reports link is 'OC-48', Server Acks report link is 'T3' Something that could provide a similar, automated analysis of a TCP stream capture is what I'm after, although I doubt a standard packet capture will be able to provided as many metric as web100 stack can. Sam
There is a Java version of xplot available now called jPlot. It works in largely the same way. http://www.tcptrace.org/jPlot/ Regards, Tim -----Original Message----- From: Sam Stickland [mailto:sam_mailinglists@spacething.org] Sent: Thursday, July 17, 2008 11:53 AM To: Matt Cable Cc: nanog@nanog.org Subject: Re: Analysing traces for performance bottlenecks Matt Cable wrote:
Kevin Oberman <oberman <at> es.net> writes
tcptrace is old and pretty basic, but it can provide a LOT if information. Combined with xplot, the graphs often point to the exact nature of a TCP problem, but you need a really good understanding of TCP to figure anything out.
Wireshark also provides tcptrace-like graphs ("Statistics -> TCP Stream Graph -> Time Sequence Graph (tcptrace)"). They're not quite as pretty, but are just as effective at tracking down all sorts of TCP problems, provided, as Kevin said, you have a really good understanding of how TCP behaves
Thanks for all the replies so far. While the TCP graphs are useful they are very difficult to read in Wireshark - they really need to be displayed in xplot, but this requires an X11 setup? I've found NDT: http://e2epi.internet2.edu/ndt/ This uses a java applet hosted on a web100 patched linux server to record network diagnostics from connecting clients. A typical report might look like this: Web100 reports the Round trip time = 122.15 msec; the Packet size = 1260 Bytes; and No packet loss was observed. C2S throughput test: Packet queuing detected: 1.09% S2C throughput test: Packet queuing detected: 1.32% This connection is receiver limited 84.33% of the time. Increasing the the client's receive buffer (63.0 KB) will improve performance This connection is sender limited 1.70% of the time. Increasing the NDT server's send buffer (127.0 KB) will improve performance This connection is network limited 13.96% of the time. The theoretical network limit is 7869.69 Mbps The NDT server has a 127.0 KByte buffer which limits the throughput to 16.37 Mbps Your PC/Workstation has a 63.0 KByte buffer which limits the throughput to 4.09 Mbps The network based flow control limits the throughput to 8.73 Mbps Client Data reports link is 'OC-48', Client Acks report link is 'OC-12' Server Data reports link is 'OC-48', Server Acks report link is 'T3' Something that could provide a similar, automated analysis of a TCP stream capture is what I'm after, although I doubt a standard packet capture will be able to provided as many metric as web100 stack can. Sam ***************************************************************** This message has originated from R. L. Polk & Co., 26955 Northwestern Highway, Southfield, MI 48033. R. L. Polk & Co. sends various types of email communications. If this email message concerns the potential licensing of a Polk product or service, and you do not wish to receive further emails regarding Polk products, forward this email to Do_Not_Send@polk.com with the word "remove" in the subject line. The email and any files transmitted with it are confidential and intended solely for the individual or entity to whom they are addressed. If you have received this email in error, please delete this message and notify the Polk System Administrator at postmaster@polk.com. *****************************************************************
What about gnuplot? Maybe it provides something more than xplot? http://www.gnuplot.info/ -- Tim Sanderson, network administrator tims@donet.com -----Original Message----- From: Bulger, Tim [mailto:Tim_Bulger@polk.com] Sent: Thursday, July 17, 2008 12:14 PM To: Sam Stickland; Matt Cable Cc: nanog@nanog.org Subject: RE: Analysing traces for performance bottlenecks There is a Java version of xplot available now called jPlot. It works in largely the same way. http://www.tcptrace.org/jPlot/ Regards, Tim -----Original Message----- From: Sam Stickland [mailto:sam_mailinglists@spacething.org] Sent: Thursday, July 17, 2008 11:53 AM To: Matt Cable Cc: nanog@nanog.org Subject: Re: Analysing traces for performance bottlenecks Matt Cable wrote:
Kevin Oberman <oberman <at> es.net> writes
tcptrace is old and pretty basic, but it can provide a LOT if information. Combined with xplot, the graphs often point to the exact nature of a TCP problem, but you need a really good understanding of TCP to figure anything out.
Wireshark also provides tcptrace-like graphs ("Statistics -> TCP Stream Graph -> Time Sequence Graph (tcptrace)"). They're not quite as pretty, but are just as effective at tracking down all sorts of TCP problems, provided, as Kevin said, you have a really good understanding of how TCP behaves
Thanks for all the replies so far. While the TCP graphs are useful they are very difficult to read in Wireshark - they really need to be displayed in xplot, but this requires an X11 setup? I've found NDT: http://e2epi.internet2.edu/ndt/ This uses a java applet hosted on a web100 patched linux server to record network diagnostics from connecting clients. A typical report might look like this: Web100 reports the Round trip time = 122.15 msec; the Packet size = 1260 Bytes; and No packet loss was observed. C2S throughput test: Packet queuing detected: 1.09% S2C throughput test: Packet queuing detected: 1.32% This connection is receiver limited 84.33% of the time. Increasing the the client's receive buffer (63.0 KB) will improve performance This connection is sender limited 1.70% of the time. Increasing the NDT server's send buffer (127.0 KB) will improve performance This connection is network limited 13.96% of the time. The theoretical network limit is 7869.69 Mbps The NDT server has a 127.0 KByte buffer which limits the throughput to 16.37 Mbps Your PC/Workstation has a 63.0 KByte buffer which limits the throughput to 4.09 Mbps The network based flow control limits the throughput to 8.73 Mbps Client Data reports link is 'OC-48', Client Acks report link is 'OC-12' Server Data reports link is 'OC-48', Server Acks report link is 'T3' Something that could provide a similar, automated analysis of a TCP stream capture is what I'm after, although I doubt a standard packet capture will be able to provided as many metric as web100 stack can. Sam ***************************************************************** This message has originated from R. L. Polk & Co., 26955 Northwestern Highway, Southfield, MI 48033. R. L. Polk & Co. sends various types of email communications. If this email message concerns the potential licensing of a Polk product or service, and you do not wish to receive further emails regarding Polk products, forward this email to Do_Not_Send@polk.com with the word "remove" in the subject line. The email and any files transmitted with it are confidential and intended solely for the individual or entity to whom they are addressed. If you have received this email in error, please delete this message and notify the Polk System Administrator at postmaster@polk.com. *****************************************************************
Tim Sanderson wrote:
What about gnuplot? Maybe it provides something more than xplot? http://www.gnuplot.info/
On Thu, 17 Jul 2008, Sam Stickland wrote:
Something that could provide a similar, automated analysis of a TCP stream capture is what I'm after, although I doubt a standard packet capture will be able to provided as many metric as web100 stack can.
There are several similar tools designed for ISP customer care centers to help analyze network problems reported by customers. Customer calls with complaint, agent clicks on tool and asks customer to try what is broken, tool analyzes traffic and guesses what's wrong. CC agent then reads from the script for fixing problem X. They tend to be expensive and are designed for point-click call center agents. The back-end analysis some of these systems attempt is impressive, even if the output is overly simplistic. The one I'm most familar with was Adlex <http://www.adlex.com/> which has been purchased by CompuWare. But there are several others.
participants (8)
-
Bulger, Tim -
Kevin Oberman -
Matt Cable -
Randy Bush -
Sam Stickland -
Sean Donelan -
Tim Eberhard -
Tim Sanderson