Re: 10Gb iPerf kit?
I have not tried doing that myself, but the only thing that would even be possible that I know of is thunderbolt. A new MacBook Pro and one of these maybe: http://www.sonnettech.com/product/echoexpresssel_10gbeadapter.html -Randy ----- On Nov 10, 2014, at 7:26 PM, Daniel Rohan drohan@gmail.com wrote:
We're looking for a semi-portable solution to validate 10Gb customer circuits and hitting walls surrounding PCI lanes and the amount of data laptops can push via their busses. We'd prefer to not have techs lugging around server equipment for these tests.
Anyone out there testing 10gbE with iPerf? If so, what are you using?
Thanks,
Dan
why doesn't a tbird do this for you? On Mon, Nov 10, 2014 at 7:35 PM, Randy Carpenter <rcarpen@network1.net> wrote:
I have not tried doing that myself, but the only thing that would even be possible that I know of is thunderbolt.
A new MacBook Pro and one of these maybe: http://www.sonnettech.com/product/echoexpresssel_10gbeadapter.html
-Randy
----- On Nov 10, 2014, at 7:26 PM, Daniel Rohan drohan@gmail.com wrote:
We're looking for a semi-portable solution to validate 10Gb customer circuits and hitting walls surrounding PCI lanes and the amount of data laptops can push via their busses. We'd prefer to not have techs lugging around server equipment for these tests.
Anyone out there testing 10gbE with iPerf? If so, what are you using?
Thanks,
Dan
I gotta wonder. How reliable is iPerf over something like RFC2544 or Y.1564? Especially at those speeds? I just picked up a couple of Accedian’s RFC2544/Y.1564 boxes to use as loopbacks to our field Exfos. We’ll probably wind up buying a few more Accedian boxes for the field where we don’t need to spend the money on an Exfo. One of the Accedian boxes is arguably less than what you’d pay for a TB MBP and that Sonnet adapter.
On Nov 10, 2014, at 7:38 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
why doesn't a tbird do this for you?
On Mon, Nov 10, 2014 at 7:35 PM, Randy Carpenter <rcarpen@network1.net> wrote:
I have not tried doing that myself, but the only thing that would even be possible that I know of is thunderbolt.
A new MacBook Pro and one of these maybe: http://www.sonnettech.com/product/echoexpresssel_10gbeadapter.html
-Randy
----- On Nov 10, 2014, at 7:26 PM, Daniel Rohan drohan@gmail.com wrote:
We're looking for a semi-portable solution to validate 10Gb customer circuits and hitting walls surrounding PCI lanes and the amount of data laptops can push via their busses. We'd prefer to not have techs lugging around server equipment for these tests.
Anyone out there testing 10gbE with iPerf? If so, what are you using?
Thanks,
Dan
On 11/11/14 00:49, Jason Lixfeld wrote:
I gotta wonder. How reliable is iPerf over something like RFC2544 or Y.1564? Especially at those speeds?
Apples and oranges? iperf tests a TCP connection (over v4 or v6) and 2544 runs Ethernet tests. Yes, both test throughput, but in very different ways. Arguably you only need one or the other -- but you work with what you have.
Anyone out there testing 10gbE with iPerf? If so, what are you using?
As you say, server hardware is heavy to lug around -- but you can fit quite a lot of CPU in a Mini-ITX/uATX board these days, and newer boards even come with PCI-E 3.0 capable chipsets (Intel Z87). Given the right NIC (Intel, if I was buying one) and a recent Linux distro... I'd be surprised if a recent quad core CPU couldn't generate 10G based on what I've seen our servers do. -- Tom
On 11/11/2014, at 1:35 PM, Randy Carpenter <rcarpen@network1.net> wrote:
I have not tried doing that myself, but the only thing that would even be possible that I know of is thunderbolt. A new MacBook Pro and one of these maybe: http://www.sonnettech.com/product/echoexpresssel_10gbeadapter.html
Or one of these ones for dual-10Gbit links (one for out of band management or internet?): http://www.sonnettech.com/product/twin10g.html I haven't tried one myself, but they're relatively cheap (for 10gig) so not that much outlay to grab one and try it (esp if you already have an Apple laptop you can test with). I've done loads of 1Gbit testing using the entry-level MacBook Air and a Thunderbolt Gigabit Ethernet adapter though, and I disagree with Saku's statement of 'You cannot use UDPSocket like iperf does, it just does not work, you are lucky if you reliably test 1Gbps'. I find iperf testing at 1Gbit on Mac Air with Thunderbolt Eth extremely reliable (always 950+mbit/sec TCP on a good network, and easy to push right to the 1gbit limit with UDP. Pete
On (2014-12-07 09:24 +1300), Pete Mundy wrote: Hey,
I've done loads of 1Gbit testing using the entry-level MacBook Air and a Thunderbolt Gigabit Ethernet adapter though, and I disagree with Saku's statement of 'You cannot use UDPSocket like iperf does, it just does not work, you are lucky if you reliably test 1Gbps'. I find iperf testing at 1Gbit on Mac Air with Thunderbolt Eth extremely reliable (always 950+mbit/sec TCP on a good network, and easy to push right to the 1gbit limit with UDP.
In my experience majority of people using iperf in UDP mode do not monitor for packet loss. And once they start doing, they notice they can't go very fast. For me 1 packet loss due to host issues is absolutely too much,I need flat 0, and in optimal scenario, I'd get microsecond resolution jitter statistics. Granted I've not tried on OSX, perhaps by default it has deeper buffers for UDP. But on Linux, top of the shelf Cisco UCS server with 10GE interfaces running RHEL, and 1Gbps is simply too much, you'll get packet loss. You can observe the packets arriving, but they'll be registered as UDP errors on 'netstat -s', because your program isn't picking them up fast enough. Things iperf could do to perform better - setsockopt for deeper buffers - recvmmsg to pick up multiple messages with single context switch - raw sockets to reduce overhead But ultimately you're still going to be very far from 10Gbps, which is doable, if you'll use something like DPDK. -- ++ytti
I find nuttcp very useful in those situations. Be sure to use one of the recent betas, I have been using 7.2.1 for UDP with excellent results (decent loss stats and jitter calc) http://nuttcp.net/nuttcp/beta/nuttcp-7.2.1.c As I understand it, it's still developed, 7.3.2 is now out. M On 7 Dec 2014 16:49, "Saku Ytti" <saku@ytti.fi> wrote:
On (2014-12-07 09:24 +1300), Pete Mundy wrote:
Hey,
I've done loads of 1Gbit testing using the entry-level MacBook Air and a Thunderbolt Gigabit Ethernet adapter though, and I disagree with Saku's statement of 'You cannot use UDPSocket like iperf does, it just does not work, you are lucky if you reliably test 1Gbps'. I find iperf testing at 1Gbit on Mac Air with Thunderbolt Eth extremely reliable (always 950+mbit/sec TCP on a good network, and easy to push right to the 1gbit limit with UDP.
In my experience majority of people using iperf in UDP mode do not monitor for packet loss. And once they start doing, they notice they can't go very fast. For me 1 packet loss due to host issues is absolutely too much,I need flat 0, and in optimal scenario, I'd get microsecond resolution jitter statistics.
Granted I've not tried on OSX, perhaps by default it has deeper buffers for UDP. But on Linux, top of the shelf Cisco UCS server with 10GE interfaces running RHEL, and 1Gbps is simply too much, you'll get packet loss. You can observe the packets arriving, but they'll be registered as UDP errors on 'netstat -s', because your program isn't picking them up fast enough.
Things iperf could do to perform better - setsockopt for deeper buffers - recvmmsg to pick up multiple messages with single context switch - raw sockets to reduce overhead
But ultimately you're still going to be very far from 10Gbps, which is doable, if you'll use something like DPDK.
-- ++ytti
On 06/12/2014 20:24, Pete Mundy wrote:
I've done loads of 1Gbit testing using the entry-level MacBook Air and a Thunderbolt Gigabit Ethernet adapter though, and I disagree with Saku's statement of 'You cannot use UDPSocket like iperf does, it just does not work, you are lucky if you reliably test 1Gbps'. I find iperf testing at 1Gbit on Mac Air with Thunderbolt Eth extremely reliable (always 950+mbit/sec TCP on a good network, and easy to push right to the 1gbit limit with UDP.
i've found that the tb gigabit ethernet adapter starts dropping udp packets at around 600-650mbit/sec on iperf (mbp, MC976xx/A model). Nick
From: pete@fiberphone.co.nz Subject: Re: 10Gb iPerf kit? Date: Sun, 7 Dec 2014 09:24:41 +1300 To: nanog@nanog.org
On 11/11/2014, at 1:35 PM, Randy Carpenter <rcarpen@network1.net> wrote:
I have not tried doing that myself, but the only thing that would even be possible that I know of is thunderbolt. A new MacBook Pro and one of these maybe: http://www.sonnettech.com/product/echoexpresssel_10gbeadapter.html
Or one of these ones for dual-10Gbit links (one for out of band management or internet?):
http://www.sonnettech.com/product/twin10g.html
I haven't tried one myself, but they're relatively cheap (for 10gig) so not that much outlay to grab one and try it (esp if you already have an Apple laptop you can test with).
How would you use it? with iperf still?I don't think you will go nearly close to 14.8Mpps per port this way.Unless you are talking about bandwidth testing with full sized packet frames and low pps rate. I personally tested a 1Gbit/s port over a MBP retina 15 thunderbot gbe with BCM5701 chipset. I had only 220kpps on a single TX flow.Later I tried another adapter with a marvel yukon mini port. Had better pps rate, but nothing beyond 260kpps.
I've done loads of 1Gbit testing using the entry-level MacBook Air and a Thunderbolt Gigabit Ethernet adapter though, and I disagree with Saku's statement of 'You cannot use UDPSocket like iperf does, it just does not work, you are lucky if you reliably test 1Gbps'. I find iperf testing at 1Gbit on Mac Air with Thunderbolt Eth extremely reliable (always 950+mbit/sec TCP on a good network, and easy to push right to the 1gbit limit with UDP. Again, with 64byte packet size? Or are you talking MTU? With MTU size you can try whatever you want and it will seem to be reliable. A wget/ftp download of a 1GB file will provide similar results, but I dont think this is useful anyway since it won't test anything close to rfc2544 or at least an ordinary internet traffic profile with a mix of 600bytes pkg size combined with a lower rate of smaller packets (icmp/udp, ping/dns/ntp/voice/video). I am also interested in a cheap and reliable method to test 10GbE connections. So far I haven't found something I trust.
Pete
Can't help with faster adapters, but I believe there are some underlying architectural issues here as to why the speeds are hard to achieve, and why some people can and others maybe can't achieve them. For Carrier Ethernet, I believe most of these are covered in RFC2444 and the related RFC6815. Even with bit speeds up to spec, traffic speeds are impacted non-linearly by customer protocols including the usual suspect, TCP. This is documented in ITU-T Y.1564, clearly enough for simple folk like me. A good example for your corkboard is slide (page) 28 of the excellent 20140409-Tierney-100G-experience-Internet2-Summit.pdf, included as part of a report on 100GE performance test methodologies. Which is how I stumbled across it. Roy *Roy Hirst* | 425-556-5773 | 425-324-0941 cell XKL LLC | 12020 113th Ave NE, Suite 100 | Kirkland, WA 98034 | USA On 12/7/2014 8:48 AM, Teleric Team wrote:
From: pete@fiberphone.co.nz Subject: Re: 10Gb iPerf kit? Date: Sun, 7 Dec 2014 09:24:41 +1300 To: nanog@nanog.org
On 11/11/2014, at 1:35 PM, Randy Carpenter <rcarpen@network1.net> wrote:
I have not tried doing that myself, but the only thing that would even be possible that I know of is thunderbolt. A new MacBook Pro and one of these maybe: http://www.sonnettech.com/product/echoexpresssel_10gbeadapter.html Or one of these ones for dual-10Gbit links (one for out of band management or internet?):
http://www.sonnettech.com/product/twin10g.html
I haven't tried one myself, but they're relatively cheap (for 10gig) so not that much outlay to grab one and try it (esp if you already have an Apple laptop you can test with).
How would you use it? with iperf still?I don't think you will go nearly close to 14.8Mpps per port this way.Unless you are talking about bandwidth testing with full sized packet frames and low pps rate. I personally tested a 1Gbit/s port over a MBP retina 15 thunderbot gbe with BCM5701 chipset. I had only 220kpps on a single TX flow.Later I tried another adapter with a marvel yukon mini port. Had better pps rate, but nothing beyond 260kpps.
I've done loads of 1Gbit testing using the entry-level MacBook Air and a Thunderbolt Gigabit Ethernet adapter though, and I disagree with Saku's statement of 'You cannot use UDPSocket like iperf does, it just does not work, you are lucky if you reliably test 1Gbps'. I find iperf testing at 1Gbit on Mac Air with Thunderbolt Eth extremely reliable (always 950+mbit/sec TCP on a good network, and easy to push right to the 1gbit limit with UDP. Again, with 64byte packet size? Or are you talking MTU? With MTU size you can try whatever you want and it will seem to be reliable. A wget/ftp download of a 1GB file will provide similar results, but I dont think this is useful anyway since it won't test anything close to rfc2544 or at least an ordinary internet traffic profile with a mix of 600bytes pkg size combined with a lower rate of smaller packets (icmp/udp, ping/dns/ntp/voice/video). I am also interested in a cheap and reliable method to test 10GbE connections. So far I haven't found something I trust. Pete
The information contained in this e-mail message may be privileged, confidential and protected from disclosure. If you are not the intended recipient, any dissemination, distribution or copying is strictly prohibited. If you think that you have received this e-mail message in error, please e-mail the sender at the above e-mail address.
For RFC2444, please read RFC2544, and forgive the spam. *Roy Hirst* | 425-556-5773 | 425-324-0941 cell XKL LLC | 12020 113th Ave NE, Suite 100 | Kirkland, WA 98034 | USA On 12/8/2014 8:29 AM, Roy Hirst wrote:
Can't help with faster adapters, but I believe there are some underlying architectural issues here as to why the speeds are hard to achieve, and why some people can and others maybe can't achieve them. For Carrier Ethernet, I believe most of these are covered in RFC2444 and the related RFC6815. Even with bit speeds up to spec, traffic speeds are impacted non-linearly by customer protocols including the usual suspect, TCP. This is documented in ITU-T Y.1564, clearly enough for simple folk like me. A good example for your corkboard is slide (page) 28 of the excellent 20140409-Tierney-100G-experience-Internet2-Summit.pdf, included as part of a report on 100GE performance test methodologies. Which is how I stumbled across it. Roy
*Roy Hirst* | 425-556-5773 | 425-324-0941 cell XKL LLC | 12020 113th Ave NE, Suite 100 | Kirkland, WA 98034 | USA
On 12/7/2014 8:48 AM, Teleric Team wrote:
From: pete@fiberphone.co.nz Subject: Re: 10Gb iPerf kit? Date: Sun, 7 Dec 2014 09:24:41 +1300 To: nanog@nanog.org
On 11/11/2014, at 1:35 PM, Randy Carpenter <rcarpen@network1.net> wrote:
I have not tried doing that myself, but the only thing that would even be possible that I know of is thunderbolt. A new MacBook Pro and one of these maybe: http://www.sonnettech.com/product/echoexpresssel_10gbeadapter.html Or one of these ones for dual-10Gbit links (one for out of band management or internet?):
http://www.sonnettech.com/product/twin10g.html
I haven't tried one myself, but they're relatively cheap (for 10gig) so not that much outlay to grab one and try it (esp if you already have an Apple laptop you can test with).
How would you use it? with iperf still?I don't think you will go nearly close to 14.8Mpps per port this way.Unless you are talking about bandwidth testing with full sized packet frames and low pps rate. I personally tested a 1Gbit/s port over a MBP retina 15 thunderbot gbe with BCM5701 chipset. I had only 220kpps on a single TX flow.Later I tried another adapter with a marvel yukon mini port. Had better pps rate, but nothing beyond 260kpps.
I've done loads of 1Gbit testing using the entry-level MacBook Air and a Thunderbolt Gigabit Ethernet adapter though, and I disagree with Saku's statement of 'You cannot use UDPSocket like iperf does, it just does not work, you are lucky if you reliably test 1Gbps'. I find iperf testing at 1Gbit on Mac Air with Thunderbolt Eth extremely reliable (always 950+mbit/sec TCP on a good network, and easy to push right to the 1gbit limit with UDP. Again, with 64byte packet size? Or are you talking MTU? With MTU size you can try whatever you want and it will seem to be reliable. A wget/ftp download of a 1GB file will provide similar results, but I dont think this is useful anyway since it won't test anything close to rfc2544 or at least an ordinary internet traffic profile with a mix of 600bytes pkg size combined with a lower rate of smaller packets (icmp/udp, ping/dns/ntp/voice/video). I am also interested in a cheap and reliable method to test 10GbE connections. So far I haven't found something I trust. Pete
The information contained in this e-mail message may be privileged, confidential and protected from disclosure. If you are not the intended recipient, any dissemination, distribution or copying is strictly prohibited. If you think that you have received this e-mail message in error, please e-mail the sender at the above e-mail address.
The information contained in this e-mail message may be privileged, confidential and protected from disclosure. If you are not the intended recipient, any dissemination, distribution or copying is strictly prohibited. If you think that you have received this e-mail message in error, please e-mail the sender at the above e-mail address.
On 8/12/2014, at 5:48 AM, Teleric Team <teleric-lists@outlook.com> wrote:
Again, with 64byte packet size? Or are you talking MTU?
You have a very good point. My 1gig tests have always been for throughput rather than pps, and therefore was using MTU sized packets and though would be much less load on the CPU. Pete
participants (10)
-
Christopher Morrow -
Jason Lixfeld -
Matthew Walster -
Nick Hilliard -
Pete Mundy -
Randy Carpenter -
Roy Hirst -
Saku Ytti -
Teleric Team -
Tom Hill