100G QSFP28 DAC cables - experience
Hi folks, I'm wondering if anyone have (either positive or negative) experience with 100G QSFP28 DAC cables? Is there anyone who is using 100G DAC in large scale and would recommend it (which means there are no issues compared to SR4 links)? I'm thinking about cables with lenght up to 1m, not more. We have had quite bad experience with 10G DAC in the past - but I do not want to be slave of the past. Thank you for your thoughts! Jiri
On Wed 2017-Sep-06 09:17:39 +0200, Jiri Prochazka <jiri@cdn77.com> wrote:
Hi folks,
I'm wondering if anyone have (either positive or negative) experience with 100G QSFP28 DAC cables?
Is there anyone who is using 100G DAC in large scale and would recommend it (which means there are no issues compared to SR4 links)?
I'm thinking about cables with lenght up to 1m, not more.
We have had quite bad experience with 10G DAC in the past - but I do not want to be slave of the past.
Thank you for your thoughts!
Jiri
We're deploying a decent chunk of 100G QSFP28 at the moment, but it's a mix of: - a handful of 100G QSFP28 copper DACs for some switch peerlinks - a bit >100x 100G QSFP28 AOC for interswitch links - a lot more 100G QSFP28 -> 4x25G SFP28 copper breakouts We're only a few weeks in at this point, so mileage may vary in the long run etc. The copper peerlinks are mostly 1M with some 3M. We've had no issues with them so far. The AOC interswitch links vary more in length, but some of those are >10M (hence AOC rather than copper). We've faced no issues with those. Granted, there is BGP with BFD running across those, so those should help in terms of liveness checks and such. I mention that because where we _have_ had issues are on the 100G -> 4x25G copper breakouts. Those are for 25G edge connectivity. It's a decent sample size with a bit north of 600x 25G ports. The trouble we've had there have been with some links showing link up on the switch and server side but actually failing to pass any traffic, so we need to stuff some >L1 liveness checks on there to ensure those links are good while we sort out the root issue. It is not yet clear if this is a cable fault, driver issue, or something firmware-ish on the NICs. Also, fun fact: 25G only made its way into the 802.3ad bonding mode driver in the Linux kernel in March this year[1]. -- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal [1]https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
We're using a mix as well, some QSFP28 AOC, others DAC. One thing that you need to keep in mind about the DACs is going to be the bend radius. These things are girthy af, so make sure to either overestimate your runs slightly, or buy one to test first. On Thu, Sep 14, 2017 at 11:54 AM, Hugo Slabbert <hugo@slabnet.com> wrote:
On Wed 2017-Sep-06 09:17:39 +0200, Jiri Prochazka <jiri@cdn77.com> wrote:
Hi folks,
I'm wondering if anyone have (either positive or negative) experience with 100G QSFP28 DAC cables?
Is there anyone who is using 100G DAC in large scale and would recommend it (which means there are no issues compared to SR4 links)?
I'm thinking about cables with lenght up to 1m, not more.
We have had quite bad experience with 10G DAC in the past - but I do not want to be slave of the past.
Thank you for your thoughts!
Jiri
We're deploying a decent chunk of 100G QSFP28 at the moment, but it's a mix of:
- a handful of 100G QSFP28 copper DACs for some switch peerlinks - a bit >100x 100G QSFP28 AOC for interswitch links - a lot more 100G QSFP28 -> 4x25G SFP28 copper breakouts
We're only a few weeks in at this point, so mileage may vary in the long run etc.
The copper peerlinks are mostly 1M with some 3M. We've had no issues with them so far.
The AOC interswitch links vary more in length, but some of those are >10M (hence AOC rather than copper). We've faced no issues with those. Granted, there is BGP with BFD running across those, so those should help in terms of liveness checks and such.
I mention that because where we _have_ had issues are on the 100G -> 4x25G copper breakouts. Those are for 25G edge connectivity. It's a decent sample size with a bit north of 600x 25G ports. The trouble we've had there have been with some links showing link up on the switch and server side but actually failing to pass any traffic, so we need to stuff some >L1 liveness checks on there to ensure those links are good while we sort out the root issue. It is not yet clear if this is a cable fault, driver issue, or something firmware-ish on the NICs.
Also, fun fact: 25G only made its way into the 802.3ad bonding mode driver in the Linux kernel in March this year[1].
-- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal
[1]https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/ linux.git/commit/?id=19ddde1eeca1ee81f4add5e04da66055e09281ac
+1 on this... I'd go so far as to say skip the copper, and just go with active-optical for short-run interconnects.
Tyler Conrad <mailto:tyler@tgconrad.com> September 14, 2017 at 2:12 PM We're using a mix as well, some QSFP28 AOC, others DAC. One thing that you need to keep in mind about the DACs is going to be the bend radius. These things are girthy af, so make sure to either overestimate your runs slightly, or buy one to test first.
Hugo Slabbert <mailto:hugo@slabnet.com> September 14, 2017 at 12:54 PM
On Wed 2017-Sep-06 09:17:39 +0200, Jiri Prochazka <jiri@cdn77.com> wrote:
We're deploying a decent chunk of 100G QSFP28 at the moment, but it's a mix of:
- a handful of 100G QSFP28 copper DACs for some switch peerlinks - a bit >100x 100G QSFP28 AOC for interswitch links - a lot more 100G QSFP28 -> 4x25G SFP28 copper breakouts
We're only a few weeks in at this point, so mileage may vary in the long run etc.
The copper peerlinks are mostly 1M with some 3M. We've had no issues with them so far.
10M (hence AOC rather than copper). We've faced no issues with
The AOC interswitch links vary more in length, but some of those are those. Granted, there is BGP with BFD running across those, so those should help in terms of liveness checks and such.
I mention that because where we _have_ had issues are on the 100G -> 4x25G copper breakouts. Those are for 25G edge connectivity. It's a decent sample size with a bit north of 600x 25G ports. The trouble we've had there have been with some links showing link up on the switch and server side but actually failing to pass any traffic, so we need to stuff some >L1 liveness checks on there to ensure those links are good while we sort out the root issue. It is not yet clear if this is a cable fault, driver issue, or something firmware-ish on the NICs.
Also, fun fact: 25G only made its way into the 802.3ad bonding mode driver in the Linux kernel in March this year[1].
Jiri Prochazka <mailto:jiri@cdn77.com> September 6, 2017 at 3:17 AM Hi folks,
I'm wondering if anyone have (either positive or negative) experience with 100G QSFP28 DAC cables?
Is there anyone who is using 100G DAC in large scale and would recommend it (which means there are no issues compared to SR4 links)?
I'm thinking about cables with lenght up to 1m, not more.
We have had quite bad experience with 10G DAC in the past - but I do not want to be slave of the past.
Thank you for your thoughts!
Jiri
-- -- Regards, -- Brant I. Stevens, Principal & Consulting Architect branto@argentiumsolutions.com d:212.931.8566, x101. m:917.673.6536. f:917.525.4759. http://argentiumsolutions.com
They're pretty fragile assemblies too, I ruined about 30 of them lacing them in, they need fish-paper around each cable so you don't crush the conductors when lacing. -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Brant Ian Stevens Sent: Thursday, September 14, 2017 2:05 PM To: Tyler Conrad Cc: NANOG Subject: Re: 100G QSFP28 DAC cables - experience +1 on this... I'd go so far as to say skip the copper, and just go with active-optical for short-run interconnects.
Tyler Conrad <mailto:tyler@tgconrad.com> September 14, 2017 at 2:12 PM We're using a mix as well, some QSFP28 AOC, others DAC. One thing that you need to keep in mind about the DACs is going to be the bend radius. These things are girthy af, so make sure to either overestimate your runs slightly, or buy one to test first.
Hugo Slabbert <mailto:hugo@slabnet.com> September 14, 2017 at 12:54 PM
On Wed 2017-Sep-06 09:17:39 +0200, Jiri Prochazka <jiri@cdn77.com> wrote:
We're deploying a decent chunk of 100G QSFP28 at the moment, but it's a mix of:
- a handful of 100G QSFP28 copper DACs for some switch peerlinks - a bit >100x 100G QSFP28 AOC for interswitch links - a lot more 100G QSFP28 -> 4x25G SFP28 copper breakouts
We're only a few weeks in at this point, so mileage may vary in the long run etc.
The copper peerlinks are mostly 1M with some 3M. We've had no issues with them so far.
10M (hence AOC rather than copper). We've faced no issues with
The AOC interswitch links vary more in length, but some of those are those. Granted, there is BGP with BFD running across those, so those should help in terms of liveness checks and such.
I mention that because where we _have_ had issues are on the 100G -> 4x25G copper breakouts. Those are for 25G edge connectivity. It's a decent sample size with a bit north of 600x 25G ports. The trouble we've had there have been with some links showing link up on the switch and server side but actually failing to pass any traffic, so we need to stuff some >L1 liveness checks on there to ensure those links are good while we sort out the root issue. It is not yet clear if this is a cable fault, driver issue, or something firmware-ish on the NICs.
Also, fun fact: 25G only made its way into the 802.3ad bonding mode driver in the Linux kernel in March this year[1].
Jiri Prochazka <mailto:jiri@cdn77.com> September 6, 2017 at 3:17 AM Hi folks,
I'm wondering if anyone have (either positive or negative) experience with 100G QSFP28 DAC cables?
Is there anyone who is using 100G DAC in large scale and would recommend it (which means there are no issues compared to SR4 links)?
I'm thinking about cables with lenght up to 1m, not more.
We have had quite bad experience with 10G DAC in the past - but I do not want to be slave of the past.
Thank you for your thoughts!
Jiri
-- -- Regards, -- Brant I. Stevens, Principal & Consulting Architect branto@argentiumsolutions.com d:212.931.8566, x101. m:917.673.6536. f:917.525.4759. http://argentiumsolutions.com
+1 on skipping the copper and going all optical. In my experience, the fiber is much less likely to have strange faults. We also experienced copper links that did not perform well but were showing up. It seems the optical modules are much more mature in terms of going offline properly if the signals are not within the parameters. Steven Naslund Chicago IL -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Jameson, Daniel Sent: Thursday, September 14, 2017 3:29 PM To: Brant Ian Stevens; Tyler Conrad Cc: NANOG Subject: RE: 100G QSFP28 DAC cables - experience They're pretty fragile assemblies too, I ruined about 30 of them lacing them in, they need fish-paper around each cable so you don't crush the conductors when lacing. -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Brant Ian Stevens Sent: Thursday, September 14, 2017 2:05 PM To: Tyler Conrad Cc: NANOG Subject: Re: 100G QSFP28 DAC cables - experience +1 on this... I'd go so far as to say skip the copper, and just go with active-optical for short-run interconnects.
Tyler Conrad <mailto:tyler@tgconrad.com> September 14, 2017 at 2:12 PM We're using a mix as well, some QSFP28 AOC, others DAC. One thing that you need to keep in mind about the DACs is going to be the bend radius. These things are girthy af, so make sure to either overestimate your runs slightly, or buy one to test first.
Hugo Slabbert <mailto:hugo@slabnet.com> September 14, 2017 at 12:54 PM
On Wed 2017-Sep-06 09:17:39 +0200, Jiri Prochazka <jiri@cdn77.com> wrote:
We're deploying a decent chunk of 100G QSFP28 at the moment, but it's a mix of:
- a handful of 100G QSFP28 copper DACs for some switch peerlinks - a bit >100x 100G QSFP28 AOC for interswitch links - a lot more 100G QSFP28 -> 4x25G SFP28 copper breakouts
We're only a few weeks in at this point, so mileage may vary in the long run etc.
The copper peerlinks are mostly 1M with some 3M. We've had no issues with them so far.
10M (hence AOC rather than copper). We've faced no issues with
The AOC interswitch links vary more in length, but some of those are those. Granted, there is BGP with BFD running across those, so those should help in terms of liveness checks and such.
I mention that because where we _have_ had issues are on the 100G -> 4x25G copper breakouts. Those are for 25G edge connectivity. It's a decent sample size with a bit north of 600x 25G ports. The trouble we've had there have been with some links showing link up on the switch and server side but actually failing to pass any traffic, so we need to stuff some >L1 liveness checks on there to ensure those links are good while we sort out the root issue. It is not yet clear if this is a cable fault, driver issue, or something firmware-ish on the NICs.
Also, fun fact: 25G only made its way into the 802.3ad bonding mode driver in the Linux kernel in March this year[1].
Jiri Prochazka <mailto:jiri@cdn77.com> September 6, 2017 at 3:17 AM Hi folks,
I'm wondering if anyone have (either positive or negative) experience with 100G QSFP28 DAC cables?
Is there anyone who is using 100G DAC in large scale and would recommend it (which means there are no issues compared to SR4 links)?
I'm thinking about cables with lenght up to 1m, not more.
We have had quite bad experience with 10G DAC in the past - but I do not want to be slave of the past.
Thank you for your thoughts!
Jiri
-- -- Regards, -- Brant I. Stevens, Principal & Consulting Architect branto@argentiumsolutions.com d:212.931.8566, x101. m:917.673.6536. f:917.525.4759. http://argentiumsolutions.com
On 9/6/17 00:17, Jiri Prochazka wrote:
Hi folks,
I'm wondering if anyone have (either positive or negative) experience with 100G QSFP28 DAC cables? I found the ones we tested to be substantially more finicky particularly at 5 meter then 10gig dacs, adding 4 x 25 sfp28 breakout on the other end probably isn't a help. We went with AOCs for the equivalent (3, 5 meter) lengths. Is there anyone who is using 100G DAC in large scale and would recommend it (which means there are no issues compared to SR4 links)?
I'm thinking about cables with lenght up to 1m, not more.
We have had quite bad experience with 10G DAC in the past - but I do not want to be slave of the past.
10g dacs were less reliable then I would have preferred for 3 5 7 meter distances but they were tolerable especially if you leave the bundles alone, 25g was sufficiently less so at least in testing that we didn't try it in the field, being at the limits at 5m was a sufficient problemthat it ruled out some adjacent rack arrangements. so we're all optical for 25/100
Thank you for your thoughts!
Jiri
participants (7)
-
Brant Ian Stevens -
Hugo Slabbert -
Jameson, Daniel -
Jiri Prochazka -
joel jaeggli -
Naslund, Steve -
Tyler Conrad