I’m in the process of choosing hardware for a 30 story building. If anyone has experience with this I’d appreciate any tips. There are two fiber pairs running up the building riser. I need to put a POE switch on each floor using this fiber. The idea is to cut the fiber at each floor and insert a switch and daisy chain the switches together using one pair, and using the other pair as the failover side of the ring going back to the source so if one device fails it doesn’t take the whole string down. The problem here is how many switches can be strung together and I would not try more than 3 to 5. This is not something I typically do (stacking switches). I have fears of STP and/or RSTP issue stacking past Ethernet switch to switch limits (if they still exist??) Is there a device with a similar protocol as the old 3com (now HP IDF) stacking capability via fiber? I’d like to use something inexpensive as its to power ubiquiti wifi on each floor. Ideally if you know something I don’t about ubiquiti switches that can do this I’d appreciate knowing. Norman
Hey Norman, I'm in the middle of a construction project where we've got 50 data rooms in one building. I've researched a lot of different options and we ended up with just home runs. A couple of items to consider and think about: 1. Have you looked into the incremental additional costs associated with more fiber? Smallest that we've seen trunk fiber is 6 strands of SM fiber. When you look at pre-termed fiber with MTP/MPO connectors, it might be worthwhile to have home runs to each floor. 2. L2 rings IMHO seem pretty brittle. I know there are L2 ring products like Juniper BTI, which use ERPS and not strictly STP/RSTP to move blocking ports, and those seem a little better although it's mostly statically configured. With one pair going up the stack and another coming down, what happens when a device in the middle releases some magic smoke? Do you have 2 devices at each landing? Now that I've (maybe?) shot holes in some arguments, my personal preference, in a situation like this, would be to use passive DWDM. With an OADM at each floor, you can easily terminate all the wavelengths at the home run location and have "dedicated" L1 paths to each floor. You could use the 2nd pair of fiber to even set up some form of link-agg if you so choose. Another non-optimal option might also be something like GPON, although I've stayed away from that. Passive DWDM would also give you room for expansion and you could have 10Gbe optics at each floor which would give you the full bandwidth at each landing. Hope this helps. I'd be curious if anyone else has ever used DWDM in an intra-building scenario. Thanks, Abhi ________________________________ From: NANOG <nanog-bounces@nanog.org> on behalf of Norman Jester <nj@jester.mx> Sent: Tuesday, February 25, 2020 8:32 PM To: nanog@nanog.org <nanog@nanog.org> Subject: Hi-Rise Building Fiber Suggestions I’m in the process of choosing hardware for a 30 story building. If anyone has experience with this I’d appreciate any tips. There are two fiber pairs running up the building riser. I need to put a POE switch on each floor using this fiber. The idea is to cut the fiber at each floor and insert a switch and daisy chain the switches together using one pair, and using the other pair as the failover side of the ring going back to the source so if one device fails it doesn’t take the whole string down. The problem here is how many switches can be strung together and I would not try more than 3 to 5. This is not something I typically do (stacking switches). I have fears of STP and/or RSTP issue stacking past Ethernet switch to switch limits (if they still exist??) Is there a device with a similar protocol as the old 3com (now HP IDF) stacking capability via fiber? I’d like to use something inexpensive as its to power ubiquiti wifi on each floor. Ideally if you know something I don’t about ubiquiti switches that can do this I’d appreciate knowing. Norman
On 2/25/20 10:48 PM, Abhi Devireddy wrote:
L2 rings IMHO seem pretty brittle. I know there are L2 ring products like Juniper BTI, which use ERPS and not strictly STP/RSTP to move blocking ports, and those seem a little better although it's mostly statically configured.
For a strict ring topology like this, I'd certainly consider E-RPS or similar over [R]STP if you were going to do this all at L2. It's a lot more "predictable" in my experience insofar as it's harder to shoot yourself in the foot by mistuning some knob somewhere and getting behavior you don't expect. That said, I'd be loathe to put 30+ switches on a ring even within the same building unless I had little other choice. One thing that I haven't seen explored in this thread is the idea of doing things at L3. If you've got 4 fibers, you can use Bi-Di optics to build a partial mesh/ladder/braid as you go up the building. That can be a mess at L2 with (R)STP, but carving out an IGP area and doing it at L3 is often not nearly so ugly. If you've got L3 switches (which are cheap, these days), it may be a good option, though it may also be annoying from an IP subnetting POV unless you overlay it with something like an IPv4-in-IPv6 core (MAP, 464XLAT, etc.) or an L2-in-IP overlay like VXLAN both of which substantially increase the conplexity of the situation. Using CWDM or DWDM with 8-16 channels on either 2 or 4 trunks across your 4 fibers to do a more conventional home-run with multi-chassis LAG or similar is another reasonable option. I'd avoid stacking 30+ switches even where you have stacking support over fiber especially if the switches aren't in a physical stack. Too much opportunity for split-brain scenarios IMO. I'd contribute to the "see really hard if you can just drop more fiber down the riser" echo chamber. -- Brandon Martin
Should consider DWDM or GPON and in those look at passive optical technologies that can benefit the project.
On Feb 25, 2020, at 9:33 PM, Norman Jester <nj@jester.mx> wrote:
I’m in the process of choosing hardware for a 30 story building. If anyone has experience with this I’d appreciate any tips.
There are two fiber pairs running up the building riser. I need to put a POE switch on each floor using this fiber.
The idea is to cut the fiber at each floor and insert a switch and daisy chain the switches together using one pair, and using the other pair as the failover side of the ring going back to the source so if one device fails it doesn’t take the whole string down.
The problem here is how many switches can be strung together and I would not try more than 3 to 5. This is not something I typically do (stacking switches). I have fears of STP and/or RSTP issue stacking past Ethernet switch to switch limits (if they still exist??)
Is there a device with a similar protocol as the old 3com (now HP IDF) stacking capability via fiber?
I’d like to use something inexpensive as its to power ubiquiti wifi on each floor. Ideally if you know something I don’t about ubiquiti switches that can do this I’d appreciate knowing.
Norman
If you are limited on fiber runs, how about using 10Gb BiDi optics to limit a ring to say two sets of 15 switches. Tim On 2/25/20 8:21 PM, Bradley Burch wrote:
Should consider DWDM or GPON and in those look at passive optical technologies that can benefit the project.
On Feb 25, 2020, at 9:33 PM, Norman Jester <nj@jester.mx> wrote:
I’m in the process of choosing hardware for a 30 story building. If anyone has experience with this I’d appreciate any tips.
There are two fiber pairs running up the building riser. I need to put a POE switch on each floor using this fiber.
The idea is to cut the fiber at each floor and insert a switch and daisy chain the switches together using one pair, and using the other pair as the failover side of the ring going back to the source so if one device fails it doesn’t take the whole string down.
The problem here is how many switches can be strung together and I would not try more than 3 to 5. This is not something I typically do (stacking switches). I have fears of STP and/or RSTP issue stacking past Ethernet switch to switch limits (if they still exist??)
Is there a device with a similar protocol as the old 3com (now HP IDF) stacking capability via fiber?
I’d like to use something inexpensive as its to power ubiquiti wifi on each floor. Ideally if you know something I don’t about ubiquiti switches that can do this I’d appreciate knowing.
Norman
How would that work to solve Norman's problem? That sounds like a lot of money spending, and setup time, for nothing. Ryan On Feb 25 2020, at 8:21 pm, Bradley Burch <bradley@wifastnetworks.com> wrote:
Should consider DWDM or GPON and in those look at passive optical technologies that can benefit the project.
On Feb 25, 2020, at 9:33 PM, Norman Jester <nj@jester.mx> wrote: I’m in the process of choosing hardware for a 30 story building. If anyone has experience with this I’d appreciate any tips.
There are two fiber pairs running up the building riser. I need to put a POE switch on each floor using this fiber. The idea is to cut the fiber at each floor and insert a switch and daisy chain the switches together using one pair, and using the other pair as the failover side of the ring going back to the source so if one device fails it doesn’t take the whole string down. The problem here is how many switches can be strung together and I would not try more than 3 to 5. This is not something I typically do (stacking switches). I have fears of STP and/or RSTP issue stacking past Ethernet switch to switch limits (if they still exist??) Is there a device with a similar protocol as the old 3com (now HP IDF) stacking capability via fiber? I’d like to use something inexpensive as its to power ubiquiti wifi on each floor. Ideally if you know something I don’t about ubiquiti switches that can do this I’d appreciate knowing. Norman
DWDM can be done fairly cheap. Some combination of MUXes and OADM modules along the way. One possible solution is: First floor: https://www.fs.com/products/35887.html Every floor between: https://www.fs.com/products/70427.html Top floor: https://www.fs.com/products/35887.html Every floor gets 10G to aggregation switches on the top floor and bottom floor. The aggregation switches directly connect via the second pair. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Ryan Hamel" <ryan@rkhtech.org> To: "Bradley Burch" <bradley@wifastnetworks.com> Cc: nanog@nanog.org Sent: Tuesday, February 25, 2020 10:45:05 PM Subject: Re: Hi-Rise Building Fiber Suggestions How would that work to solve Norman's problem? That sounds like a lot of money spending, and setup time, for nothing. Ryan On Feb 25 2020, at 8:21 pm, Bradley Burch <bradley@wifastnetworks.com> wrote: Should consider DWDM or GPON and in those look at passive optical technologies that can benefit the project. <blockquote> On Feb 25, 2020, at 9:33 PM, Norman Jester <nj@jester.mx> wrote: I’m in the process of choosing hardware for a 30 story building. If anyone has experience with this I’d appreciate any tips. There are two fiber pairs running up the building riser. I need to put a POE switch on each floor using this fiber. The idea is to cut the fiber at each floor and insert a switch and daisy chain the switches together using one pair, and using the other pair as the failover side of the ring going back to the source so if one device fails it doesn’t take the whole string down. The problem here is how many switches can be strung together and I would not try more than 3 to 5. This is not something I typically do (stacking switches). I have fears of STP and/or RSTP issue stacking past Ethernet switch to switch limits (if they still exist??) Is there a device with a similar protocol as the old 3com (now HP IDF) stacking capability via fiber? I’d like to use something inexpensive as its to power ubiquiti wifi on each floor. Ideally if you know something I don’t about ubiquiti switches that can do this I’d appreciate knowing. Norman </blockquote>
After 30 add/drops you may lose too much power. There is a minimum 1.4dB per passthru and 1.3dB per add/drop, 3.5dB per MUX at the ends. With these SFP+ modules: https://www.fs.com/products/31238.html it looks like you would have a 19-20 dB budget to work with. You may be able to get 10 add/drops without amplification. But they have amps too: https://www.fs.com/products/72284.html I'd definitely contact sales and talk to one of their engineers so they can design a complete working solution for you. Are you sure you can't pull more fiber? It may be cheaper. On Wed, Feb 26, 2020 at 07:42:23AM -0600, Mike Hammett wrote:
DWDM can be done fairly cheap. Some combination of MUXes and OADM modules along the way. One possible solution is:
First floor: https://www.fs.com/products/35887.html Every floor between: https://www.fs.com/products/70427.html Top floor: https://www.fs.com/products/35887.html
Every floor gets 10G to aggregation switches on the top floor and bottom floor. The aggregation switches directly connect via the second pair.
----- Mike Hammett Intelligent Computing Solutions
Midwest Internet Exchange
The Brothers WISP
----- Original Message -----
From: "Ryan Hamel" <ryan@rkhtech.org> To: "Bradley Burch" <bradley@wifastnetworks.com> Cc: nanog@nanog.org Sent: Tuesday, February 25, 2020 10:45:05 PM Subject: Re: Hi-Rise Building Fiber Suggestions
How would that work to solve Norman's problem? That sounds like a lot of money spending, and setup time, for nothing.
Ryan
On Feb 25 2020, at 8:21 pm, Bradley Burch <bradley@wifastnetworks.com> wrote:
Should consider DWDM or GPON and in those look at passive optical technologies that can benefit the project.
<blockquote>
On Feb 25, 2020, at 9:33 PM, Norman Jester <nj@jester.mx> wrote:
I’m in the process of choosing hardware for a 30 story building. If anyone has experience with this I’d appreciate any tips.
There are two fiber pairs running up the building riser. I need to put a POE switch on each floor using this fiber.
The idea is to cut the fiber at each floor and insert a switch and daisy chain the switches together using one pair, and using the other pair as the failover side of the ring going back to the source so if one device fails it doesn’t take the whole string down.
The problem here is how many switches can be strung together and I would not try more than 3 to 5. This is not something I typically do (stacking switches). I have fears of STP and/or RSTP issue stacking past Ethernet switch to switch limits (if they still exist??)
Is there a device with a similar protocol as the old 3com (now HP IDF) stacking capability via fiber?
I’d like to use something inexpensive as its to power ubiquiti wifi on each floor. Ideally if you know something I don’t about ubiquiti switches that can do this I’d appreciate knowing.
Norman
</blockquote>
I would use single fiber CWDM muxing and OADM, then you can get it down to 7-8 switches per fiber. CWDM single fiber has a max of 9 channels and the optics are typically less expensive. Erich Kaiser The Fusion Network erich@gotfusion.net On Wed, Feb 26, 2020 at 7:43 AM Mike Hammett <nanog@ics-il.net> wrote:
DWDM can be done fairly cheap. Some combination of MUXes and OADM modules along the way. One possible solution is:
First floor: https://www.fs.com/products/35887.html Every floor between: https://www.fs.com/products/70427.html Top floor: https://www.fs.com/products/35887.html
Every floor gets 10G to aggregation switches on the top floor and bottom floor. The aggregation switches directly connect via the second pair.
----- Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> <https://www.linkedin.com/company/intelligent-computing-solutions> <https://twitter.com/ICSIL> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix> <https://www.linkedin.com/company/midwest-internet-exchange> <https://twitter.com/mdwestix> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> ------------------------------ *From: *"Ryan Hamel" <ryan@rkhtech.org> *To: *"Bradley Burch" <bradley@wifastnetworks.com> *Cc: *nanog@nanog.org *Sent: *Tuesday, February 25, 2020 10:45:05 PM *Subject: *Re: Hi-Rise Building Fiber Suggestions
How would that work to solve Norman's problem? That sounds like a lot of money spending, and setup time, for nothing.
Ryan
On Feb 25 2020, at 8:21 pm, Bradley Burch <bradley@wifastnetworks.com> wrote:
Should consider DWDM or GPON and in those look at passive optical technologies that can benefit the project.
On Feb 25, 2020, at 9:33 PM, Norman Jester <nj@jester.mx> wrote:
I’m in the process of choosing hardware for a 30 story building. If anyone has experience with this I’d appreciate any tips.
There are two fiber pairs running up the building riser. I need to put a POE switch on each floor using this fiber.
The idea is to cut the fiber at each floor and insert a switch and daisy chain the switches together using one pair, and using the other pair as the failover side of the ring going back to the source so if one device fails it doesn’t take the whole string down.
The problem here is how many switches can be strung together and I would not try more than 3 to 5. This is not something I typically do (stacking switches). I have fears of STP and/or RSTP issue stacking past Ethernet switch to switch limits (if they still exist??)
Is there a device with a similar protocol as the old 3com (now HP IDF) stacking capability via fiber?
I’d like to use something inexpensive as its to power ubiquiti wifi on each floor. Ideally if you know something I don’t about ubiquiti switches that can do this I’d appreciate knowing.
Norman
I'd say a pair of Juniper switches on each floor, with their virtual-chassis capability. Terminate the top/bottom floor of fiber 1 into switch 1, and the other into switch two. Create an LACP bond between each floors switches, tag the necessary VLANs, and put the VLAN SVIs onto the first pair of switches at the building electrical/telecom room. The same thing can be done with MLAG across many switch vendors, but that will require additional configuration. On Feb 25 2020, at 6:32 pm, Norman Jester <nj@jester.mx> wrote:
I’m in the process of choosing hardware for a 30 story building. If anyone has experience with this I’d appreciate any tips.
There are two fiber pairs running up the building riser. I need to put a POE switch on each floor using this fiber. The idea is to cut the fiber at each floor and insert a switch and daisy chain the switches together using one pair, and using the other pair as the failover side of the ring going back to the source so if one device fails it doesn’t take the whole string down. The problem here is how many switches can be strung together and I would not try more than 3 to 5. This is not something I typically do (stacking switches). I have fears of STP and/or RSTP issue stacking past Ethernet switch to switch limits (if they still exist??) Is there a device with a similar protocol as the old 3com (now HP IDF) stacking capability via fiber? I’d like to use something inexpensive as its to power ubiquiti wifi on each floor. Ideally if you know something I don’t about ubiquiti switches that can do this I’d appreciate knowing. Norman
Sent from my iPhone
On Feb 25, 2020, at 18:34, Norman Jester <nj@jester.mx> wrote:
I’m in the process of choosing hardware for a 30 story building. If anyone has experience with this I’d appreciate any tips.
There are two fiber pairs running up the building riser. I need to put a POE switch on each floor using this fiber.
In my experience with retrofitting existing structures, if you have access to the riser at each floor as it sounds like you do, you would typically drop in a new duct, blow micro duct through it with a branch for each floor, have an MDF or two In a utility spaces and them you have the ability to reconfigure the fiber as necessary to meet your present and future needs. You didn’t specify if the existing fiber is single or multi-mode however it is unlikely that the was enough slack built into two fiber runs to make 30 additional splices so that approach seems dubious as a premise. As you correctly surmise daisy chaining 30 switches is not an advisable network design practice.
The idea is to cut the fiber at each floor and insert a switch and daisy chain the switches together using one pair, and using the other pair as the failover side of the ring going back to the source so if one device fails it doesn’t take the whole string down.
The problem here is how many switches can be strung together and I would not try more than 3 to 5. This is not something I typically do (stacking switches). I have fears of STP and/or RSTP issue stacking past Ethernet switch to switch limits (if they still exist??)
Is there a device with a similar protocol as the old 3com (now HP IDF) stacking capability via fiber?
I’d like to use something inexpensive as its to power ubiquiti wifi on each floor. Ideally if you know something I don’t about ubiquiti switches that can do this I’d appreciate knowing.
Norman
Sent: Wednesday, February 26, 2020 4:46 AM To: Norman Jester <nj@jester.mx>
Sent from my iPhone
On Feb 25, 2020, at 18:34, Norman Jester <nj@jester.mx> wrote:
I’m in the process of choosing hardware for a 30 story building. If anyone has experience with this I’d appreciate any tips.
There are two fiber pairs running up the building riser. I need to put a POE switch on each floor using this fiber.
In my experience with retrofitting existing structures, if you have access to the riser at each floor as it sounds like you do, you would typically drop in a new duct, blow micro duct through it with a branch for each floor, have an MDF or two In a utility spaces and them you have the ability to reconfigure the fiber as necessary to meet your present and future needs.
You didn’t specify if the existing fiber is single or multi-mode however it is unlikely that the was enough slack built into two fiber runs to make 30 additional splices so that approach seems dubious as a premise.
As you correctly surmise daisy chaining 30 switches is not an advisable network design practice.
+1 to that, Put your own fiber in and do a star topology to an MDF device. adam
Joel Jaeggli Sent: Wednesday, February 26, 2020 4:46 AM
There are two fiber pairs running up the building riser. I need to put a POE switch on each floor using this fiber.
You didn’t specify if the existing fiber is single or multi-mode however
On that note would you gents recommend single-mode or multimode fiber for buildings? adam
Only single mode ever. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: adamv0025@netconsultings.com To: "Joel Jaeggli" <joelja@bogus.com>, "Norman Jester" <nj@jester.mx> Cc: nanog@nanog.org Sent: Wednesday, February 26, 2020 9:43:00 AM Subject: RE: Hi-Rise Building Fiber Suggestions
Joel Jaeggli Sent: Wednesday, February 26, 2020 4:46 AM
There are two fiber pairs running up the building riser. I need to put a POE switch on each floor using this fiber.
You didn’t specify if the existing fiber is single or multi-mode however
On that note would you gents recommend single-mode or multimode fiber for buildings? adam
On Wed, Feb 26, 2020 at 4:55 PM <adamv0025@netconsultings.com> wrote:
On that note would you gents recommend single-mode or multimode fiber for buildings?
adam
Single mode fiber for all new installs. There are only few uses cases where multimode still saves a little money (100G optics) but otherwise there are only downsides in my opinion. Single mode on the other hand will always work no matter what application. Regards, Baldur
On 26/Feb/20 17:43, adamv0025@netconsultings.com wrote:
On that note would you gents recommend single-mode or multimode fiber for buildings?
Single-mode, for sure. More predictable characteristics when you climb up the capacity scale, e.g., 10Gbps to 40Gbps to 100Gbps. We use plenty of multi-mode, but only in the data centre, between our own kit, for racks within the same cage. Between floors, multi-mode should be good for anywhere from 50m - 300m (100Gbps all the way down to 1Gbps), but why risk it? Mark.
On 2020-02-26 11:14, Randy Bush wrote:
We use plenty of multi-mode, but only in the data centre, between our own kit, for racks within the same cage.
so you have to stock both single and multi? hmmm
randy
I'd expect that from the ToR -> Servers would be MMF, but that other infrastructure cabling would be SMF. Even using aftermarket optics, putting single-mode transceivers in every server and access port would quickly become cost-prohibitive, would it not? -- Coy Hile coy.hile@coyhile.com
On Wed, Feb 26, 2020 at 11:20 AM Coy Hile <coy.hile@coyhile.com> wrote:
On 2020-02-26 11:14, Randy Bush wrote:
We use plenty of multi-mode, but only in the data centre, between our own kit, for racks within the same cage.
so you have to stock both single and multi? hmmm
randy
I'd expect that from the ToR -> Servers would be MMF, but that other infrastructure cabling would be SMF. Even using aftermarket optics, putting single-mode transceivers in every server and access port would quickly become cost-prohibitive, would it not?
Cisco GLC-SX-MM Compatible 1000BASE-SX SFP 850nm 550m DOM Transceiver Module - $6.00 - https://www.fs.com/products/11774.html Cisco SFP-GE-L Compatible 1000BASE-LX/LH SFP 1310nm 10km DOM Transceiver Module - $7.00 - https://www.fs.com/products/12622.html Yup, it is $1.00 more for SM, and you need 2 per link, but unless you are doing *lots* that's likely not cost-prohibitive. The delta on 10G is a bit more ($21 vs $18), but still not crazy-pants territory... Of course, sometimes you don't have the option of SM - you are connecting some someone else than they only do MM, or you are connecting to a piece of kit which doesn't have replaceable optics, or you have legacy cabling which is MM, or... but, the cost of the optics these days is not really the limiting factor.
-- Coy Hile coy.hile@coyhile.com
-- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf
It really depends on what you're interconnecting. Some NICs don't support SM optics, so even if you would like to run SM everywhere, it's not necessarily possible depending on the equipment. For example, I had issues with some SolarFlare cards which happily take 10G-SR MM but won't take 10G-LR SM. Filip On 26 February 2020 5:30:56 pm GMT+01:00, Warren Kumari <warren@kumari.net> wrote:
On Wed, Feb 26, 2020 at 11:20 AM Coy Hile <coy.hile@coyhile.com> wrote:
On 2020-02-26 11:14, Randy Bush wrote:
We use plenty of multi-mode, but only in the data centre, between
our
own kit, for racks within the same cage.
so you have to stock both single and multi? hmmm
randy
I'd expect that from the ToR -> Servers would be MMF, but that other infrastructure cabling would be SMF. Even using aftermarket optics, putting single-mode transceivers in every server and access port would quickly become cost-prohibitive, would it not?
Cisco GLC-SX-MM Compatible 1000BASE-SX SFP 850nm 550m DOM Transceiver Module - $6.00 - https://www.fs.com/products/11774.html Cisco SFP-GE-L Compatible 1000BASE-LX/LH SFP 1310nm 10km DOM Transceiver Module - $7.00 - https://www.fs.com/products/12622.html
Yup, it is $1.00 more for SM, and you need 2 per link, but unless you are doing *lots* that's likely not cost-prohibitive. The delta on 10G is a bit more ($21 vs $18), but still not crazy-pants territory...
Of course, sometimes you don't have the option of SM - you are connecting some someone else than they only do MM, or you are connecting to a piece of kit which doesn't have replaceable optics, or you have legacy cabling which is MM, or... but, the cost of the optics these days is not really the limiting factor.
-- Coy Hile coy.hile@coyhile.com
-- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf
-- Sent from my mobile device. Please excuse my brevity.
On 2/26/20 11:43 AM, Filip Hruska wrote:
Some NICs don't support SM optics, so even if you would like to run SM everywhere, it's not necessarily possible depending on the equipment. For example, I had issues with some SolarFlare cards which happily take 10G-SR MM but won't take 10G-LR SM.
Is this because they're dumb, or is there some serious reason? AFAIK, the electrical side of the SFP+ is the same for all 10G-R PHYs. My philosophy has generally been that all fixed infrastructure installed in this era might as well be single-mode. If I'm just dropping a patch cord in a raceway or similar, I'll use multi-mode in many cases. On the fixed side, I have enough trouble convincing folks that APC and UPC plugs are different let alone trying to explain why you can plug SM optics into MMF and it will (generally) work while the other way around does not beyond a few meters and, since SMF can't be gotten rid of entirely in fixed infrastructure, I'll take the normalization where I can get it. -- Brandon Martin
On 26/Feb/20 18:56, Brandon Martin wrote:
On the fixed side, I have enough trouble convincing folks that APC and UPC plugs are different
On that note, I migrated our network from DC to AC in 2007, and that was a major philosophical drama. At current job, all Transport kit runs DC for historical reasons. But when I joined in 2012, I got rid of DC for IP kit and went with AC. Nowadays, data centres either don't have DC, charge a premium for it, or you have to build your own DC plant which connects to - you guessed it - AC. Mark.
On 26/Feb/20 18:30, Warren Kumari wrote:
Of course, sometimes you don't have the option of SM - you are connecting some someone else than they only do MM, or you are connecting to a piece of kit which doesn't have replaceable optics, or you have legacy cabling which is MM, or... but, the cost of the optics these days is not really the limiting factor.
We are not flexible on use of SM for customer links. Over the years, plenty of customers have requested use of MM, and we have always insisted our conditions against that. Mark.
Warren Kumari Sent: Wednesday, February 26, 2020 4:31 PM
On Wed, Feb 26, 2020 at 11:20 AM Coy Hile <coy.hile@coyhile.com> wrote:
On 2020-02-26 11:14, Randy Bush wrote:
We use plenty of multi-mode, but only in the data centre, between our own kit, for racks within the same cage.
so you have to stock both single and multi? hmmm
randy
I'd expect that from the ToR -> Servers would be MMF, but that other infrastructure cabling would be SMF. Even using aftermarket optics, putting single-mode transceivers in every server and access port would quickly become cost-prohibitive, would it not?
Cisco GLC-SX-MM Compatible 1000BASE-SX SFP 850nm 550m DOM Transceiver Module - $6.00 - https://www.fs.com/products/11774.html Cisco SFP-GE-L Compatible 1000BASE-LX/LH SFP 1310nm 10km DOM Transceiver Module - $7.00 - https://www.fs.com/products/12622.html
Yup, it is $1.00 more for SM, and you need 2 per link, but unless you are doing *lots* that's likely not cost-prohibitive. The delta on 10G is a bit more ($21 vs $18), but still not crazy-pants territory...
Unfortunately I'm currently stuck with 40G breakout into 10G towards the floor switches, which is $39 vs $299 -hence my question regarding SMF vs MMF MPO/MTP for the risers. Alternatively killing a 40G port with single 10G. adam
On 26/Feb/20 18:19, Coy Hile wrote:
I'd expect that from the ToR -> Servers would be MMF, but that other infrastructure cabling would be SMF.
I've been designing in-data-centre cabling between routers with MM since 2007. Back then, there was a real material saving in doing that, relegating SM to backbone links and customer connections. Even then, while the savings are not as wild given it makes more sense to buy optics from 3rd party suppliers in 2020 than you could in 2007, savings are savings. Operationally, there is zero issues. As long as you don't exceed the distance limits of MM, you'll do okay.
Even using aftermarket optics, putting single-mode transceivers in every server and access port would quickly become cost-prohibitive, would it not?
The only servers we have running some part of the network infrastructure are our VMware-based CSR1000v RR's. As they are out-of-path RR's, a copper optic running Cat-6 is fine. Other servers are running basic services that don't require fibre, e.g., DNS, TACACS+, e.t.c. I imagine that if any of our servers needed to go fibre, they'd also run MM. The only part of our network where SM is the standard (and you won't find any MM) is our terrestrial and marine Transport infrastructure. Mark.
1G --- MM $6/ea SM $7/ea 10G --- MM $18/ea SM $24/ea DAC $9.50-$18/pair (length dependent) 25G --- MM $39/ea SM $59/ea DAC $23-$51/pair (length dependent) Not a significant price difference from SM to MM, but DAC is even cheaper. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Coy Hile" <coy.hile@coyhile.com> To: "Randy Bush" <randy@psg.com> Cc: nanog@nanog.org Sent: Wednesday, February 26, 2020 10:19:37 AM Subject: Re: Hi-Rise Building Fiber Suggestions On 2020-02-26 11:14, Randy Bush wrote:
We use plenty of multi-mode, but only in the data centre, between our own kit, for racks within the same cage.
so you have to stock both single and multi? hmmm
randy
I'd expect that from the ToR -> Servers would be MMF, but that other infrastructure cabling would be SMF. Even using aftermarket optics, putting single-mode transceivers in every server and access port would quickly become cost-prohibitive, would it not? -- Coy Hile coy.hile@coyhile.com
On 26/Feb/20 18:33, Mike Hammett wrote:
1G --- MM $6/ea SM $7/ea
10G --- MM $18/ea SM $24/ea DAC $9.50-$18/pair (length dependent)
25G --- MM $39/ea SM $59/ea DAC $23-$51/pair (length dependent)
Not a significant price difference from SM to MM, but DAC is even cheaper.
When you're buying thousands or 10's of thousands every year, that price difference adds up. Mark.
When you're buying thousands or tens of thousands, you're also not shopping off of the FiberStore web site. Also, DACs are even cheaper. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mark Tinka" <mark.tinka@seacom.mu> To: nanog@nanog.org Sent: Wednesday, February 26, 2020 11:05:46 AM Subject: Re: Hi-Rise Building Fiber Suggestions On 26/Feb/20 18:33, Mike Hammett wrote: 1G --- MM $6/ea SM $7/ea 10G --- MM $18/ea SM $24/ea DAC $9.50-$18/pair (length dependent) 25G --- MM $39/ea SM $59/ea DAC $23-$51/pair (length dependent) Not a significant price difference from SM to MM, but DAC is even cheaper. When you're buying thousands or 10's of thousands every year, that price difference adds up. Mark.
since we're at this layer, should i worry about going 3m with dacs at low speed, i.e. 10g? may need to do runs to neighbor rack. randy
nah. We do up to 10m on knockoff 40G DACs in production. It's no problem. On Wed, Feb 26, 2020 at 11:44 AM Randy Bush <randy@psg.com> wrote:
since we're at this layer, should i worry about going 3m with dacs at low speed, i.e. 10g? may need to do runs to neighbor rack.
randy
Randy Bush writes:
since we're at this layer, should i worry about going 3m with dacs at low speed, i.e. 10g? may need to do runs to neighbor rack.
No, 3m is totally fine for passive DAC, never had any issues with those. (5m should also be fine, we just have less experience with that because most we use DAC mostly for server/ToR cabling, usually using QSFP(28) to SFP+/SFP28 break-out cables.) -- Simon.
Once upon a time, Coy Hile <coy.hile@coyhile.com> said:
I'd expect that from the ToR -> Servers would be MMF, but that other infrastructure cabling would be SMF. Even using aftermarket optics, putting single-mode transceivers in every server and access port would quickly become cost-prohibitive, would it not?
If you are looking to save a buck on the ToR->server connection, why not just use DAC cables? -- Chris Adams <cma@cmadams.net>
On 26/Feb/20 18:14, Randy Bush wrote:
so you have to stock both single and multi? hmmm
Optics are dirt cheap. We don't pay the equipment vendors for their flavour :-). That said, stocking MM and SM is cheaper than stocking just SM, because we can reliably predict when/where we shall use either. For MM, it is always on the same floor, within a space of 1 - 10 racks, all nicely pinched together. In the old days, equipment vendors always shipped MM ports before SM anyway. Over 13 years ago, some line cards only shipped with MM for 10Gbps ports for almost 6 months before SM options were available. We had the same issue with our first 100Gbps ports as little as 2 years ago, for MPO cables. For customer circuits, SM pricing can be scoffed at. What is more important to us is that the optics are multi-rate. And even more important now, is that our 3rd party optics suppliers can allow us to code and re-code optics to our heart's content. Mark.
What is more important to us is that the optics are multi-rate. And even more important now, is that our 3rd party optics suppliers can allow us to code and re-code optics to our heart's content.
i wish flexoptix did 400g DACs. we have two boxes from the same ODM with interfaces whose sole pupose is to interconnect the two boxes, and the optics are coded for different vendors. unbelievable. randy
On 26/Feb/20 18:30, Randy Bush wrote:
i wish flexoptix did 400g DACs. we have two boxes from the same ODM with interfaces whose sole pupose is to interconnect the two boxes, and the optics are coded for different vendors. unbelievable.
I'm certain someone from Flex has NANOG chatter on "Promiscuous Mode" :-). Mark.
Randy Bush wrote on 26/02/2020 16:14:
We use plenty of multi-mode, but only in the data centre, between our own kit, for racks within the same cage.
so you have to stock both single and multi? hmmm
in-cabinet multimode can make sense, as long as you keep the stock types contained, i.e. highly restricted number of transceiver and cable types. SR4 + MTP vs LR4 + SMF is a good example here. Nick
Also, Juniper switches will stack over fiber. I have deployed Virtual Chassis over multiple IDFs. The VC ports can be (and highly suggested) to be in a ring. https://www.juniper.net/documentation/en_US/junos/topics/concept/virtual-cha... https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/... On 2/25/20 6:32 PM, Norman Jester wrote:
I’m in the process of choosing hardware for a 30 story building. If anyone has experience with this I’d appreciate any tips.
There are two fiber pairs running up the building riser. I need to put a POE switch on each floor using this fiber.
The idea is to cut the fiber at each floor and insert a switch and daisy chain the switches together using one pair, and using the other pair as the failover side of the ring going back to the source so if one device fails it doesn’t take the whole string down.
The problem here is how many switches can be strung together and I would not try more than 3 to 5. This is not something I typically do (stacking switches). I have fears of STP and/or RSTP issue stacking past Ethernet switch to switch limits (if they still exist??)
Is there a device with a similar protocol as the old 3com (now HP IDF) stacking capability via fiber?
I’d like to use something inexpensive as its to power ubiquiti wifi on each floor. Ideally if you know something I don’t about ubiquiti switches that can do this I’d appreciate knowing.
Norman
I do not recommend doing that, it's 30 members in a single stack. Mine was only two, directly connected to each other. Treat your control plane like your L2, don't extend it farther than necessary. Ryan On Feb 25 2020, at 9:00 pm, Tim Požár <pozar@lns.com> wrote:
Also, Juniper switches will stack over fiber. I have deployed Virtual Chassis over multiple IDFs. The VC ports can be (and highly suggested) to be in a ring.
https://www.juniper.net/documentation/en_US/junos/topics/concept/virtual-cha... https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/... On 2/25/20 6:32 PM, Norman Jester wrote:
I’m in the process of choosing hardware for a 30 story building. If anyone has experience with this I’d appreciate any tips.
There are two fiber pairs running up the building riser. I need to put a POE switch on each floor using this fiber. The idea is to cut the fiber at each floor and insert a switch and daisy chain the switches together using one pair, and using the other pair as the failover side of the ring going back to the source so if one device fails it doesn’t take the whole string down. The problem here is how many switches can be strung together and I would not try more than 3 to 5. This is not something I typically do (stacking switches). I have fears of STP and/or RSTP issue stacking past Ethernet switch to switch limits (if they still exist??) Is there a device with a similar protocol as the old 3com (now HP IDF) stacking capability via fiber? I’d like to use something inexpensive as its to power ubiquiti wifi on each floor. Ideally if you know something I don’t about ubiquiti switches that can do this I’d appreciate knowing. Norman
On 2/25/20 6:32 PM, Norman Jester wrote: I’m in the process of choosing hardware for a 30 story building. If anyone has experience with this I’d appreciate any tips.
There are two fiber pairs running up the building riser. I need to put a POE switch on each floor using this fiber.
The idea is to cut the fiber at each floor and insert a switch and daisy chain the switches together using one pair, and using the other pair as the failover side of the ring going back to the source so if one device fails it doesn’t take the whole string down.
The problem here is how many switches can be strung together and I would not try more than 3 to 5.
Yeah… I’d regenerate every five L2 devices as well. Which just means going up to L3 periodically. Would it work for you to use the first pair for daisy-chaining switches on each floor that’s not a multiple of five, and then put the switches on the floors that are multiples of five into router mode, with a switch-group facing their own floor, but routed ports facing other floors? Then use the second pair as an “express” lane between the exit, floor 10, and floor 20, to keep L3 hop-sounds down and provide some redundancy? -Bill
If you can go fully dynamically routed, Layer 3 only, this problem becomes much, much easier to solve given the constraints you mention. Among others, Ruckus switches will stack over fiber, but nowhere near 30 units. I think the max is 12 and I would not recommend going over 8. If you need L2, consider running it on an overlay, even if that overlay is just GRE. Again, rings are child's play if you can eliminate the L2 aspect. On Tue, Feb 25, 2020 at 8:32 PM Norman Jester <nj@jester.mx> wrote:
I’m in the process of choosing hardware for a 30 story building. If anyone has experience with this I’d appreciate any tips.
There are two fiber pairs running up the building riser. I need to put a POE switch on each floor using this fiber.
The idea is to cut the fiber at each floor and insert a switch and daisy chain the switches together using one pair, and using the other pair as the failover side of the ring going back to the source so if one device fails it doesn’t take the whole string down.
The problem here is how many switches can be strung together and I would not try more than 3 to 5. This is not something I typically do (stacking switches). I have fears of STP and/or RSTP issue stacking past Ethernet switch to switch limits (if they still exist??)
Is there a device with a similar protocol as the old 3com (now HP IDF) stacking capability via fiber?
I’d like to use something inexpensive as its to power ubiquiti wifi on each floor. Ideally if you know something I don’t about ubiquiti switches that can do this I’d appreciate knowing.
Norman
On 26/Feb/20 04:32, Norman Jester wrote:
The idea is to cut the fiber at each floor and insert a switch and daisy chain the switches together using one pair, and using the other pair as the failover side of the ring going back to the source so if one device fails it doesn’t take the whole string down.
Curious - blowing more fibre through the riser is a no-go? 30 floors worth of kit with only two spans is tricky any way you look at it. It would work, yes, but complexity will go up quite a bit. Of course, if there is absolutely no chance to run more fibre up the building, that is that. But if there is, just asking... Mark.
At the very minimum use bidirectional modules so you will have four channels. That way you would only have 15 switches on a chain. Also be sure to configured your STP weight so the cut will be in the middle. So one fiber will normally be transmitting to 7 switches, the other fiber to the other 8 switches. This is still inferior to the WDM solutions proposed, but I fear you have multimode fiber and might not have that choice. Regards, Baldur On Wed, Feb 26, 2020 at 3:33 AM Norman Jester <nj@jester.mx> wrote:
I’m in the process of choosing hardware for a 30 story building. If anyone has experience with this I’d appreciate any tips.
There are two fiber pairs running up the building riser. I need to put a POE switch on each floor using this fiber.
The idea is to cut the fiber at each floor and insert a switch and daisy chain the switches together using one pair, and using the other pair as the failover side of the ring going back to the source so if one device fails it doesn’t take the whole string down.
The problem here is how many switches can be strung together and I would not try more than 3 to 5. This is not something I typically do (stacking switches). I have fears of STP and/or RSTP issue stacking past Ethernet switch to switch limits (if they still exist??)
Is there a device with a similar protocol as the old 3com (now HP IDF) stacking capability via fiber?
I’d like to use something inexpensive as its to power ubiquiti wifi on each floor. Ideally if you know something I don’t about ubiquiti switches that can do this I’d appreciate knowing.
Norman
participants (24)
-
Abhi Devireddy -
adamv0025＠netconsultings.com -
Baldur Norddahl -
Bill Woodcock -
Bradley Burch -
Brandon Martin -
Chris Adams -
Chuck Anderson -
Coy Hile -
Fearghas Mckay -
Filip Hruska -
Hunter Fuller -
Jared Mauch -
Joel Jaeggli -
Kaiser, Erich -
Mark Tinka -
Mike Hammett -
Nick Hilliard -
Norman Jester -
Randy Bush -
Ryan Hamel -
Simon Leinen -
Tim Požár -
Warren Kumari