Hey there Any ISP using Juniper Fusion Provider Edge? https://www.juniper.net/documentation/en_US/junos/information-products/pathw... I am trying to chat with an engineer besides Juniper engineers to understand how buggy (or not) this is to go on production for a medium size ISP. Any feedback good/bad appreciated. -- Mehmet +1-424-298-1903
Fusion has made a lot more sense since Juniper changed the licensing model from every switch AND the MX to just the MX. We've deployed it in some of our sites. It is very cool from a forwarding plane perspective, but from a control plane standpoint it's very...meh. For example, you can't get SNMP oids for light levels or even read them right from the CLI. You have to log into the satellite switch like you would log into an FPC just to get light levels. That's probably the dumbest thing we've dealt with though. I've also heard you can have them do local L2 forwarding, which can be nice for latency and conserving uplink bandwidth, but we don't do any L2 that way so I wouldn't know the implications. From what we can tell though, it does give you Trio L3 performance and features with a MUCH cheaper port cost which is exactly what we were looking for, the extended reach of the chassis was just a fantastic bonus. We also REALLY like that we can have one pair of MX dists for a whole data center with hundreds of thousands of square feet of raised floor and deploy QFX5100 or EX4300 switches in every pod and haul back over just a few pairs of fiber. Saves a lot of time because all that's required to turn up a new connection is a cross connect in the pod. It also allows us to offer copper ports very far away from the MX device, which would normally require media converters. We've wanted to experiment with doing this over dark fiber in the metro as well, but we want to feel out any kinks locally before we add additional failure modes. Very interested in hearing about other's experiences with Fusion, good, bad, and ugly. -Matt On Mon, Dec 17, 2018 at 12:08 PM Mehmet Akcin <mehmet@akcin.net> wrote:
Hey there
Any ISP using Juniper Fusion Provider Edge?
https://www.juniper.net/documentation/en_US/junos/information-products/pathw...
I am trying to chat with an engineer besides Juniper engineers to understand how buggy (or not) this is to go on production for a medium size ISP.
Any feedback good/bad appreciated. -- Mehmet +1-424-298-1903
Hi there, About Juniper Fusion PE and our experience with it.
For example, you can't get SNMP oids for light levels or even read them right from the CLI. Sure it’s possible but also with a big „meh“. Here is how: "show interfaces diagnostics optics satellite <interface>“ (<- on the MX) BUT at least with MX Junos 16.1R7 and aligned SAT Image aka SNOS these values are wrong by a pretty big offset. Juniper promised they already fixed it but we can’t confirm (at least not in MX Junos 16.1). Soon we will take a look at MX Junos 17.3 with aligned sat image.
I've also heard you can have them do local L2 forwarding, which can be nice for latency and conserving uplink bandwidth, but we don't do any L2 that way so I wouldn't know the implications Same thing here … we don’t really need it. At least it’s on the roadmap and/or already implemented with higher Junos/SNOS versions.
From what we can tell though, it does give you Trio L3 performance and features with a MUCH cheaper port cost which is exactly what we were looking for, the extended reach of the chassis was just a fantastic bonus. Yep, that is really amazing and the reason we use it on many MXes. You can get rid of almost all ports you want (restrictions apply tho).
We also REALLY like that we can have one pair of MX dists for a whole data center with hundreds of thousands of square feet of raised floor and deploy QFX5100 or EX4300 switches in every pod and haul back over just a few pairs of fiber. Saves a lot of time because all that's required to turn up a new connection is a cross connect in the pod. It also allows us to offer copper ports very far away from the MX device, which would normally require media converters. We use Junos PE NOT as a replacement for any switch and/or ip fabrics within a datacenter. We use it to get rid of many customer/client ports (mainly 1G and 10G ports) which were directly connected to our MXes before. Atm I would not recommend using any closed fabrics beyond that scope. If it works for you … ok.
We've wanted to experiment with doing this over dark fiber in the metro as well, but we want to feel out any kinks locally before we add additional failure modes. At the moment? Don’t do it. If you run mpls on so called „core router/dwdm/wan facing ports“ you have to know that this is totally not supported on extended satellite ports. It’s not even on the roadmap. I already started to „collect“ some other ISPs to push juniper towards this feature because technically there no real reason why fusion should NOT be capable of pushing some mpls labels on already tagged 802.1br packets.
Best regards, Vincentz
Am 17.12.2018 um 19:26 schrieb Matt Erculiani <merculiani@gmail.com <mailto:merculiani@gmail.com>>:
Fusion has made a lot more sense since Juniper changed the licensing model from every switch AND the MX to just the MX.
We've deployed it in some of our sites. It is very cool from a forwarding plane perspective, but from a control plane standpoint it's very...meh. For example, you can't get SNMP oids for light levels or even read them right from the CLI. You have to log into the satellite switch like you would log into an FPC just to get light levels. That's probably the dumbest thing we've dealt with though. I've also heard you can have them do local L2 forwarding, which can be nice for latency and conserving uplink bandwidth, but we don't do any L2 that way so I wouldn't know the implications. From what we can tell though, it does give you Trio L3 performance and features with a MUCH cheaper port cost which is exactly what we were looking for, the extended reach of the chassis was just a fantastic bonus.
We also REALLY like that we can have one pair of MX dists for a whole data center with hundreds of thousands of square feet of raised floor and deploy QFX5100 or EX4300 switches in every pod and haul back over just a few pairs of fiber. Saves a lot of time because all that's required to turn up a new connection is a cross connect in the pod. It also allows us to offer copper ports very far away from the MX device, which would normally require media converters.
We've wanted to experiment with doing this over dark fiber in the metro as well, but we want to feel out any kinks locally before we add additional failure modes.
Very interested in hearing about other's experiences with Fusion, good, bad, and ugly.
-Matt
On Mon, Dec 17, 2018 at 12:08 PM Mehmet Akcin <mehmet@akcin.net <mailto:mehmet@akcin.net>> wrote: Hey there
Any ISP using Juniper Fusion Provider Edge?
https://www.juniper.net/documentation/en_US/junos/information-products/pathw... <https://www.juniper.net/documentation/en_US/junos/information-products/pathway-pages/junos-fusion/junos-fusion.html>
I am trying to chat with an engineer besides Juniper engineers to understand how buggy (or not) this is to go on production for a medium size ISP.
Any feedback good/bad appreciated. -- Mehmet +1-424-298-1903
I've, generally, always been wary about these "satellite" systems since Cisco introduced them with the ASR9000v back in 2012. My whole thing is it being a closed system that requires the same vendor hardware from start to finish. Back then, wiring things up redundantly where all ports are in action was not possible, not sure where things stand now but I guess there are solutions to that. Ultimately, I didn't want to be in a position where I am no longer satisfied with the solution, but my only recourse is to either lose the investment or continue with the same vendor just to save face. Recently, we had to vote with our feet as Juniper behaved badly on their EX4550's and EX4600's. So we are now switching to Arista, including getting rid of the old gear as it can't scale anymore. If we had gone this satellite route, we'd have fewer options to maneuver. The vendors love to push these systems because they say it will simplify your life - and in essence, it probably will - but this is a proper lock-in of note, and no vendor hates that. Mark. On 17/Dec/18 20:26, Matt Erculiani wrote:
Fusion has made a lot more sense since Juniper changed the licensing model from every switch AND the MX to just the MX.
We've deployed it in some of our sites. It is very cool from a forwarding plane perspective, but from a control plane standpoint it's very...meh. For example, you can't get SNMP oids for light levels or even read them right from the CLI. You have to log into the satellite switch like you would log into an FPC just to get light levels. That's probably the dumbest thing we've dealt with though. I've also heard you can have them do local L2 forwarding, which can be nice for latency and conserving uplink bandwidth, but we don't do any L2 that way so I wouldn't know the implications. From what we can tell though, it does give you Trio L3 performance and features with a MUCH cheaper port cost which is exactly what we were looking for, the extended reach of the chassis was just a fantastic bonus.
We also REALLY like that we can have one pair of MX dists for a whole data center with hundreds of thousands of square feet of raised floor and deploy QFX5100 or EX4300 switches in every pod and haul back over just a few pairs of fiber. Saves a lot of time because all that's required to turn up a new connection is a cross connect in the pod. It also allows us to offer copper ports very far away from the MX device, which would normally require media converters.
We've wanted to experiment with doing this over dark fiber in the metro as well, but we want to feel out any kinks locally before we add additional failure modes.
Very interested in hearing about other's experiences with Fusion, good, bad, and ugly.
-Matt
On Mon, Dec 17, 2018 at 12:08 PM Mehmet Akcin <mehmet@akcin.net <mailto:mehmet@akcin.net>> wrote:
Hey there
Any ISP using Juniper Fusion Provider Edge?
https://www.juniper.net/documentation/en_US/junos/information-products/pathw...
I am trying to chat with an engineer besides Juniper engineers to understand how buggy (or not) this is to go on production for a medium size ISP.
Any feedback good/bad appreciated. -- Mehmet +1-424-298-1903
Hi there, About Juniper Fusion PE and our experience with it.
For example, you can't get SNMP oids for light levels or even read them right from the CLI. Sure it’s possible but also with a big „meh“. Here is how: "show interfaces diagnostics optics satellite <interface>“ (<- on the MX) BUT at least with MX Junos 16.1R7 and aligned SAT Image aka SNOS these values are wrong by a pretty big offset. Juniper promised they already fixed it but we can’t confirm (at least not in MX Junos 16.1). Soon we will take a look at MX Junos 17.3 with aligned sat image.
I've also heard you can have them do local L2 forwarding, which can be nice for latency and conserving uplink bandwidth, but we don't do any L2 that way so I wouldn't know the implications Same thing here … we don’t really need it. At least it’s on the roadmap and/or already implemented with higher Junos/SNOS versions.
From what we can tell though, it does give you Trio L3 performance and features with a MUCH cheaper port cost which is exactly what we were looking for, the extended reach of the chassis was just a fantastic bonus. Yep, that is really amazing and the reason we use it on many MXes. You can get rid of almost all ports you want (restrictions apply tho).
We also REALLY like that we can have one pair of MX dists for a whole data center with hundreds of thousands of square feet of raised floor and deploy QFX5100 or EX4300 switches in every pod and haul back over just a few pairs of fiber. Saves a lot of time because all that's required to turn up a new connection is a cross connect in the pod. It also allows us to offer copper ports very far away from the MX device, which would normally require media converters. We use Junos PE NOT as a replacement for any switch and/or ip fabrics within a datacenter. We use it to get rid of many customer/client ports (mainly 1G and 10G ports) which were directly connected to our MXes before. Atm I would not recommend using any closed fabrics beyond that scope. If it works for you … ok.
We've wanted to experiment with doing this over dark fiber in the metro as well, but we want to feel out any kinks locally before we add additional failure modes. At the moment? Don’t do it. If you run mpls on so called „core router/dwdm/wan facing ports“ you have to know that this is totally not supported on extended satellite ports. It’s not even on the roadmap. I already started to „collect“ some other ISPs to push juniper towards this feature because technically there no real reason why fusion should NOT be capable of pushing some mpls labels on already tagged 802.1br packets.
Best regards, Vincentz — PS: some have received this mail multiple times because I’ve send it from the wrong account … time for vacation I guess.
Am 17.12.2018 um 19:26 schrieb Matt Erculiani <merculiani@gmail.com>:
Fusion has made a lot more sense since Juniper changed the licensing model from every switch AND the MX to just the MX.
We've deployed it in some of our sites. It is very cool from a forwarding plane perspective, but from a control plane standpoint it's very...meh. For example, you can't get SNMP oids for light levels or even read them right from the CLI. You have to log into the satellite switch like you would log into an FPC just to get light levels. That's probably the dumbest thing we've dealt with though. I've also heard you can have them do local L2 forwarding, which can be nice for latency and conserving uplink bandwidth, but we don't do any L2 that way so I wouldn't know the implications. From what we can tell though, it does give you Trio L3 performance and features with a MUCH cheaper port cost which is exactly what we were looking for, the extended reach of the chassis was just a fantastic bonus.
We also REALLY like that we can have one pair of MX dists for a whole data center with hundreds of thousands of square feet of raised floor and deploy QFX5100 or EX4300 switches in every pod and haul back over just a few pairs of fiber. Saves a lot of time because all that's required to turn up a new connection is a cross connect in the pod. It also allows us to offer copper ports very far away from the MX device, which would normally require media converters.
We've wanted to experiment with doing this over dark fiber in the metro as well, but we want to feel out any kinks locally before we add additional failure modes.
Very interested in hearing about other's experiences with Fusion, good, bad, and ugly.
-Matt
On Mon, Dec 17, 2018 at 12:08 PM Mehmet Akcin <mehmet@akcin.net <mailto:mehmet@akcin.net>> wrote: Hey there
Any ISP using Juniper Fusion Provider Edge?
https://www.juniper.net/documentation/en_US/junos/information-products/pathw... <https://www.juniper.net/documentation/en_US/junos/information-products/pathway-pages/junos-fusion/junos-fusion.html>
I am trying to chat with an engineer besides Juniper engineers to understand how buggy (or not) this is to go on production for a medium size ISP.
Any feedback good/bad appreciated. -- Mehmet +1-424-298-1903
There is a fundamental product limitation. CoS on Cascade port for MX is not officially supported as well QFX acting as AD. I agree with those who perceive all these approaches as proprietary lock-in (disguised as cheap). From: NANOG <nanog-bounces@nanog.org> On Behalf Of Vincentz Petzholtz Sent: Wednesday, December 19, 2018 9:01 AM To: nanog <nanog@nanog.org> Subject: [EXTERNAL] Re: JunOS Fusion Provider Edge Hi there, About Juniper Fusion PE and our experience with it. For example, you can't get SNMP oids for light levels or even read them right from the CLI. Sure it’s possible but also with a big „meh“. Here is how: "show interfaces diagnostics optics satellite <interface>“ (<- on the MX) BUT at least with MX Junos 16.1R7 and aligned SAT Image aka SNOS these values are wrong by a pretty big offset. Juniper promised they already fixed it but we can’t confirm (at least not in MX Junos 16.1). Soon we will take a look at MX Junos 17.3 with aligned sat image. I've also heard you can have them do local L2 forwarding, which can be nice for latency and conserving uplink bandwidth, but we don't do any L2 that way so I wouldn't know the implications Same thing here … we don’t really need it. At least it’s on the roadmap and/or already implemented with higher Junos/SNOS versions. From what we can tell though, it does give you Trio L3 performance and features with a MUCH cheaper port cost which is exactly what we were looking for, the extended reach of the chassis was just a fantastic bonus. Yep, that is really amazing and the reason we use it on many MXes. You can get rid of almost all ports you want (restrictions apply tho). We also REALLY like that we can have one pair of MX dists for a whole data center with hundreds of thousands of square feet of raised floor and deploy QFX5100 or EX4300 switches in every pod and haul back over just a few pairs of fiber. Saves a lot of time because all that's required to turn up a new connection is a cross connect in the pod. It also allows us to offer copper ports very far away from the MX device, which would normally require media converters. We use Junos PE NOT as a replacement for any switch and/or ip fabrics within a datacenter. We use it to get rid of many customer/client ports (mainly 1G and 10G ports) which were directly connected to our MXes before. Atm I would not recommend using any closed fabrics beyond that scope. If it works for you … ok. We've wanted to experiment with doing this over dark fiber in the metro as well, but we want to feel out any kinks locally before we add additional failure modes. At the moment? Don’t do it. If you run mpls on so called „core router/dwdm/wan facing ports“ you have to know that this is totally not supported on extended satellite ports. It’s not even on the roadmap. I already started to „collect“ some other ISPs to push juniper towards this feature because technically there no real reason why fusion should NOT be capable of pushing some mpls labels on already tagged 802.1br packets. Best regards, Vincentz — PS: some have received this mail multiple times because I’ve send it from the wrong account … time for vacation I guess. Am 17.12.2018 um 19:26 schrieb Matt Erculiani <merculiani@gmail.com<mailto:merculiani@gmail.com>>: Fusion has made a lot more sense since Juniper changed the licensing model from every switch AND the MX to just the MX. We've deployed it in some of our sites. It is very cool from a forwarding plane perspective, but from a control plane standpoint it's very...meh. For example, you can't get SNMP oids for light levels or even read them right from the CLI. You have to log into the satellite switch like you would log into an FPC just to get light levels. That's probably the dumbest thing we've dealt with though. I've also heard you can have them do local L2 forwarding, which can be nice for latency and conserving uplink bandwidth, but we don't do any L2 that way so I wouldn't know the implications. From what we can tell though, it does give you Trio L3 performance and features with a MUCH cheaper port cost which is exactly what we were looking for, the extended reach of the chassis was just a fantastic bonus. We also REALLY like that we can have one pair of MX dists for a whole data center with hundreds of thousands of square feet of raised floor and deploy QFX5100 or EX4300 switches in every pod and haul back over just a few pairs of fiber. Saves a lot of time because all that's required to turn up a new connection is a cross connect in the pod. It also allows us to offer copper ports very far away from the MX device, which would normally require media converters. We've wanted to experiment with doing this over dark fiber in the metro as well, but we want to feel out any kinks locally before we add additional failure modes. Very interested in hearing about other's experiences with Fusion, good, bad, and ugly. -Matt On Mon, Dec 17, 2018 at 12:08 PM Mehmet Akcin <mehmet@akcin.net<mailto:mehmet@akcin.net>> wrote: Hey there Any ISP using Juniper Fusion Provider Edge? https://www.juniper.net/documentation/en_US/junos/information-products/pathw... I am trying to chat with an engineer besides Juniper engineers to understand how buggy (or not) this is to go on production for a medium size ISP. Any feedback good/bad appreciated. -- Mehmet +1-424-298-1903 This email is from Equinix (EMEA) B.V. or one of its associated companies in the territory from where this email has been sent. This email, and any files transmitted with it, contains information which is confidential, is solely for the use of the intended recipient and may be legally privileged. If you have received this email in error, please notify the sender and delete this email immediately. Equinix (EMEA) B.V.. Registered Office: Amstelplein 1, 1096 HA Amsterdam, The Netherlands. Registered in The Netherlands No. 57577889.
QFX10k is the AD in Fusion Datacenter. In a Fusion Edge setup it is MX. On Fri, Dec 21, 2018 at 8:21 PM Nikos Leontsinis < Nikos.Leontsinis@eu.equinix.com> wrote:
There is a fundamental product limitation. CoS on Cascade port for MX is not officially supported as well QFX acting as AD.
I agree with those who perceive all these approaches as proprietary lock-in (disguised as cheap).
*From:* NANOG <nanog-bounces@nanog.org> *On Behalf Of *Vincentz Petzholtz *Sent:* Wednesday, December 19, 2018 9:01 AM *To:* nanog <nanog@nanog.org> *Subject:* [EXTERNAL] Re: JunOS Fusion Provider Edge
Hi there,
About Juniper Fusion PE and our experience with it.
For example, you can't get SNMP oids for light levels or even read them right from the CLI.
Sure it’s possible but also with a big „meh“. Here is how:
"show interfaces diagnostics optics satellite <interface>“ (<- on the MX)
BUT at least with MX Junos 16.1R7 and aligned SAT Image aka SNOS these values are wrong
by a pretty big offset. Juniper promised they already fixed it but we can’t confirm (at least not in MX Junos 16.1).
Soon we will take a look at MX Junos 17.3 with aligned sat image.
I've also heard you can have them do local L2 forwarding, which can be nice for latency and conserving uplink bandwidth, but we don't do any L2 that way so I wouldn't know the implications
Same thing here … we don’t really need it. At least it’s on the roadmap and/or already implemented with higher Junos/SNOS versions.
From what we can tell though, it does give you Trio L3 performance and features with a MUCH cheaper port cost which is exactly what we were looking for, the extended reach of the chassis was just a fantastic bonus.
Yep, that is really amazing and the reason we use it on many MXes. You can get rid of almost all ports you want (restrictions apply tho).
We also REALLY like that we can have one pair of MX dists for a whole data center with hundreds of thousands of square feet of raised floor and deploy QFX5100 or EX4300 switches in every pod and haul back over just a few pairs of fiber. Saves a lot of time because all that's required to turn up a new connection is a cross connect in the pod. It also allows us to offer copper ports very far away from the MX device, which would normally require media converters.
We use Junos PE NOT as a replacement for any switch and/or ip fabrics within a datacenter. We use it to get rid of many customer/client ports (mainly 1G and 10G ports)
which were directly connected to our MXes before. Atm I would not recommend using any closed fabrics beyond that scope. If it works for you … ok.
We've wanted to experiment with doing this over dark fiber in the metro as well, but we want to feel out any kinks locally before we add additional failure modes.
At the moment? Don’t do it. If you run mpls on so called „core router/dwdm/wan facing ports“ you have to know that this is totally not supported on extended satellite ports.
It’s not even on the roadmap. I already started to „collect“ some other ISPs to push juniper towards this feature because technically there no
real reason why fusion should NOT be capable of pushing some mpls labels on already tagged 802.1br packets.
Best regards,
Vincentz
—
PS: some have received this mail multiple times because I’ve send it from the wrong account … time for vacation I guess.
Am 17.12.2018 um 19:26 schrieb Matt Erculiani <merculiani@gmail.com>:
Fusion has made a lot more sense since Juniper changed the licensing model from every switch AND the MX to just the MX.
We've deployed it in some of our sites. It is very cool from a forwarding plane perspective, but from a control plane standpoint it's very...meh. For example, you can't get SNMP oids for light levels or even read them right from the CLI. You have to log into the satellite switch like you would log into an FPC just to get light levels. That's probably the dumbest thing we've dealt with though. I've also heard you can have them do local L2 forwarding, which can be nice for latency and conserving uplink bandwidth, but we don't do any L2 that way so I wouldn't know the implications. From what we can tell though, it does give you Trio L3 performance and features with a MUCH cheaper port cost which is exactly what we were looking for, the extended reach of the chassis was just a fantastic bonus.
We also REALLY like that we can have one pair of MX dists for a whole data center with hundreds of thousands of square feet of raised floor and deploy QFX5100 or EX4300 switches in every pod and haul back over just a few pairs of fiber. Saves a lot of time because all that's required to turn up a new connection is a cross connect in the pod. It also allows us to offer copper ports very far away from the MX device, which would normally require media converters.
We've wanted to experiment with doing this over dark fiber in the metro as well, but we want to feel out any kinks locally before we add additional failure modes.
Very interested in hearing about other's experiences with Fusion, good, bad, and ugly.
-Matt
On Mon, Dec 17, 2018 at 12:08 PM Mehmet Akcin <mehmet@akcin.net> wrote:
Hey there
Any ISP using Juniper Fusion Provider Edge?
https://www.juniper.net/documentation/en_US/junos/information-products/pathw...
I am trying to chat with an engineer besides Juniper engineers to understand how buggy (or not) this is to go on production for a medium size ISP.
Any feedback good/bad appreciated.
--
Mehmet +1-424-298-1903
This email is from Equinix (EMEA) B.V. or one of its associated companies in the territory from where this email has been sent. This email, and any files transmitted with it, contains information which is confidential, is solely for the use of the intended recipient and may be legally privileged. If you have received this email in error, please notify the sender and delete this email immediately. Equinix (EMEA) B.V.. Registered Office: Amstelplein 1, 1096 HA Amsterdam, The Netherlands. Registered in The Netherlands No. 57577889.
Hi Mehmet, This has been discussed on the Juniper-NSP list several times, here's a couple of examples: https://puck.nether.net/pipermail/juniper-nsp/2018-November/036673.html https://puck.nether.net/pipermail/juniper-nsp/2018-April/035397.html https://puck.nether.net/pipermail/juniper-nsp/2016-June/032964.html Cheers, James.
thank you this is very useful to know Mehmet On Wed, Dec 19, 2018 at 1:46 AM James Bensley <jwbensley@gmail.com> wrote:
Hi Mehmet,
This has been discussed on the Juniper-NSP list several times, here's a couple of examples:
https://puck.nether.net/pipermail/juniper-nsp/2018-November/036673.html https://puck.nether.net/pipermail/juniper-nsp/2018-April/035397.html https://puck.nether.net/pipermail/juniper-nsp/2016-June/032964.html
Cheers, James.
participants (10)
-
Aaron Gould -
James Bensley -
Mark Tinka -
Matt Erculiani -
Mehmet Akcin -
Melchior Aelmans -
Nikos Leontsinis -
Sander Steffann -
Vincentz Petzholtz -
Vincentz Petzholtz