MX204 Virtual Chassis Setup
Hello, Does the MX204 support virtual chassis setup? Regards, Paschal Masha
Paschal, It is not supported, nor is it recommended for redundancy in a routed setup. Please describe your (desired) topology, that way the community can discuss alternatives. Thanks, Ryan Hamel ________________________________ From: NANOG <nanog-bounces+ryan=rkhtech.org@nanog.org> on behalf of Pascal Masha <pascalmasha@gmail.com> Sent: Monday, August 21, 2023 7:41 AM To: nanog@nanog.org <nanog@nanog.org> Subject: MX204 Virtual Chassis Setup Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments. Hello, Does the MX204 support virtual chassis setup? Regards, Paschal Masha
On 8/21/23 16:51, Ryan Hamel wrote:
Paschal,
It is not supported, nor is it recommended for redundancy in a routed setup. Please describe your (desired) topology, that way the community can discuss alternatives.
Sounds like the OP wants to build a chassis-based system out of non-redundant hardware. Mark.
No, but they do however work just great as an active-active pair of routers when cross linked and iBGP peered to each other and everything downstream connected to each one. Chris On Mon, Aug 21, 2023 at 9:43 AM Pascal Masha <pascalmasha@gmail.com> wrote:
Hello,
Does the MX204 support virtual chassis setup?
Regards, Paschal Masha
Thanks just wanted to know whether it was a supported feature. On Tue, 22 Aug 2023, 21:00 Chris, <chris@noskillz.com> wrote:
No, but they do however work just great as an active-active pair of routers when cross linked and iBGP peered to each other and everything downstream connected to each one.
Chris
On Mon, Aug 21, 2023 at 9:43 AM Pascal Masha <pascalmasha@gmail.com> wrote:
Hello,
Does the MX204 support virtual chassis setup?
Regards, Paschal Masha
On 8/23/23 08:00, Pascal Masha wrote:
Thanks just wanted to know whether it was a supported feature.
What would have been nice is if Juniper oversubscribed the face plate of this platform, as most people are more likely to run out of ports than they would the 400Gbps forwarding capacity of Trio. In some cases, we deploy more of these in the same PoP just because we need more ports, not because we need more capacity; and a chassis would not make sense for the function, yet. Mark.
What would have been nice is if Juniper oversubscribed the face plate of this platform, as most people are more likely to run out of ports than they would the 400Gbps forwarding capacity of Trio.
You're restricted to 400G because they did fixed lane allocations to the EA chip on the PFE to each port group. Doing an MRATE setup to let you access all 480G would have increased electrical complexity, and dramatically increased the price point of the box. There are tradeoffs. The more flexibility you want, the more expensive the box is going to be. I'm not sure they allow oversubscription on anything in the MX line anymore honestly. I could be wrong, I've been face down in a specific subset of equipment for a while, someone please correct me if I am. On Wed, Aug 23, 2023 at 2:11 AM Mark Tinka <mark@tinka.africa> wrote:
On 8/23/23 08:00, Pascal Masha wrote:
Thanks just wanted to know whether it was a supported feature.
What would have been nice is if Juniper oversubscribed the face plate of this platform, as most people are more likely to run out of ports than they would the 400Gbps forwarding capacity of Trio.
In some cases, we deploy more of these in the same PoP just because we need more ports, not because we need more capacity; and a chassis would not make sense for the function, yet.
Mark.
Does Fusion not make sense in this case? I've not had a ton of experience with it, but it does well to add a crazy port count to an otherwise very port limited device. -Matt On Wed, Aug 23, 2023 at 9:01 AM Tom Beecher <beecher@beecher.cc> wrote:
What would have been nice is if Juniper oversubscribed the face plate of
this platform, as most people are more likely to run out of ports than they would the 400Gbps forwarding capacity of Trio.
You're restricted to 400G because they did fixed lane allocations to the EA chip on the PFE to each port group. Doing an MRATE setup to let you access all 480G would have increased electrical complexity, and dramatically increased the price point of the box. There are tradeoffs. The more flexibility you want, the more expensive the box is going to be.
I'm not sure they allow oversubscription on anything in the MX line anymore honestly. I could be wrong, I've been face down in a specific subset of equipment for a while, someone please correct me if I am.
On Wed, Aug 23, 2023 at 2:11 AM Mark Tinka <mark@tinka.africa> wrote:
On 8/23/23 08:00, Pascal Masha wrote:
Thanks just wanted to know whether it was a supported feature.
What would have been nice is if Juniper oversubscribed the face plate of this platform, as most people are more likely to run out of ports than they would the 400Gbps forwarding capacity of Trio.
In some cases, we deploy more of these in the same PoP just because we need more ports, not because we need more capacity; and a chassis would not make sense for the function, yet.
Mark.
-- Matt Erculiani, NREMT ERCUL-ARIN
On 8/23/23 17:14, Matt Erculiani wrote:
Does Fusion not make sense in this case? I've not had a ton of experience with it, but it does well to add a crazy port count to an otherwise very port limited device.
In small edge PoP's, we attach an Arista 1U switch with tons of 1/10Gbps ports to an MX204 via 802.1Q. Works a treat. I've never been convinced by vendor-specific satellite systems :-). On another note, the potential issue we might run into is pressure on control plane memory on the MX204 for us that run BGP Add-Paths. You can always upgrade the RE on an MX240/480/960, but the MX204 is fixed (and last time I checked, fiddling with Juniper RE memory was generally frowned upon). Luckily, the MX10003 ships with 64GB of RAM, since it is now EoL. The MX304 ships with 128GB of RAM, so anybody running Add-Paths on that box won't have an issue there. Mark.
On another note, the potential issue we might run into is pressure on control plane memory on the MX204 for us that run BGP Add-Paths. You can always upgrade the RE on an MX240/480/960, but the MX204 is fixed (and last time I checked, fiddling with Juniper RE memory was generally frowned upon).
In my experience and testing with them, you have a decent bit of headroom past the published RIB/FIB limits before they'll fall over. On Fri, Aug 25, 2023 at 11:35 AM Mark Tinka <mark@tinka.africa> wrote:
On 8/23/23 17:14, Matt Erculiani wrote:
Does Fusion not make sense in this case? I've not had a ton of experience with it, but it does well to add a crazy port count to an otherwise very port limited device.
In small edge PoP's, we attach an Arista 1U switch with tons of 1/10Gbps ports to an MX204 via 802.1Q. Works a treat. I've never been convinced by vendor-specific satellite systems :-).
On another note, the potential issue we might run into is pressure on control plane memory on the MX204 for us that run BGP Add-Paths. You can always upgrade the RE on an MX240/480/960, but the MX204 is fixed (and last time I checked, fiddling with Juniper RE memory was generally frowned upon).
Luckily, the MX10003 ships with 64GB of RAM, since it is now EoL.
The MX304 ships with 128GB of RAM, so anybody running Add-Paths on that box won't have an issue there.
Mark.
On 8/25/23 19:16, Tom Beecher wrote:
In my experience and testing with them, you have a decent bit of headroom past the published RIB/FIB limits before they'll fall over.
They are holding up pretty well for us, mainly because we do a lot more BGP on MX480's than on MX204's. We use the MX204's mainly for peering and CDN gateways. Where we use them for edge customers, it's a handful of BGP sessions. On MX480 16GB RE's running two full BGP feeds but hundreds of customer sessions, Add-Paths really eats into RAM. We've had to upgrade some of the busier routers from 16GB to 64GB RE's, especially on later versions of code where ROV can also bite into memory on boxes carrying lots of BGP sessions. Mark.
No VC here, unsure if it works, but yeah, we like them and deploy them in pairs for metro-e (ce) and cbh for vlans carried over mpls pw Reliable for us Aaron
On Aug 25, 2023, at 4:40 PM, Mark Tinka <mark@tinka.africa> wrote:
On 8/25/23 19:16, Tom Beecher wrote:
In my experience and testing with them, you have a decent bit of headroom past the published RIB/FIB limits before they'll fall over.
They are holding up pretty well for us, mainly because we do a lot more BGP on MX480's than on MX204's. We use the MX204's mainly for peering and CDN gateways. Where we use them for edge customers, it's a handful of BGP sessions.
On MX480 16GB RE's running two full BGP feeds but hundreds of customer sessions, Add-Paths really eats into RAM. We've had to upgrade some of the busier routers from 16GB to 64GB RE's, especially on later versions of code where ROV can also bite into memory on boxes carrying lots of BGP sessions.
Mark.
On MX480 16GB RE's running two full BGP feeds but hundreds of customer sessions, Add-Paths really eats into RAM.
It would, sure. Instead of storing a single prefix/next-hop with flags in memory, you now have to store every prefix/next-hop that you are announcing as well. On Fri, Aug 25, 2023 at 5:39 PM Mark Tinka <mark@tinka.africa> wrote:
On 8/25/23 19:16, Tom Beecher wrote:
In my experience and testing with them, you have a decent bit of headroom past the published RIB/FIB limits before they'll fall over.
They are holding up pretty well for us, mainly because we do a lot more BGP on MX480's than on MX204's. We use the MX204's mainly for peering and CDN gateways. Where we use them for edge customers, it's a handful of BGP sessions.
On MX480 16GB RE's running two full BGP feeds but hundreds of customer sessions, Add-Paths really eats into RAM. We've had to upgrade some of the busier routers from 16GB to 64GB RE's, especially on later versions of code where ROV can also bite into memory on boxes carrying lots of BGP sessions.
Mark.
On 8/26/23 00:54, Tom Beecher wrote:
It would, sure. Instead of storing a single prefix/next-hop with flags in memory, you now have to store every prefix/next-hop that you are announcing as well.
Indeed. But it has been worth it. The load balancing from PE-to-PE has been fantastic, especially when coupled with BGP Multipath. No more messing about with LOCAL_PREF for multi-homed customers, and it works just as well with different (but equal-length) AS_PATH's. Mark.
On 8/23/23 17:01, Tom Beecher wrote:
I'm not sure they allow oversubscription on anything in the MX line anymore honestly. I could be wrong, I've been face down in a specific subset of equipment for a while, someone please correct me if I am.
On the new ACX line, yes. If I look at the MPC7E, MPC10E, MX10003 and MX304, no oversubscription is allowed. Even the LC2103 MPC on the MX10003 which has more ports than Trio capacity, won't let you use more than 1.2Tbps (3x Trio 3 chips on it). We don't mess around with any other MX products, so not sure (although we are still yet to deploy the MPC10E's and the MX304). Mark.
On Wednesday, 23 August, 2023 16:33, "Mark Tinka" <mark@tinka.africa> said: [faceplate oversubscription]
On the new ACX line, yes.
Not Trio, and different PLM :)
We don't mess around with any other MX products, so not sure (although we are still yet to deploy the MPC10E's and the MX304).
MX304 (well, strictly LMIC16) has the same restriction, and a need for another entry in the magic port checker (https://apps.juniper.net/home/port-checker/index.html) for restrictions beyond "SUM(port-speeds) <= 1.6T". They make sense once you've looked at the block diagram for the thing and followed the lines, but things like "4x10G breakout can only go in odd-numbered ports, and you have to leave the corresponding next-lowest even-numbered port empty" are not instantly obvious. Thanks, Tim.
some of these port capabilities are weird to me. like on the ACX7100-48L you can do 4x100 or 8x50, but ONLY one 40g ?! me@7100> show chassis pic pic-slot 0 fpc-slot 0 | find 400 48 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 4x10G 3x100G 49 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 4x10G 3x100G 50 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 4x10G 3x100G 51 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 4x10G 3x100G 52 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 4x10G 3x100G 53 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 4x10G 3x100G 54 NA 1x10G On 8/23/2023 11:29 AM, tim@pelican.org wrote:
On Wednesday, 23 August, 2023 16:33, "Mark Tinka" <mark@tinka.africa> said:
[faceplate oversubscription]
On the new ACX line, yes. Not Trio, and different PLM :)
We don't mess around with any other MX products, so not sure (although we are still yet to deploy the MPC10E's and the MX304). MX304 (well, strictly LMIC16) has the same restriction, and a need for another entry in the magic port checker (https://apps.juniper.net/home/port-checker/index.html) for restrictions beyond "SUM(port-speeds) <= 1.6T".
They make sense once you've looked at the block diagram for the thing and followed the lines, but things like "4x10G breakout can only go in odd-numbered ports, and you have to leave the corresponding next-lowest even-numbered port empty" are not instantly obvious.
Thanks, Tim.
-- -Aaron
Aaron Gould писал(а) 2023-08-23 12:38:
some of these port capabilities are weird to me. like on the ACX7100-48L you can do 4x100 or 8x50, but ONLY one 40g ?!
me@7100> show chassis pic pic-slot 0 fpc-slot 0 | find 400 48 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 4x10G 3x100G 49 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 4x10G 3x100G 50 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 4x10G 3x100G 51 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 4x10G 3x100G 52 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 4x10G 3x100G 53 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 4x10G 3x100G 54 NA 1x10G
Probably because 40G is a product 10G lanes. There are only 4 lanes available, and the speed of a single lane can vary. So, 40G is the max speed for the lowest single lane's speed. Kind regards, Andrey
I sincerely doubt there is much demand for *new* 40G these days. Look at the population of 40G members on major IXes. People have either one 10G, 2 x 10G, or 100G. 40G was a dead-end 9 years ago and much so more now. On Wed, Aug 23, 2023 at 9:38 AM Aaron Gould <aaron1@gvtc.com> wrote:
some of these port capabilities are weird to me. like on the ACX7100-48L you can do 4x100 or 8x50, but ONLY one 40g ?!
me@7100> show chassis pic pic-slot 0 fpc-slot 0 | find 400 48 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 4x10G 3x100G 49 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 4x10G 3x100G 50 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 4x10G 3x100G 51 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 4x10G 3x100G 52 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 4x10G 3x100G 53 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 4x10G 3x100G 54 NA 1x10G
On 8/23/2023 11:29 AM, tim@pelican.org wrote:
On Wednesday, 23 August, 2023 16:33, "Mark Tinka" <mark@tinka.africa> said:
[faceplate oversubscription]
On the new ACX line, yes. Not Trio, and different PLM :)
We don't mess around with any other MX products, so not sure (although we are still yet to deploy the MPC10E's and the MX304). MX304 (well, strictly LMIC16) has the same restriction, and a need for another entry in the magic port checker ( https://apps.juniper.net/home/port-checker/index.html) for restrictions beyond "SUM(port-speeds) <= 1.6T".
They make sense once you've looked at the block diagram for the thing and followed the lines, but things like "4x10G breakout can only go in odd-numbered ports, and you have to leave the corresponding next-lowest even-numbered port empty" are not instantly obvious.
Thanks, Tim.
-- -Aaron
On 8/27/23 04:52, Eric Kuhnke wrote:
I sincerely doubt there is much demand for *new* 40G these days.
Look at the population of 40G members on major IXes.
People have either one 10G, 2 x 10G, or 100G.
40G was a dead-end 9 years ago and much so more now.
We have customers that sometimes ask for 40Gbps interconnects. I always tell our Pre-Sales team that those are the ones who "led the way", back in the day. Sadly, they were a bit too early :-). Mark.
Well, or they simply found a potential deal on hardware that came with 40 gig ports. 40 gigs is still a lot of bits to a lot of people. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mark Tinka" <mark@tinka.africa> To: nanog@nanog.org Sent: Saturday, August 26, 2023 10:59:36 PM Subject: Re: MX204 Virtual Chassis Setup On 8/27/23 04:52, Eric Kuhnke wrote:
I sincerely doubt there is much demand for *new* 40G these days.
Look at the population of 40G members on major IXes.
People have either one 10G, 2 x 10G, or 100G.
40G was a dead-end 9 years ago and much so more now.
We have customers that sometimes ask for 40Gbps interconnects. I always tell our Pre-Sales team that those are the ones who "led the way", back in the day. Sadly, they were a bit too early :-). Mark.
On 8/28/23 03:05, Mike Hammett wrote:
Well, or they simply found a potential deal on hardware that came with 40 gig ports. 40 gigs is still a lot of bits to a lot of people.
For internal use, sure. But when connecting to another AS, the chances of them supporting 40Gbps in one or more places is inconsistent to slim. Exchange points may be an exception. Mark.
(Enterprise AS for context) This hasn’t been my experience in the US, however we mostly deal in tier 2 markets (I.e. Detroit, Miami, Dallas, etc…) and we have plenty of 40G private interconnects. I don’t doubt 40G is going away, I’ve just never had trouble using it around here. The only time we’ve been asked to run something other than 40G was because we like to run our ports very hot (latency insensitive traffic) and some networks do not tolerate consistently high utilization of their ports. Different story in Japan, it’s 100G+ or nothing. You just have to find someone willing to peer with you in the first place… Sent from my iPhone
On Aug 27, 2023, at 23:43, Mark Tinka <mark@tinka.africa> wrote:
On 8/28/23 03:05, Mike Hammett wrote:
Well, or they simply found a potential deal on hardware that came with 40 gig ports. 40 gigs is still a lot of bits to a lot of people.
For internal use, sure.
But when connecting to another AS, the chances of them supporting 40Gbps in one or more places is inconsistent to slim.
Exchange points may be an exception.
Mark.
On 8/28/23 07:55, Daniel Marks wrote:
(Enterprise AS for context)
This hasn’t been my experience in the US, however we mostly deal in tier 2 markets (I.e. Detroit, Miami, Dallas, etc…) and we have plenty of 40G private interconnects. I don’t doubt 40G is going away, I’ve just never had trouble using it around here.
This would be expected, since most 40Gbps hardware I have seen in Europe and Africa is in the enterprise and exchange point space. If service providers have them, I'd imagine that inventory is far lower than 100Gbps. Mark.
Look at the population of 100G ports at the SIX in Seattle as well. I think there's a total of maybe four 40G members out of hundreds. 100G really is the new 10. On Sun, Aug 27, 2023, 10:56 PM Daniel Marks via NANOG <nanog@nanog.org> wrote:
(Enterprise AS for context)
This hasn’t been my experience in the US, however we mostly deal in tier 2 markets (I.e. Detroit, Miami, Dallas, etc…) and we have plenty of 40G private interconnects. I don’t doubt 40G is going away, I’ve just never had trouble using it around here.
The only time we’ve been asked to run something other than 40G was because we like to run our ports very hot (latency insensitive traffic) and some networks do not tolerate consistently high utilization of their ports.
Different story in Japan, it’s 100G+ or nothing. You just have to find someone willing to peer with you in the first place…
Sent from my iPhone
On Aug 27, 2023, at 23:43, Mark Tinka <mark@tinka.africa> wrote:
On 8/28/23 03:05, Mike Hammett wrote:
Well, or they simply found a potential deal on hardware that came with 40 gig ports. 40 gigs is still a lot of bits to a lot of people.
For internal use, sure.
But when connecting to another AS, the chances of them supporting 40Gbps in one or more places is inconsistent to slim.
Exchange points may be an exception.
Mark.
Sounds more like a Huawei roadmap oops, didn't mean to mention names :) On Sun, 27 Aug 2023, 07:02 Mark Tinka, <mark@tinka.africa> wrote:
On 8/27/23 04:52, Eric Kuhnke wrote:
I sincerely doubt there is much demand for *new* 40G these days.
Look at the population of 40G members on major IXes.
People have either one 10G, 2 x 10G, or 100G.
40G was a dead-end 9 years ago and much so more now.
We have customers that sometimes ask for 40Gbps interconnects. I always tell our Pre-Sales team that those are the ones who "led the way", back in the day. Sadly, they were a bit too early :-).
Mark.
On 8/23/23 18:29, tim@pelican.org wrote:
Not Trio, and different PLM :)
Yes, aware... I was just speaking in general for what is likely to be a very popular platform :-).
MX304 (well, strictly LMIC16) has the same restriction, and a need for another entry in the magic port checker (https://apps.juniper.net/home/port-checker/index.html) for restrictions beyond "SUM(port-speeds) <= 1.6T".
Yep. That trick they did where you can live with one RE and get 3 MIC's in the MX304 is... well, I guess everyone will have their own opinion.
They make sense once you've looked at the block diagram for the thing and followed the lines, but things like "4x10G breakout can only go in odd-numbered ports, and you have to leave the corresponding next-lowest even-numbered port empty" are not instantly obvious.
They do take some getting used to. But this is what comes with all the flexibility operators often seek. Mark.
participants (13)
-
Aaron Gould -
Aaron1
-
Andrey Kostin -
Chris -
Daniel Marks -
Eric Kuhnke -
Mark Tinka -
Matt Erculiani -
Mike Hammett -
Pascal Masha -
Ryan Hamel -
tim＠pelican.org -
Tom Beecher