Does anyone have any experience with peering at this NAP, the doc's I read say I can connect to the NAP in three flavors ATM, POS, and Ethernet. But the technical side of the house tells me I must co-locate and can only connect via ethernet for peering. Also any way to find out who is currently peering today, their web page doesn't tell me much. I would also be curios to know if there is a peermaker type site or how elegant it is to setup peering with someone at this NAP. Thanks...
Currently, it is an Ethernet NAP, similar to PAIX. I doubt you'll see any more ATM NAPs. Basically, you can pull whatever kind of ckt you want into the facility, then terminate it on your own router. Then you run Ethernet to the switching fabric. I don't think they have a peermaker app, but it has been discussed for NOTA, so I'm sure they'll build one if there is demand. The only real reason to have one at an ethernet NAP is for building private VLANs, which may or may not get heavily used. As NOTA is run as a collaboration between Terremark and an industry consortium (which you become a member of when you come into the NAP), they tend to be quite responsive. BTW, the alternative to placing your equipment at a NAP like this is to get a metro ethernet like from someone like Yipes or Telseon, and locate your gear somewhere else. Epik provides metro ethernet into that facility, and others may as well. Of course, you would still need to have a router somewhere in the Miami metro area. - Daniel Golding -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Vinny India Sent: Wednesday, August 22, 2001 2:49 PM To: nanog@merit.edu Subject: Maimi Nap of the America's Does anyone have any experience with peering at this NAP, the doc's I read say I can connect to the NAP in three flavors ATM, POS, and Ethernet. But the technical side of the house tells me I must co-locate and can only connect via ethernet for peering. Also any way to find out who is currently peering today, their web page doesn't tell me much. I would also be curios to know if there is a peermaker type site or how elegant it is to setup peering with someone at this NAP. Thanks...
At 15:06 22/08/01, Daniel Golding wrote:
Currently, it is an Ethernet NAP, similar to PAIX. I doubt you'll see any more ATM NAPs. Basically, you can pull whatever kind of ckt you want into the facility, then terminate it on your own router. Then you run Ethernet to the switching fabric.
I agree that ATM NAPs are probably on their way out over time. GigE Ethernet switches are a whole lot cheaper than ATM switches and are currently scaling much better than ATM switches. For example, at least one vendor has a GigE switch that has 256 GB of non-blocking switch fabric and can support up to 192 GigE ports or 1440 10/100baseT ports (or some mixture in between if preferable). 10 GigE products will likely be available from multiple vendors by end of this calendar year. Folks building Ethernet based NAPs might want to consider building them with Ethernet equipment that can support non-IEEE-standard large frame sizes (e.g. 4K is popular with POS-oriented folks; 9K is popular with folks who run server farms) in the interest of side-stepping PMTU-Discovery/IP-Fragmentation issues. Many GigE or 10 GigE vendors support the 9K MTU on switched Ethernet ports (jumbo frames generally won't work if the link is half-duplex, btw, because of adverse interactions with CSMA/CD timers). Similarly, folks connecting at such NAPs might want to find out if large non-IEEE-standard frames are supported in the switch fabric. If yes, those folks might want to think about whether they want to deploy IP routers that can support such large frames (e.g. Juniper's current GigE interface reportedly supports 9K frames if configured that way) for the same reasons. Your mileage might vary, this is just something to mull over. Any flames to /dev/null please. Ran rja@inet.org
On Wed, 22 Aug 2001, RJ Atkinson wrote: It's no rocket science. You get your highend GigE switch, you set it to 4470 MTU. You use either 7200 with PA-GE, or GSR with 1GE (avoid 3GE here, 2450 MTU doesn't cut it). Juniper supports jumbos as well. Most vendors do. It works well in my experience, but my experience is also that most people want to talk using Ciscos anyway (at least over here). Apart from the obvious speed issue FDDI worked well. I see no major difference to what GigE has to offer. The only thing it lacks is the A+B protection, and I don't see that as such a big issue, I mean, how often is it the NAP equipment that fails compared to all other fault factors? We are currently using SRP OC12 in Stockholm with 22 nodes connected, as an exchange point. My personal opinion is that SRP is no good way to connect a lot of people with. Packets have to traverse a lot of nodes to get to where they're supposed to, with all that latency and inefficient use of bandwidth. Price is also a heavy issue with SRP, especially the OC48 cards listing at something like $100k and you have to get a GSR to put it in. ATM is an option. Also expensive. GigE is an option. Cheap, efficient, supported by loads and loads of vendors. Just remember to do jumbos and you're fine. What else is there? POS, but that doesn't scale very well when you're going fully meshed. What else is there? I can't think of anything. -- Mikael Abrahamsson email: swmike@swm.pp.se
At 16:22 22/08/01, Mikael Abrahamsson wrote:
You get your highend GigE switch, you set it to 4470 MTU. You use either 7200 with PA-GE, or GSR with 1GE (avoid 3GE here, 2450 MTU doesn't cut it).
Maybe 4470, but some operators prefer ~9K. Different operators might have different circumstances and hence different opinions. If I were running an exchange point, I'd configure the exchange switch for ~9K and leave the choice of MTU on the connecting device up to my customers -- because that would maximise the potential customer base and I am a capitalist. Btw, some old GSR 1 GigE interfaces did not support anything above IEEE-standard MTU. I infer from your note that there are now some GSR interface cards that do support larger than IEEE-standard MTU. I haven't used any of the supposed new cards, but deployed several of the older ones that didn't support anything above IEEE-standard MTU on GigE. Or maybe it was an IOS thing that changed in the meantime. Cheers, Ran rja@inet.org
I just have to speek up that this is all very well and good, but it's also a good way to make a NAP that doesn't work. _ALL_ devices on a layer-2 fabric need to have the same MTU. That means if there are any FastEthernet or Ethernet connected members 1500 bytes is it. It also means if you pick a larger value (4470, 9k) _ALL_ members must use the same value. If you don't, the behavior is simple. A 9k MTU GigE arps for a 1500 byte FastEthernet host. Life is good. The TCP handshake completes, life is good. TCP starts to send a packet, putting a 9k frame on the wire. Depending the switch, the switch either drops it as over MTU for the FastEthernet, or the FastEthernet card cuts it off at 1500 bytes, and counts it as an errored frame (typically with a jabber or two afterwards) and no data flows. A larger MTU is a fine plan, but make sure if you try that anywhere that the switch is set for the larger sizes and all devices are capable of that larger frame size, or you're in trouble. -- Leo Bicknell - bicknell@ufp.org Systems Engineer - Internetworking Engineer - CCIE 3440 Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
On Wed, 22 Aug 2001, Leo Bicknell wrote:
_ALL_ devices on a layer-2 fabric need to have the same MTU. That means if there are any FastEthernet or Ethernet connected members 1500 bytes is it. It also means if you pick a larger value (4470, 9k) _ALL_ members must use the same value.
If you don't, the behavior is simple. A 9k MTU GigE arps for a 1500 byte FastEthernet host. Life is good. The TCP handshake completes, life is good. TCP starts to send a packet, putting a 9k frame on the wire. Depending the switch, the switch either drops it as over MTU for the FastEthernet, or the FastEthernet card cuts it off at 1500 bytes, and counts it as an errored frame (typically with a jabber or two afterwards) and no data flows.
Well, the reasoning "why" is a bit more complex than that... The TCP handshake will result in the FE host saying "hey, I can do a max 1460 byte mss". The other host with a larger MTU won't send larger packets than remote MSS + 40 bytes header over that TCP connection, end of story. Now, sure, you certainly have to have agreements between devices in various contexts, but what is and isn't a "working" configuration and why is a bit more complex. A can't-go-wrong simplification, of course, is "always make sure all devices on the same L2 have the same MTU"...
On Wed, Aug 22, 2001 at 05:24:12PM -0700, Marc Slemko wrote:
If you don't, the behavior is simple. A 9k MTU GigE arps for a 1500 byte FastEthernet host. Life is good. The TCP handshake completes, life is good. TCP starts to send a packet, putting a 9k frame on the wire. Depending the switch, the switch either drops it as over MTU for the FastEthernet, or the FastEthernet card cuts it off at 1500 bytes, and counts it as an errored frame (typically with a jabber or two afterwards) and no data flows.
Well, the reasoning "why" is a bit more complex than that... The TCP handshake will result in the FE host saying "hey, I can do a max 1460 byte mss". The other host with a larger MTU won't send larger packets than remote MSS + 40 bytes header over that TCP connection, end of story.
I was assuming the TCP connection was between two end stations, not to the device itself. Consider: host---GE---exchange router--GE--<fabric>--FE--exchange router---GE---host The two hosts talk, get MSS of 9k (ish), SYN, SYN+ACK, ACK are all small and pass, the first data packet tries to be 9k, and gets lost between the fabric and the exit (FE) router. Yes, if the host talked to the FE device itself MSS should prevent any issues (eg, for BGP sessions across the exchange). -- Leo Bicknell - bicknell@ufp.org Systems Engineer - Internetworking Engineer - CCIE 3440 Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
On Wed, 22 Aug 2001, Marc Slemko wrote:
Now, sure, you certainly have to have agreements between devices in various contexts, but what is and isn't a "working" configuration and why is a bit more complex. A can't-go-wrong simplification, of course, is "always make sure all devices on the same L2 have the same MTU"...
One approach would be separate VLANs for ge and fe peers. Just set the mtu separately for the virtual interface in each separate VLAN. -Dan -- [-] Omae no subete no kichi wa ore no mono da. [-]
In article <Pine.BSF.4.20.0108221717430.8992-100000@alive.znep.com>, Marc Slemko <marcs@znep.com> wrote:
On Wed, 22 Aug 2001, Leo Bicknell wrote:
_ALL_ devices on a layer-2 fabric need to have the same MTU. That means if there are any FastEthernet or Ethernet connected members 1500 bytes is it. It also means if you pick a larger value (4470, 9k) _ALL_ members must use the same value.
If you don't, the behavior is simple. A 9k MTU GigE arps for a 1500 byte FastEthernet host. Life is good. The TCP handshake completes, life is good. TCP starts to send a packet, putting a 9k frame on the wire. Depending the switch, the switch either drops it as over MTU for the FastEthernet, or the FastEthernet card cuts it off at 1500 bytes, and counts it as an errored frame (typically with a jabber or two afterwards) and no data flows.
Well, the reasoning "why" is a bit more complex than that... The TCP handshake will result in the FE host saying "hey, I can do a max 1460 byte mss". The other host with a larger MTU won't send larger packets than remote MSS + 40 bytes header over that TCP connection, end of story.
So it's simply waiting for a routing vendor that sets the MTU per endpoint based on the MSS in the TCP handshake for the BGP session. Mike. -- "Answering above the the original message is called top posting. Sometimes also called the Jeopardy style. Usenet is Q & A not A & Q." -- Bob Gootee
On Thu, 23 Aug 2001, Miquel van Smoorenburg wrote:
So it's simply waiting for a routing vendor that sets the MTU per endpoint based on the MSS in the TCP handshake for the BGP session.
Or at least where you can somehow specify different MTUs for different IPs within a subnet. Then that could be a part of any BGP peer setup that you also include the MTU you can handle. -- Mikael Abrahamsson email: swmike@swm.pp.se
At 19:43 22/08/01, Leo Bicknell wrote:
_ALL_ devices on a layer-2 fabric need to have the same MTU. That means if there are any FastEthernet or Ethernet connected members 1500 bytes is it. It also means if you pick a larger value (4470, 9k) _ALL_ members must use the same value.
Only devices exchanging frames with each other need have the same MTU. One could easily imagine one 802.1q VLAN with MTU N and a different 802.1q VLAN with MTU Y all on the same switch, to give a trivial example. Some exchanges already use VLANs with their GigE switches for a variety of business reasons. Ran rja@inet.org
On Wed, 22 Aug 2001, RJ Atkinson wrote:
Only devices exchanging frames with each other need have the same MTU. One could easily imagine one 802.1q VLAN with MTU N and a different 802.1q VLAN with MTU Y all on the same switch, to give a trivial example. Some exchanges already use VLANs with their GigE switches for a variety of business reasons.
Which is exactly how we've done it with our testbed here in Stockholm. One VLAN where you are REQUIRED to handle 4470 MTU to be able to join, and one MTU 1500 vlan. There are some equipment that'll fragment IP when doing L2 forwarding to a lesser MTU port (FE for instance) (one of the old Cabletron companies) just like the old DEC gigaswitches used to do, but I prefer to have everybody in the same L2 domain talk the same size MTU. Seems much cleaner, and most vendors support at least 4470MTU on their GigE ports these days. -- Mikael Abrahamsson email: swmike@swm.pp.se
At 02:31 23/08/01, Mikael Abrahamsson wrote:
...most vendors support at least 4470MTU on their GigE ports these days.
True. As near as I can tell, nearly every vendor is supporting ~9K MTU on GigE/10 GigE ports. Not all of the vendors with ~9K MTU support permit configuration of 4470 byte MTU -- a number of them have binary configuration knobs for ~1500 byte/~9K byte MTU, though clearly a number of vendors do permit configuration of a 4470 MTU. In some cases that configuration constraint might just be a software issue. In other cases, there might be hardware issues. The MTU size is related to certain other timer parameters and isn't really entirely an independent variable, btw. Folks buying GigE interfaces might want to investigate these factors as part of their product evaluations. Ran rja@inet.org
Leo Bicknell wrote:
I just have to speek up that this is all very well and good, but it's also a good way to make a NAP that doesn't work. [...] If you don't, the behavior is simple. A 9k MTU GigE arps for a 1500 byte FastEthernet host. Life is good. The TCP handshake completes, life is good. TCP starts to send a packet, putting a
Furthermore... Larger frames would be nice if all hosts supported them, but the problem is that the that most end hosts cannot and probably will not ever support so called jumbo frames. What does having 9K ethernet frame support at a NAP get us? If one end of the connection, all those 1500 MTU end hosts, don't support large frames fragmentation may have to occur somewhere and that would probably be worse. With fragmentation, the performance problem is pushed closer out to the edge and the edge is probably where the performance benefit is needed so it could be a step in the wrong direction. Perhaps the one good approach to jumbo frames is to make use of the networking layer and ensure hosts are doing Path MTU discovery to avoid fragmentation. John
On Thu, 23 Aug 2001, John Kristoff wrote:
not ever support so called jumbo frames. What does having 9K ethernet frame support at a NAP get us? If one end of the connection, all those
You want to at least support 1600 or so, "baby jumbos" so people can tunnel stuff without fragmentation if they like to. If you support 1600, you're very likely to be able to support 4470 or larger, so why not do that? Anyhow, you're removing one more reason why some people can say "ethernet is lame for NAPs". If NAPs do not support jumbos, then end systems will never support them. If all internet backbone infrastructure supports larger frames, then there is at least a possibility that end systems will in the long run. A lot of servers are being connected via GigE nowadays, and with jumbos being a possibility, why not support it? -- Mikael Abrahamsson email: swmike@swm.pp.se
Mikael Abrahamsson wrote:
If NAPs do not support jumbos, then end systems will never support them.
Many end systems will never support jumbo frames, period. There are lots of 10/100 Mb/s ethernet hosts that will not ever be upgraded.
A lot of servers are being connected via GigE nowadays, and with jumbos being a possibility, why not support it?
Certainly. In a nutshell, it might be best to take steps to avoid fragmentation elsewhere in the network. Perhaps a rule of thumb that should be stressed is to use jumbo frames if you know for sure the other end system(s) support it, otherwise default to 1500. John
At 10:18 23/08/01, John Kristoff wrote:
Certainly. In a nutshell, it might be best to take steps to avoid fragmentation elsewhere in the network. Perhaps a rule of thumb that should be stressed is to use jumbo frames if you know for sure the other end system(s) support it, otherwise default to 1500.
Or just use Path MTU Discovery. For IP traffic that is traversing more than one layer-2 network, the variety in network technologies (e.g. ATM, SMDS, POS, Radio, Satellite, other) means that even ~1500 byte IP frames might not always work end-to-end. For example, I know of several commercial IP/SATCOM systems that have an MTU of 576 bytes. Many ISPs, not all, seem to try to engineer an end-to-end MTU of 4470 (that number chosen for historical reasons relating to FDDI). Some ISPs use a smaller MTU number and some use a higher MTU number. Ran rja@inet.org
In article <5.1.0.14.2.20010823102412.01d94dd0@10.30.15.2>, RJ Atkinson <rja@inet.org> wrote:
At 10:18 23/08/01, John Kristoff wrote:
Certainly. In a nutshell, it might be best to take steps to avoid fragmentation elsewhere in the network. Perhaps a rule of thumb that should be stressed is to use jumbo frames if you know for sure the other end system(s) support it, otherwise default to 1500.
Or just use Path MTU Discovery.
Ofcourse, for path MTU discovery to work, a router must know that the outgoing link has a smaller MTU than the packet so that it can send back an ICMP error to the sender. On a GigE interface with jumbo frames where the other side on the same ethernet might not support jumbo frames that is probably not the case. The packet will simply get dropped and the sender will never know why it was dropped. Mike. -- "Answering above the the original message is called top posting. Sometimes also called the Jeopardy style. Usenet is Q & A not A & Q." -- Bob Gootee
At 11:14 23/08/01, Miquel van Smoorenburg wrote:
On a GigE interface with jumbo frames where the other side on the same ethernet might not support jumbo frames that is probably not the case. The packet will simply get dropped and the sender will never know why it was dropped.
Folks sending anything larger than IEEE standard frame sizes on any Ethernet need to be sure to re-configure all the equipment accordingly and test the whole subnet out thoroughly. For now, there seems to be no substitute for that (and yes, it is a fair bit of work). As near as I can tell, all switches ship with IEEE-standard MTU as the default, so require manual operator reconfiguration to support any larger frame size. Ran rja@inet.org
On Thu, Aug 23, 2001 at 09:18:09AM -0500, John Kristoff wrote:
Mikael Abrahamsson wrote:
If NAPs do not support jumbos, then end systems will never support them.
Many end systems will never support jumbo frames, period. There are lots of 10/100 Mb/s ethernet hosts that will not ever be upgraded.
3Com Vortex/Boomerang 10/100 Mb/s claims to support FIDDI size packets 4.5K. Texas Instruments ThunderLAN 10/100Mbps PCI Network Controller can support 65536 bytes. Details http://www.scyld.com/expert/100mbps.html Any one has experience with TI chipset ? for those who interested in IPv6, Path MTU Discovery is not in IPv6 stack. To have end to end jumbo frames in native v6 we NAPs in the path should support jumbo frames. -antony
At 11:37 23/08/01, Antony Antony wrote:
for those who interested in IPv6, Path MTU Discovery is not in IPv6 stack. To have end to end jumbo frames in native v6 we NAPs in the path should support jumbo frames.
More precisely, with IPv6 networks, either: (1) Path MTU Discovery is ALWAYS enabled and cannot be disabled XOR (2) the sending system never sends an IP packet larger than 1280 bytes. [Source: RFC-2460, Section 5, Page 24] Ran rja@inet.org
On Thu, Aug 23, 2001 at 12:04:36PM -0400, RJ Atkinson wrote:
At 11:37 23/08/01, Antony Antony wrote:
for those who interested in IPv6, Path MTU Discovery is not in IPv6 stack. To have end to end jumbo frames in native v6 we NAPs in the path should support jumbo frames.
More precisely, with IPv6 networks, either:
Sorry. I ment to say routers don't perform fragmentation of packets they're forwarding. If a packet is to be forwarded onto a link whose link MTU is less than the size of the packet, the node must discard the packet and send an ICMP Packet Too Big message to the packet's Source Address. ICMP error message also contains MTU of the link. Host can retransmit with size obtained from the ICMP message. To have jumbo frames in IPv6 from end to end and path is via NAP, NAPs should support it. Or am I still mistaken ?
[Source: RFC-2460, Section 5, Page 24] Same RFC section 4.4 and 4.5
-antony
On Thu, 23 Aug 2001, Antony Antony wrote:
3Com Vortex/Boomerang 10/100 Mb/s claims to support FIDDI size packets 4.5K.
They do.
for those who interested in IPv6, Path MTU Discovery is not in IPv6 stack. To have end to end jumbo frames in native v6 we NAPs in the path should support jumbo frames.
What? That is incorrect! Any IPv6 host which can sent over the minimum MTU *MUST* support Path MTU Discovery. It can get away without it if it always uses the minimum MTU, which it will tell the other end about directly, thus no issue.
On Thu, 23 Aug 2001, John Kristoff wrote:
Mikael Abrahamsson wrote:
If NAPs do not support jumbos, then end systems will never support them.
Many end systems will never support jumbo frames, period. There are lots of 10/100 Mb/s ethernet hosts that will not ever be upgraded.
While not a standard feature, many 100mbit nics support jumbo on full-duplex links and some 100tx switches do as well.
Certainly. In a nutshell, it might be best to take steps to avoid fragmentation elsewhere in the network. Perhaps a rule of thumb that should be stressed is to use jumbo frames if you know for sure the other end system(s) support it, otherwise default to 1500.
Path MTU discovery works fine. (except for broken firewalls) Your complaint about end systems never supporting jumbos is a moot point because if neither end supports them, then PMTU discovery isn't even an issue.
At 09:57 23/08/01, John Kristoff wrote:
Furthermore... Larger frames would be nice if all hosts supported them, but the problem is that the that most end hosts cannot and probably will not ever support so called jumbo frames.
WindowsNT/Windows2000 [1] and a lot of UNIX servers/hosts do support 9K frames today. Most GigE PCI NIC cards support them. Most of the commodity GigE ASICs support the 9K MTU. You are correct that not all hosts/servers support them today. In any event, any size of jumbo Ethernet frame will only work over an all-switched layer-2 network. My guess is that the trend over time will be for more and more hosts to support the ~9K MTU. YMMV.
What does having 9K ethernet frame support at a NAP get us?
Some folks run their end-to-end network with a 9K MTU. So having it at the NAP means they avoid potential fragmentation in their network. Certainly my employer would prefer a WAN network provider that supported the 9K MTU because it would improve NFS performance among our several sites as compared with a smaller end-to-end MTU.
Perhaps the one good approach to jumbo frames is to make use of the networking layer and ensure hosts are doing Path MTU discovery to avoid fragmentation.
Path MTU Discovery is curiously controversial in some circles. My own experience is that PMTUD works well today (not necessarily true 5 years ago). So I agree that ensuring Path MTU Discovery is deployed is generally clever. Past experience is that many vocal folks will disagree with this view. Ran rja@inet.org [1] Someone at Microsoft has told me that use of ~9K frames is how Microsoft got their high WinNT network throughput for the SuperComputing conference demo a few years back. I'm also told the POS links used in that demo had also been configured for a ~9K MTU and worked fine.
> Currently, it is an Ethernet NAP, similar to PAIX. If you like PAIX and want to be in Miami, it seems simplest to just use the PAIX Miami switch-fabric i the Switch & Data facility... http://www.switchanddata.com/locations/pdf/miami_site_specs.pdf Then once NOTA comes online, you can just buy a VLAN over to there, or peer with whichever NOTA folks connect to the PAIX switch-fabric there, assuming it goes in. I think this is the same situation as in the San Francisco area, where the PAIX switch-fabric appears in Palo Alto, San Francisco, and down at MAE-West/AMES. -Bill
participants (13)
-
Antony -
Antony Antony
-
Bill Woodcock -
Dan Hollis -
Daniel Golding -
Greg Maxwell -
John Kristoff -
Leo Bicknell -
Marc Slemko -
Mikael Abrahamsson -
miquels＠cistron-office.nl -
RJ Atkinson -
Vinny India