A few time already I've wished for a fully standardized and vendor interoperable way of doing a true point to point ethernet link. It would work just like an old leased line or synchronous serial interface and completely do away with ARP, MAC addresses and all that stuff. Obviously no switches in between would be allowed. Each side would run in "promiscuous mode" where every ethernet frame is received and passed up to the network stack (just like on a serial link). Since MAC addresses are useless they can be scrapped and only the ethertype field remains. This increases the effective MTU by 12 bytes. The framing overhead goes away and the packet can directly be directly placed on the wire without taking a detour through L3->L2 lookup and encapsulation step. More importantly one can specify the just the outgoing interface again instead of the next hop: ip route 10.0.0.0 255.0.0.0 g0/1 Do you think this is useful? Maybe vendors will hear me/us. -- Andre
We're halfway there (OK, a bit less, they've messed up OSPF) with the unnumbered VLAN interfaces. http://wiki.nil.com/Unnumbered_Ethernet_VLAN_interfaces What's missing is the removal of MAC layer header, but that would require modifications to the NIC chipsets (= expensive). Ivan http://www.ioshints.info/about http://blog.ioshints.info/
-----Original Message----- From: Andre Oppermann [mailto:nanog-list@nrg4u.com] Sent: Wednesday, July 08, 2009 12:01 PM To: nanog@nanog.org Subject: Point to Point Ethernet
A few time already I've wished for a fully standardized and vendor interoperable way of doing a true point to point ethernet link.
It would work just like an old leased line or synchronous serial interface and completely do away with ARP, MAC addresses and all that stuff. Obviously no switches in between would be allowed. Each side would run in "promiscuous mode" where every ethernet frame is received and passed up to the network stack (just like on a serial link). Since MAC addresses are useless they can be scrapped and only the ethertype field remains. This increases the effective MTU by 12 bytes.
The framing overhead goes away and the packet can directly be directly placed on the wire without taking a detour through L3->L2 lookup and encapsulation step.
More importantly one can specify the just the outgoing interface again instead of the next hop:
ip route 10.0.0.0 255.0.0.0 g0/1
Do you think this is useful? Maybe vendors will hear me/us.
-- Andre
Do you think this is useful? Maybe vendors will hear me/us.
-- Andre
We also need functional remote loop testing, of the "remote hands guy plugs in a loopback plug" or "I send remote-triggered loop" type. David Barak Need Geek Rock? Try The Franchise: http://www.listentothefranchise.com
There are lots of great little cable testers that can "loop" an Ethernet link or even blink the switchport (this one is copper only): http://www.jdsu.com/products/communications-test-measurement/products/a-z-pr oduct-list/lanscaper.html The remote-triggered is harder, but there are a number of switches I have seen that have some form of line testing built in, so that might be close to a decent solution. One example is the "Integrated Cable Test" and Optical Transceiver Diagnostics in the Dell PowerConnect switches. -Scott -----Original Message----- From: David Barak [mailto:thegameiam@yahoo.com] Sent: Wednesday, July 08, 2009 9:47 AM To: 'Andre Oppermann'; nanog@nanog.org; Ivan Pepelnjak Subject: RE: Point to Point Ethernet
Do you think this is useful? Maybe vendors will hear me/us.
-- Andre
We also need functional remote loop testing, of the "remote hands guy plugs in a loopback plug" or "I send remote-triggered loop" type. David Barak Need Geek Rock? Try The Franchise: http://www.listentothefranchise.com
Visit the Accedian website to learn about Ethernet demarcation and related standards. They market me heavily (and it apparently works). Frank -----Original Message----- From: David Barak [mailto:thegameiam@yahoo.com] Sent: Wednesday, July 08, 2009 8:47 AM To: 'Andre Oppermann'; nanog@nanog.org; Ivan Pepelnjak Subject: RE: Point to Point Ethernet
Do you think this is useful? Maybe vendors will hear me/us.
-- Andre
We also need functional remote loop testing, of the "remote hands guy plugs in a loopback plug" or "I send remote-triggered loop" type. David Barak Need Geek Rock? Try The Franchise: http://www.listentothefranchise.com
On Wed, Jul 8, 2009 at 6:01 AM, Andre Oppermann<nanog-list@nrg4u.com> wrote:
Do you think this is useful?
Andre, Some thoughts on this: 1. What's the point of increasing the max MTU from 9000 to 9012? If we want a higher MTU, why not just ask for one in the next standard? 2. Why do we need to save 12 bytes per packet by eliminating the MAC address? Why not just get the next larger ethernet size? 3. Could we be better served by using RFC1918 addresses that we define as link-local and asking the router vendors to support a "link local" config option that causes it to use the address from loopback0 for any communications it initiates over that interface instead of using the interface IP address? In other words, if your loop0 is 199.33.224.1 and your g0/0 is 10.3.2.1/30 the link-local option would cause the router to send any host-unreachable messages out g0/0 from 199.33.225.1 instead of 10.3.2.1. Likewise, pings and snmp traps would originate from 199.33.224.1. Only packets to 10.3.2.2 would originate from 10.3.2.1. Such a software-only change would be less expensive to implement than custom ethernet hardware and it would be applicable on all interface types, not just ethernet. And of course we already have tools to prevent such link-local addresses from entering the routing protocol. At a software level, we could also declare a specific remote address as the point-to-point destination so that we could use the interface name as shorthand elsewhere in the config if that proves desirable. 4. L3->L2 lookup is a pretty negligible cost. It's cached for a good long while. And you already have tools to hardcode it if so desired. With Ciscos at least, you can even hardcode addresses to ffff.ffff.ffff though this causes some unexpected behavior. Regards, Bill -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Wed, 8 Jul 2009, William Herrin wrote:
1. What's the point of increasing the max MTU from 9000 to 9012? If we want a higher MTU, why not just ask for one in the next standard?
To me the only reason for this would be to lessen overhead on small packets. Also, afaik standard payload MTU is 1500 for ethernet, anything else is vendor extension, outside the standard. Ethernet overhead compared to HDLC is pretty big... -- Mikael Abrahamsson email: swmike@swm.pp.se
On Jul 8, 2009, at 10:37 AM, Mikael Abrahamsson wrote:
On Wed, 8 Jul 2009, William Herrin wrote:
1. What's the point of increasing the max MTU from 9000 to 9012? If we want a higher MTU, why not just ask for one in the next standard?
From what I have been told, IEEE 802 refuses to make a Jumbo frame standard, for backwards compatibility reasons. Joe St Sauver's jumbo frame site : http://darkwing.uoregon.edu/~joe/jumbo-clean-gear.html shows what a mess this is. There isn't a standard now, and if you "ask for one in the next standard" you may be in for a long wait.
To me the only reason for this would be to lessen overhead on small packets. Also, afaik standard payload MTU is 1500 for ethernet, anything else is vendor extension, outside the standard.
Ethernet overhead compared to HDLC is pretty big...
-- Mikael Abrahamsson email: swmike@swm.pp.se
Regards Marshall AmericaFree.TV
On Wed, Jul 8, 2009 at 10:50 AM, Marshall Eubanks<tme@americafree.tv> wrote:
On Jul 8, 2009, at 10:37 AM, Mikael Abrahamsson wrote:
On Wed, 8 Jul 2009, William Herrin wrote:
1. What's the point of increasing the max MTU from 9000 to 9012? If we want a higher MTU, why not just ask for one in the next standard? To me the only reason for this would be to lessen overhead on small packets.
Mikael, At the cost of low-volume production run hardware which is A. much more expensive (because of the low volume), B. restricted to a few supported routers and C. less thoroughly tested. I don't see how you come out ahead in that calculation.
Also, afaik standard payload MTU is 1500 for ethernet, anything else is vendor extension, outside the standard.
From what I have been told, IEEE 802 refuses to make a Jumbo frame standard, for backwards compatibility reasons.
Marshall, My understanding is that 9000 is a standard for GigE and up but for compatibility with earlier ethernets it's not the default. You have to explicitly configure it and you must configure it the same on every host and switch within the broadcast zone. For a point to point link, this should be trivial. Or am I mistaken? I gather from your list that not everything which supports gige also supports jumbo frames but that most things do. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Wed, Jul 8, 2009 at 11:07 AM, <sthaug@nethelp.no> wrote:
My understanding is that 9000 is a standard for GigE and up but for compatibility with earlier ethernets it's not the default.
Your understanding is wrong. The only IEEE standard is 1500 bytes.
Steinar, I 'spose I could have consulted wikipedia and gotten the answer: "The IEEE 802 standards committee does not recognize jumbo frames, as doing so would remove interoperability with other 802 protocols, including 802.5 Token Ring and 802.11 Wireless LAN. The use of 9,000 bytes as preferred size for jumbo frames arose from discussions within the Joint Engineering Team of Internet2 and the U.S. federal government networks. Their recommendation has been adopted by all other national research and education networks. In order to meet this mandatory purchasing criterion, manufacturers have in turn adopted 9,000 bytes as the conventional jumbo frame size." So 9000 for gige and up is a convention, not a standard. My bad. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Wed, 8 Jul 2009, William Herrin wrote:
At the cost of low-volume production run hardware which is A. much more expensive (because of the low volume), B. restricted to a few supported routers and C. less thoroughly tested. I don't see how you come out ahead in that calculation.
The only way to do it would be to make this a standard in the next evolution of Ethernet, perhaps 400GE. I don't see this happening though. But the only REASON to do it, would be to lessen overhead for small packets. I don't see how you can not see this.
My understanding is that 9000 is a standard for GigE and up but for compatibility with earlier ethernets it's not the default. You have to explicitly configure it and you must configure it the same on every host and switch within the broadcast zone. For a point to point link, this should be trivial.
No, IEEE says only 1500 payload MTU. This was discussed in 40GE and 100GE, and IEEE left the framesize the same way it's always been.
I gather from your list that not everything which supports gige also supports jumbo frames but that most things do.
Yes, but that doesn't make it standard. It makes it common. -- Mikael Abrahamsson email: swmike@swm.pp.se
1. What's the point of increasing the max MTU from 9000 to 9012? If we want a higher MTU, why not just ask for one in the next standard?
To me the only reason for this would be to lessen overhead on small packets. Also, afaik standard payload MTU is 1500 for ethernet, anything else is vendor extension, outside the standard.
Ethernet overhead compared to HDLC is pretty big...
Also, there would be small point in getting rid of the normal Ethernet headers if you still needed to use the standard Ethernet preamble and inter frame gap (a total of 20 bytes). These were really only needed for 10 Mbps Ethernet. I find it highly unlikely that such a change would be accepted - also I don't really see the big point. Steinar Haug, Nethelp consulting, sthaug@nethelp.no
My first thought was that there's really no use ripping the guts out of a protocol whose core mechanisms are aimed at dealing with the complexities of operating on a shared medium only to use it in an environment in which none of those complexities exist. But, if interfaces would be made to support both Ethernet II and some new Addressless Ethernet (or some other moniker) frames, the additional costs, real or administrative, wouldn't be outstanding. You might want to first try proving that reducing the Ethernet frame overhead by 2/3 and, in turn, reducing the average frame size by 12 / [average frame size] percent is worthwhile. Or try making the frame overhead reduction argument only a small piece of the larger argument for getting rid of multi-access cruft in point-to-point environments. But good luck pushing the principle argument of making things "as simple as possible but no simpler" without good technical and (hence) business cases. Stephen Kratzer Network Engineer CTI Networks, Inc. On Wednesday 08 July 2009 06:01:20 Andre Oppermann wrote:
A few time already I've wished for a fully standardized and vendor interoperable way of doing a true point to point ethernet link.
It would work just like an old leased line or synchronous serial interface and completely do away with ARP, MAC addresses and all that stuff. Obviously no switches in between would be allowed. Each side would run in "promiscuous mode" where every ethernet frame is received and passed up to the network stack (just like on a serial link). Since MAC addresses are useless they can be scrapped and only the ethertype field remains. This increases the effective MTU by 12 bytes.
The framing overhead goes away and the packet can directly be directly placed on the wire without taking a detour through L3->L2 lookup and encapsulation step.
More importantly one can specify the just the outgoing interface again instead of the next hop:
ip route 10.0.0.0 255.0.0.0 g0/1
Do you think this is useful? Maybe vendors will hear me/us.
Speaking from a personal interest, has the Point-to-Point Protocol stopped being useful? After all, PPP over Sonet/SDH was specifically designed for just this case. Once upon a time, it worked well for intra-site connections, as originally specified in RFC1619: PPP encapsulation over high speed private point-to-point links, such as intra-campus single-mode fiber which may already be installed and unused. It was only after others crammed stuff in to make it work on inferior quality long distance links that it became more expensive and complicated. Still, it has all the testing and link configuration mentioned. And very low overhead. And works on copper wiring, too....
Speaking from a personal interest, has the Point-to-Point Protocol stopped being useful?
After all, PPP over Sonet/SDH was specifically designed for just this case.
Absolutely, and it still works great for that purpose. However, given a provider backbone with Ethernet being the underlying technology, I don't see why anybody would want to use PPP on top of Ethernet instead of just plain Ethernet. After all we're *not* talking about DSL or dialup here, there's no need to authenticate users etc. Steinar Haug, Nethelp consulting, sthaug@nethelp.no
More importantly one can specify the just the outgoing interface again instead of the next hop:
ip route 10.0.0.0 255.0.0.0 g0/1
Do you think this is useful? Maybe vendors will hear me/us.
No. What makes Ethernet useful and successful is that, unlike most other network/interconnection technologies, it is cheap and plentiful, out of necessity. Any time you have something that is high volume, demand creates pressure to produce more cheaply, and costs slide as volumes spike. There are absolutely "better" choices for point to point circuits, and there are certainly ways to make a point to point version of Ethernet, but doing so invariably seems to involve a lot of reworking of the underlying mechanisms, which means that you've just invented a way to make your special-case Ethernet ... expensive. Anything that can be done exclusively in software, without modifying the silicon, is probably the only practical way to accomplish this. However, then you're still leaving more cans of worms to open, because then there'll be someone who wants to be able to do "something like" this but have vlan support so they can break out an expensive Foundry port on inexpensive switches and still do your new point to point protocol, etc. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
On 08.07.2009 18:04, Joe Greco wrote:
More importantly one can specify the just the outgoing interface again instead of the next hop:
ip route 10.0.0.0 255.0.0.0 g0/1
Do you think this is useful? Maybe vendors will hear me/us.
No. What makes Ethernet useful and successful is that, unlike most other network/interconnection technologies, it is cheap and plentiful, out of necessity. Any time you have something that is high volume, demand creates pressure to produce more cheaply, and costs slide as volumes spike.
There are absolutely "better" choices for point to point circuits, and there are certainly ways to make a point to point version of Ethernet, but doing so invariably seems to involve a lot of reworking of the underlying mechanisms, which means that you've just invented a way to make your special-case Ethernet ... expensive.
Anything that can be done exclusively in software, without modifying the silicon, is probably the only practical way to accomplish this. However, then you're still leaving more cans of worms to open, because then there'll be someone who wants to be able to do "something like" this but have vlan support so they can break out an expensive Foundry port on inexpensive switches and still do your new point to point protocol, etc.
This shouldn't require any hardware change. The PHY doesn't care at all and the MAC (Media Access Controller) can be told to ignore the DST MAC address and pass up any packet. So it ends up being only a software change. That is disabling the ARP lookup and the L3->L2 mapping table. I'm not attached to losing the ethernet header as long as a fixed MAC address as destination can be put there and every such incoming packet is accepted. -- Andre
On Wed, 08 Jul 2009 06:01:20 -0400, Andre Oppermann <nanog-list@nrg4u.com> wrote:
... completely do away with ARP, MAC addresses and all that stuff.
Removing "all that stuff" means it's no longer ethernet.
Do you think this is useful? Maybe vendors will hear me/us.
No. I do not. Ethernet is not a point-to-point technology. It is a multi-point (broadcast, bus, etc.) technology with DECADES of optimization and adoption. No one has gotten IEEE to adopt a larger frame size, and you want to drop *fundamental* elements of ethernet?!? --Ricky
On Wed, 8 Jul 2009, Ricky Beam wrote:
Ethernet is not a point-to-point technology. It is a multi-point (broadcast, bus, etc.) technology with DECADES of optimization and adoption. No one has gotten IEEE to adopt a larger frame size, and you want to drop *fundamental* elements of ethernet?!?
I think the latest suggestion was to do away with the mechanisms, not change the frame format. It's like when you take a /30, run isis/ospf over it and tell the routing protocol it's a point to point link so it doesn't have to create a node for the "multi access network" that really isn't there. Same way here, putting the ethernet link in "p2p" mode would mean it wouldnt do arp anymore, didn't care about source or destination MACs, it just installed static ARP entry for other end and sent out packets, other end would be in promisc mode and accept anything. I don't see much gain from this though, and it's another way things can be configured wrong and cause havoc if you connect this interface to a LAN. -- Mikael Abrahamsson email: swmike@swm.pp.se
The fundamental disconnect here is that a bunch of Layer 3 guys are trying to define Layer 2. History shows us that Layer 2 winds up being IEEE, and Layer 3 IETF. ITU-T and others write long "standards" that wind up not being so, due to too many "options", while spending lots of money and keeping the airlines, rental car companies, and meeting space vendors in business. If you want "Real Ethernet" (IE multiple access, not point to point) in a metro framework, then why increase the likelihood of collisions by using jumbo frames? If you want to use Ethernet in point to point, then do it, just make sure your optics are up to the task, and the endpoints are configured properly. If what you're looking for is carrier Ethernet with the sort of "craft" interfaces and features you're used to from the telco world, then you may want to talk to Ipitek. (I've done some consulting for them, but am in no way affiliated or compensated for sales.)
-----Original Message----- From: Mikael Abrahamsson [mailto:swmike@swm.pp.se] Sent: Wednesday, July 08, 2009 11:33 AM To: Ricky Beam Cc: nanog@nanog.org Subject: Re: Point to Point Ethernet
On Wed, 8 Jul 2009, Ricky Beam wrote:
Ethernet is not a point-to-point technology. It is a multi-point (broadcast, bus, etc.) technology with DECADES of optimization and adoption. No one has gotten IEEE to adopt a larger frame size, and you want to drop *fundamental* elements of ethernet?!?
I think the latest suggestion was to do away with the mechanisms, not change the frame format. It's like when you take a /30, run isis/ospf over it and tell the routing protocol it's a point to point link so it doesn't have to create a node for the "multi access network" that really isn't there.
Same way here, putting the ethernet link in "p2p" mode would mean it wouldnt do arp anymore, didn't care about source or destination MACs, it just installed static ARP entry for other end and sent out packets, other end would be in promisc mode and accept anything.
I don't see much gain from this though, and it's another way things can be configured wrong and cause havoc if you connect this interface to a LAN.
-- Mikael Abrahamsson email: swmike@swm.pp.se
On Wed, Jul 8, 2009 at 6:01 AM, Andre Oppermann<nanog-list@nrg4u.com> wrote:
Do you think this is useful? Maybe vendors will hear me/us.
They sort of did a few decades back, created HDLC (5 bytes minimum) and PPP (6 bytes minimum) for P2P links. I think you're at risk of over-thinking this problem working in reverse from ethernet to something slightly-less-than-ethernet. Further, if we want to get truly sizable improvement from 'ethernet like p2p paradigm' we can *drop the damn IFG and preample.* http://sd.wareonearth.com/~phil/net/overhead/ Best case, you blow 12 bytes on IFG in gig, 20 bytes on fast-e/slow-e. No matter how you slice it, it's not getting better than what we've already got (i.e. p2p link prots). Though, I do somewhat relate to your disgust and general sentiments. In 2009 I have cheap asics that can recover clock from line code alone and we're not doing CSMA/CD, so what's the freaking point of IFG and preamble? ./rhetorical (see lanhy vs. wanphy) -Tk
Best case, you blow 12 bytes on IFG in gig, 20 bytes on fast-e/slow-e.
As far as I know Gig and 10 Gig (with LAN PHY) are exactly the same as 10 and 100 Mbps in this respect, i.e. 8 bytes of preamble and 12 bytes of IFG. So you always have an overhead of 20 bytes, no matter what. 10 Gig with WAN PHY is a whole different ballgame, of course. Steinar Haug, Nethelp consulting, sthaug@nethelp.no
Overhead shmoverhead. Seriously, we're fighting over the non-issue. It's not the "wasted" 0.02% of bandwidth (@ 1Gbps) that's the issue. It's the utility of a "come as you are" "plug and play" network that "Ethernet" (which really loosely means all IEEE 802 protocols) provides, which the current carrier networks do not. If I read the thread correctly, what you really are asking for is the ability to plug your IEEE compliant gig/10gig switch into a carrier port and just have it ARP and respond for valid IP addresses on the segment, as opposed to all the back and forth provisioning, truck rolls, and interaction with bell-head union workers that the current system requires. Now, HOW to accomplish that is an interesting discussion, and the first valid result will probably be a great business. That doesn't require breaking Ethernet, using promiscuous mode, or much except the carriers stopping trying to throw their legacy circuit-switched requirements onto a packet switched network. There's plenty of fiber in the ground. Light dark stuff with the new network, plug it into IEEE 802* compliant layer 2, and IETF compliant layer 3 infrastructure; and leave the dying Bellcore/ITU network on the old copper and SONET.
-----Original Message----- From: sthaug@nethelp.no [mailto:sthaug@nethelp.no] Sent: Wednesday, July 08, 2009 11:34 PM To: tkapela@gmail.com Cc: nanog@nanog.org Subject: Re: Point to Point Ethernet
Best case, you blow 12 bytes on IFG in gig, 20 bytes on fast-e/slow-e.
As far as I know Gig and 10 Gig (with LAN PHY) are exactly the same as 10 and 100 Mbps in this respect, i.e. 8 bytes of preamble and 12 bytes of IFG. So you always have an overhead of 20 bytes, no matter what.
10 Gig with WAN PHY is a whole different ballgame, of course.
Steinar Haug, Nethelp consulting, sthaug@nethelp.no
For the sake of my knowledge (and perhaps that of some others on the list), I would like to ask if the current work on standards by IETF, ITU and IEEE not a step to address the issue of seamlessly using Ethernet in the metro/core? IETF is working on GMPLS Ethernet Label Switching (GELS), which proposes to replace the Ethernet control plane (MAC learning, spanning tree etc) by the GMPLS control plane. This would provide explicitly routed Ethernet LSPs. ITU seems to be working on Transport MPLS (T-MPLS), and IEEE seems to be at work on the Provider Backbone Bridge (PBB) standard. Granted the difficulties and faults with the standardization process, my question is more concerned with the technical nature. Thanks and best regards On Thu, Jul 9, 2009 at 11:51 AM, Tomas L. Byrnes <tomb@byrneit.net> wrote:
Overhead shmoverhead.
Seriously, we're fighting over the non-issue. It's not the "wasted" 0.02% of bandwidth (@ 1Gbps) that's the issue. It's the utility of a "come as you are" "plug and play" network that "Ethernet" (which really loosely means all IEEE 802 protocols) provides, which the current carrier networks do not.
If I read the thread correctly, what you really are asking for is the ability to plug your IEEE compliant gig/10gig switch into a carrier port and just have it ARP and respond for valid IP addresses on the segment, as opposed to all the back and forth provisioning, truck rolls, and interaction with bell-head union workers that the current system requires.
Now, HOW to accomplish that is an interesting discussion, and the first valid result will probably be a great business.
That doesn't require breaking Ethernet, using promiscuous mode, or much except the carriers stopping trying to throw their legacy circuit-switched requirements onto a packet switched network.
There's plenty of fiber in the ground. Light dark stuff with the new network, plug it into IEEE 802* compliant layer 2, and IETF compliant layer 3 infrastructure; and leave the dying Bellcore/ITU network on the old copper and SONET.
-----Original Message----- From: sthaug@nethelp.no [mailto:sthaug@nethelp.no] Sent: Wednesday, July 08, 2009 11:34 PM To: tkapela@gmail.com Cc: nanog@nanog.org Subject: Re: Point to Point Ethernet
Best case, you blow 12 bytes on IFG in gig, 20 bytes on fast-e/slow-e.
As far as I know Gig and 10 Gig (with LAN PHY) are exactly the same as 10 and 100 Mbps in this respect, i.e. 8 bytes of preamble and 12 bytes of IFG. So you always have an overhead of 20 bytes, no matter what.
10 Gig with WAN PHY is a whole different ballgame, of course.
Steinar Haug, Nethelp consulting, sthaug@nethelp.no
-- Muhammad Saqib Ilyas PhD Student, Computer Science and Engineering Lahore University of Management Sciences
On Wed, 8 Jul 2009, Tomas L. Byrnes wrote:
There's plenty of fiber in the ground. Light dark stuff with the new network, plug it into IEEE 802* compliant layer 2, and IETF compliant layer 3 infrastructure; and leave the dying Bellcore/ITU network on the old copper and SONET.
Have you built an nationwide greenfield network based on dark fibre and ethernet? I have. Don't misrepresent the problems that might arise from this. It might be easier to do compared to a SONET network, but it's still a can of worms and you definitely want a lot of the "bellhead" stuff you're ridiculing. -- Mikael Abrahamsson email: swmike@swm.pp.se
I frequently run into scenarios where two devices (two routers, or a router and a host) need a point-to-point connection to each other with a capacity of (much) more than 10 Gbps. For cost reasons, Ethernet is often used. Since more than 10 Gbps is needed, we end up with multiple parallel 10GE point-to-point connections. Because the devices often don't support LAG or have limitations on the number of links in a LAG, we often cannot use LAG at all or cannot put all 10GE links in a single LAG group. So, we end up with multiple parallel layer-3 point-to-point connections where each connections is either an Ethernet or a LAG group. Furthermore, in order to conserve IP addresses, there is a desire to make these interfaces unnumbered. The involved devices have a numbered loopback interface whose address is used as the "donor" for the unnumbered Ethernet / LAG interfaces. Most router vendors already support unnumbered point-to-point Ethernet, see for example: http://www.juniper.net/techpubs/en_US/junos9.5/information-products/topic-co... However, there are some interoperability issues / open questions related to point-to-point unnumbered Ethernet, see for example: http://forums.juniper.net/jnet/board/message?board.id=JUNOS&message.id=130 http://forums.juniper.net/jnet/board/message?board.id=switch&thread.id=835 I would be very interested in some standards (i.e. an IETF BCP) to describe the best current practices for these applications of Ethernet. I am not particularly interested in re-inventing a new flavor of Ethernet for this application. All that is needed, in my opinion, is some clarifications or best practices on how to use the existing standards to create point-to-point unnumbered Ethernet connections. PS -- I am also aware of some esoteric BRAS applications of Ethernet where one side is numbered and the other side is unnumbered.
Cayle, This may be partial hijack of the thread or even a trivial query but I ask this since you mentioned "For cost reasons, Ethernet is often used". We hear this argument all the time. The standard unabridged reason I have learned is the ubiquity of Ethernet devices, whatever that means. Can you say why precisely the cost of Ethernet is low compared to other viable alternatives? Zartash On Thu, Jul 9, 2009 at 7:29 PM, Cayle Spandon <cayle.spandon@gmail.com>wrote:
I frequently run into scenarios where two devices (two routers, or a router and a host) need a point-to-point connection to each other with a capacity of (much) more than 10 Gbps.
For cost reasons, Ethernet is often used.
Since more than 10 Gbps is needed, we end up with multiple parallel 10GE point-to-point connections.
Because the devices often don't support LAG or have limitations on the number of links in a LAG, we often cannot use LAG at all or cannot put all 10GE links in a single LAG group.
So, we end up with multiple parallel layer-3 point-to-point connections where each connections is either an Ethernet or a LAG group.
Furthermore, in order to conserve IP addresses, there is a desire to make these interfaces unnumbered.
The involved devices have a numbered loopback interface whose address is used as the "donor" for the unnumbered Ethernet / LAG interfaces. Most router vendors already support unnumbered point-to-point Ethernet, see for example:
http://www.juniper.net/techpubs/en_US/junos9.5/information-products/topic-co...
However, there are some interoperability issues / open questions related to point-to-point unnumbered Ethernet, see for example:
http://forums.juniper.net/jnet/board/message?board.id=JUNOS&message.id=130
http://forums.juniper.net/jnet/board/message?board.id=switch&thread.id=835
I would be very interested in some standards (i.e. an IETF BCP) to describe the best current practices for these applications of Ethernet.
I am not particularly interested in re-inventing a new flavor of Ethernet for this application. All that is needed, in my opinion, is some clarifications or best practices on how to use the existing standards to create point-to-point unnumbered Ethernet connections.
PS -- I am also aware of some esoteric BRAS applications of Ethernet where one side is numbered and the other side is unnumbered.
On Fri, 10 Jul 2009, Zartash Uzmi wrote:
Can you say why precisely the cost of Ethernet is low compared to other viable alternatives?
The components going into ethernet devices are cheaper because of high volume, but it's also that the SONET/SDH stuff is grossly overpriced "because we can" by short sighted vendors. There are cheap ethernet ports for cheap platforms, there are basically no cheap SONET/SDH ports anywhere. POS is technically better compared to Ethernet for carrier links imho, but for instance Cisco price their SPA-TENGE-XFP at 1/6 the cost of SPA-OC192-XFP. I know quite a lot of people who would gladly pay more for POS, but not that much more. -- Mikael Abrahamsson email: swmike@swm.pp.se
Prices of terrestrial SDH/SONET cards are very low for transport providers. For customers I believe there is a greater divergenc between the Ethernet and SONET/SDH costs. A strong hunch based on what clients tell me Cisco charges for SONET/SDH interfaces. Roderick S. Beck Director of European Sales Hibernia Atlantic 13-15, rue Sedaine, 75011 Paris http://www.hiberniaatlantic.com Wireless: 33+6+8692+5357. French Landline: 33+1+4355+8224 AOL Messenger: GlobalBandwidth rod.beck@hiberniaatlantic.com info@globalwholesalebandwidht.com ``Unthinking respect for authority is the greatest enemy of truth.'' Albert Einstein.
Prices of terrestrial SDH/SONET cards are very low for transport providers. For customers I believe there is a greater divergenc between the Ethernet and SONET/SDH costs.
A strong hunch based on what clients tell me Cisco charges for SONET/SDH interfaces.
I doubt a lot of people would think that SDH/SONET cards for *routers* are inexpensive. And yes, I have a reasonable idea of what kind of discounts are available out there... Steinar Haug, Nethelp consulting, sthaug@nethelp.no
Prices of terrestrial SDH/SONET cards are very low for transport providers. For customers I believe there is a greater divergenc between the Ethernet and SONET/SDH costs.
A strong hunch based on what clients tell me Cisco charges for SONET/SDH interfaces.
I doubt a lot of people would think that SDH/SONET cards for *routers* are inexpensive. And yes, I have a reasonable idea of what kind of discounts are available out there... Steinar Haug, Nethelp consulting, sthaug@nethelp.no Transport in my world is not Layer 3. It's Layer 2: Ethernet, SDH, SONET, waves. But my clients are mostly Layer 3. So you and I are in fundamental agreement. Roderick S. Beck Director of European Sales Hibernia Atlantic
On Thu, 09 Jul 2009 16:33:10 -0400, Zartash Uzmi <zartash@gmail.com> wrote:
... Can you say why precisely the cost of Ethernet is low compared to other viable alternatives?
Volume. Economies of scale. Etc. Ethernet is cheap because it's everywhere, and built into almost everything. (however, the likes of Cisco and Juniper still charge insane amounts for line cards, be they ethernet, T1, or OC48.) Given the choice of buying a $4k DS3 card or just plugging into an existing, builtin ethernet port, which do you think most people will choose? And it doesn't take a multi-thousand dollar Router(tm) to deal with ethernet -- a 200$ "trash" PC can handle routing (and NAT) for a great deal of traffic. (in many cases, *better* than the high priced kit.) Case in point, our voice/data line from TW enters the building as fiber (along with about 4000 other circuits), crawls up the inside of the building as HDSL (single pair) to a box in my "closet" where 8 POTS line and an ethernet are handed to me. The ethernet runs to a switch and then to the firewall. If it wasn't handed to me as ethernet, I'd need a router to turn it into ethernet. (On the other wall... $8k worth of gear to turn a DS3 into ethernet. Yes, the Optera Metro shelf at the other end of that DS3 could just as easily be an ethernet port -- but that would require TW and VZB to play nice with each other; it was enough of a pain to get the DS3 to work. But that's miles off topic.) --Ricky
Once upon a time, Ricky Beam <jfbeam@gmail.com> said:
Ethernet is cheap because it's everywhere, and built into almost everything. (however, the likes of Cisco and Juniper still charge insane amounts for line cards, be they ethernet, T1, or OC48.) Given the choice of buying a $4k DS3 card or just plugging into an existing, builtin ethernet port, which do you think most people will choose?
Also, if you are plugging in a lower-speed link, you can plug ethernet in a <$1000 switch and trunk it to a router, while a mux for T1/T3/OCx circuits costs a lot more. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
Chris Adams wrote:
Once upon a time, Ricky Beam <jfbeam@gmail.com> said:
Ethernet is cheap because it's everywhere, and built into almost everything. (however, the likes of Cisco and Juniper still charge insane amounts for line cards, be they ethernet, T1, or OC48.) Given the choice of buying a $4k DS3 card or just plugging into an existing, builtin ethernet port, which do you think most people will choose?
Also, if you are plugging in a lower-speed link, you can plug ethernet in a <$1000 switch and trunk it to a router, while a mux for T1/T3/OCx circuits costs a lot more.
I just ordered a circuit to be delivered over Ethernet - Verizon just plugged a pair of STM-1's into an ISG5100 and it's suddenly ridiculously cheaper. ~Seth
Hate to say it, but also some of the cost on the circuits can be blamed on uncle Sam. ATM circuits are currently tariffed that same way are voice circuits. These tariffs are not charged to Ethernet because it is a 'data circuit'. At least that was the case a little while back. -- ----------------- Brian Raaen Network Engineer email: /braaen@zcorum.com/ <mailto:braaen@zcorum.com> // Seth Mattinen wrote:
Chris Adams wrote:
Once upon a time, Ricky Beam <jfbeam@gmail.com> said:
Ethernet is cheap because it's everywhere, and built into almost everything. (however, the likes of Cisco and Juniper still charge insane amounts for line cards, be they ethernet, T1, or OC48.) Given the choice of buying a $4k DS3 card or just plugging into an existing, builtin ethernet port, which do you think most people will choose?
Also, if you are plugging in a lower-speed link, you can plug ethernet in a <$1000 switch and trunk it to a router, while a mux for T1/T3/OCx circuits costs a lot more.
I just ordered a circuit to be delivered over Ethernet - Verizon just plugged a pair of STM-1's into an ISG5100 and it's suddenly ridiculously cheaper.
~Seth
Brian Raaen wrote:
Hate to say it, but also some of the cost on the circuits can be blamed on uncle Sam. ATM circuits are currently tariffed that same way are voice circuits. These tariffs are not charged to Ethernet because it is a 'data circuit'. At least that was the case a little while back.
Are you sure it's "Uncle Sam"? My experience is that voice tariffs are always cheaper than data; telco's mantra is still "I Smell Dollars Now". The telcos were mightily pissed when we redesigned protocols to pass over voice circuits instead of requiring data circuits. Usually, non-tariffed lines seem to be much more expensive, as the account manager says "Oh, that special order will have to be approved by HQ".
Brian Raaen wrote:
Hate to say it, but also some of the cost on the circuits can be blamed on uncle Sam. ATM circuits are currently tariffed that same way are voice circuits. These tariffs are not charged to Ethernet because it is a 'data circuit'. At least that was the case a little while back. ? Are you sure it's "Uncle Sam"? My experience is that voice tariffs are always cheaper than data; telco's mantra is still "I Smell Dollars Now".
The telcos were mightily pissed when we redesigned protocols to pass over voice circuits instead of requiring data circuits.
Usually, non-tariffed lines seem to be much more expensive, as the account manager says "Oh, that special order will have to be approved by HQ".
Strictly speaking, it's not Uncle Sam, but the PUCs who review the tariffs. I would view it fundamentally as a lack of competition. Who can provide ATM backhaul from central offices? In many cases just the incumbent. Roderick S. Beck Director of European Sales Hibernia Atlantic
It should be noted that this usually isn't recommened. Dropping an ethernet circuit directly into a switch (even if it is laywer 3) can create design issues, esecially later when you need to scale the network. One big issue that is often overlooked is many swithces do not support traffic shaping. Some support policing, but shaping can be far more efficient. There are some nortel switches that do this, but I haven't seen many in the wild. -----Original Message----- From: Chris Adams [mailto:cmadams@hiwaay.net] Sent: Friday, July 10, 2009 10:39 AM To: nanog@nanog.org Subject: Re: Point to Point Ethernet Once upon a time, Ricky Beam <jfbeam@gmail.com> said:
Ethernet is cheap because it's everywhere, and built into almost everything. (however, the likes of Cisco and Juniper still charge insane amounts for line cards, be they ethernet, T1, or OC48.) Given the choice of buying a $4k DS3 card or just plugging into an existing,
builtin ethernet port, which do you think most people will choose?
Also, if you are plugging in a lower-speed link, you can plug ethernet in a <$1000 switch and trunk it to a router, while a mux for T1/T3/OCx circuits costs a lot more. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
if it does away with MAC addresses, in what sense is it an Ethernet link? I'll go with EPON or a variety of those. Not sure what messing with the Mac frame actually buys. On Jul 8, 2009, at 3:01 AM, Andre Oppermann wrote:
A few time already I've wished for a fully standardized and vendor interoperable way of doing a true point to point ethernet link.
It would work just like an old leased line or synchronous serial interface and completely do away with ARP, MAC addresses and all that stuff. Obviously no switches in between would be allowed. Each side would run in "promiscuous mode" where every ethernet frame is received and passed up to the network stack (just like on a serial link). Since MAC addresses are useless they can be scrapped and only the ethertype field remains. This increases the effective MTU by 12 bytes.
The framing overhead goes away and the packet can directly be directly placed on the wire without taking a detour through L3->L2 lookup and encapsulation step.
More importantly one can specify the just the outgoing interface again instead of the next hop:
ip route 10.0.0.0 255.0.0.0 g0/1
Do you think this is useful? Maybe vendors will hear me/us.
-- Andre
participants (26)
-
Andre Oppermann -
Anton Kapela -
Brian Raaen -
Cayle Spandon -
Chris Adams -
David Barak -
Dylan Ebner -
Frank Bulk -
Fred Baker -
Ivan Pepelnjak -
Joe Greco -
Joel Jaeggli -
Marshall Eubanks -
Mikael Abrahamsson -
Randy Bush -
Ricky Beam -
Rod Beck -
Saqib Ilyas -
Scott Berkman -
Seth Mattinen -
Stephen Kratzer -
sthaug＠nethelp.no -
Tomas L. Byrnes -
William Allen Simpson -
William Herrin -
Zartash Uzmi