Peering VLANs and MAC addresses
Hi , We are unable to resolve a problem with our peering exchange connection and would like any assistance. Our peering setup is a follows: - Our peering exchange connection goes into switch A - Switch A has a dark fibre connection to switch B, which is in a different PoP - Our peering router is connected to switch B We use spanning tree across our network to allow the VLANs connectivity across our network. The peering exchange has an MoU that only 1 MAC address should be visible on their switch. However they see 2 MAC addresses on our port. - MAC address of Peering router - MAC address of the port they are connected to on switch A Is there any way to prevent switch A from presenting the interface MAC address? Or is this a symptom of spanning tree that cannot be stopped? Your input will be most welcome. The config on switch A is as follows: interface GigabitEthernet0/5 description Peering Link switchport access vlan 148 switchport mode access speed nonegotiate storm-control broadcast level 5.00 no cdp enable spanning-tree portfast spanning-tree bpdufilter enable spanning-tree guard root Regards Simon Brilus
On Wed, 2005-11-09 at 10:39 +0000, Simon Brilus wrote:
The peering exchange has an MoU that only 1 MAC address should be visible on their switch. However they see 2 MAC addresses on our port.
- MAC address of Peering router - MAC address of the port they are connected to on switch A
Is there any way to prevent switch A from presenting the interface MAC address? Or is this a symptom of spanning tree that cannot be stopped?
Has it been confirmed that the violating frames are in fact STP BPDUs? Could be all kinds of other autoconfig crap (CDP, VTP, etc.). Don't know your VLAN setup and I certainly don't know that much about Cisco L2 features. Is it possible to have topology groups with a master VLAN (the one that does STP) and member VLANs that don't speak STP? If so, that may be a way to keep STP traffic from coming out of your IX port (barring vendor bugs).
Your input will be most welcome.
Is it absolutely necessary to run STP on the edge of the IX? -- Steven
Hi Simon, so you have: IX---SwitchA---SwitchB---Router why not disable spanning tree? There is no redundancy here anyway so disable it in that particular VLAN. Steve On Wed, 9 Nov 2005, Simon Brilus wrote:
Hi ,
We are unable to resolve a problem with our peering exchange connection and would like any assistance. Our peering setup is a follows:
- Our peering exchange connection goes into switch A - Switch A has a dark fibre connection to switch B, which is in a different PoP - Our peering router is connected to switch B
We use spanning tree across our network to allow the VLANs connectivity across our network.
The peering exchange has an MoU that only 1 MAC address should be visible on their switch. However they see 2 MAC addresses on our port.
- MAC address of Peering router - MAC address of the port they are connected to on switch A
Is there any way to prevent switch A from presenting the interface MAC address? Or is this a symptom of spanning tree that cannot be stopped?
Your input will be most welcome.
The config on switch A is as follows:
interface GigabitEthernet0/5 description Peering Link switchport access vlan 148 switchport mode access speed nonegotiate storm-control broadcast level 5.00 no cdp enable spanning-tree portfast spanning-tree bpdufilter enable spanning-tree guard root
Regards
Simon Brilus
On 9-Nov-2005, at 16:35, Randy Bush wrote:
IX---SwitchA---SwitchB---Router
ok, i gotta ask. you folk really do this on exchanges?
I seem to think I've seen people doing this at most exchanges ISC has installed an F-root node at. The motivation is usually the avoidance of either expense ports on exchange switches, or the avoidance of expensive ports on routers (trunking the IX in as a vlan to a router on an existing port). I'm not saying that the practice is good, or recommended, or without peril. But it's certainly not isolated to the UK. Joe
I'm not saying that the practice is good, or recommended, or without peril. But it's certainly not isolated to the UK.
perhaps it should be :-) as folk from all over read this list, i just could not let discussion of how to do something that is generally broken and quite ill-advised go without raising a red flag. i would recommend all my competitors do this, except, as it is being done to exchange meshes, it is damaging to all. randy
On Wed, 2005-11-09 at 11:55 -1000, Randy Bush wrote:
[IX---SwitchA---SwitchB---Router] I'm not saying that the practice is good, or recommended, or without peril. But it's certainly not isolated to the UK.
perhaps it should be :-)
as folk from all over read this list, i just could not let discussion of how to do something that is generally broken and quite ill-advised go without raising a red flag.
What is the problem with this for the IXP, assuming proper safeguards are in place which are best practice anyway (BPDU filters, port security, ...)? Which rule would you suggest for the IXP? The naive "connect only routers" wouldn't do of course in nowaday's world of hybrids. Robert
What is the problem with this for the IXP, assuming proper safeguards are in place which are best practice anyway (BPDU filters, port security, ...)?
Hello Robert :)
Which rule would you suggest for the IXP? The naive "connect only routers" wouldn't do of course in nowaday's world of hybrids.
I think the 'connect only routers' adage is probably a good conservative motto to stick to. There are situations where connecting switches and hybrids to IXPs is certainly more efficient and better suited, but only if you know what you're doing or have a good reason for it. As I understand it, most IXPs are pretty well protected against guff coming from switches these days anyway, but it still doesn't make sense in my mind to have a free for all on what people can connect. At least this adage might make someone who might not be experienced in what they're doing think twice and ask someone who knows better before doing it (as indeed it seems to have done in this case).
Robert
Cheers, Chris. -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.362 / Virus Database: 267.12.8/163 - Release Date: 08/11/2005
On Wed, Nov 09, 2005 at 11:59:38PM -0000, Chris Roberts wrote:
I think the 'connect only routers' adage is probably a good conservative motto to stick to. There are situations where connecting switches and hybrids to IXPs is certainly more efficient and better suited, but only if you know what you're doing or have a good reason for it. As I understand it, most IXPs are pretty well protected against guff coming from switches these days anyway, but it still doesn't make sense in my mind to have a free for all on what people can connect. At least this adage might make someone who might not be experienced in what they're doing think twice and ask someone who knows better before doing it (as indeed it seems to have done in this case).
There is no technical reason why you can't hook up as many switches as you need, is there any real difference between a L3 switch and a L3 router (except for its internal architecture and maybe a couple of 0's at the end of the price :P). There are only good products, and bad products, smart people, and stupid people. Stupid people running bad products will find a way to leak stupid stuff to the IX and screw things up royally, regardless of the type of product connected. Smart people running good products USUALLY won't, no matter how many layer 2 and 3 switches stand between a router and an IX port. Of course I think part of the qualification for being considered a smart person involves being able to connect switches to IX's without blowing anything up, so those results might be a little biased. "Only connecting routers" is really just attempting to mitigate the effects of stupid people by forcing them to run configurations so simple "even a monkey could pull it off". -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
[ this seems to have been in an edit buffer for a while ] fantasies about using 1918 space without leaking. and folk never leaking igp<->bgp, and pigs flying, and cash falling from the sky.
Of course I think part of the qualification for being considered a smart person involves being able to connect switches to IX's without blowing anything up
i don't give a <bleep> if they're considered smart, green, or gardenia. people who think they're smart and l33t probably do more damage than those of us who are 0ld, st00pid, and 1azy. cleverness is how disasters are made. testosterone poisoning is often fatal, but it's impolite to take your neighbors with you. in a world where over 90% of ix peers' net configs are not rigidly automated, <bleep> is gonna happen. and it has, time and again. the only stuff that makes me feel at all safe is what mike hughes of linx described, or something even stricter, but i bow to mike's experience. and folk wonder why the grown-ups use pnis for anything important. randy
Randy Bush wrote:
the only stuff that makes me feel at all safe is what mike hughes of linx described, or something even stricter, but i bow to mike's experience.
and folk wonder why the grown-ups use pnis for anything important.
Isn't this due to the fact their engineering scale is bigger? There's little point connecting to an IX fabric if you want to peer with 3 others and all of those at 10G. The assumption that big carriers don't join IXs because they're somehow unreliable or toy-like doesn't seem fair. Many folk have the highest respect for the operational efficacy of membership organisations like the LINX, and that is borne out by their willingness to join and put traffic across the exchanges. Will
On Nov 11, 2005, at 4:06 AM, Will Hargrave wrote:
Randy Bush wrote:
the only stuff that makes me feel at all safe is what mike hughes of linx described, or something even stricter, but i bow to mike's experience. and folk wonder why the grown-ups use pnis for anything important.
Isn't this due to the fact their engineering scale is bigger? There's little point connecting to an IX fabric if you want to peer with 3 others and all of those at 10G. The assumption that big carriers don't join IXs because they're somehow unreliable or toy- like doesn't seem fair.
Who said "big carriers" don't join IXes? There are plenty of networks who have more traffic than some "teir ones" at IXes. Hell, RANDY has a presence at least one IX.
Many folk have the highest respect for the operational efficacy of membership organisations like the LINX, and that is borne out by their willingness to join and put traffic across the exchanges.
Apparently those folk are not 'grown-ups'. The 100+ Gbps passing over NAPs like AMS-IX and LINX are apparently just childish endeavors. Or perhaps Randy is saying that Internet traffic in general is not "important"? -- TTFN, patrick
Who said "big carriers" don't join IXes? There are plenty of networks who have more traffic than some "teir ones" at IXes. Hell, RANDY has a presence at least one IX.
well, one of my routers does :-) and it moves almost 50kb/sec! i have spent <long enough i don't want to count> years trying to get large isps to peer openly (originally pushed by asp). the costs of managing in the presence of bad behavior on public meshes is often used as a reason not to peer publicly. randy
On Nov 11, 2005, at 9:33 AM, Randy Bush wrote:
Who said "big carriers" don't join IXes? There are plenty of networks who have more traffic than some "teir ones" at IXes. Hell, RANDY has a presence at least one IX.
well, one of my routers does :-) and it moves almost 50kb/sec!
:-)
i have spent <long enough i don't want to count> years trying to get large isps to peer openly (originally pushed by asp). the costs of managing in the presence of bad behavior on public meshes is often used as a reason not to peer publicly.
TouchƩ. No one is claiming that Bad Stuff does not happen. However, NAPs these days are stable, scalable, and useful. Picking your peers carefully - which is a requirement for public or private peering - helps. We are just saying that "grown-ups" _DO_ use public IXes. (Plus people like me. :) And anyone who claims that IXes are bad places to trade traffic are ... not being very grown up. -- TTFN, patrick
NAPs these days are stable, scalable, and useful.
IXs (there were only four NAPs, and i'm too old and lazy to play droid terminology drift) have pretty much always been scalable (for the then current meaning of scale) and useful. though i have admiration and sympathy for folk such as steve, keith, mike, et alia, who have had the task of scaling them (it always looks easy from the sidelines). but, wearing a different hat, i am aware of issues on some of the large exchanges where one is suspicious of layer two origins.
Picking your peers carefully - which is a requirement for public or private peering - helps.
this does not isolate one from the majority of layer two messes, the subject of this thread. on the six, i had just such a problem caused by a sloppy-but-arrogant weenie with whom i had no desire to peer. and guess what caused it, switch on the ix. randy
On Wed, 9 Nov 2005, Robert Kiessling wrote:
Which rule would you suggest for the IXP? The naive "connect only routers" wouldn't do of course in nowaday's world of hybrids.
I've been following this with interest: * How do you differentiate between a switch/router and a router? A lot of people are deploying C76xx as peering routers these days because of the cheapness of the ports (including 10G) on that particular problem. * A lot of people with vendor J kit take a dot1q encaps GigE to a switch and break out 100M ports, because 100M ports aren't very cost effective on the M-series platform. The rule on LINX always used to be "no switches", until the switch/router convergence dilemma. There was also an issue of keeping business (especially business we wanted to keep!) on the IX, that wanted to connect through LAN extensions and various other more colourful methods, so we had to find a way of permitting this while mitigating the risks. There's a fairly competitive IX environment in London, and the other exchanges didn't seem to be too bothered by switches, especially if it helped them attract participants from the larger exchange ;). So, the rule at LINX is now "one port, one source MAC, any more and we shut you down". This deals with all the usual evils, such as ad-hoc extensions (or resold connections) to the peering platform (a prime concern), incoming STP/CDP/VTP, etc., and broken routers/switches (which end up spewing crap - C75xx used to melt spectacularly in this way). At the same time it permits switch/routers, and various LAN/Ethernet extension services, as long as they are sensibly managed and configured. We used to police this policy semi-manually, but now the switch vendors do decent hardware-based port-security/mac-locking functionality, so that does it for us, and actually does it pretty well. - The switch learns the first address received on the interface, which should be the first ingress frame (usually an ARP generated by the router sending a BGP Open), and remembers it (with a 3 minute ageing time). - This has the affect of applying an acl to the port (in hardware), which permits traffic from the "good" address, and drops frames from other addresses. - Should more than 100 different source MACs be learned (99 of which will be filtered and dropped) on the interface, the port will then log a critical violation and shut the port down. It works pretty well, it prevents all the usual badness we'd normally associate with switches on the IXP. So at the end of the day, it looks like we've been able to find a happy medium, maintaining decent "hygiene", while being able to let people indulge in deploying switches if they so choose. Cheers, Mike
On Thu, 2005-11-10 at 00:01 +0000, Mike Hughes wrote:
* How do you differentiate between a switch/router and a router?
Ooh, that's easy: just look at the crap they spew towards the peering fabric. :-)
A lot of people are deploying C76xx as peering routers ...
<rant> ... which should be prohibited by law. Actually, C76xx should be prohibited by law. </rant>
... At the same time it permits switch/routers, and various LAN/Ethernet extension services, as long as they are sensibly managed and configured.
Right. It does give complications wrt. trouble-shooting connectivity though. Switch port up but no traffic (L2 black-holing), a single L2 provider with a loop or misconfiguration causing port security violations on all of their customers' ports, communication channels when those problems come up, etc. Having said that, so far it's still pretty manageable. -- Steven
Steven Bakker wrote:
A lot of people are deploying C76xx as peering routers ...
<rant> ... which should be prohibited by law. Actually, C76xx should be prohibited by law. </rant>
i know the current sport de jour in nanog is vendor bashing - but what specifically do you see as faults in the c6500/7600? off-list or on-list - i don't mind - although don't want to violate any nanog policies. cheers, lincoln.
A lot of people are deploying C76xx as peering routers ...
<rant> ... which should be prohibited by law. Actually, C76xx should be prohibited by law. </rant>
I've done my share of Cisco bashing in the past - but I have to say that 6500/7600 worked pretty well as peering routers at my previous employer. Great capacity, hardware forwarding, hardware ACLs and policing. Handled Gbps sized DDoS attacks just fine. 6500/7600 as MPLS PE routers is another story altogether... Steinar Haug, Nethelp consulting, sthaug@nethelp.no
[ the voice of experience speaks ]
We used to police this policy semi-manually, but now the switch vendors do decent hardware-based port-security/mac-locking functionality, so that does it for us, and actually does it pretty well.
- The switch learns the first address received on the interface, which should be the first ingress frame (usually an ARP generated by the router sending a BGP Open), and remembers it (with a 3 minute ageing time).
- This has the affect of applying an acl to the port (in hardware), which permits traffic from the "good" address, and drops frames from other addresses.
- Should more than 100 different source MACs be learned (99 of which will be filtered and dropped) on the interface, the port will then log a critical violation and shut the port down.
It works pretty well, it prevents all the usual badness we'd normally associate with switches on the IXP.
So at the end of the day, it looks like we've been able to find a happy medium, maintaining decent "hygiene", while being able to let people indulge in deploying switches if they so choose.
thanks! this approaches reassuring. why does it tolerate 100 macs? at first blush, i would think three or four would be a bad enough sign. randy
On Wed, 9 Nov 2005, Randy Bush wrote:
thanks! this approaches reassuring. why does it tolerate 100 macs? at first blush, i would think three or four would be a bad enough sign.
It's a balance to avoid unduly penalising a genuine mistake, or being too severe against some poor guy with a router which is still forwarding but has an interface in it's death throes (and is occasionally generating bursts of crap frames), and making his problems even worse. In our experience, you either see a handful of macs caused by there being a shakily configured switch-router attached, a slightly larger number of macs caused by something being broken, or a couple of hundred due to either a physical loop being applied or leaking other vlans (true badness). It's also a relatively sensible default when you apply the "restrict" behaviour. Cheers, Mike
* randy@psg.com (Randy Bush) [Thu 10 Nov 2005, 03:35 CET]:
[ the voice of experience speaks ] [..] thanks! this approaches reassuring. why does it tolerate 100 macs? at first blush, i would think three or four would be a bad enough sign.
I've seen several cases where a router goes bonkers and spews a bunch of broken frames - more than four but usually less than a hundred. The frames get dropped but the port doesn't get shut down. I have a hunch that it's connected with bad memory in the router, a pointer going awry somewhere, using some payload to fill an Ethernet frame header. Usually it goes away by itself without further outaged conditions. So this could be a reason to set the limit at 100 instead of 4... another could be that the default in Foundry MG8 firmware is 128. Also, AMS-IX implemented port security almost three years ago (we've presented about it at AMS-IX Technical Meetings, RIPE meetings and Euro-IX Forums). -- Niels. -- "Calling religion a drug is an insult to drugs everywhere. Religion is more like the placebo of the masses." -- MeFi user boaz
Mike, All, I know the changes the LINX has implemented, and I am curious... and this might affect other folk as well. What is better - the LINX approach (blocking the port, trying again in x minutes when too many MACs were seen) or the Equinix approach (we hardcode your MAC per VLAN/ per port if untagged, all else we just drop)? Alexander
On Thu, 10 Nov 2005, Alexander Koch wrote:
I know the changes the LINX has implemented, and I am curious... and this might affect other folk as well.
What is better - the LINX approach (blocking the port, trying again in x minutes when too many MACs were seen) or the Equinix approach (we hardcode your MAC per VLAN/ per port if untagged, all else we just drop)?
Much of a muchness really. With the former approach, it's easier for the participants to effect changes to their IX equipment without having to ask the IX operator to clear the locking/reconfigure the static MAC. The protection against badness is pretty equal, whatever you do. Cheers, Mike
On Nov 9, 2005, at 7:49 PM, Christopher L. Morrow wrote:
On Wed, 9 Nov 2005, Robert Kiessling wrote:
Which rule would you suggest for the IXP? The naive "connect only routers" wouldn't do of course in nowaday's world of hybrids.
yick hybrids...
I like watching some of those hybrids boot though. It is fun to watch all the different layers of processors send their little boot messages to the console. It is so very reassuring to know that it takes an 8 layered cake to make my router function. I had a few of 7609s at one network and I was, actually, quite happy with the performance. We had one failure of an OC12 card and that was it as I recall. Of course this tells me very little about performance in "bad" situations. If we had had a network meltdown it would have been interesting to observe the behavior of those switches in comparison to the other equipment. It felt very weird building VLANs inside a router (and I made sure we only did it once so I would not wake up at night and scream). We used RPR+ which was pretty nice. And you pretty much can't shake a stick at an interface card without it popping up with an Ethernet interface. I should probably summarize a bit... a. I still don't really like hybrids and feel they are too disjointed in their approach to survive in the long term. b. The 7609 hybrid was not nearly as bad as I feared. c. I would likely NOT purchase a hybrid if I could avoid it. d. You will probably find that a hybrid is the most cost effective option and will have to fight your finance folks who know a thing or two about routers and switches! Well, I should say they know whatever the vendor is telling them behind your back<grin>. e. You may not have to fight with finance as you may personally realize that you want another toy and purchasing the hybrid leaves you with enough capital to purchase the other toy and take the potential issues as a risk.
participants (17)
-
Alexander Koch -
Blaine Christian -
Chris Roberts -
Christopher L. Morrow -
Joe Abley -
Lincoln Dale -
Mike Hughes -
Niels Bakker -
Patrick W. Gilmore -
Randy Bush -
Richard A Steenbergen -
Robert Kiessling -
Simon Brilus -
Stephen J. Wilcox -
Steven Bakker -
sthaugļ¼ nethelp.no -
Will Hargrave