[Fwd:] Nvidia NICs with duplicate mac addresses
Forwarded to NANOG in the interests of wider awareness... having been there and torn out my already scarce hair, duplicate MAC addresses can really mess up your day... --- Just when you thought this couldn't happen any more... Copying from a different email list... mac address 04:4b:80:80:80:03, was showing up in multiple places across the network. I googled the mac address and discovered that other people are having the same issue with this mac address. Below are some links describing the problem: http://forums.nvidia.com/index.php?showtopic=22148 http://www.nvnews.net/vbulletin/archive/index.php/t-73469.html I just wanted everyone to know about this problem in case you run across similar slow "connectivity" issues. I believe the network card is made by NVIDIA.
On Fri, Sep 05, 2008 at 10:32:46AM -0400, Robert E. Seastrom wrote:
mac address 04:4b:80:80:80:03, was showing up in multiple places across the network. I googled the mac address and discovered that other people are having the same issue with this mac address. Below are some links describing the problem:
http://forums.nvidia.com/index.php?showtopic=22148 http://www.nvnews.net/vbulletin/archive/index.php/t-73469.html
I just wanted everyone to know about this problem in case you run across similar slow "connectivity" issues. I believe the network card is made by NVIDIA.
FWIW, IEEE's OUI lookup page says that's not assigned to *anyone*. http://standards.ieee.org/regauth/oui/index.shtml Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com '87 e24 St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274 Those who cast the vote decide nothing. Those who count the vote decide everything. -- (Josef Stalin)
Date: Fri, 5 Sep 2008 10:36:24 -0400 From: "Jay R. Ashworth" <jra@baylink.com>
On Fri, Sep 05, 2008 at 10:32:46AM -0400, Robert E. Seastrom wrote:
mac address 04:4b:80:80:80:03, was showing up in multiple places across the network. I googled the mac address and discovered that other people are having the same issue with this mac address. Below are some links describing the problem:
http://forums.nvidia.com/index.php?showtopic=22148 http://www.nvnews.net/vbulletin/archive/index.php/t-73469.html
I just wanted everyone to know about this problem in case you run across similar slow "connectivity" issues. I believe the network card is made by NVIDIA.
FWIW, IEEE's OUI lookup page says that's not assigned to *anyone*.
No, the IEEE page does not say anything. For many years IEEE considered OUIs to be confidential by default. IEEE would list the OUI only if a box was checked on the application for the OUI. At some point the default was changes to make the OUIs public, but existing "private" entries remain private. That said, it turns out the nVidia's OUI is 00:04:4B. So it looks like the addressing was completely hosed and the MAC was shifted over 1 byte. What mess! This used to happen in the early days of Ethernet, but I have not seen it in some time. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751
On Fri, Sep 05, 2008 at 08:02:27AM -0700, Kevin Oberman wrote:
From: "Jay R. Ashworth" <jra@baylink.com>
FWIW, IEEE's OUI lookup page says that's not assigned to *anyone*.
No, the IEEE page does not say anything. For many years IEEE considered OUIs to be confidential by default. IEEE would list the OUI only if a box was checked on the application for the OUI. At some point the default was changes to make the OUIs public, but existing "private" entries remain private.
Ah, then you're correct; I characterised it worng. Thanks for the clarification; I hadn't realised that. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com '87 e24 St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274 Those who cast the vote decide nothing. Those who count the vote decide everything. -- (Josef Stalin)
This reminds me of a story I was told a while back that there was a batch of 3com NIC's that all went out with the same MAC from the factory. I never found out if that was a rumor/urban legend or the truth. Anyone know firsthand or have an article about that? -Scott -----Original Message----- From: Robert E. Seastrom [mailto:rs@seastrom.com] Sent: Friday, September 05, 2008 10:33 AM To: nanog@nanog.org Subject: [Fwd:] Nvidia NICs with duplicate mac addresses Forwarded to NANOG in the interests of wider awareness... having been there and torn out my already scarce hair, duplicate MAC addresses can really mess up your day... --- Just when you thought this couldn't happen any more... Copying from a different email list... mac address 04:4b:80:80:80:03, was showing up in multiple places across the network. I googled the mac address and discovered that other people are having the same issue with this mac address. Below are some links describing the problem: http://forums.nvidia.com/index.php?showtopic=22148 http://www.nvnews.net/vbulletin/archive/index.php/t-73469.html I just wanted everyone to know about this problem in case you run across similar slow "connectivity" issues. I believe the network card is made by NVIDIA.
Does this MAC present itself all the time, or just during boot? Marvel makes a NIC prevalent in some Dell systems, that presents MAC 0c00.0000.0000 during its startup process. If you run port security, and several people boot their computer within the cam table expiration period, port security will disable the port. You can work around it but it is time consuming in large networks where port security are enabled. Robert D. Scott Robert@ufl.edu Senior Network Engineer 352-273-0113 Phone CNS - Network Services 352-392-2061 CNS Receptionist University of Florida 352-392-9440 FAX Florida Lambda Rail 352-294-3571 FLR NOC Gainesville, FL 32611 321-663-0421 Cell -----Original Message----- From: Robert E. Seastrom [mailto:rs@seastrom.com] Sent: Friday, September 05, 2008 10:33 AM To: nanog@nanog.org Subject: [Fwd:] Nvidia NICs with duplicate mac addresses Forwarded to NANOG in the interests of wider awareness... having been there and torn out my already scarce hair, duplicate MAC addresses can really mess up your day... --- Just when you thought this couldn't happen any more... Copying from a different email list... mac address 04:4b:80:80:80:03, was showing up in multiple places across the network. I googled the mac address and discovered that other people are having the same issue with this mac address. Below are some links describing the problem: http://forums.nvidia.com/index.php?showtopic=22148 http://www.nvnews.net/vbulletin/archive/index.php/t-73469.html I just wanted everyone to know about this problem in case you run across similar slow "connectivity" issues. I believe the network card is made by NVIDIA.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Robert D. Scott wrote:
Does this MAC present itself all the time, or just during boot?
Marvel makes a NIC prevalent in some Dell systems, that presents MAC 0c00.0000.0000 during its startup process. If you run port security, and several people boot their computer within the cam table expiration period, port security will disable the port. You can work around it but it is time consuming in large networks where port security are enabled.
I know that this doesn't apply here, but a year or so ago I had a client that had issues with port security continually dropping an end user's PC. The problem was the MAC address kept changing from Realtek to Cisco. Sometimes the same NIC would present both MACs at the same time. It turned out the box was apparently infected with something. Never could find anything specific (even when booting the Windows box from Knoppix and scanning for unusual files) except for some large ADS files that where apparently encrypted. A clean wipe and complete rebuild of the box fixed the problem. Jon - -- Jon R. Kibler Chief Technical Officer Advanced Systems Engineering Technology, Inc. Charleston, SC USA o: 843-849-8214 c: 843-224-2494 s: 843-564-4224 My PGP Fingerprint is: BAA2 1F2C 5543 5D25 4636 A392 515C 5045 CF39 4253 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkjBZLQACgkQUVxQRc85QlMUpwCfQXrML+jZ8Lkwh3z2QuvldWh6 6+YAn3eqq2GBv7qof+urEGtibAKQf/6m =un9B -----END PGP SIGNATURE----- ================================================== Filtered by: TRUSTEM.COM's Email Filtering Service http://www.trustem.com/ No Spam. No Viruses. Just Good Clean Email.
The Marvel NIC presents the MAC as what we believe to be part of dot1x negotiation. These were new Dells out of the box, not yet infected. If we disable dot1x on the NIC the problem goes away. http://www.cisco.com/en/US/docs/switches/lan/catalyst3750e_3560e/software/re lease/12.2_44_se/release/notes/OL14629.html#wp885689 Cisco's Recommendation: Replace the NIC Robert D. Scott Robert@ufl.edu Senior Network Engineer 352-273-0113 Phone CNS - Network Services 352-392-2061 CNS Receptionist University of Florida 352-392-9440 FAX Florida Lambda Rail 352-294-3571 FLR NOC Gainesville, FL 32611 321-663-0421 Cell -----Original Message----- From: Jon Kibler [mailto:Jon.Kibler@aset.com] Sent: Friday, September 05, 2008 12:56 PM To: nanog@nanog.org Cc: 'Robert E. Seastrom' Subject: Re: [Fwd:] Nvidia NICs with duplicate mac addresses -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Robert D. Scott wrote:
Does this MAC present itself all the time, or just during boot?
Marvel makes a NIC prevalent in some Dell systems, that presents MAC 0c00.0000.0000 during its startup process. If you run port security, and several people boot their computer within the cam table expiration period, port security will disable the port. You can work around it but it is time consuming in large networks where port security are enabled.
I know that this doesn't apply here, but a year or so ago I had a client that had issues with port security continually dropping an end user's PC. The problem was the MAC address kept changing from Realtek to Cisco. Sometimes the same NIC would present both MACs at the same time. It turned out the box was apparently infected with something. Never could find anything specific (even when booting the Windows box from Knoppix and scanning for unusual files) except for some large ADS files that where apparently encrypted. A clean wipe and complete rebuild of the box fixed the problem. Jon - -- Jon R. Kibler Chief Technical Officer Advanced Systems Engineering Technology, Inc. Charleston, SC USA o: 843-849-8214 c: 843-224-2494 s: 843-564-4224 My PGP Fingerprint is: BAA2 1F2C 5543 5D25 4636 A392 515C 5045 CF39 4253 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkjBZLQACgkQUVxQRc85QlMUpwCfQXrML+jZ8Lkwh3z2QuvldWh6 6+YAn3eqq2GBv7qof+urEGtibAKQf/6m =un9B -----END PGP SIGNATURE----- ================================================== Filtered by: TRUSTEM.COM's Email Filtering Service http://www.trustem.com/ No Spam. No Viruses. Just Good Clean Email.
On Fri, Sep 05, 2008 at 10:32:46AM -0400, Robert E. Seastrom wrote:
Forwarded to NANOG in the interests of wider awareness... having been there and torn out my already scarce hair, duplicate MAC addresses can really mess up your day...
Out of curiosity, does this happen often enough we might want to consider an automated means to negotiate out of the problem state (e.g. detect collisions and negotiate MAC address by DHCP)? It would take years to deploy but might save thousands of hairs. -- Ash bugud-gul durbatuluk agh burzum-ishi krimpatul. Why settle for the lesser evil? https://secure.isc.org/store/t-shirt/ -- David W. Hankins "If you don't do it right the first time, Software Engineer you'll just have to do it again." Internet Systems Consortium, Inc. -- Jack T. Hankins
"David W. Hankins" <David_Hankins@isc.org> writes:
On Fri, Sep 05, 2008 at 10:32:46AM -0400, Robert E. Seastrom wrote:
Forwarded to NANOG in the interests of wider awareness... having been there and torn out my already scarce hair, duplicate MAC addresses can really mess up your day...
Out of curiosity, does this happen often enough we might want to consider an automated means to negotiate out of the problem state (e.g. detect collisions and negotiate MAC address by DHCP)?
It would take years to deploy but might save thousands of hairs.
The same DHCP server (ip helper-address blah) serves my office, my home, and the colo. Can you give me an idea of a good heuristic for telling the difference between moving my laptop around and finding MAC address collisions? Or are you suggesting that you hand out a MAC address along with an IP address when the client DHCPs and the client then changes it? -r
On Fri, Sep 05, 2008 at 01:19:44PM -0400, Robert E. Seastrom wrote:
The same DHCP server (ip helper-address blah) serves my office, my home, and the colo. Can you give me an idea of a good heuristic for telling the difference between moving my laptop around and finding MAC address collisions?
The theoretically conforming client supplies two pieces of information; Its DHCPv4 client-identifier-option (per RFC 4361), which will be different even on systems with identical MAC addresses (unless you are very lucky). The 'chaddr' BOOTP header contents ("its current MAC address"), which is a return-address for unicast replies (before the client has an IP address, ARP doesn't work). The DHCPv4 client-identifier tells us it's a specific interface on your laptop, no matter where it boots. Context from the packet (giaddr (relay-agent address on that network), the interface it was received upon) is used in normal DHCPv* operation (since its dawn) to find a broadcast domain. From there, we find subnets on that domain, and dynamic addresses within that subnet that are appropriate. This is how we locate configuration when your laptop connects to different wires in normal operation. The new trick is to detect two clients with the same MAC address, but with different client identifiers, that are active on the same broadcast domain. It's just a simple database lookup to detect a collision. The solution is to:
Or are you suggesting that you hand out a MAC address along with an IP address when the client DHCPs and the client then changes it?
Precisely. Negotiate a new address in a broadcast reply. I suppose you could just as easily always allocate a MAC dynamically, but this seems invasive. An alternative solution is to deny the client from receiving a config, signalling an error to the operator, so the first client remains operational. But this can have false positives. It'd take years to deploy tho. Many clients today send no identifier and just use their chaddr contents. Their MAC is their ID, so there is no way to detect collisions. Most others send a client-id that is identical to their chaddr contents, which is approximately useless. You'd also be suffering MAC changes on systems that boot multiple operating systems without releasing their previous lease (because the clients send inconsistent identifiers, and even with DUID-based id's, I think this is not going to change). This is in addition to today, where such clients consume multiple IPv4 addresses. Insult to injury. -- Ash bugud-gul durbatuluk agh burzum-ishi krimpatul. Why settle for the lesser evil? https://secure.isc.org/store/t-shirt/ -- David W. Hankins "If you don't do it right the first time, Software Engineer you'll just have to do it again." Internet Systems Consortium, Inc. -- Jack T. Hankins
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Robert E. Seastrom wrote:
Just when you thought this couldn't happen any more...
Copying from a different email list...
mac address 04:4b:80:80:80:03, was showing up in multiple places across the network.
That's why I stick to ARCNET :) ... alec - -- `____________ / Alec Berry \______________________________ | Senior Partner and Director of Technology \ | PGP/GPG key 0xE8E9030F | | http://alec.restontech.com/#PGP | |-------------------------------------------| | RestonTech, Ltd. | | http://www.restontech.com/ | | Phone: (703) 234-2914 | \___________________________________________/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIwWLuREO1P+jpAw8RAm32AKCL5p2yYfeVcUR7BJ16HfS6Rs3XjwCePlc2 wh3Vf0a6MgCe48BICDJufu0= =+YLd -----END PGP SIGNATURE-----
participants (8)
-
Alec Berry
-
David W. Hankins
-
Jay R. Ashworth
-
Jon Kibler
-
Kevin Oberman
-
Robert D. Scott
-
Robert E. Seastrom
-
Scott Berkman