I have come to believe the Intel 82574L is the worst Ethernet chip in the universe. We had horrible issues with it (random bursts of dropped packets showing in ifconfig). We ended up simply putting a card based on a different chip into our systems and all our issues went away. -----Original Message----- From: Blake Dunlap [mailto:ikiris@gmail.com] Sent: Wednesday, February 06, 2013 3:40 PM To: Kristian Kielhofner Cc: nanog@nanog.org Subject: Re: Interesting debugging: Specific packets cause some Intel gigabit ethernet controllers to reset Wow, you just solved my issue with my firewall. On Wed, Feb 6, 2013 at 2:33 PM, Kristian Kielhofner <kris@kriskinc.com>wrote:

Over the year I've read some interesting (horrifying?) tales of debugging on NANOG. It seems I finally have my own to contribute:

http://blog.krisk.org/2013/02/packets-of-death.html

The strangest issue I've experienced, that's for sure.

-- Kristian Kielhofner

Hi, Yes I had that issue, it was a firmware problem... and a timed one too :( We had a customer with a few Raid5 of 3 drives, once 1 drive go bad he had about 20m before another drive would. And they where bricked btw, you couldn't just upload the new firmware. Wasn't an happy weekend. ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 02/06/13 15:47, Jay Ashworth wrote:

----- Original Message -----

...

From: "Kristian Kielhofner" <kris@kriskinc.com> Over the year I've read some interesting (horrifying?) tales of debugging on NANOG. It seems I finally have my own to contribute:

http://blog.krisk.org/2013/02/packets-of-death.html

The strangest issue I've experienced, that's for sure. FWIW, I had a similar situation crop up a couple of years ago with *five different* Seagate SATA drives: they grew some specific type of bad spot on the drive which, if you even tried to read it, would *knock the drive adapter off line until powercycle*; even a reboot didn't clear it.

Nice writeup.

Cheers, -- jra

----- Original Message -----

From: "Harry Hoffman" <hhoffman@ip-solutions.net>

On a similar vein here's some fun reading:

http://travisgoodspeed.blogspot.com/2011/09/remotely-exploiting-phy-layer.ht...

Really? That environment does not have out-of-band framing, which can't be duplicated by the data inside a framed packet? Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274

I just want you to know that this was the best piece of technical debugging I've read in years. Absolutely awesome. Thank you so much for sharing what I can only imagine was an endless series of nightmares. I've done debugging like this before and I can only say: I feel your pain and I wish I documented my previous efforts. Great writing sir. Cheers, Joshua Joshua Goldbard VP of Marketing, 2600hz 116 Natoma Street, Floor 2 San Francisco, CA, 94104 415.886.7923 | j@2600hz.com<mailto:j@2600hz.com> On Feb 8, 2013, at 12:50 PM, Kristian Kielhofner <kris@kriskinc.com<mailto:kris@kriskinc.com>> wrote: Update with a response to the statement from Intel: http://blog.krisk.org/2013/02/packets-of-death-update.html On Wed, Feb 6, 2013 at 3:33 PM, Kristian Kielhofner <kris@kriskinc.com> wrote: Over the year I've read some interesting (horrifying?) tales of debugging on NANOG. It seems I finally have my own to contribute: http://blog.krisk.org/2013/02/packets-of-death.html The strangest issue I've experienced, that's for sure. -- Kristian Kielhofner -- Kristian Kielhofner