February 2009 - Test - lists.nanog.org

Global Blackhole Service
by Jens Ott - PlusServer AG 16 Feb '09

16 Feb '09

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, in the last 24 hours we received two denial of service attacks with something like 6-8GBit volume. It did not harm us too much, but e.g. one of our upstreams got his Amsix-Port exploded. With our upstreams we have remote-blackhole sessions running where we announce /32 prefixes to blackhole at their edge, but this does not work with our peers. Also our Decix-Port received something like 2Gbit extra-traffic during this DoS. I can imagine, that for some peers, especially for the once having only a thin fiber (e.g. 1GBit) to Decix, it's not to funny having it flooded with a DoS and that they might be interested in dropping such traffic at their edge. Well I could discuss with my peers (at least the once who might get in trouble with such issue) to do some individual config for some blackhole-announcement, but most probably I'm not the only one receiving DoS and who would be interested in such setup. Therefore I had the following idea: Why not taking one of my old routers and set it up as blackhole-service. Then everyone who is interested could set up a session to there and 1.) announce /32 (/128) routes out of his prefixes to blackhole them 2.) receive all the /32 (/128) announcements from the other peers with the IPs they want to have blackholed and rollout the blackhole to their network. My questions to all of you: - - What do you think about such service? - - Would you/your ASN participate in such a service? - - Do you see some kind of usefull feature in such a service? - - Do you have any comments? Thank you for telling me your opinions and best regards - -- =================================================================== Jens Ott Leiter Network Management Tel: +49 22 33 - 612 - 3501 Fax: +49 22 33 - 612 - 53501 E-Mail: j.ott(a)plusserver.de GPG-Fingerprint: 808A EADF C476 FABE 2366 8402 31FD 328C C2CA 7D7A PlusServer AG Daimlerstraße 9-11 50354 Hürth Germany HRB 58428 / Amtsgericht Köln, USt-ID DE216 740 823 Vorstand: Jochen Berger, Frank Gross, Jan Osthues, Thomas Strohe Aufsichtsratsvorsitz: Claudius Schmalschläger =================================================================== -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmVilwACgkQMf0yjMLKfXpNuQCeKcicthIadISe7I+Xs5ZNHS+1 0qUAnRDkOY9/6kokq3Hf68BRQFfkP3xy =jKUA -----END PGP SIGNATURE-----

21 28

Looking for GBLX contact + check your ACL for 66.249.96.0/20 please :)
by Swen Wulf 16 Feb '09

16 Feb '09

Hello, We have been assigned 66.249.96.0/20 in April of 2008. Little did we know that this network was used by a spammer in 2004 (Lightwave Transit) and with this is deeply embedded in a lot of ACL's that seem to be unmaintained. If a GLBX engineer could take a look, it seems traffic originating from 66.249.96.0/20 can't reach some website when passing thru GBLX. Other prefixes we announce from our AS (22442) such as 66.60.224.0/20 don't have this problem. Customer with source of 66.249.106.112/27 can't get to the website of www.ups.com (72.246.89.243 (akamai)). core1#traceroute 72.246.89.243 source 66.249.106.113 Type escape sequence to abort. Tracing the route to a72-246-89-243.deploy.akamaitechnologies.com (72.246.89.243) 1 ge-1-11.r03.hstntx01.us.bb.gin.ntt.net (128.241.5.1) 0 msec 0 msec 0 msec 2 xe-0-1-0.r20.hstntx01.us.bb.gin.ntt.net (129.250.2.228) [AS 2914] 0 msec 0 msec 0 msec 3 p64-1-3-0.r20.dllstx09.us.bb.gin.ntt.net (129.250.3.129) [AS 2914] 8 msec 8 msec 8 msec 4 po-2.r03.dllstx09.us.bb.gin.ntt.net (129.250.4.38) [AS 2914] 4 msec 4 msec 8 msec 5 xe-0.globalcrossing.dllstx09.us.bb.gin.ntt.net (129.250.8.190) [AS 2914] 4 msec 8 msec 4 msec 6 te7-1-10G.ar2.DAL2.gblx.net (67.16.129.117) [AS 3549] 8 msec 12 msec 4 msec 7 * * * 8 * * And from a different source IP from the same router core1#traceroute 72.246.89.243 source 66.60.233.1 Type escape sequence to abort. Tracing the route to a72-246-89-243.deploy.akamaitechnologies.com (72.246.89.243) 1 ge-1-11.r03.hstntx01.us.bb.gin.ntt.net (128.241.5.1) 4 msec 0 msec 0 msec 2 xe-0-1-0.r20.hstntx01.us.bb.gin.ntt.net (129.250.2.228) [AS 2914] 0 msec 4 msec 0 msec 3 p64-1-3-0.r20.dllstx09.us.bb.gin.ntt.net (129.250.3.129) [AS 2914] 8 msec 8 msec 8 msec 4 po-2.r03.dllstx09.us.bb.gin.ntt.net (129.250.4.38) [AS 2914] 4 msec 8 msec * 5 xe-0.globalcrossing.dllstx09.us.bb.gin.ntt.net (129.250.8.190) [AS 2914] 4 msec 8 msec 8 msec 6 te7-1-10G.ar2.DAL2.gblx.net (67.16.129.117) [AS 3549] 4 msec 8 msec 8 msec 7 PCCW-Global-Chicago-Dallas.te3-2.ar2.DAL2.gblx.net (64.211.211.186) [AS 3549] 200 msec 24 msec 212 msec 8 * * * 9 I tried an email to their puck noc email address, but without being a customer I can't open a ticket. Thanks, Swen Wulf Network Security Manager Email: swulf(a)phonoscope.com Phone: 832-615-7743 (direct) Fax: 832-213-0110 --- * Network Operations Center (NOC): 713-272-4600 (24x7x365) --- Phonoscope - moving at the speed of light

1 0

3/11 (invalid or corrupt AS path)
by Ozar 16 Feb '09

16 Feb '09

I am starting to see random BGP neighbor messages from multiple neighbors on different boxes. %BGP-3-NOTIFICATION: received from neighbor X.X.X.X 3/11 (invalid or corrupt AS path) 516 bytes I dont see much documentation on this, and we are in the process of opening a TAC case, just curious if anyone else has seen these and may be able to shed some light. Thanks

3 2

Littering the global BGP - AS 47868
by Ran Liebermann 16 Feb '09

16 Feb '09

Hi all, We're seeing some garbage being thrown into the global BGP table by AS 47868. %BGP-6-ASPATH: Long AS path 3549 3257 29113 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 received from 208.49.128.49: Has more than 255 AS This is hitting some routers with older software versions. Is there someone here from AS 29113 (Sloane Park Property Trust in the Czech Repulic) that can filter this from the source? Cheers, -- Ran Liebermann VP Engineering, PurePeak ranl(a)purepeak.com http://purepeak.com

1 0

Re: Linux Router: TCP slow, UDP fast
by Chris 15 Feb '09

15 Feb '09

Thanks, Karl, Allen and Nickola. I failed-over to another router last night and briefly had full expected throughput but this morning despite dropping providers and moving between routers again for trial and error I still see _outbound_ TCP at about the same 300 - 600kbps per session. I eliminated conntract modules firstly, then iptables as a whole. I've eliminated TSO and checksumming (which caused very sticky connections) on the e1000 NIC. The failover router has a slightly older kernel and was working before Christmas so it's not most likely not kernel versions. I've also tried removing FIB_TRIE as a stab in the dark with no success. And the failover router connects using FE not GE so I've eliminated NICs and connection speeds to a front-facing switch. The only constant is the front-facing switch (it's negotiating perfectly at FD though) so all I can think of is removing that from the equation. It's definitely only _outbound_ TCP getting buffered though ! I've pushed 92Mbps on a FE link with UDP and uploaded at 16Mbps on a 16Mbps link. Any last ideas appreciated before causing headaches removing switches would be appreciated. Thanks, Chris

2 1

Re: Linux Router: TCP slow, UDP fast
by Chris 15 Feb '09

15 Feb '09

Thanks for the suggestions everyone. I've got to the bottom of the problem now (I'm sure there will be a collective sigh of relief from the list because of the noise this thread generated :-)). I installed two brand new, low spec, 3Com switches one at the 'front' of the network and one 'behind' the routers. They are the same model, same latest firmware, same config (saved to and then copied off disk) and only their IPs were different. The front switch was the problem. As two final tests before removing it we switched off unicast/multicast broadcast control and flow control (and simultaneously on the same port on the switch behind the routers) because there were pause frames showing but not a massive amount in terms of percentage. The switch behind the routers however is serving the same bandwidth equally well ! We've put an ancient switch in place of the front switch and its been working perfectly so far. My lesson learned is too change as little as possible at once ! That said, recent network changes were spread about a month apart and this very odd issue was far easier to dismiss than believe due to its bizarre nature. Especially when providers have changes in their network conditions as testing is done. I really appreciate all the input and have learnt loads, possibly just not in the way I would have liked to :-) Doubles all round, Chris

1 0

Re: Linux Router: TCP slow, UDP fast
by Mathias Wolkert 15 Feb '09

15 Feb '09

I would turn off ethernet flow control. Maybe you already have. It can be really mean on tcp's own flow control if the switch has an issue of some kind (load). /Tias 15 feb 2009 kl. 10.24 skrev Chris <chris(a)ghostbusters.co.uk>: > Thanks, Karl, Allen and Nickola. > I failed-over to another router last night and briefly had full > expected > throughput but this morning despite dropping providers and moving > between > routers again for trial and error I still see _outbound_ TCP at > about the > same 300 - 600kbps per session. > > I eliminated conntract modules firstly, then iptables as a whole. I've > eliminated TSO and checksumming (which caused very sticky > connections) on > the e1000 NIC. > > The failover router has a slightly older kernel and was working before > Christmas so it's not most likely not kernel versions. I've also tried > removing FIB_TRIE as a stab in the dark with no success. And the > failover > router connects using FE not GE so I've eliminated NICs and connection > speeds to a front-facing switch. > > The only constant is the front-facing switch (it's negotiating > perfectly at > FD though) so all I can think of is removing that from the equation. > > It's definitely only _outbound_ TCP getting buffered though ! I've > pushed > 92Mbps on a FE link with UDP and uploaded at 16Mbps on a 16Mbps link. > > Any last ideas appreciated before causing headaches removing > switches would > be appreciated. > > Thanks, > > Chris

1 0

Re: Global Blackhole Service
by Nuno Vieira - nfsi telecom 15 Feb '09

15 Feb '09

Hi Suresh, But in the meanwhile, a decade later, it does not longer exist. At least, i can't reach that host, and i was unable to find working documentation on google of how about this project works, today. In fact, the first link that google gave out, says that this project is dead at least 2 years ago. http://www.dnsbl.com/2007/02/status-of-rblmapsvixcom-invalid-domain.html I think that we all have a real opportunity here for make something that can be useful to all. And, we are not talk of spam here, but, to mitigate time, money and patience consuming DDoS attacks, which often are easier to mitigate only at the Source and at the Destination, while at Destinatation, sink is the only viable solution that is out there today. regards, --- Nuno Vieira nfsi telecom, lda. nuno.vieira(a)nfsi.pt Tel. (+351) 21 949 2300 - Fax (+351) 21 949 2301 http://www.nfsi.pt/ ----- "Suresh Ramasubramanian" <ops.lists(a)gmail.com> wrote: > On Fri, Feb 13, 2009 at 8:27 PM, Jens Ott - PlusServer AG > <j.ott(a)plusserver.de> wrote: > > - - What do you think about such service? > > - - Would you/your ASN participate in such a service? > > - - Do you see some kind of usefull feature in such a service? > > - - Do you have any comments? > > Ah. rbl.maps.vix.com from about a decade back when it was available > as > a bgp feed. But only for ddos sources. > > srs

10 14

Re: Linux Router: TCP slow, UDP fast
by Chris 14 Feb '09

14 Feb '09

Thanks for all the answers. I'm currently going down the path of looking at IPtables' conntrack slowing the forwarding rate. If I can't find any more docs then I'll boot the router with a kernel without any IPtables built-in and see if that's it. As Lee said "rollback" ! That's the last change to the box. If I can rule out the logging of traffic from conntrack is slowing down the forwarding then I can look into hardware further ;-) Chris

2 2

Linux Router: TCP slow, UDP fast
by Chris 14 Feb '09

14 Feb '09

Hi All, I'm losing the will to live with this networking headache ! Please feel free to point me at a Linux list if NANOG isn't suitable. I'm at a loss where else to ask. I've diagnosed some traffic oddities and after lots of head-scratching, reading and trial and error I can say with certainty that: With and without shaping and over different bandwidth providers using the e1000 driver for an Intel PRO/1000 MT Dual Port Gbps NIC (82546EB) I can replicate full, expected throughput with UDP but consistently only get 300kbps - 600kbps throughput _per connection_ for outbound TCP (I couldn't find a tool I trusted to replicate ICMP traffic). Multiple connections are cumulative and increase incrementally at roughly 300kbps - 600kbps. Inbound seems slightly erratic in holding a consistent speed but manages 15Mbps as expected, a far cry from 300kbps to 600kbps. The router is Quad Core sitting at no load and there's very little traffic being forwarded back and forth. The NIC's kernel parameters are set at default as 'built-in'. NAPI is not enabled though (enabling it requires a reboot which is a problem as this box is in production). The only other change to the box is that over Christmas IPtables (ip_conntrack and its associated modules mainly) was loaded into the kernel as 'built-in'. There's no sign of packet loss on any tests and I upped the conntrack max_connections size suitably for the amount of RAM. Has anyone come across IPtables without any rules loaded causing throughput issues ? I've also changed the following kernel parameters with no luck: net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216 net.ipv4.tcp_no_metrics_save = 1 net.core.netdev_max_backlog = 2500 echo 0 > /proc/sys/net/ipv4/tcp_window_scaling It feels to me like a buffer limit is being reached 'per connection'. The throughput spikes at around 1.54Mbps and TCP backs off to about 300kbps - 600kbps or so. What am I missing ? Is NAPI that essential for such low traffic ? A very similar build moved far higher throughput on cheap NICs. MTU is at 1500, txqueuelen is 1000. Any help would be massively appreciated ! Chris

5 13