operational: icmp echo out of control?
Hello, I've observed that our border routers are getting pinged a fair bit. I measured on one router and saw: 5 per second, seems consistent throughout the day, roughly 40 different sources every 15 seconds I took a look at the varied sources and discovered that the sites are well connected and those that resolve resolve to akamai names. This isn't more than a nuisance for me, but I run a small net. Should I conclude that an ISP with a population 10 times bigger would have their border routers getting pinged at 10 times the rate I see? If so, should we care, or just ignore it? Thanks, -mark
On Thu, May 23, 2002 at 10:05:08AM -0700, Mark Kent wrote:
This isn't more than a nuisance for me, but I run a small net. Should I conclude that an ISP with a population 10 times bigger would have their border routers getting pinged at 10 times the rate I see? If so, should we care, or just ignore it?
I can't speak for the specific rate at which our devices get pinged, but the issue doesn't really worry me. On Juniper (and probably on Cisco, but I can't speak for it) you can apply a policer to ICMP on the loopback interface and it will take effect for traffic addressed to any interface on the router. So if it gets to be out of control, some will start to get dropped. No need to watch over it and worry about adding filters and such. -c
On Thu, 23 May 2002, Mark Kent wrote:
I've observed that our border routers are getting pinged
5 per second, seems consistent throughout the day, roughly 40 different sources every 15 seconds
I took a look at the varied sources and discovered that the sites are well connected and those that resolve resolve to akamai names.
Congratulations! You've just discovered akamai load balancing. Block it if you want your customers to get akamai content from the less optimal source. ;-} -bryan bradsby NOC: 512-475-2432 Texas State Government Net -- Experience is what you get when you were expecting something else. ======================================================================
On Thu, May 23, 2002 at 10:05:08AM -0700, Mark Kent wrote:
I've observed that our border routers are getting pinged a fair bit. I measured on one router and saw:
5 per second, seems consistent throughout the day, roughly 40 different sources every 15 seconds
I took a look at the varied sources and discovered that the sites are well connected and those that resolve resolve to akamai names.
This isn't more than a nuisance for me, but I run a small net. Should I conclude that an ISP with a population 10 times bigger would have their border routers getting pinged at 10 times the rate I see? If so, should we care, or just ignore it?
I can't speak as to what exactly Akamai is doing, but this kind of probing for "performance" reasons is becoming increasingly common as more people jump on the "optimized routing" bandwagon. Not only do you have operational networks originating these probes on their own (InterNAP, Digital Island, Akamai, others), but you now have companies making boxes which "optimize routing" in part by doing these probes from every one of their customers. Right now it's mostly noise, but it has the potential to get way out of hand. A packet or two an hour probably wouldn't hurt anyone, but 5 packets/sec is personally what I would consider to be an acceptable amount of data to be directed at any specific host or router. Not only can this many probes trigger ICMP rate limiting and ruin the data for the prober and others, it is just plain unnecessary. Path latency doesn't change much, you can determine this with very few probes. Reachability does not need to be continuously probed, you can take cues from other data to decide if you need to re-probe. Packet loss cannot be reliably determined without a lot more packets than it is reasonable to send. Much like web spidering, some simple common sense can help keep probes from becoming a hassle: * Control the rate of your probes to a given destination. * Don't allow your probes to continuously hit a destination. * If you are using traceroute-style probes, extra care must be taken as if you were pinging every host along the path. * If at all possible, only target destinations you actually exchange traffic with. For example, get a netflow feed. * Make sure a DoS attack cannot provoke your system into probing innocent third parties. * Consider what is the smallest unit of "distinct network topology" you need to map. A very reasonable number would be a /24. * Source your probes from an IP which resolves to something that can explain what your probe is doing, and a webpage for people to read more about what you are doing and why (such as how it benefits them). * Have an "opt out" option for networks who REALLY don't like probes. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
RAS> Date: Thu, 23 May 2002 16:36:23 -0400 RAS> From: Richard A Steenbergen [ moderate snipping throughout ] RAS> I can't speak as to what exactly Akamai is doing, but this RAS> kind of probing for "performance" reasons is becoming RAS> increasingly common as more people jump on the "optimized RAS> routing" bandwagon. Perhaps most maddening is that ICMP echo/response hardly reflects real-world performance. (At least I don't usually tunnel my HTTP, SMTP, and FTP packets through ICMP, but perhaps I'm just being weird again.) RAS> Not only do you have operational networks originating these RAS> probes on their own (InterNAP, Digital Island, Akamai, RAS> others), but you now have companies making boxes which RAS> "optimize routing" in part by doing these probes from every RAS> one of their customers. I'd hope that they're having the IP stack communicate timing info to the apps. The information is superior, and it doesn't require any additional packets. RAS> Path latency doesn't change much, you can determine this RAS> with very few probes. Reachability does not need to be RAS> continuously probed, you can take cues from other data to RAS> decide if you need to re-probe. Packet loss cannot be RAS> reliably determined without a lot more packets than it is RAS> reasonable to send. RAS> Much like web spidering, some simple common sense can help RAS> keep probes from becoming a hassle: Hmmmm.... anyone recall the number of the RFC that says many of the same things? RAS> * If at all possible, only target destinations you actually RAS> exchange traffic with. For example, get a netflow feed. Which, IMHO, is the sane way anyhow. Why spend bandwidth and CPU munching on a point that exchanges 0.00000001% of one's traffic? It's silly. Now, if it's an outlier that shows performance worse than, say, 3 sigma slower than average, that might be another story. -- Eddy Brotsman & Dreger, Inc. - EverQuick Internet Division Phone: +1 (316) 794-8922 Wichita/(Inter)national Phone: +1 (785) 865-5885 Lawrence ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap <blacklist@brics.com> To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to <blacklist@brics.com>, or you are likely to be blocked.
RAS> I can't speak as to what exactly Akamai is doing, but this I should add that Akamai contacted me with minutes of my initial post to ask for more data and they said that they are looking into it... leaving me with the impression that what I was seeing was not typical. -mark
Its important to note a point entioned here that vendors are building boxes to do this as well. I ran a 3dns pair for a while and wow the mail that came in from people with firewalls or simply watching for probes. F5 was opening all sorts of half opened connections and wierd ports other than ones involving dns. I believed they called it iquerry. On Thu, 23 May 2002, E.B. Dreger wrote:
RAS> Date: Thu, 23 May 2002 16:36:23 -0400 RAS> From: Richard A Steenbergen
[ moderate snipping throughout ]
RAS> I can't speak as to what exactly Akamai is doing, but this RAS> kind of probing for "performance" reasons is becoming RAS> increasingly common as more people jump on the "optimized RAS> routing" bandwagon.
Perhaps most maddening is that ICMP echo/response hardly reflects real-world performance. (At least I don't usually tunnel my HTTP, SMTP, and FTP packets through ICMP, but perhaps I'm just being weird again.)
RAS> Not only do you have operational networks originating these RAS> probes on their own (InterNAP, Digital Island, Akamai, RAS> others), but you now have companies making boxes which RAS> "optimize routing" in part by doing these probes from every RAS> one of their customers.
I'd hope that they're having the IP stack communicate timing info to the apps. The information is superior, and it doesn't require any additional packets.
RAS> Path latency doesn't change much, you can determine this RAS> with very few probes. Reachability does not need to be RAS> continuously probed, you can take cues from other data to RAS> decide if you need to re-probe. Packet loss cannot be RAS> reliably determined without a lot more packets than it is RAS> reasonable to send.
RAS> Much like web spidering, some simple common sense can help RAS> keep probes from becoming a hassle:
Hmmmm.... anyone recall the number of the RFC that says many of the same things?
RAS> * If at all possible, only target destinations you actually RAS> exchange traffic with. For example, get a netflow feed.
Which, IMHO, is the sane way anyhow. Why spend bandwidth and CPU munching on a point that exchanges 0.00000001% of one's traffic? It's silly. Now, if it's an outlier that shows performance worse than, say, 3 sigma slower than average, that might be another story.
-- Eddy
Brotsman & Dreger, Inc. - EverQuick Internet Division Phone: +1 (316) 794-8922 Wichita/(Inter)national Phone: +1 (785) 865-5885 Lawrence
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap <blacklist@brics.com> To: blacklist@brics.com Subject: Please ignore this portion of my mail signature.
These last few lines are a trap for address-harvesting spambots. Do NOT send mail to <blacklist@brics.com>, or you are likely to be blocked.
Its important to note a point entioned here that vendors are building boxes to do this as well. I ran a 3dns pair for a while and wow the mail that came in from people with firewalls or simply watching for probes. F5 was opening all sorts of half opened connections and wierd ports other than ones involving dns. I believed they called it iquerry.
draft-ietf-dnsext-obsolete-iquery-03.txt randy
The problem here is that other types of probes raise IDS alarms on way too many networks - the next-best method is to probe HTTP ports, but we don't want to have to pull down thousands of web pages just to get performance stats. So, they send a SYN, wait for the ACK, record the latency and send a FIN. Sounds benign, but you'd be surprised how klaxons go off in response to this. -C
Perhaps most maddening is that ICMP echo/response hardly reflects real-world performance. (At least I don't usually tunnel my HTTP, SMTP, and FTP packets through ICMP, but perhaps I'm just being weird again.)
Chris Woodfield wrote:
...the next-best method is to probe HTTP ports, but we don't want to have to pull down thousands of web pages just to get performance stats.
Why not just passively measure the time it takes to send actual traffic to actual clients? It shouldn't take too much talent to use the aggregate data for an evolving best guess for each client request. KL
On Tue, May 28, 2002 at 01:05:19PM -0400, Chris Woodfield wrote:
The problem here is that other types of probes raise IDS alarms on way too many networks - the next-best method is to probe HTTP ports, but we don't want to have to pull down thousands of web pages just to get performance stats. So, they send a SYN, wait for the ACK, record the latency and send a FIN. Sounds benign, but you'd be surprised how klaxons go off in response to this.
So what, someone sneezes on an ethernet cable and IDS alarms go off. :) Theoretically, ICMP Echo should be less intrusive for performance measuring since it is clearly only for this purpose, whereas doing an actual TCP handshake could easily be mistaken for a port scan. But for so many network admins, all they know is "ICMP bad". Besides, if you thought ICMP was inaccurate, a TCP handshake is far worse. First, you have no way of knowing if the TCP replies are the originals, or if they have been retransmitted, so any single sample is pretty useless for measuring reply time. Second, for most cases with TCP opens, you are measuring the interaction time of the application process, not response time from the kernel. Depending on how the application is written, and what event model it uses, it is very likely to be doing other things before and after it gets notified that there is a new connection, then does the accept(). Internet path measurement isn't simple, you're probably best off trying to get as much data as possible passively. :) -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
RAS> Date: Tue, 28 May 2002 14:43:25 -0400 RAS> From: Richard A Steenbergen RAS> Theoretically, ICMP Echo should be less intrusive for RAS> performance measuring since it is clearly only for this RAS> purpose, whereas doing an actual TCP handshake could easily And less accurate for asymmetric traffic, under-representing the return flow. RAS> be mistaken for a port scan. But for so many network admins, RAS> all they know is "ICMP bad". That'll be the day when someone calls abuse saying "I'm being attacked by ICMP unreachables!" ;-) RAS> Besides, if you thought ICMP was inaccurate, a TCP handshake RAS> is far worse. First, you have no way of knowing if the TCP RAS> replies are the originals, or if they have been RAS> retransmitted, so any single sample is pretty useless for RAS> measuring reply time. Second, for most cases with TCP opens, RAS> you are measuring the interaction time of the application RAS> process, not response time from the kernel. Depending on how Ah, yes. Reminds me of the "benchmark" where a company that writes mailserver software tested HELO response time. Gee, my MXen spend more time actually sending and receiving messages! RAS> the application is written, and what event model it uses, it RAS> is very likely to be doing other things before and after it RAS> gets notified that there is a new connection, then does the RAS> accept(). If it gets that far. I suspect that accept filters would give artificially fast results. Then there are issues such as delayed ACK. And if one is measuring throughput, does the session ever make it past slow start? And just how many frames do they send during slow start? (Tunable under FreeBSD...) Once ECN becomes more common, does one consider those points outliers or useful data? RAS> Internet path measurement isn't simple, you're probably best RAS> off trying to get as much data as possible passively. :) Less intrusive, and chances are that anything too small for statistical validity is insignificant, anyway. -- Eddy Brotsman & Dreger, Inc. - EverQuick Internet Division Phone: +1 (316) 794-8922 Wichita/(Inter)national Phone: +1 (785) 865-5885 Lawrence ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap <blacklist@brics.com> To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to <blacklist@brics.com>, or you are likely to be blocked.
Also sprach E.B. Dreger
RAS> be mistaken for a port scan. But for so many network admins, RAS> all they know is "ICMP bad".
That'll be the day when someone calls abuse saying "I'm being attacked by ICMP unreachables!" ;-)
"That'll be..."? Future tense? Hrmm... -- Jeff McAdams Email: jeffm@iglou.com Head Network Administrator Voice: (502) 966-3848 IgLou Internet Services (800) 436-4456
At 03:21 PM 28/05/2002 -0400, Jeff Mcadams wrote:
Also sprach E.B. Dreger
RAS> be mistaken for a port scan. But for so many network admins, RAS> all they know is "ICMP bad".
That'll be the day when someone calls abuse saying "I'm being attacked by ICMP unreachables!" ;-)
"That'll be..."? Future tense? Hrmm...
Here is a sample posting from the "future"... It was much worse when my email address was the ARIN contact and I would get people screaming at me asking why I was hacking their machine :-( ---------begin annoying auto-robot-program---------- The attempt was detected by the personal firewall running on my machine, and I am quite concerned about it. If you are in fact responsible for this network, please do the following: 1) Research the access attempt(s), 2) Inform the responsible parties to discontinue access attempts, 3) Reply to me with your findings. If you are not responsible for this network, please forward this message to the person who is, or, if you do not know who this person is, please get back to me with that information as well. Thank you. The access attempt(s) are shown below, including the date and time, port number, TCP or UDP indicator, and, if known, a service name associated with the port. Jeu 09 mai 2002 15:30:22, Port 3, ICMP, Destination Unreachable Jeu 09 mai 2002 15:30:21, Port 3, ICMP, Destination Unreachable Jeu 09 mai 2002 15:30:10, Port 3, ICMP, Destination Unreachable Jeu 09 mai 2002 15:30:09, Port 3, ICMP, Destination Unreachable
On Tue, May 28, 2002 at 03:36:08PM -0400, Mike Tancsa wrote:
Jeu 09 mai 2002 15:30:22, Port 3, ICMP, Destination Unreachable Jeu 09 mai 2002 15:30:21, Port 3, ICMP, Destination Unreachable Jeu 09 mai 2002 15:30:10, Port 3, ICMP, Destination Unreachable Jeu 09 mai 2002 15:30:09, Port 3, ICMP, Destination Unreachable
I don't know whats worse, those crappy personal firewalls that make every packet look like a life or death assault, or the idiots who send abuse email demanding that you do something for them or they will sue and/or hax0r you. I've seen supposed "security professionals" for theoretically clued places like NASA send abuse complaints over traceroutes they've originated, and people complain about "port 80 hacking attempts" then flatly refuse to admit they visited website. At best, it's annoying clutter. Is it any wonder that legitimate emails about ongoing DoS attacks are completely ignored or responded to a week later? At worst, it can get innocent people in trouble and cost them a lot of time, effort, and potentially money. These false abuse reports are FAR too common, and the net equivilent of crying wolf. In my opinion, it is the responsability of these personal firewall makers to at least make an EFFORT to warn their users about this. So far, I havn't seen it. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
On Tue, 28 May 2002 16:01:12 EDT, Richard A Steenbergen said:
I don't know whats worse, those crappy personal firewalls that make every packet look like a life or death assault, or the idiots who send abuse email demanding that you do something for them or they will sue and/or hax0r you.
It's common enough that it's got it's own acronym. IWF - Idiot With Firewall.
On Tue, 28 May 2002 16:16:08 -0400 Valdis.Kletnieks@vt.edu wrote:
It's common enough that it's got it's own acronym. IWF - Idiot With Firewall.
We call them OZZADs and here is how we respond: http://condor.depaul.edu/~jkristof/technotes/incident-response.html John
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Mike Tancsa Sent: Tuesday, May 28, 2002 3:36 PM To: Jeff Mcadams Cc: nanog@merit.edu Subject: Re: operational: icmp echo out of control? [deleted] The access attempt(s) are shown below, including the date and time, port number, TCP or UDP indicator, and, if known, a service name associated with the port. Jeu 09 mai 2002 15:30:22, Port 3, ICMP, Destination Unreachable Jeu 09 mai 2002 15:30:21, Port 3, ICMP, Destination Unreachable Jeu 09 mai 2002 15:30:10, Port 3, ICMP, Destination Unreachable Jeu 09 mai 2002 15:30:09, Port 3, ICMP, Destination Unreachable ---- Speaking of IWFs... my personal pet peeve is when people forward you IDS or Firewall / some other packet log and neglect to provide a timezone and confirmation the clock was sync'd to a reliable source lately. For those of us that have the capability of pouring through a few billion packets to help identify _real_ issues, every few minutes represents millions of packets. Explaining the significance of this goes something like this: NOC: "When was your clock was synchronized with a network time source?" IWF: "huh?" [repeat variations of this for at least 5 minutes] Deepak Jain AiNET
[ On Tuesday, May 28, 2002 at 19:17:32 (+0000), E.B. Dreger wrote: ]
Subject: Re: operational: icmp echo out of control?
RAS> be mistaken for a port scan. But for so many network admins, RAS> all they know is "ICMP bad".
That'll be the day when someone calls abuse saying "I'm being attacked by ICMP unreachables!" ;-)
You mean that hasn't happened to you yet? Where have you been? :-) -- Greg A. Woods +1 416 218-0098; <gwoods@acm.org>; <g.a.woods@ieee.org>; <woods@robohack.ca> Planix, Inc. <woods@planix.com>; VE3TCP; Secrets of the Weird <woods@weird.com>
participants (15)
-
Bryan Bradsby -
Chris Woodfield -
Clayton Fiske -
Deepak Jain -
E.B. Dreger -
Jeff Mcadams -
John Kristoff -
Kevin Loch -
Mark Kent -
Mike Tancsa -
Randy Bush -
Richard A Steenbergen -
Scott Granados -
Valdis.Kletnieks＠vt.edu -
woods＠weird.com