Using Mobile Phone email addys for monitoring
Hello folks, First off, apologies if this is off topic. I'm hoping that system and network monitoring tip are enough of a common issue that this falls under the group's charter. We've traditionally used mobile phone email addresses for system notifications, but over the past 6-12 months, it seems to have become increasingly sketchy. For instance, if an application fails to contact a certain service on a certain server, it sends an email (through it's own SMTP service, to avoid a chicken-and-egg prob if/when our main SMTP service fails) to 1234567890@tmomail.net. (Obviously, that was a fake number.) More and more, I'm getting less and less of these notifications. It seems especially prevalent when MANY things are sent at once; if, for example, a central piece fails, and dependent pieces suddenly fail as well. I try to telnet to mailx.tmomail.net port 25 and get sometimes good, sometimes laggy, and sometimes no response. T-Mobile, support levels all the way up to 3 tell me that it's not them, and everything should work wonderfully. Is SMTP to a mobile phone a fundamentally flawed way to do this? Anyone else have any issues, past or present, with this kind of thing? Thanks, Rick Kunkel
Anyone else have any issues, past or present, with this kind of thing?
It takes ~ 7 minutes from the time Nagios sends an email sms to AT&T to the time it hits my phone. I'm using @mobile.mycingular.com because mmode.com stopped working (which results in at least two txt pages vs. the one I was used to).
Is SMTP to a mobile phone a fundamentally flawed way to do this?
I'm beginning to think it is!
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
It takes ~ 7 minutes from the time Nagios sends an email sms to AT&T to the time it hits my phone. I'm using @mobile.mycingular.com because mmode.com stopped working (which results in at least two txt pages vs. the one I was used to).
Is SMTP to a mobile phone a fundamentally flawed way to do this?
I'm beginning to think it is!
It's more effective to spend the money on SMS messages. Mobile providers are forced to use very aggressive anti spam measures, which can add significant delays in message delivery. Regards, Ken - -- Ken Simpson CEO, MailChannels Fax: +1 604 677 6320 Web: http://mailchannels.com MailChannels - Reliable Email Delivery (tm) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFG4G6G2YHPr/ypq5QRAlG1AJ9/UGJwjzm1sAn5MUQpnGxRqMYtAACfaeh1 FVWwE0HDF6XdYMNz8d/zS7w= =+xQP -----END PGP SIGNATURE-----
Ken Simpson wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
It takes ~ 7 minutes from the time Nagios sends an email sms to AT&T to the time it hits my phone. I'm using @mobile.mycingular.com because mmode.com stopped working (which results in at least two txt pages vs. the one I was used to).
Is SMTP to a mobile phone a fundamentally flawed way to do this? I'm beginning to think it is!
It's more effective to spend the money on SMS messages. Mobile providers are forced to use very aggressive anti spam measures, which can add significant delays in message delivery.
Recommendations on software and modems?
On Thu, 6 Sep 2007, matthew zeier wrote:
Recommendations on software and modems? Couple of options:
Dedicated cell phone connected via serial cable and gnokii-like software Analog modem and voice line and TAP software (like sendpage or qpage) Technically, SNPP is the appropriate solution, but might be overkill if you just have a single host sending messages. -alex [not nanog mlc blah blah]
On Thu, Sep 06, 2007 at 05:51:38PM -0400, Alex Pilosov wrote: [...]
Analog modem and voice line and TAP software (like sendpage or qpage)
I like the TAP route with qpage. I was starting to get spam via my provider's e-mail to SMS gateway. They were kind enough to disable it, and we use TAP to send messages. I've never had any serious delays when using this method. Now, if I could just convince them that the messages are "on net" rather than out of network...
On Thu, Sep 06, 2007 at 02:22:10PM -0700, matthew zeier wrote:
Ken Simpson wrote:
It's more effective to spend the money on SMS messages. Mobile providers are forced to use very aggressive anti spam measures, which can add significant delays in message delivery.
Recommendations on software and modems?
We use Intercel modems with the bog-stock smstools package, and it works fine. - Matt
Ken Simpson wrote:
It's more effective to spend the money on SMS messages. Mobile providers are forced to use very aggressive anti spam measures, which can add significant delays in message delivery.
Recommendations on software and modems?
We've been having fun with these foxbox devices: http://www.acmesystems.it/?id=70 Low power, solid state embedded linux devices. Sending sms's is as easy as wget --post-data. We have 2 monitoring servers and 1 foxbox connected to each via crossover ethernet, and we're using a different cellphone provider for each. // nick
Am 06.09.2007 um 23:22 schrieb matthew zeier:
Ken Simpson wrote:
It's more effective to spend the money on SMS messages. Mobile providers are forced to use very aggressive anti spam measures, which can add significant delays in message delivery.
Recommendations on software and modems?
gsmd (from gsmlib http://www.pxh.de/fs/gsmlib/), a couple of small scripts, and a Siemens MC35i, working nicely with nagios at $work. gsmlib should be able to talk to most any GSM device with a serial port (data cards included), since the AT commands for sending and receiving SMS are standardized. Stefan -- Stefan Bethke <stb@lassitu.de> Fon +49 170 346 0140
* matthew zeier <mrz@velvet.org> [2007-09-06 23:39]:
It's more effective to spend the money on SMS messages. Mobile providers are forced to use very aggressive anti spam measures, which can add significant delays in message delivery. Recommendations on software and modems?
the UMTS PC-Cards you can get rather cheap these days show up as usb controllers with usb-cereal converters behind. add a little kermit magic and you're done (this is on OpenBSD). unlock at boottime (replace 0000 with your PIN): #!/usr/local/bin/kermit + set line /dev/ttyU0 if failure exit 1 set carrier-watch off set input echo on lineout "AT+CPIN?" input 10 "+CPIN: SIM PIN" if failure exit 1 input 10 OK if failure exit 1 lineout "AT+CPIN=0000" input 20 OK if failure exit 1 lineout "AT+CPIN?" input 10 "+CPIN: SIM PIN2" if failure exit 1 input 10 OK if failure exit 1 exit 0 send an sms: parameters: number "message" (+49177... is the SMSC, replace by your provider's one) #!/usr/local/bin/kermit + set line /dev/ttyU0 if failure exit 1 set carrier-watch off lineout ATZ input 10 OK if failure exit 1 lineout AT+CSCA="+491770610000" input 10 OK if failure exit 1 lineout AT+CMGF=1 input 10 OK if failure exit 1 lineout AT+CMGS="\%1" input 10 > lineout \%2 output \26 input 100 ok if failure exit 1 exit 0 of course I have some shell around it for failure handling (retries) etc -- Henning Brauer, hb@bsws.de, henning@openbsd.org BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam
On Thu, Sep 06, 2007 at 02:12:34PM -0700, matthew zeier wrote:
Anyone else have any issues, past or present, with this kind of thing?
as much as i hate to say it, verizon has been extremely reliable on the smtp<->sms gateway. been using them for paging for 3 years or so now and never had a significant (detected) failure or latency. if you don't like this way of doing things you could get an gprs modem and originate sms directly from the computer. t. -- _____________________________________________________________________ todd underwood +1 603 643 9300 x101 renesys corporation general manager babbledog todd@renesys.com http://www.renesys.com/blog
On 9/6/07, Todd Underwood <todd-nanog@renesys.com> wrote:
as much as i hate to say it, verizon has been extremely reliable on the smtp<->sms gateway. been using them for paging for 3 years or so now and never had a significant (detected) failure or latency.
if you don't like this way of doing things you could get an gprs modem and originate sms directly from the computer.
I miss the old days of hanging a modem off the side of the monitoring server and having it send a coded numeric page when something died. Seemed like that was much more reliable. I'd second the recommendation to go the modem route. Or, if you want to mess around just trying things without spending any money, try doing something like sending the alerts to a Gmail account, on which you have forwarding set up to go to the gateway. Or have a shell/perl/fetchmail script on another box offside download that Gmail message and feed it to the SMS mail gateway. I'd also perhaps set up (light, not excessive) monitoring of the provider's inbound SMTP gateway for a bit, to see if you can prove if it's your issue or theirs. If you can't reach their SMTP gateway consistently, then try it from another connection on another network (assuming you've got nine shell accounts around the world like most admins seem to), and if the reliability data is similar, you know the provider's got the problem, and you really need to either pound on the provider, switch providers, or go the modem route. I also have the feeling that unless you get a hold of somebody who actually works on those servers at T-Mobile, their support people are going to tell you it's working, no matter what, because it's a complex thing that they have no visibility into. I ran into similar fun conversations with Verizon Wireless in the past, before finally getting to the actual right person to tell me what I needed to know. BTW, how I handle it nowadays, believe it or not, is that the alerts go to a Hotmail account that my Windows Mobile phone accesses. (I know, I know, people complain about being able to mail to Hotmail -- I must be one of the lucky ones who knows the secret code to figure it out.) Has worked fine so far, put it in place a number of weeks ago. Regards, Al Iverson -- Al Iverson on Spam and Deliverability, see http://www.spamresource.com News, stats, info, and commentary on blacklists: http://www.dnsbl.com My personal website: http://www.aliverson.com -- Chicago, IL, USA Remove "lists" from my email address to reach me faster and directly.
On Thu, Sep 06, 2007 at 02:12:34PM -0700, matthew zeier wrote:
Anyone else have any issues, past or present, with this kind of thing?
It takes ~ 7 minutes from the time Nagios sends an email sms to AT&T to the time it hits my phone. I'm using @mobile.mycingular.com because mmode.com stopped working (which results in at least two txt pages vs. the one I was used to).
Is SMTP to a mobile phone a fundamentally flawed way to do this?
I'm beginning to think it is!
Some mobile phones you can talk to via AT commandset, either via USB cable or something else. (eg: I have used a Nokia 6230 with usb cable.. you can also use bluetooth). If you pay $5 or whatnot for unlimited SMS on a el-cheapo plan, it might work better than using the SMTP gateway (when tied to Nagios, etc..) as you can send SMS messages with the AT commandset. - Jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
At 05:29 PM 9/6/2007, Jared Mauch wrote:
On Thu, Sep 06, 2007 at 02:12:34PM -0700, matthew zeier wrote:
Anyone else have any issues, past or present, with this kind of thing?
It takes ~ 7 minutes from the time Nagios sends an email sms to
AT&T to the
time it hits my phone. I'm using @mobile.mycingular.com because mmode.com stopped working (which results in at least two txt pages vs. the one I was used to).
Is SMTP to a mobile phone a fundamentally flawed way to do this?
I'm beginning to think it is!
Some mobile phones you can talk to via AT commandset, either via USB cable or something else. (eg: I have used a Nokia 6230 with usb cable.. you can also use bluetooth). If you pay $5 or whatnot for unlimited SMS on a el-cheapo plan, it might work better than using the SMTP gateway (when tied to Nagios, etc..) as you can send SMS messages with the AT commandset.
Assuming, for the moment, that there's a cell signal available in your data center... Not always the case, unfortunately.
On Thu, Sep 06, 2007 at 02:12:34PM -0700, matthew zeier wrote:
Anyone else have any issues, past or present, with this kind of
Hi All, Our experience with using the e-mail-to-SMS gateways provided by AT&T/Cingular and T-Mobile: AT&T: Messages come through with very little delay (even during alert storms). T-Mobile: 10-15 messages/hour are allowed through...then T-Mobile refuses the IP for about an hour. -J -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Daniel Senie Sent: Thursday, September 06, 2007 4:09 PM To: Jared Mauch; matthew zeier Cc: Rick Kunkel; nanog@merit.edu Subject: Re: Using Mobile Phone email addys for monitoring At 05:29 PM 9/6/2007, Jared Mauch wrote: thing?
It takes ~ 7 minutes from the time Nagios sends an email sms to
AT&T to the
time it hits my phone. I'm using @mobile.mycingular.com because mmode.com stopped working (which results in at least two txt pages vs. the one I was used to).
Is SMTP to a mobile phone a fundamentally flawed way to do this?
I'm beginning to think it is!
Some mobile phones you can talk to via AT commandset, either via USB cable or something else. (eg: I have used a Nokia 6230 with usb cable.. you can also use bluetooth). If you pay $5 or whatnot for unlimited SMS on a el-cheapo plan, it might work better than using the SMTP gateway (when tied to Nagios, etc..) as you can send SMS messages with the AT commandset.
Assuming, for the moment, that there's a cell signal available in your data center... Not always the case, unfortunately. !SIG:46e0923b62578058632379!
On Thu, Sep 06, 2007 at 06:04:44PM -0600, Jason J. W. Williams wrote:
Hi All,
Our experience with using the e-mail-to-SMS gateways provided by AT&T/Cingular and T-Mobile:
AT&T: Messages come through with very little delay (even during alert storms).
As long as you're sourced from ARIN IP space. If sourced from for exmaple, APNIC space, you may have mixed results or lack of service. This has been my experience, even when there is proper rDNS, SPF, and other such methods being used. This means you need to relay via an alias elsewhere. - Jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
:> Some mobile phones you can talk to via AT commandset, either :>via USB cable or something else. (eg: I have used a Nokia 6230 with usb :>cable.. you can also use bluetooth). If you pay $5 or whatnot for unlimited :>SMS on a el-cheapo plan, it might work better than using the SMTP gateway :>(when tied to Nagios, etc..) as you can send SMS messages with the AT :>commandset. : :Assuming, for the moment, that there's a cell signal available in :your data center... Not always the case, unfortunately. I recall a datacenter in BOS that went so far as to nearly eliminate RF using corrugated aluminum inside the walls (you know who you are :) The simple answer is that it depends on how critical such notifications are. Address it as you would your upstream connectivity, and make it as redunant as is justified. For my meager purposes, smtp is usually fine. For truly critical issues, my nms will use a dedicated phone line to dial a handful of on-call techs, with no more info than caller-id. If that id shows up on their phones, immediate investigation is needed. It's embarrassingly primitive, but it's never failed. Cheers, Brian
matthew zeier wrote:
Anyone else have any issues, past or present, with this kind of thing?
It takes ~ 7 minutes from the time Nagios sends an email sms to AT&T to the time it hits my phone. I'm using @mobile.mycingular.com because mmode.com stopped working (which results in at least two txt pages vs. the one I was used to).
Is SMTP to a mobile phone a fundamentally flawed way to do this?
I'm beginning to think it is!
Any place I've worked thats used sms alerting has ended up using either sms-tools (http://smstools.meinemullemaus.de/) or gnokki (www.gnokii.org) with a bit of scripting to send the sms ourselves. Vince
matthew zeier wrote:
Anyone else have any issues, past or present, with this kind of thing?
It takes ~ 7 minutes from the time Nagios sends an email sms to AT&T to the time it hits my phone. I'm using @mobile.mycingular.com because mmode.com stopped working (which results in at least two txt pages vs. the one I was used to).
Is SMTP to a mobile phone a fundamentally flawed way to do this?
I'm beginning to think it is!
I've never had a problem with Sprint (###@messaging.sprintpcs.com) accepting on their gateway. Although it has always accepted messages, sometimes there was an hour or two delay before the it hit the phone. Also, if you send too many messages too fast it'll stop talking to you for a bit (450 errors) or throttle the SMS delivery. The delay I normally see is under 30 seconds. If you want to be fancy and take the internet out of the equation, you can use festival with Asterisk to have it call you and speak the messages. (Bonus points for a "press 1 to acknowledge this problem, 2 to escalate, etc." IVR tree.) ~Seth
On Sep 6, 2007, at 5:39 PM, Seth Mattinen wrote:
matthew zeier wrote:
Anyone else have any issues, past or present, with this kind of thing? It takes ~ 7 minutes from the time Nagios sends an email sms to AT&T to the time it hits my phone. I'm using @mobile.mycingular.com because mmode.com stopped working (which results in at least two txt pages vs. the one I was used to). Is SMTP to a mobile phone a fundamentally flawed way to do this? I'm beginning to think it is!
I've never had a problem with Sprint (###@messaging.sprintpcs.com) accepting on their gateway. Although it has always accepted messages, sometimes there was an hour or two delay before the it hit the phone. Also, if you send too many messages too fast it'll stop talking to you for a bit (450 errors) or throttle the SMS delivery. The delay I normally see is under 30 seconds.
If you want to be fancy and take the internet out of the equation, you can use festival with Asterisk to have it call you and speak the messages. (Bonus points for a "press 1 to acknowledge this problem, 2 to escalate, etc." IVR tree.)
I've had issues sometimes with Sprint throttling email -> SMS, especially if the message all look very similar. Also had the stop delivering some emails (our trouble ticket notification system specifically, they would accept the message and effectively bit bucket it). We've had very good luck using qpage (http://www.qpage.org/) to send messages via TAP. It has worked for years to our Nextel's (NPA-NXX-NOTE), and I now do the same on my Treo from Sprint. Just need to locate a TAP terminal for your carrier. We have nagios (and various other bits of software) submit pages to a qpage daemon, and it handles delivery via dedicated modem, which is nice in case all of your upstreams decided to die @ the exact same time. -Patrick -- Patrick Muldoon Network/Software Engineer INOC (http://www.inoc.net) PGPKEY (http://www.inoc.net/~doon) Key ID: 0x370D752C "Life's disappointments are harder to take when you don't know any swear words." --Calvin & Hobbes
Anyone else have any issues, past or present, with this kind of thing?
It takes ~ 7 minutes from the time Nagios sends an email sms to AT&T to the time it hits my phone. I'm using @mobile.mycingular.com because mmode.com stopped working (which results in at least two txt pages vs. the one I was used to).
Is SMTP to a mobile phone a fundamentally flawed way to do this?
I'm beginning to think it is!
It appears that device messaging in general is getting more difficult. We use SNPP and TAP paging to drive paging to actual pagers. Years ago, I experimented with using cell phones instead of pagers, and the reliability of the service offered by cell phone companies was all over the map, despite the fact that a phone ought to make a fairly ideal pager, being two-way capable, rechargeable, etc. Slow and non-deliveries were about ~50%. These days, we're seeing that problem with our pager service, where the pager is a confirmed delivery pager, like the PF1500. In this model, the pager network knows where it last saw the pager, so there's no multistate or nationwide broadcasting of pages - the local tower speaks to the pager, which confirms. If it fails to confirm, the network queues the message, and when the pager reappears, rebroadcasts. This even handles the case where the tower is too distant to hear the pager, since the page is still sent in the last seen area. Unfortunately, we've noticed a degradation in service quality on the part of the paging network, with problems ranging from busies on the TAP dial pool, to other really stupid stuff. It used to be that I could be in a basement or other RF-nasty environment, come on out, and pages would be retransmitted to me within a few minutes. Now, I can drive around areas near towers, not get pages, or, for more fun, and this is great, get near a different tower, get *new* pages, followed an hour or two (or twelve) later by *old* pages. I think I mostly despise the UI on the PF1500 anyways. I'd rather be able to dismiss a page with a single keystroke, and overall I preferred the way the Mot Adv Elite used to work. Anyways, this is an interesting and useful topic, which I'm watching with some interest. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
On Thu, 2007-09-06 at 14:12 -0700, matthew zeier wrote:
Anyone else have any issues, past or present, with this kind of thing?
It takes ~ 7 minutes from the time Nagios sends an email sms to AT&T to the time it hits my phone. I'm using @mobile.mycingular.com because mmode.com stopped working (which results in at least two txt pages vs. the one I was used to).
try using @txt.att.net ;-) -Jim P.
On Sep 6, 2007, at 1:46 PM, Rick Kunkel wrote:
Is SMTP to a mobile phone a fundamentally flawed way to do this?
Yes, IMHO - too many things to fail, including potentially your own DCN, the SMTP gateway service from the mobile operator, et. al. I'd strongly recommend a direct NMS-to-SMS gateway, etc., OOB. And of course, multiple methods in event of failure of one of them. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // 408.527.6376 voice I don't sound like nobody. -- Elvis Presley
On 9/6/07, Rick Kunkel <kunkel@w-link.net> wrote: [snip] We've traditionally used mobile phone email addresses for system
notifications, but over the past 6-12 months, it seems to have become increasingly sketchy.
[snip] Is SMTP to a mobile phone a fundamentally flawed way to do this?
Anyone else have any issues, past or present, with this kind of thing?
We tend to avoid the whole SMTP mess and deliver messages to mobiles and pagers via a modem and the provider's TAP gateway. It works quite well with Verizon and AT&T/Cingular, but I've no experience with T-Mobile. It also avoids the whole mess of failing to alert when your monitoring box has a bad NIC, cable, switchport, etc - of course considering that you trade those for problems with a serial port, cable, modem, or phone line... But it gives us a big (and perhaps false) warm fuzzy that our alerting is 'out of band' relative to our upstream Internet connections. The folks at Avtech have a nice index of TAP gateway numbers at http://www.avtech.com/Support/TAP/index.htm Hope you find this useful... --D
Once upon a time, Duane Waddle <duane.waddle@gmail.com> said:
We tend to avoid the whole SMTP mess and deliver messages to mobiles and pagers via a modem and the provider's TAP gateway. It works quite well with Verizon and AT&T/Cingular, but I've no experience with T-Mobile.
T-Mobile dropped their TAP access several years ago. The only way to send messages to T-Mobile phones is SMTP or SMS. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
Once upon a time, Duane Waddle <duane.waddle@gmail.com> said:
We tend to avoid the whole SMTP mess and deliver messages to mobiles and pagers via a modem and the provider's TAP gateway. It works quite well with Verizon and AT&T/Cingular, but I've no experience with T-Mobile.
T-Mobile dropped their TAP access several years ago.
Well, good, because they were pretty cruddy at it. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
On Thu, Sep 06, 2007 at 01:46:18PM -0700, Rick Kunkel wrote:
For instance, if an application fails to contact a certain service on a certain server, it sends an email (through it's own SMTP service, to avoid a chicken-and-egg prob if/when our main SMTP service fails) to
[...]
Is SMTP to a mobile phone a fundamentally flawed way to do this?
Anyone else have any issues, past or present, with this kind of thing?
Consider what other points of failure there are for your notification e-mails, other than your main SMTP server. I've got: * Failure of your Internet link * DNS failure at your end * SMTP failure at the other end * Failure of *their* Internet link * Some sort of SMTP blacklisting at their end There's probably some I've missed there, too. Notification of outages needs to be as robust as possible, and SMTP to an off-site location is about as fragile as they come these days. The only thing I spec for SMS notifications is a GSM modem physically connected to the monitoring box. There's still points of failure, but they're a lot fewer than SMTP to some third party. True paranoids (as we all should be) monitor their monitoring box, and it might be permissible to use an SMTP to SMS gateway for that monitoring, as long as you're monitoring all the appropriate things so that wide-scale failures (such as power loss) still get to you via your GSM modem (mmm, local UPSen). - Matt Professional Paranoid
GSM/GPRS modems are cheap; so are SMS messages. The answer should be clear... On 9/6/07, Matthew Palmer <mpalmer@hezmatt.org> wrote:
The only thing I spec for SMS notifications is a GSM modem physically connected to the monitoring box. There's still points of failure, but they're a lot fewer than SMTP to some third party.
True paranoids (as we all should be) monitor their monitoring box, and it might be permissible to use an SMTP to SMS gateway for that monitoring, as long as you're monitoring all the appropriate things so that wide-scale failures (such as power loss) still get to you via your GSM modem (mmm, local UPSen).
- Matt Professional Paranoid
Is it flawed? It depends on your business requirements. If seconds, milliseconds, or even microseconds matter to your mission critical apps (think real-time trading networks) then you would want a 24x7 staffed NOC using an enterpise monitoring system - something like Openview. You wouldn't want to rely on anything that sends emails. Brian Knoll -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Rick Kunkel Sent: Thursday, September 06, 2007 3:46 PM To: nanog@merit.edu Subject: Using Mobile Phone email addys for monitoring Hello folks, First off, apologies if this is off topic. I'm hoping that system and network monitoring tip are enough of a common issue that this falls under the group's charter. We've traditionally used mobile phone email addresses for system notifications, but over the past 6-12 months, it seems to have become increasingly sketchy. For instance, if an application fails to contact a certain service on a certain server, it sends an email (through it's own SMTP service, to avoid a chicken-and-egg prob if/when our main SMTP service fails) to 1234567890@tmomail.net. (Obviously, that was a fake number.) More and more, I'm getting less and less of these notifications. It seems especially prevalent when MANY things are sent at once; if, for example, a central piece fails, and dependent pieces suddenly fail as well. I try to telnet to mailx.tmomail.net port 25 and get sometimes good, sometimes laggy, and sometimes no response. T-Mobile, support levels all the way up to 3 tell me that it's not them, and everything should work wonderfully. Is SMTP to a mobile phone a fundamentally flawed way to do this? Anyone else have any issues, past or present, with this kind of thing? Thanks, Rick Kunkel
On 9/6/07, Rick Kunkel <kunkel@w-link.net> wrote:
We've traditionally used mobile phone email addresses for system notifications, but over the past 6-12 months, it seems to have become increasingly sketchy.
Rick, I've had good results with vzw.blackberry.net (Verizon Wireless + Blackberry) in the Washington DC area. A secondary monitoring server outside the network will send a message if the network problem is serious enough to break the path from within the network to vzw.blackberry.net. I also have a network monitoring system that's smart enough to track dependencies so it doesn't page me about the http service being down if it has already paged me because it can't ping the router that the host sits behind. As a result, I very rarely get a notification flood. It also keeps notifying until I ack it so if the first message doesn't go through, the second does. Regards, Bill Herrin -- William D. Herrin herrin@dirtside.com bill@herrin.us 3005 Crane Dr. Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Thu, Sep 06, 2007 at 01:46:18PM -0700, Rick Kunkel wrote: [snip]
Is SMTP to a mobile phone a fundamentally flawed way to do this?
Yes - think of the dependency chain involved. Years ago, hacking hylafax (or similar DTMF sources) to dial directly to pagers was a commonplace solution. Cheers, Joe -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE
I used a wide range of alerting methods. The most reliable that I have found (at least for Cingular/AT&T phones) has been TAP. Since this way the monitoring server can have its own dedicated modem / phone line (separate from the PBX). Thereby you no longer have to use any of the monitored equipment to relay failures notifications (i.e. chicken and the egg). Because a TAP solution is total separate, one thing we had to do was a daily "test page". Basically every day at noon, a fake check would get tripped on the monitoring system and send an alert to some of the folks that maintained the monitoring server. This way if there was a breakdown in the monitoring system they would (hopefully) notice that they did not get the daily page and fix the problem. Regards, Adam Stasiniewicz -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Rick Kunkel Sent: Thursday, September 06, 2007 3:46 PM To: nanog@merit.edu Subject: Using Mobile Phone email addys for monitoring Hello folks, First off, apologies if this is off topic. I'm hoping that system and network monitoring tip are enough of a common issue that this falls under the group's charter. We've traditionally used mobile phone email addresses for system notifications, but over the past 6-12 months, it seems to have become increasingly sketchy. For instance, if an application fails to contact a certain service on a certain server, it sends an email (through it's own SMTP service, to avoid a chicken-and-egg prob if/when our main SMTP service fails) to 1234567890@tmomail.net. (Obviously, that was a fake number.) More and more, I'm getting less and less of these notifications. It seems especially prevalent when MANY things are sent at once; if, for example, a central piece fails, and dependent pieces suddenly fail as well. I try to telnet to mailx.tmomail.net port 25 and get sometimes good, sometimes laggy, and sometimes no response. T-Mobile, support levels all the way up to 3 tell me that it's not them, and everything should work wonderfully. Is SMTP to a mobile phone a fundamentally flawed way to do this? Anyone else have any issues, past or present, with this kind of thing? Thanks, Rick Kunkel
As an experiment, I wanted to try to summarize all the answers given on this question, hope this helps someone. Suggestions given: * modem and TAP gateway ** TAP numbers at http://www.avtech.com/Support/TAP/index.htm ** Software: sendpage or qpage * Mobile phone with a serial port and AT commandset ** Software: sms-tools gnokii gsmd ** Issues: not reliable because of battery drain * Purpose-made GSM/CDMA modems ** Software: same as above ** Manufacturers: Intercel, Sierra 750 (PCMCIA), Falcom Samba 75 (USB) * Purpose-made GSM-IP modems ** Manufacturers: http://www.acmesystems.it/?id=70 * Pages via DTMF ** Hylafax/asterisk -alex [for mlc]
On Fri, 2007-09-07 at 19:54 -0400, Alex Pilosov wrote:
As an experiment, I wanted to try to summarize all the answers given on this question, hope this helps someone.
Suggestions given:
* modem and TAP gateway ** TAP numbers at http://www.avtech.com/Support/TAP/index.htm ** Software: sendpage or qpage
* Mobile phone with a serial port and AT commandset ** Software: sms-tools gnokii gsmd ** Issues: not reliable because of battery drain
s/serial/usb/ to solve the power problem ;-) -Jim P.
On 9/7/07, Alex Pilosov <alex@pilosoft.com> wrote:
* Purpose-made GSM/CDMA modems ** Software: same as above ** Manufacturers: Intercel, Sierra 750 (PCMCIA), Falcom Samba 75 (USB)
Does anyone have experience and/or opinion (positive or negative) with the Multi-Tech cellular modems? http://www.multitech.com/PRODUCTS/Families/MultiModemCDMA/ I've been quite pleased with their standard analog kit, so I'm hoping it translates well into this product line.
I would never trust SMTP for all the reasons already mentioned. Primarily if my network is dead, I still want to get paged about it. Relying on the import policy of another organization in the hostile port 25 environment is also bad voodoo. We've used a mix of TAP and SMS for many years with varying success. When the cell providers started dropping their TAP gateways, we went through a few GSM modems. First a Nokia on a cable, but the thing would do dumb stuff like charge the battery, and with the cable connected go ahead and drain the cells unless someone walked by daily and reseated the power cord. Avoid tethering phones, you will likely run into something to drive you nuts. Next was a Sierra 750 PCMCIA GSM modem, which supported the standard AT command set (I forget the ANSI spec). That was fine once overcoming the motherboard not assigning an IRQ, but once a week it'd stop responding to commands and have to be reseated. The final, ultimately reliable setup, which I recommend, was a Falcom Samba 75 USB GSM modem (quad band) talking to smstools. With unlimited SMS plans, two modems on separate networks, and some cron scripts, they could also monitor each other every hour to ensure they were sending and receiving pages. We also did a daily "paging system is up on X" to oncall similar to what another poster mentioned. Also we configured smstools to call an error script which'd send warnings another way (IRC in our case) if the modem wasn't responding as expected (failed to send, receive or init). On 9/6/07, Rick Kunkel <kunkel@w-link.net> wrote:
Hello folks,
First off, apologies if this is off topic. I'm hoping that system and network monitoring tip are enough of a common issue that this falls under the group's charter.
We've traditionally used mobile phone email addresses for system notifications, but over the past 6-12 months, it seems to have become increasingly sketchy.
For instance, if an application fails to contact a certain service on a certain server, it sends an email (through it's own SMTP service, to avoid a chicken-and-egg prob if/when our main SMTP service fails) to 1234567890@tmomail.net. (Obviously, that was a fake number.) More and more, I'm getting less and less of these notifications.
It seems especially prevalent when MANY things are sent at once; if, for example, a central piece fails, and dependent pieces suddenly fail as well.
I try to telnet to mailx.tmomail.net port 25 and get sometimes good, sometimes laggy, and sometimes no response. T-Mobile, support levels all the way up to 3 tell me that it's not them, and everything should work wonderfully.
Is SMTP to a mobile phone a fundamentally flawed way to do this?
Anyone else have any issues, past or present, with this kind of thing?
Thanks,
Rick Kunkel
Is SMTP to a mobile phone a fundamentally flawed way to do this?
Yes. It takes too long and nobody is responsible for making sure it is fast. SMS is better, even though it can also suffer from delays, because somebody is in charge of making sure it is fast. The worst delays I witnessed were over an hour during the terror attacks on London a couple of years ago. We have an SDK http://web21c.bt.com/ that lets people build SMS sending into their applications for event notification, so if you are in Europe, this may be useful. I know other companies have software for sending SMS that works with a PCMCIA GSM card plugged into a computer. If guaranteed turnaround time is important, then becoming an official customer is the best way of communicating those needs. Even better if your supplier is selling you the SMS service specifically for event notification. Outside of having a supplier who offers an SLA on SMS delivery time, I'm not sure that it is wise to rely on SMS for notifying your first responders. --Michael Dillon
On 9/6/07, Rick Kunkel <kunkel@w-link.net> wrote:
Hello folks,
It seems especially prevalent when MANY things are sent at once; if, for example, a central piece fails, and dependent pieces suddenly fail as well.
I'd probably recommend implementing some sort of parent/child system to reduce the number of messages you receive. Couple that with an acknowledgment syetem to re-page every so often to remind you, and it works out really well. That way you're not overloading the person receiving the alerts, and, especially with SMS, it costs less to send them.
Thanks,
Rick Kunkel
-- Jason 'XenoPhage' Frisvold XenoPhage0@gmail.com http://blog.godshell.com
participants (31)
-
Al Iverson -
Alex Pilosov -
Alexander Harrowell -
Brian Knoll (TTNET) -
Brian Wallingford -
Chris Adams -
Daniel Senie -
Duane Waddle -
Henning Brauer -
Jared Mauch -
Jason Frisvold -
Jason J. W. Williams -
Jim Popovitch -
Joe Greco -
Joe Provo -
John Osmon -
Ken Simpson -
Kevin Blackham -
Matthew Palmer -
matthew zeier -
michael.dillon＠bt.com -
nick.nauwelaerts＠thomson.com -
Patrick Muldoon -
Rick Kunkel -
Roland Dobbins -
Seth Mattinen -
Stasiniewicz, Adam -
Stefan Bethke -
Todd Underwood -
Vince -
William Herrin