On 08.12.24 12:43, Jay Hennigan wrote:

Matthew Black wrote:

...

I've had difficulties reaching anyone with a brain at my DSL provider Verizon California. Switch to a local ISP with local tech support.

bingo. i have multiple offices. in each case, i buy layers one and two from the copper/fiber monopoly and layer three from local folk with clue and caring: lavanet (hawai`i), infinitiy internet (pnw), and iij (tokyo, and yes i work for iij). local packet pushers with clue are not only better at layer three support and delivery, but they carry more weight with the hellco to get your layer one and two problem fixed. randy

Etaoin Shrdlu wrote:

Randy Bush wrote:

...

On 08.12.24 12:43, Jay Hennigan wrote:

...

Matthew Black wrote:

...

I've had difficulties reaching anyone with a brain at my DSL provider Verizon California.

Switch to a local ISP with local tech support.

...

bingo.

Uh, ditto? Having left SoCal a couple of years ago, my data is a bit stale. However, I happily used XO+Covad in three separate locations (in SoCal). DSLExtreme also has (or at least had) a good reputation. Verizon sucks. In fact, since you are in the Long Beach area, they suck even more than they do other places. Vote with your feet.

Layer 3 should indeed never be bought based on price or how big the provider is , but based on service. And if none of the options work out, you can always start your own. Verizon, no matter where in the world, always seems to be a nightmare to deal with. I second the suggestion to vote with your feet. Kind regards, Martin List-Petersen -- Airwire - Ag Nascadh Pobal an Iarthar http://www.airwire.ie Phone: 091-865 968

Roy wrote:

Etaoin Shrdlu wrote:

...

...However, I happily used XO+Covad in three separate locations (in SoCal). DSLExtreme also has (or at least had) a good reputation. Verizon sucks. In fact, since you are in the Long Beach area, they suck even more than they do other places. Vote with your feet.

I am pretty sure that COVAD is offshore now....

Might be, but the quality of customer service was the issue, I believe, not just where it was located (at least I hope that wasn't the only objection). I think Mr. Black has already made plain that cost is an issue, in any case. I used to have the lowest business class they provided (even though it was just to my house). Currently, I am the only customer for my local ISP with the service level I have, going to a residential address. We all spend our $$$ on what's important to us. Packets are important to me. I like 'em. -- Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. Brian W. Kernighan

You cannot blame the script readers. Whether they are next door to you or in India, the employer has decided to provide low training levels and script based support. You might consider calling their sales department. Explain that you are technically oriented and experienced and that the level of support they provide is inadequate for you and that you understand how networks run. Ask if it is possible to give you access to higher level support since this would result in less time being wasted by you and them when there are problems. They may suggest you get a business account.

On Wed, Dec 24, 2008 at 09:43:20AM -0800, Jay Hennigan wrote:

Matthew Black wrote:

...

I've had difficulties reaching anyone with a brain at my DSL provider Verizon California.

Switch to a local ISP with local tech support.

Actually, and I know this kind of experience is really subjective, but lately I have been getting better service from residents of India via web-based chat tools than I have been getting from residents of the US via telephone. At the same company. My impression as a customer is that only one of these two individuals genuinely wanted to do or keep the job they were given, and desired to do it well. That's really what you should be looking for, locality is irrelevant. -- Ash bugud-gul durbatuluk agh burzum-ishi krimpatul. Why settle for the lesser evil? https://secure.isc.org/store/t-shirt/ -- David W. Hankins "If you don't do it right the first time, Software Engineer you'll just have to do it again." Internet Systems Consortium, Inc. -- Jack T. Hankins

On Thu Dec 25, 2008 at 04:54:37PM -0500, Martin Hannigan wrote:

As long as the language skills are good [...]

Because, generally, this is not the case. Oh, and when there's 3 fibre cuts between you and India, and your voice gets shrunk to a 9kbps VoIP channel, it's doubly bad. Simon

Martin Hannigan wrote:

Hi Jay:

Is there really anything wrong with sending first-level technical support offshore?

Macs are macs, Windows is windows and mail is mail whether you're in Mumbai or Memphis. As long as the language skills are good and the people are well trained, it should be mostly irrelevant, IMHO.

In and of itself and setting aside patriotic/nationalistic issues, probably not, assuming adequate technical and product knowledge and language skills. I suppose that it would be possible that if it were done well enough one wouldn't be able to tell. However, there is something about dealing with a local company that adds value. People seem to care more about their community and neighbors than a random, barely understandable voice on a G.729 8k codec at the other end of a satellite link. I have generally found dealing with most offshore tech support to be very frustrating. The language issues are burdensome, some accents so thick as to be barely understandable, and the lack of clue and scripted menu-driven responses are obvious and usually of no value. I wouldn't be calling if the problem could be solved by reading the documentation and some judicious web searching. There are some exceptions, including Cisco TAC which is very good. I've talked to Cisco engineers in Australia and Europe on occasion. I've had mixed results with Linksys support, which I believe is in the Philippines. Dealing with one offshore AT&T billing representative who was clearly a non-English speaker was extremely painful. The latency and nonsense of the person's responses suggested either some type of auto-translator or satellite link, or both. The person wasn't capable of getting the hint when I asked after several minutes of frustration what the "A" in "AT&T" stood for, and in fact claimed to have no idea. I suspect that this level of disservice may be deliberate so that people will pay bogus charges on bills because the frustration level of disputing them is intentionally high. -- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV

Martin Hannigan wrote:

...

Hi Jay:

Is there really anything wrong with sending first-level technical support offshore?

...

Macs are macs, Windows is windows and mail is mail whether you're in Mumbai or Memphis. As long as the language skills are good and the people are well trained, it should be mostly irrelevant, IMHO.

In and of itself and setting aside patriotic/nationalistic issues, probably not, assuming adequate technical and product knowledge and language skills. I suppose that it would be possible that if it were done well enough one wouldn't be able to tell.

Sure. Blaming off-shore tech support is pretty easy stuff, but the reality is that the trouble is more along the line of appropriate training. For example, we maintain a Road Runner connection at the house, which has been generally flawless over the years, with some notable exceptions. I'll skip the DHCP-server-allocating-an-IP-address-from-a-netblk-recently- vanished-from-the-global-routing-table story. Just *try* explaining that to a tier 1... apparently my UNIX box was one of only a very few boxes that hadn't re-DHCP'd in a year or two :-) At one point, Road Runner introduced their "turbo" service here for a mere $10/month more. Since it's nice to be able to download the occasional ISO at high speed, and because it included a greater upstream speed, it was a no-brainer. Worked great for maybe about a year. Then, suddenly, one day, I began to see the modem crash anytime a largish amount of data was being pushed through it. Spend time characterizing the problem. Spend time on the phone. Get told the modem must be bad, get a replacement. You know the runaround, so I'll omit the gory details. After a replacement modem and the same problem, having spent several hours over the period of two days on it, start raising enough noise through both the local and national support services, talked to even the supposedly clueful people who were puzzled, and one finally suggested calling some direct line to "a network engineer." Well, that actually turned out to be TWC Business. The guy was a bit puzzled why I was calling *him*, but a brief explanation sufficed, and within a minute or two he had the problem located ... the modem had been only marginally sufficient for Turbo, and they had changed <something> on the local cable that had broken it. Needed a *different* kind of modem. Told me what to demand from the local cableco store, provided a ticket number and everything. Some discussion suggested that the RR people were highly script-oriented and not necessarily capable of complicated problem solving. It appears that the TWC Business tier 1 people actually have a fair amount of technical training and clue, and resources to tap if that's not good enough. Further, he was bright enough to let me know that they had a "better than turbo" package available with a higher upstream speed, for only a little more, that'd make me a business customer, so I'd never have to deal with Road Runner again. Based on this one experience, we were more than happy to sign an annual contract and pay just $10/mo more, and have direct access to people who know what words like "DHCP" and "route" actually mean. I did ask, and all the local people are, in fact, local. It's a matter of training and technical knowledge. None of them was really putting together the fact that the modem was sketchy for the service class we had. My point is that you not only need the language skills and a good phone connection, but also a reasonable process to deal with knowledgeable people. I understand the need to provide scripted support, but there should also be a reasonable path to determine that someone has an exceptional problem and isn't being well-served by the script.

However, there is something about dealing with a local company that adds value. People seem to care more about their community and neighbors than a random, barely understandable voice on a G.729 8k codec at the other end of a satellite link.

I have generally found dealing with most offshore tech support to be very frustrating. The language issues are burdensome, some accents so thick as to be barely understandable, and the lack of clue and scripted menu-driven responses are obvious and usually of no value. I wouldn't be calling if the problem could be solved by reading the documentation and some judicious web searching.

That'll be the typical problem for this audience, yes.

There are some exceptions, including Cisco TAC which is very good. I've talked to Cisco engineers in Australia and Europe on occasion. I've had mixed results with Linksys support, which I believe is in the Philippines.

Dealing with one offshore AT&T billing representative who was clearly a non-English speaker was extremely painful. The latency and nonsense of the person's responses suggested either some type of auto-translator or satellite link, or both. The person wasn't capable of getting the hint when I asked after several minutes of frustration what the "A" in "AT&T" stood for, and in fact claimed to have no idea. I suspect that this level of disservice may be deliberate so that people will pay bogus charges on bills because the frustration level of disputing them is intentionally high.

Yeah, ahaha. Like the "let's charge a late fee because we didn't promptly process your payment" thing (another fun story). Reminds me of the good old days of trying to contact somebody clueful at some random network's NOC. Many of the same problems. However, the operator community seems to have made good progress towards solving this problem. So now I'm wondering why we're discussing this. :-) ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

Joe Greco wrote:

...

Sure. Blaming off-shore tech support is pretty easy stuff, but the reality is that the trouble is more along the line of appropriate training.

But, the reason that US-based $TELCO and $CABLECO use off-shore tech support is that they don't want to pay for the training and supervision to do it right in-house.

Jay, that's an interesting misstatement. It implies that they're going to be paying a lesser rate to do it right somewhere else, which typically does not seem to be what happens.

The same person diagnosing your IP routing issues may indeed be asking, "Would you like fries with that?" thirty seconds later. [1]

Does Bronco actually do that? :-)

And, for purposes of, "Would you like fries with that?", off-shore is good enough that most customers can't tell, nor do they care. It may often be better than a newbie local ten feet from you. It's the ultimate scripted application, a literal menu. People expect half-duplex-low-fi audio when talking to a tin speaker buried inside of a plastic clown. ;-)

Right.

...

Some discussion suggested that the RR people were highly script-oriented and not necessarily capable of complicated problem solving.

And they are afraid to admit (or don't realize) that they are not capable of complicated problem solving. They're following a script, just like the fast food order-takers.

Don't-realize. The number of times I've been talked down to by people who don't have any clue what the "4" in "IPv4" means is depressingly high. I do not need to reboot my Windows PC to know that the DHCP answer my UNIX box is getting from the DHCP server, dumped in gory detail, is providing an IP address in a prefix that's not appearing in the global routing table now.

Or maybe they don't have the authority to escalate it to someone with clue, even if/when they do realize they're over their heads.

That's definitely a problem.

...

It appears that the TWC Business tier 1 people actually have a fair amount of technical training and clue, and resources to tap if that's not good enough. Further, he was bright enough to let me know that they had a "better than turbo" package available with a higher upstream speed, for only a little more, that'd make me a business customer, so I'd never have to deal with Road Runner again. Based on this one experience, we were more than happy to sign an annual contract and pay just $10/mo more, and have direct access to people who know what words like "DHCP" and "route" actually mean.

I did ask, and all the local people are, in fact, local. It's a matter of training and technical knowledge. None of them was really putting together the fact that the modem was sketchy for the service class we had.

So, regardless of geographic location, using scripted clueless order-takers without the ability to escalate for customer support is a bad thing. And, scripted clueless order-takers exist solely because they're cheap, not because they provide anything remotely resembling good service. Cheap, from a US-centric perspective, generally means offshore.

The interesting thing about your experience is that your service problems resulted in an up-sell, but only because you were persistent enough to fight through the system.

Plausible interpretation, but not really accurate. An upsell would normally be convincing someone to buy something that they would not otherwise have thought to be useful; is it really an "upsell" when you fail to advertise your new service offerings on your web site, and so leave your potential business customers with the impression that the only offerings you have are the same in-excess-of-T1 prices that you offered last time they talked to you? Come to think of it, I just looked and I still can't find any solid information about the plan we've got. I *think* it's some variation on the "teleworker" package. There's a "home business solution" pkg for $100/mo that includes 15M/2M broadband, but we're paying less than that...

Furthermore, it took a person with clue to do the up-sell. How many customers and up-sell opportunities does RR lose because of their decision to go with cheap, scripted, clueless off-shore support?

... or in this case, cheap, scripted, clueless in-house support ... The thing that is really unfortunate is that I had told the agent at the time we went to Turbo that I was primarily interested in upstream speed.

...

My point is that you not only need the language skills and a good phone connection, but also a reasonable process to deal with knowledgeable people. I understand the need to provide scripted support, but there should also be a reasonable path to determine that someone has an exceptional problem and isn't being well-served by the script.

Precisely. Or for better service have reasonably clueful people at level 1 so that they can quickly and expeditiously deal with the easy problems that could be scripted.

The scripted part could (and often is) being done with IVR, no humans at all. But, please, if you do this, use DTMF menus and not that God-awful worthless "Tell-me" speech-guessing machine. And make sure that every menu has a "0-to-human-being" option.

I don't know, I've seen some relatively impressive "speech-guessing machines." It is clear that the technology still needs some work, but Amtrak's "Julie" is fairly impressive and useful. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

Skywing wrote:

I find those speech recognition menus quite annoying. American Airlines has one that's just not good enough over a lower bitrate cell voice link in a crowded situation when you're trying to determine what's the deal with cancelled flights or whatnot along with everyone else in the plane. Always have to waste a minute for it to decide that it's going to punt to a real person. It would be nice if there was a way to bypass it.

say "agent" and keep repeating that word until it understands you. That will bypass the menus, and take you to a person. -- Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. Brian W. Kernighan

Skywing <Skywing@valhallalegends.com> writes:

Always have to waste a minute for it to decide that it's going to punt to a real person. It would be nice if there was a way to bypass it.

A lot (but not all) of them are implemented in such a way that they will yield encouraging results if you start dropping f-bombs on them. Avoiding an arrest for disorderly conduct if you are on your cell phone in public is left as an exercise to the reader. -r

On Sat, 27 Dec 2008, JF Mezei wrote:

The problem with oursourced first level support is that they are totally disconnected from real time operations and wouldn't be aware of problems that network engineers are currently working on.

Not always true. Our outsourced support in India were also our first layer of network troubleshooting, and they monitored everything related to the products they supported. They were almost always the first to call the engineers (in .us and .ca) to alert them of issues. It's all about /what/ you hire them to do..... ...david --- david raistrick http://www.netmeister.org/news/learn2quote.html drais@icantclick.org http://www.expita.com/nomime.html

On Sat, 27 Dec 2008 11:53:18 +0000 Martin List-Petersen <martin@airwire.ie> wrote:

The problem is, and this was stated by the original poster, that the lads off-shore he deals with have no clue and simply stick to the script. No intention of looking what the real problem is. And that problem lies not in the call center. It is the deal, that $TELCO struck with $CALLCENTER and the procedures, that were put in place, that are the problem.

Only solution: find a provider, who's support (off-shore or not) does have a clue, has an escalation process and is willing to find a solution.

How does one find such a provider? I'm unaware of any company that lets potential customers test drive their $SERVICE call center before purchase. Even if one did, how is a potential customer supposed to evaluate the competence of said call center when customer has no clue as to what problems may arise 5 years after purchase of provider's service, whether said test drive provided an accurate and appropriate solution, and whether said call center quality will exist 5 years after purchase of the service. matthew black long beach, ca

Matthew Black wrote:

On Sat, 27 Dec 2008 11:53:18 +0000 Martin List-Petersen <martin@airwire.ie> wrote:

...

The problem is, and this was stated by the original poster, that the lads off-shore he deals with have no clue and simply stick to the script. No intention of looking what the real problem is. And that problem lies not in the call center. It is the deal, that $TELCO struck with $CALLCENTER and the procedures, that were put in place, that are the problem.

Only solution: find a provider, who's support (off-shore or not) does have a clue, has an escalation process and is willing to find a solution.

How does one find such a provider? I'm unaware of any company that lets potential customers test drive their $SERVICE call center before purchase.

Ask others for their experience :), like for example here.

Even if one did, how is a potential customer supposed to evaluate the competence of said call center when customer has no clue as to what problems may arise 5 years after purchase of provider's service, whether said test drive provided an accurate and appropriate solution, and whether said call center quality will exist 5 years after purchase of the service.

Well, if you're not any happy longer with the service, vote with your feet again and find a better option. It's as easy as that. Kind regards, Martin List-Petersen -- Airwire - Ag Nascadh Pobal an Iarthar http://www.airwire.ie Phone: 091-865 968

Skywing wrote:

Of course, in much of the US, "vote with your feet" on residential ISP service might as well be as realistic advice as "pack up and move to a different city". [Perhaps not in the OP's case, though, if they are fortunate. Which it seems like they might be.]

It isn't different here either :) Solution: if there is no alternative, it might be an idea to create one. We had to do that here and works like a treat. You might find, that you get more custom, that you wished for. Kind regards, Martin List-Petersen

- S

-----Original Message----- From: Martin List-Petersen [mailto:martin@airwire.ie] Sent: Sunday, December 28, 2008 3:59 PM To: Matthew Black Cc: nanog@nanog.org Subject: Re: What to do when your ISP off-shores tech support

Matthew Black wrote:

...

On Sat, 27 Dec 2008 11:53:18 +0000 Martin List-Petersen <martin@airwire.ie> wrote:

...

The problem is, and this was stated by the original poster, that the lads off-shore he deals with have no clue and simply stick to the script. No intention of looking what the real problem is. And that problem lies not in the call center. It is the deal, that $TELCO struck with $CALLCENTER and the procedures, that were put in place, that are the problem.

Only solution: find a provider, who's support (off-shore or not) does have a clue, has an escalation process and is willing to find a solution.

How does one find such a provider? I'm unaware of any company that lets potential customers test drive their $SERVICE call center before purchase.

Ask others for their experience :), like for example here.

...

Even if one did, how is a potential customer supposed to evaluate the competence of said call center when customer has no clue as to what problems may arise 5 years after purchase of provider's service, whether said test drive provided an accurate and appropriate solution, and whether said call center quality will exist 5 years after purchase of the service.

Well, if you're not any happy longer with the service, vote with your feet again and find a better option. It's as easy as that.

Kind regards, Martin List-Petersen

-- Airwire - Ag Nascadh Pobal an Iarthar http://www.airwire.ie Phone: 091-865 968

On Fri, 26 Dec 2008 19:10:13 -0600 (CST) Joe Greco <jgreco@ns.sol.net> wrote:

I did ask, and all the local people are, in fact, local. It's a matter of training and technical knowledge. None of them was really putting together the fact that the modem was sketchy for the service class we had.

Yup -- I've had similar fun with Comcast. Once, I was seeing 15-20% packet loss on the local loop and 90% (you read that correctly) packet duplication. The advice I received translated to "clear your IE browser cache". I demurred, and I was told that (a) generally, performance problems were solvable that way, and (b) 15% packet loss was pretty good. I escalated... Then there was the time they upgraded the firmware in my cable modem/NAT to a buggy release that didn't understand the activity timer in the NAT table. Every 30 minutes, like clockwork, my ssh sessions would die. I had to try to explain that to someone who didn't know how to spell IP, let alone TCP. Oh yes -- judging from their accents, everyone I spoke with was American. In both cases, once I reached the clueful people, things were resolved pretty quickly. (Well, not the packet duplication; that took *weeks* to resolve, but once the packet loss problem was solved I could at least get decent throughput.)

My point is that you not only need the language skills and a good phone connection, but also a reasonable process to deal with knowledgeable people. I understand the need to provide scripted support, but there should also be a reasonable path to determine that someone has an exceptional problem and isn't being well-served by the script.

Customer records often include an optional data field that says things about particular customers. I heard a story -- and I'll leave out the names, since it's second- or third-hand and it does involve people and companies most of us know -- that one very clueful person's record had a note saying more or less "if you don't understand what he's saying, he's right and you're wrong, and you should route his call immediately to Tier N, where N is large"... But getting on that list is the hard part. --Steve Bellovin, http://www.cs.columbia.edu/~smb

Assuming that what you're getting from Verizon is copper and not FIOS, there should be a number of small to medium-sized ISPs that will provide you with Layer 3 Internet Service using that copper. It will cost you a few dollars a month more, but not a lot more, and you'll not only have more chance of getting somebody with a clue to answer questions, but you'll be able to do things like running servers from home if you want. It looks like Sonic.net doesn't cover Long Beach, but Speakeasy does, and you should be able to find a range of other small clueful ISPs. The off-shore call-center business has changed a lot in the last decade; in addition to Bangalore undercutting the Nebraska and Utah call centers, there are cheaper places like the Phillipines undercutting Bangalore, and Canada's been trying to address unemployment in former fishing villages by promoting call centers (which has the advantage of good English), and VOIP has simplified work-at-home distributed call centers in the rural US. But still, if your company is outsourcing first-line support to script-readers, then they need to be good at recognizing when to get past the initial script and escalate to somebody with more training. On Thu, Dec 25, 2008 at 2:37 PM, Jay Hennigan <jay@west.net> wrote:

The person wasn't capable of getting the hint when I asked after several minutes of frustration what the "A" in "AT&T" stood for, and in fact claimed to have no idea.

Actually, for the last N years, the "A" in "AT&T" is just a letter; the company name stopped being an acronym for "American Telephone & Telegraph" even before they were bought by the Company Formerly Known As SBC. -- ---- Thanks; Bill Note that this isn't my regular email account - It's still experimental so far. And Google probably logs and indexes everything you send it.

On Thu, Dec 25, 2008 at 6:00 PM, Frank Bulk - iName.com <frnkblk@iname.com>wrote:

I don't think there would be a concern about off-shore support if we couldn't tell it was "off-shore".

You can't tell most of the time. The point that is relevant operationally is that off shoring can be a solid method to help significantly reduce costs. It can work easier for some functions than others. Level 1/Tier1 support seems like an excellent candidate for off shoring and I think that the measure is still quality of service from the provider verses if they off shore or not. Just my humble opinion. Happy Holidays! -M<

On Fri, Dec 26, 2008 at 2:01 AM, Mark Foster <blakjak@blakjak.net> wrote:

Aside from the typical Degree or Diploma that tertiary outfits offer, there's not a lot of good ways to 'break in' to the Network and Systems Operations communities other than good ol experience, working-from-the-bottom-up.

I'm working in management of software engineering now, and in my experience, the only worthwhile candidates for hiring -- who have not gone through the self-teaching and self-experimentation phases that mirror working at a helpdesk on a small scale -- have progressed through exactly this chain. They have developed the necessary instincts to know when a bug could become a serious problem at 2 a.m. on a Sunday, instincts that are an absolute prerequisite to working on software intended to be used 24/7. In software development, new college grads can be OK for non-operationally-facing applications, but they tend to have high ideals, and just haven't "had their hearts broken" by business contradictions or operational emergencies yet. On the opposite side of the spectrum, those who have gone through only regimented software processes between school and the present tend not to be aware of operational impacts at all, as they've been shielded from that aspect all along.

So as you move your Tier 1's offshore, you cut off the channel by which people can gain experience and move on up the chain...

We're seeing this more and more as time goes on. What's worse is that offshoring of software development was becoming just as rampant, resulting in the double-whammy of "engineers" not knowing the consequences of their actions, and operations caught unaware when those consequences manifest as critical problems. Many businesses have at least partially learned from this mistake the Hard Way, by losing customers when there was no one capable of fixing a critical problem within 24 or even 72 hours. Alas, this hasn't been heeded by all of the market yet. All of the above is solely my opinion, and definitely represents an experience-diluted version of my personal ideals. While I generally agree from a business perspective that offshoring of operations can be a lucrative cost-cutting measure, the key problem in most such arrangements is that the operations and systems (hardware/software/networks as applicable) are not *all* offshored at once. When these bits do not exist in relatively close proximity to each other, communications between their responsible folks grinds to a halt. -- -- Todd Vierling <tv@duh.org> <tv@pobox.com> <todd@vierling.name>

Martin Hannigan wrote:

I'm not sure if I support off shoring or not as related to quality, but there is certainly a a business case to to be made supporting it as this thread ending up pointing out. There are trade offs which matter more to some than others.

I'm quite fascinated by some of the examples given of "offshoring". Cisco use Sydney as one of their locations for around the world coverage. From our point of view (being Australians) this isn't offshore - we have a local TAC who are closer and we tend to be able to get the same group of SP TAC people everytime to deal with our issues. My experience is that, given global companies like Cisco rely on locations to provide wide language support to people everywhere that the language issue is a bit moot. Some people in the Belgium TAC are easier to understand over the phone then people in the US TAC because the US TAC people have been employed for their Spanish skills or other language skills where as many Europeans have better English skills than, well, a lot of people. Some of the people in the TAC in Australia don't have English as their first language and are tricky to explain why my GSR crashed with an IPv6 issue over the phone (but fine via email). Some people on the other end of the phone just suck no matter which country or land of origin. (I use Cisco's TAC as an example purely because I'm familar - but the example can be reused). I think offshoring is more an issue because often it's built around a lie. If I'm talking to someone in another country, then I'm okay with that but I hate it when they're forced to lie about who they are and where they are. They're representing a company I deal with and as a customer I want it to be a good experience - if a company doesn't care about the overall customer experience and looks at it as a cost to be squashed and reduced then that (as someone else has said) is really the problem. Give them the tools and desire to help me as a customer no matter where they are or which god they pray to. The offshoring I think can be a problem isn't the customer facing part, but the anonymous part where backends of companies are taken offshore where data privacy laws etc aren't the same and suddenly my private data can be compromised in a way that is out of control of the laws of the country where I live. (I'm thinking banks, health care etc). Matthew

On Fri, Dec 26, 2008 at 08:01:59PM +1300, Mark Foster wrote:

On Fri, 26 Dec 2008, Martin Hannigan wrote:

...

On Thu, Dec 25, 2008 at 6:00 PM, Frank Bulk - iName.com <frnkblk@iname.com>wrote:

...

I don't think there would be a concern about off-shore support if we couldn't tell it was "off-shore".

You can't tell most of the time.

The point that is relevant operationally is that off shoring can be a solid method to help significantly reduce costs. It can work easier for some functions than others. Level 1/Tier1 support seems like an excellent candidate for off shoring and I think that the measure is still quality of service from the provider verses if they off shore or not.

Just my humble opinion.

Hi Martin, Seemingly a rational viewpoint (what, on NANOG? Surely not!) but the problem with the gradual depletion of Level/Tier 1 support environments in your home country is the (eventual) gradual depletion of expertise available to the higher levels.

A hellovalot of the clueful engineers that i've come to know over the past few years are people who started off on Helpdesks, and moved up the tiers, to finally land in NOC type slots and from there to engineering and design, perhaps skipping some or all of the 'tiers'... but you've gotta start somewhere.

Aside from the typical Degree or Diploma that tertiary outfits offer, there's not a lot of good ways to 'break in' to the Network and Systems Operations communities other than good ol experience, working-from-the-bottom-up.

So as you move your Tier 1's offshore, you cut off the channel by which people can gain experience and move on up the chain...

(The issues around the advantages from a cultural sense of having access to people who actually know your environs, current events, etc, are probably far more obvious..)

Could offshoring be considered a 'short term fix' and be hindering our ability to employ clooful operators in a few years time? (else, are we limiting ourselves to employing immigrants from 'offshore locations' because we don't locally build the right experience?)

Mark.

The fact that work is offshored also acts as a disincentive to people who might otherwise enter the field. Instead, they pursue work that is much less likely to be offshored. Witness the increase of healthcare workers in the US who might have otherwise entered various engineering professions, but do not, because they are concerned that they may not keep their engineering jobs. --gregbo