Ross Vandegrift wrote:
On Thu, Jul 03, 2008 at 10:36:27PM -0400, Christian Koch wrote:
i definitely see value in appliances like the fcp and route science box, i just think for a smaller provider it may not be necessary - or maybe i have it backwards,and it is a better solution for a smaller provider so they don't have to waste money on highly skilled engineers? maybe i am just thinking "inside" the box at the moment, from an engineers view..if so my apologies for steering off course
We've used the FCP for quite a few years now, with "good" success. The point at which we started seeing it being worthwhile was about 4 providers. Many of the challenges weren't having qualified engineers, or knowing the nature of your traffic, it was more a matter of being able to be dynamic, aware of the impact of the prefixes/ASN's that you are making changes to, managing cost, etc. In a content heavy network, where your traffic patterns vary greatly (based on clients/visitors all over the world), just knowing your traffic isn't enough. The argument could probably be made that you could script some of this, but it still doesn't get you the same solution, so partly it depends if you need a complete solution. We reached a point that in order to monitor traffic, commits, costs, performance, etc.. That we were spending a significant amount of time to do this with an engineer (or 3). It's an ongoing thing, not a once a day change, and with all the factors involved as to why you would make a change, it becomes far less accurate doing it with an engineer (using scripts, and traffic data) than an appliance designed to do it. Some of the biggest challenges we hit using an engineer were being able to "accurately" determine the amount of data you will be shifting when a change is made, based on a prefix or ASN, also knowing what the performance impact looks like for that prefix or ASN when a change it made to send it via another provider, not to mention monitoring your current active paths to attempt to be aware of performance problems you want to make a pro-active change for.
The FCP stinks at managing blackholing. There's supposedly new code on the way to help with some of the blackhole avoidance, but I'll believe it when I see it. It can only really control the outbound path, so if someone else chooses a path to me that blackholed between us, there's not a lot it can do.
Agreed, none of the appliances I've seen are 100%, nor are they infinitely scalable. We've had numerous issues with blackhole problems and the FCP, and I too won't hold my breath for this to get resolved. Especially since in the last 5 yrs we've used this product, we've seen very little evolution in features and functionality. We are actually at the point that we're out growing the abilities of the FCP, and I'm interested in the input on this thread to try and figure out what's next. The preferred method of data collection with the FCP is SPAN/Monitor, however for our network/topology that doesn't scale well (not to mention their costs don't scale well either). They also support Netflow, but have a VERY limited ability to process it in any volume.
On the other hand, the best value of the FCP is commit management. It does a fantastic job of making sure we pay the least amount of money to our tranit providers. No more manual balancing of traffic frees up a lot of time, and having an automatic process for it means that we never exceed commit on links that we don't have to.
The FCP produces lovely graphs and charts that describe this, which is probably why people accuse it of being too PHB-friendly. But Internap wasn't stupid - one of those pretty charts is cost savings the FCP has accumulated this month vs. the natural BGP decision.
For a network with a heavy outbound bias, that quickly adds up to a decent chunk of change.
Ross
------------------------------------------------------ Tom Sands Chief Network Engineer Rackspace (210)312-4391 ------------------------------------------------------ Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at abuse@rackspace.com, and delete the original message. Your cooperation is appreciated.