I've been doing a lot of reading and thinking on what the best solution is for an application that initially requires approximately 15Mbps outgoing and 8Mbps incoming (as viewed from my router), and talks with 500000 unique hosts daily (i.e. has fairly wide coverage of the net). The application involves at least thirty machines, so colocation is likely to be cost-prohibitive. A single T3, or frac T3 isn't an option because there isn't a single provider that I can trust for the availability we want. Even ignoring availability, I seriously doubt that any provider can consistently fill a single customer's T3 pipe these days.
From stuff I've seen here and elsewhere I think the most important reason for this is congestion at NAPs making it impossible to suck (or shove) lots of bandwidth at anything but your provider's backbone. Taking all of this into account, I'm really leaning towards a solution that involves lots of small pipes to lots of providers. Essentially eliminating the need for 90% of our packets to traverse NAPs by using each backbone mostly for their own customers.
Now that I've done my homework, I'd like to hear comments from some of the more experienced folk here. I haven't considered yet the maintenance/logistical cost of managing 15 T1s to 6 or 7 providers vs. the "ease" of two frac-T3s to two providers.
From a provider's point of view, if a site wanted to connect, and was willing to sign a use-policy saying they wouldn't use the connection for transit to other providers (i.e. would only ask for customer BGP and only route to the nets you provide in BGP updates), would that site have lower costs associated with it? (that you could pass on?)
Thanks, Dean