Eg, I click on a web page to log in. The login process then kicks off a few authentication sessions with servers located halfway around the world. Then you do the data gathering, 2 phase locks, distributed file systems with the masters and lock servers all over the place. Your hellish user experience, let me SHOW YOU IT.
I know that this kind of hellish user experience exists today but in those situations, the users are not happy with it, and I don't think anybody would consider it to be a well-architected application. But let's use two concrete examples. One is a city newspaper hosted in a local ISP data center which interacts with a database at the newspaper's premises. The other is an Internet retailer hosted in the same local ISP data center which basically runs standalone except for sending shipping orders to a fulfillment center in another city, and the nightly site updates uploaded by the store owner after returning home from the dayjob. Now, move the retailer half a mile closer to the city center in distributed pod 26 of a distributed ISP data center, and move the newspaper to pod 11 which just happens to be on their own premises because they have outsourced their data center to the ISP who runs these distributed data center pods. For the life of me, I can't see any technical issue with this. Obviously, it doesn't suit the customers who can't fit into a single pod because they risk creating a scenario like you describe. I'm not foolish enough to suggest that one size fits all. All I am suggesting is that there is room for a new type of data center that mitigates the power and cooling issues by spreading out into multiple pod locations instead of clustering everything into one big blob.
Other than that minor handwaving, we are all good. Turns out that desining such distributed datacenters and setting up applications that you just handwaved away is a bit harder than it looks. I eagerly await papers on distributed database transactions with cost estimates for a distributed datacenter model vs. a traditional model.
For the big applications which cannot fit inside a single pod, I expect the Amazon EC2/S3 model to influence the way in which they approach decentralized scaling. And in that case, when these people figure out the details, then the distributed pod data center architecture can support this model just as easily as the big blob architecture. I'm not holding my breath waiting for papers because in the real world, people are going with what works. The scientific world has bought into the grid architecture, or the so-called supercomputer cluster model. Everyone else is looking to Google MapReduce/BigTable, Amazon EC2/S3, Yahoo Hadoop, XEN virtualization and related technologies.
See above. That right there doesn't quite solve most of the problems of traditional jobsets but its kind of hard to hear with the wind in my ears.
This is NANOG. Traditional jobsets here are web servers, blogs/wikis, Internet stores, Content-management sites like CNN, on-demand video, etc. The kind of things that our customers run out of the racks that they rent. Tell me again how on-demand video works better in one big data center?
I guess I can try to make it clearer by example: look at the cross-sectional bandwidth availability of a datacenter, now compare and contrast what it would take to pull it apart by a few tens of miles and conduct the cost comparison.
It would be pretty dumb to try and pull apart a big blob architecture and convert it into a distributed pod architecture. But it would be very clever to build some distributed pods and put new customers in there. --Michael Dillon