Hi folks, just musing... From an ops perspective, wonder just how much traffic caused: "This morning, our engineers sounded the alarms ... and we have installed a digital version of a traffic cop. We enacted stopgaps that we planned for last night. We had hoped we didn't have to." --Jeff Ventura, communications director for the House's chief administrator. (from http://www.cnn.com/2008/POLITICS/09/30/congress.website/index.html) Don't .govs have enough b/w or at least ability to add b/w in order to satisfy their 'public outreach/information' role? (not a rhetorical question...hehe) It also seems to me that adding load balancing, firewall, throttling, etc methods for traffic shaping might actually make the problem worse by adding yet another layer(s) of hardware/software that may be prone to bottlenecking or overloading. whaddayathink? Ernie M. Rubi Network Engineer AMPATH/CIARA Florida International Univ, Miami
I'm surprised it isn't outsourced to some managed (hosting) provider, or a CDN.. Like Akamai or LLNW. It would surely be far more efficient for their purposes. Also, if you've planned your network correctly QoS/Shaping will not negatively effect your network. You always engineer your outer edge to take a beating. Sargun Dhillon 925.202.9485 deCarta sdhillon@decarta.com www.decarta.com -----Original Message----- From: Ernie Rubi [mailto:ernesto@cs.fiu.edu] Sent: Tue 9/30/2008 21:41 To: nanog@nanog.org Subject: 143.228.0.0/16 and house.gov Hi folks, just musing... From an ops perspective, wonder just how much traffic caused: "This morning, our engineers sounded the alarms ... and we have installed a digital version of a traffic cop. We enacted stopgaps that we planned for last night. We had hoped we didn't have to." --Jeff Ventura, communications director for the House's chief administrator. (from http://www.cnn.com/2008/POLITICS/09/30/congress.website/index.html) Don't .govs have enough b/w or at least ability to add b/w in order to satisfy their 'public outreach/information' role? (not a rhetorical question...hehe) It also seems to me that adding load balancing, firewall, throttling, etc methods for traffic shaping might actually make the problem worse by adding yet another layer(s) of hardware/software that may be prone to bottlenecking or overloading. whaddayathink? Ernie M. Rubi Network Engineer AMPATH/CIARA Florida International Univ, Miami
Ernie Rubi wrote:
From an ops perspective, wonder just how much traffic caused:
"This morning, our engineers sounded the alarms ...
More of a case of a worldwide press conference broadcasted live by news networms around the world when Nancy Pelosi stated that "as of now, the recovery plan is available for everyone to read at the following web address". The web site was immediatly overloaded and remained so for hours. Normally, this web site would have received just a few visits from the public at large every day. All of a sudden, millions of people tried to get to it at the same time.
Some political action groups probably decided to step up the astroturfing. You know, enter your email address here and we'll send out some boilerplate nonsense to a bunch of congressmen and senators. Block or firewall the worst of them, whether left or right leaning, and I guess that should leave the servers clear for real users ... --srs On Wed, Oct 1, 2008 at 10:11 AM, Ernie Rubi <ernesto@cs.fiu.edu> wrote:
Hi folks, just musing...
From an ops perspective, wonder just how much traffic caused:
"This morning, our engineers sounded the alarms ... and we have installed a digital version of a traffic cop. We enacted stopgaps that we planned for last night. We had hoped we didn't have to." --Jeff Ventura, communications director for the House's chief administrator. (from http://www.cnn.com/2008/POLITICS/09/30/congress.website/index.html)
Don't .govs have enough b/w or at least ability to add b/w in order to satisfy their 'public outreach/information' role? (not a rhetorical question...hehe)
It also seems to me that adding load balancing, firewall, throttling, etc methods for traffic shaping might actually make the problem worse by adding yet another layer(s) of hardware/software that may be prone to bottlenecking or overloading.
whaddayathink?
Ernie M. Rubi Network Engineer AMPATH/CIARA Florida International Univ, Miami
-- Suresh Ramasubramanian (ops.lists@gmail.com)
On Wed, Oct 01, 2008 at 12:41:12AM -0400, Ernie Rubi wrote:
Hi folks, just musing...
From an ops perspective, wonder just how much traffic caused:
"This morning, our engineers sounded the alarms ... and we have installed a digital version of a traffic cop. We enacted stopgaps that we planned for last night. We had hoped we didn't have to." --Jeff Ventura, communications director for the House's chief administrator. (from http://www.cnn.com/2008/POLITICS/09/30/congress.website/index.html)
Don't .govs have enough b/w or at least ability to add b/w in order to satisfy their 'public outreach/information' role? (not a rhetorical question...hehe)
What makes you thing that .gov's "have" anything at all? They have to buy any bandwidth they have (other than strictly internal bandwidth) from ISP's. If the IT budget doesn't allow for it, the IT department can't buy it. If the projected need is much lower than this surge, then they would not have budgeted for it. The USGOV, contrary to some folks' belief, does not own the Internet. Some ISP's are able to quickly add bandwidth if the line is set up for it, but I think the IT department would have had to have an existing active relationship with the ISP to be able to know whom to ask. -- Joe Yao Qinetiq NA / Analex Contractor
Are you saying that the house.gov site is not in a large data center with direct fiber connectivity along with many of the other large federal web sites (with alternative hot sites ready to go at a moment's notice, of course)? As someone who has been to different government data centers, I can tell you they have huge amounts of data connectivity there in case of emergency. For a large site like house.gov, bandwidth should never be an issue. In this case I highly doubt it was the issue, but instead overloading of the hardware in place. Just my $.02... Mick -----Original Message----- From: Joseph S D Yao [mailto:jsdy@center.osis.gov] Sent: Thursday, October 02, 2008 11:54 AM To: Ernie Rubi Cc: nanog@nanog.org Subject: Re: 143.228.0.0/16 and house.gov What makes you thing that .gov's "have" anything at all? They have to buy any bandwidth they have (other than strictly internal bandwidth) from ISP's. If the IT budget doesn't allow for it, the IT department can't buy it. If the projected need is much lower than this surge, then they would not have budgeted for it. The USGOV, contrary to some folks' belief, does not own the Internet. Some ISP's are able to quickly add bandwidth if the line is set up for it, but I think the IT department would have had to have an existing active relationship with the ISP to be able to know whom to ask. -- Joe Yao Qinetiq NA / Analex Contractor
Is this really technical discussion of operation of networks? I connected the internal network of the US House of Representatives to the Internet when I worked there, and operated it through both Democratic and Republican control. I never saw any snooping by either party of the network traffic, and I had sniffers for diagnosing problems in several communication closets. I do recall unfounded accusations both ways, but it would be sad for the rumors to outlive the reality. The notorious case of intercepted cell-phone conversations had nothing to do with the data network. Not only is the data center, but so are all the committee and member offices that want it connected. Skilled professionals operate the House's network. There has been a collegial relationship among the operators of both the Senate and House networks, as well as the rest of the Legislative branch. There are good reasons, including Constitutional separation of powers, that the Legislative Branch is not managed by the Executive Branch. The independence of the two houses of Congress is more a matter of tradition, and the fact that a different party sometimes controls the other house. Bandwidth has ALWAYS been an issue because Internet access is acquired through normal business processes, and the appetite for bandwidth both to Congressional staff, and (occasionally - when something important happens) to the public. Since the source of money for these operations is Federal taxes, many readers of this list might appreciate that we have not bought more than we could justify. I will not say anything about how large or redundant the data center is for obvious reasons, beyond that I am no longer employed there and do not have the details. I really think this thread has outlived its entertainment value. John On 2008Oct2, at 12:39 PM, Mick Bergman wrote:
Are you saying that the house.gov site is not in a large data center with direct fiber connectivity along with many of the other large federal web sites (with alternative hot sites ready to go at a moment's notice, of course)? As someone who has been to different government data centers, I can tell you they have huge amounts of data connectivity there in case of emergency.
For a large site like house.gov, bandwidth should never be an issue. In this case I highly doubt it was the issue, but instead overloading of the hardware in place.
Just my $.02...
John Schnizlein wrote:
I connected the internal network of the US House of Representatives to the Internet when I worked there, and operated it through both Democratic and Republican control.
Aha, I wondered who was to blame.... Of course, my Member was on the Internet before the House, as MERIT -- the very same organization that ran/runs NANOG -- had its own POP (called an SCP in those days) in DC. Only later did we use the House net. She usually took her Mac laptop to Science and Education committee meetings. Her staff was often asked how they got her to use her own laptop, when they couldn't get their own members to read (or type) their own email. This was all pre-2001, and Blackberry mania.
I never saw any snooping by either party of the network traffic, and I had sniffers for diagnosing problems in several communication closets.
Yet, there was verified interception of both House and Senate email communications. Nobody claimed it was "on the wire" network traffic, as there were many weaknesses in the data network security design. And the vicious fight about our setting up a VPN to bypass the centrally controlled system -- as in "if you do this, we'll cut off your network access entirely" -- led all concerned to guess that there was a political reason, not a technical reason. So, I just used non-standard ports, and some other firewalling, to prevent your staff from detecting it. Also, there was the long fight about members running their own servers (as in member.house.gov), instead of relying on the central servers for connectivity (www.house.gov/member). Again, we really didn't trust the Republicans not to examine internal data.
I do recall unfounded accusations both ways, but it would be sad for the rumors to outlive the reality.
Like this verified and widely reported: "Democrats Suggest Inquiry Points to Wider Spying by G.O.P." http://query.nytimes.com/gst/fullpage.html?res=940DE4D7173AF933A25751C0A9629C8B63&sec=&spon=&pagewanted=print
The notorious case of intercepted cell-phone conversations had nothing to do with the data network.
True, but irrelevant.
I will not say anything about how large or redundant the data center is for obvious reasons, beyond that I am no longer employed there and do not have the details.
I've not even visited DC since 2002, and the old building with the page dorm was torn down that summer. But I can dig and traceroute. I'm pretty sure this isn't an ideal (or standard conforming) setup. But it shouldn't have been swamped, as seems to be akamaized. === ;; QUESTION SECTION: ;financialservices.house.gov. IN A ;; ANSWER SECTION: financialservices.house.gov. 3600 IN CNAME www.house.gov. www.house.gov. 3503 IN CNAME house.gov.edgesuite.net. house.gov.edgesuite.net. 4372 IN CNAME a1164.g.akamai.net. a1164.g.akamai.net. 20 IN A 192.122.184.19 a1164.g.akamai.net. 20 IN A 192.122.184.7 === house.gov. 900 IN SOA mercury.house.gov. dnsadmin.mail.house.gov. 1002529 3600 1800 604800 3600 house.gov. 14128 IN NS chyron.house.gov. house.gov. 14128 IN NS mercury.house.gov. mercury.house.gov. 14166 IN A 143.231.1.67 chyron.house.gov. 14149 IN A 143.228.129.38
This will be my last response on this despite whatever spin follows. On 2008Oct2, at 4:08 PM, William Allen Simpson wrote:
John Schnizlein wrote:
I connected the internal network of the US House of Representatives to the Internet when I worked there, and operated it through both Democratic and Republican control.
Aha, I wondered who was to blame....
Thank you for the compliment.
...
I never saw any snooping by either party of the network traffic, and I had sniffers for diagnosing problems in several communication closets.
Yet, there was verified interception of both House and Senate email communications. Nobody claimed it was "on the wire" network traffic, as there were many weaknesses in the data network security design.
If you know any, please send them to me privately. I can assure the community that our design and implementation got repeated review and testing from the best we could find at the time.
And the vicious fight about our setting up a VPN to bypass the centrally controlled system -- as in "if you do this, we'll cut off your network access entirely" -- led all concerned to guess that there was a political reason, not a technical reason. So, I just used non-standard ports, and some other firewalling, to prevent your staff from detecting it.
I hope no damage was produced by any inadvertent back doors opened by your VPN. Since we were not blocking applications other than IRC, I don't know what you felt you needed to get around.
Also, there was the long fight about members running their own servers (as in member.house.gov), instead of relying on the central servers for connectivity (www.house.gov/member). Again, we really didn't trust the Republicans not to examine internal data.
Although I do not recall the particular offices, I do recall that several committees and members had both email and web servers in their own offices with domains delegated to them on request. I have no idea what "long fight" you might have experienced.
I do recall unfounded accusations both ways, but it would be sad for the rumors to outlive the reality.
Like this verified and widely reported:
"Democrats Suggest Inquiry Points to Wider Spying by G.O.P." http://query.nytimes.com/gst/fullpage.html?res=940DE4D7173AF933A25751C0A9629C8B63&sec=&spon=&pagewanted=print
As I recall this was simply a case of one staffer logging into a server in a different office. As you mentioned above, not "on the wire" and not a data network security issue. As sometimes still happens, the "computer network" actually referred to a file server. This article is about activities in the Senate, which operates independently of the House - was your experience actually with respect to the Senate? John
William Allen Simpson wrote:
But I can dig and traceroute. I'm pretty sure this isn't an ideal (or standard conforming) setup. But it shouldn't have been swamped, as seems to be akamaized.
I don't have traceroutes kept, but during that night when Pelosi announced the bill was available for all to download, I tried to get to that page and it was extremely slow. Doing a traceroute didn't *seem* to end at an akamai point. My memory could be in error. Question: Is it possible to setup an akamai feed in hours once you know your website is to be swamped ? Obviously, the system managers there might not have been warned in advance that the politicians would place a huge load on their servers. But once they realised it, is it conceivable that they quickly setup an akamai feed ? Or is that something which takes weeks to setup ?
On 10/2/08, Jean-François Mezei <jfmezei@vaxination.ca> wrote:
<snip>
Question:
Is it possible to setup an akamai feed in hours once you know your website is to be swamped ?
Obviously, the system managers there might not have been warned in advance that the politicians would place a huge load on their servers. But once they realised it, is it conceivable that they quickly setup an akamai feed ? Or is that something which takes weeks to setup ?
I'm not sure about Akamai, but I believe Amazon is about to roll out CDN services as well (and I would assume they're as flexible as their other "cloud" offerings). As always, hindsight is 20/20. http://www.amazon.com/gp/html-forms-controller/aws-content-delivery-service -brandon
Pretty much no matter who you use, this can easily be done in an hour or so if people really want it to and the right techs are available. If there's a pre-existing agreement, this can go to mere minutes. The setup doesn't take long. it's usually the business stuff that drags it out. On Thu, Oct 02, 2008 at 04:20:01PM -0500, Brandon Galbraith wrote:
On 10/2/08, Jean-Fran??ois Mezei <jfmezei@vaxination.ca> wrote:
<snip>
Question:
Is it possible to setup an akamai feed in hours once you know your website is to be swamped ?
Obviously, the system managers there might not have been warned in advance that the politicians would place a huge load on their servers. But once they realised it, is it conceivable that they quickly setup an akamai feed ? Or is that something which takes weeks to setup ?
I'm not sure about Akamai, but I believe Amazon is about to roll out CDN services as well (and I would assume they're as flexible as their other "cloud" offerings). As always, hindsight is 20/20.
http://www.amazon.com/gp/html-forms-controller/aws-content-delivery-service
-brandon
--- Wayne Bouchard web@typo.org Network Dude http://www.typo.org/~web/
participants (11)
-
Brandon Galbraith -
Ernie Rubi -
Jean-François Mezei -
John Schnizlein -
Joseph S D Yao -
Laurence F. Sheldon, Jr. -
Mick Bergman -
Sargun Dhillon -
Suresh Ramasubramanian -
Wayne E. Bouchard -
William Allen Simpson