It would have been more interesting to see: -- a network weather map -- the ELK implementation -- actual cache statistics (historically steam/game downloads are not cahce'able) Thanks for the share though Sean! On Sat, Aug 1, 2015 at 9:16 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:

highlights: "happy and blinking" "two firewalls for the two att 1gig links, and two spare doing ....."

catalyst 6500's

Also the 3750 on top of the services rack is funny... because empty.

On Sat, Aug 1, 2015 at 3:27 PM, Sean Donelan <sean@donelan.com> wrote:

...

Non-work, work related information. Many NANOG geeks might be interested in this video tour of the Quakecon NOC tour. As any ISP operator knows, gamers complain faster about problems than any NMS, so you've got to admire the bravery of any NOC in the middle of a gaming convention floor.

What Powers Quakecon | Network Operations Center Tour https://www.youtube.com/watch?v=mOv62lBdlXU

-- Miano, Steven M. http://stevenmiano.com

Also, 2 Gbps for 4,400 people? Pretty lackluster compared to European events. 30C3 had 100 Gbps to the conference building. And no NAT: every host got real IP addresses (IPv4 + IPv6).

ietf, >1k people, easily fits in 10g, but tries to have two for redundancy. also no nat, no firewall, and even ipv6. but absorbing or combatting scans and other attacks cause complexity one would prefer to avoid. in praha, there was even a tkip attack, or so it is believed; turned off tkip. the quakecon net was explained very poorly. what in particular provides game-quality latency, or lack thereof? with only 2g, i guess i can understand the cache. decent bandwidth would reduce complexity. and the network is flat? randy

On Sun, Aug 2, 2015 at 7:56 AM, Niels Bakker <niels=nanog@bakker.net> wrote:

I guess a tale of punching 300-odd patchpanels is not that captivating to everybody out there.

I find this hard to believe. :) I was hoping for more 'how the network is built' (flat? segmented? any security protections so competitors can't kill off their competition?) and ideally some discussion of why the decisions made a difference. (what tradeoffs were made and why?)

...

any security protections so competitors can't kill off their competition?)

It would be interesting to learn whether they saw any DDoS attacks or cheating attempts during competitive play, or even casual non-competitive play amongst attendees.

I wonder if that would be a reason for the relatively anemic 1Gb Internet pipe-- making sure that a DDoS couldn't push enough packets through to inconvenience the LAN party. (Disclaimer: $DAYJOB did the audio/visual/lighting for QuakeCon but we had nothing to do with the network and I was utterly uninvolved in any way, so my speculation is based on no information obtained from outside my own skull.) -- Dave Pooser Cat-Herder-in-Chief, Pooserville.com

While increasing bandwidth to the endpoint isn't viable wouldn't increasing the edge bandwidth out to the ISP be a start in the right direction? I would assume this would a start to the problem if your attacks were volumetric. Once the bandwidth is there you can look at mitigation before it reaches the endpoint, in this case the computers on the floor (assuming no NAT). On 2 Aug 2015 16:51, "Roland Dobbins" <rdobbins@arbor.net> wrote:

On 2 Aug 2015, at 22:44, Dave Pooser wrote:

I wonder if that would be a reason for the relatively anemic 1Gb Internet

...

pipe-- making sure that a DDoS couldn't push enough packets through to inconvenience the LAN party.

While increasing bandwidth is not a viable DDoS defense tactic, decreasing it isn't one, either.

----------------------------------- Roland Dobbins <rdobbins@arbor.net>

It's completely reasonable when the world at large is only secondary to the local, on-net operations. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Roland Dobbins" <rdobbins@arbor.net> To: "nanog list" <nanog@nanog.org> Sent: Sunday, August 2, 2015 10:50:05 AM Subject: Re: Quakecon: Network Operations Center tour On 2 Aug 2015, at 22:44, Dave Pooser wrote:

I wonder if that would be a reason for the relatively anemic 1Gb Internet

pipe-- making sure that a DDoS couldn't push enough packets through to inconvenience the LAN party.

While increasing bandwidth is not a viable DDoS defense tactic, decreasing it isn't one, either. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>

It most certainly does. If the core of the mission is local LAN play and your Internet connection fills up.... who gives a shit? The games play on. If your 500 megabit corporate connection gets a 20 terabit DDoS, your RDP session to the finance department will continue to hum along just fine. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com ----- Original Message ----- From: "Roland Dobbins" <rdobbins@arbor.net> To: "nanog list" <nanog@nanog.org> Sent: Sunday, August 2, 2015 11:23:18 AM Subject: Re: Quakecon: Network Operations Center tour On 2 Aug 2015, at 22:56, Mike Hammett wrote:

It's completely reasonable when the world at large is only secondary to the local, on-net operations.

It has nothing to do with DDoS. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>

Not that often you see a bunch of people talking about a video you're in, especially so on NANOG. So here goes. BYOC is around 2700 seats. Total attendance was around 11,000. 2Gbps has been saturated at some point every year we have had it. Additional bandwidth is definitely a serious consideration going forward. It is a lot better than the 45mbps or less we dealt with 2010 and prior, but better doesn't mean good enough. Many games these days do depend upon online services, and forced us to look for options. AT&T has been sponsoring since then and we do appreciate it. We have had the potential for DDoS attacks on our minds. Our first option in those cases is blackhole announcements to the carrier for the targeted /32. AT&T did provide address space for us to use so the BYOC was using public IPs, and hopefully the impact of blackholing a single IP could be made minimal. Thankfully we have not yet been targeted, and we can only keep hoping it stays that way. We haven't tackled IPv6 yet since it adds complexity that our primary focus doesn't significantly benefit from yet since most games just don't support it. Our current table switches don't have an RA guard, and will probably require replacement to get ones that are capable. We also re-designed the LAN back in 2011 to break up the giant single broadcast domain down to a subnet per table switch. This has definitely gotten us some flack from the BYOC since it breaks their LAN browsers, but we thought a stable network was more important with how much games have become dependent on stable Internet connectivity. Still trying to find a good way to provide a middle ground for attendees on that one, but I'm sure everyone here would understand how insane a single broadcast domain with 2000+ hosts that aren't under your control is. We have tried to focus on latency on the LAN, however when so many games are no longer LAN oriented Internet connectivity became a dominant issue. Some traffic is routed out a separate lower capacity connection to keep saturation issues from impacting it during the event. Squid and nginx do help with caching, and thankfully Steam migrated to a http distribution method and allows for easy caching. Some other services make it more difficult, but we try our best. Before Steam changed to http distribution there were a few years they helped in providing a local mirror but that seems to have been discontinued with the migration to http. The cache pushed a little over 4Gbps of traffic at peak at the event. The core IT team which handles the network (L2 and above) is about 9 volunteers. The physical infrastructure is our IP & D team, which gets a huge team of volunteers put together in order to get that 13 miles of cable ready between Monday and Wednesday. The event is very volunteer driven, like many LAN parties across the planet. We try to reuse cable from year to year, including loading up the table runs onto a pallet to be used in making new cables out of in future years. I imagine I haven't answered everyone's questions, but hopefully that fills in some of the blanks. If this has anyone considering sponsorship interest in the event the contact email is sponsors(at)quakecon.org. Information is also available on the website http://www.quakecon.org/.

On 02/08/2015 22:59, Randy Bush wrote:

so it is heavily routed using L3 on the core 'switches'? makes a lot of sense.

Lots of switches will happily forward layer 3 packets. Nick

On 02/08/2015 23:30, Randy Bush wrote:

otoh, i did not believe in the fad of using 65xxs at the bgp global edge. while it was temporarily cheap, two years later not a lot of folk had that many boats which needed anchoring.

A juniper EX9200 is a switch and a cisco sup2t box is a router. The vendor said it so it must be true. As anchors, I would be hard put to make a choice between a 6500 and a 7500, which was a fine router in its day but alas only had a useful lifetime of a small number of years. Obsolescence happens. The distinction between layer 2 and layer 3 capable kit is not that important these days. What's important is whether the device's packet or frame forwarding capabilities are a good match for the expected workload and that the total operating cost over the depreciation period works. Nick

On Sun, Aug 2, 2015 at 9:46 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:

On Sun, Aug 2, 2015 at 6:57 PM, Nick Hilliard <nick@foobar.org> wrote:

...

As anchors, I would be hard put to make a choice between a 6500 and a 7500, which was a fine router in its day but alas only had a useful lifetime of a small number of years. Obsolescence happens.

isn't some of L3's edge still 7500's? I think some of 703/702's edges are still 7500's even.

"Last Date of Support: HW The last date to receive service and support for the product. After this date, all support services for the product are unavailable, and the product becomes obsolete. December 31, 2012" oh .. maybe they really are all gone :)

On Sun, Aug 2, 2015 at 4:59 PM, Randy Bush <randy@psg.com> wrote:

josh,

thanks for the more technical scoop. now i get it a bit better.

...

We also re-designed the LAN back in 2011 to break up the giant single broadcast domain down to a subnet per table switch.

so it is heavily routed using L3 on the core 'switches'? makes a lot of sense.

Single core switch, the Cisco 6509 VE in the video, handles routing between subnets. Table switches have an IP for management and monitoring. We have some 3750Gs for additional routing in other parts of the event.

I help with an event that has a pretty decent sized lan party as well. We're not just focused on the lan party, more of a rock concerts - huge arcade - panels - lan party type event. It was a few years ago that a mincraft "griefing" team came and attacked the network internally. At the time the BYOC LAN party I think was using 3com switches on the edge. Griefers were doing MAC flooding or something that was causing the switches to fall over. And not just the switch they were connected to it was bringing down many of them. They were doing it in spurts and the people dealing with the network thought the issue was misbehaving equipment for a bit (it seemed foreign at that time that someone from the community would be doing it.) Mind you the people running things (volunteers) are running on little sleep, had no time to build out security appliances let alone watch a bunch of logs. They're pretty smart but you know - you get a bunch of smart people together they all bicker about how to do things their way. In the end, one of the griefers friends went and told on them, and that's how they were discovered. Badges yanked and banned for life. Most of these cons and events run on surplus hardware. Granted, these days there is more and more higher end stuff being cast away. More and more 10 gig, Juniper, Force10 and other decent equipment coming into play. Getting bandwidth into the events is a pain. Huge venues are meant for large corporate events not lower budget cons and festivals. Venue pricing I believe is 750-1500$ per megabit. 100 megabit = $75,000 for the weekend. One year I rememeber there being a switch with 8 vlans on it sitting outside the back door with 8 clear modems spread out all blinking away. Geeks get creative. These days, a random family next door gets their business class FiOS paid for the entire year (with a good TV package) in return for a weekend or two a year of it being slammed. But that isn't keeping up with demand. I think sponsorship is in our future as far as bandwidth goes. Internally, the hotels charge for any ports. So if you need cross connects between rooms, it's pretty expensive. And it's managed by them so running tagged traffic is a no go an other things. So out comes miles of fiber and rolls of gaffers tape every year. And miles of cat5. The lan party is fairly concentrated, but other departments all have other network needs. HD video streams outbound, voip telephones, ARTNet, etc. It's crazy. But I guess it's a good way to keep skills sharp and learn new things. Also, Steam and others should make a caching server solution similar to what exists in Apple OSX server. - Ethan

Venue Internet is the bane of events. Crazy expensive. Almost as expensive as a laborer in Chicago to move your box from the truck to your booth. ;-) ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Ethan" <telmnstr@757.org> To: nanog@nanog.org Sent: Monday, August 3, 2015 9:58:35 AM Subject: Re: Quakecon: Network Operations Center tour I help with an event that has a pretty decent sized lan party as well. We're not just focused on the lan party, more of a rock concerts - huge arcade - panels - lan party type event. It was a few years ago that a mincraft "griefing" team came and attacked the network internally. At the time the BYOC LAN party I think was using 3com switches on the edge. Griefers were doing MAC flooding or something that was causing the switches to fall over. And not just the switch they were connected to it was bringing down many of them. They were doing it in spurts and the people dealing with the network thought the issue was misbehaving equipment for a bit (it seemed foreign at that time that someone from the community would be doing it.) Mind you the people running things (volunteers) are running on little sleep, had no time to build out security appliances let alone watch a bunch of logs. They're pretty smart but you know - you get a bunch of smart people together they all bicker about how to do things their way. In the end, one of the griefers friends went and told on them, and that's how they were discovered. Badges yanked and banned for life. Most of these cons and events run on surplus hardware. Granted, these days there is more and more higher end stuff being cast away. More and more 10 gig, Juniper, Force10 and other decent equipment coming into play. Getting bandwidth into the events is a pain. Huge venues are meant for large corporate events not lower budget cons and festivals. Venue pricing I believe is 750-1500$ per megabit. 100 megabit = $75,000 for the weekend. One year I rememeber there being a switch with 8 vlans on it sitting outside the back door with 8 clear modems spread out all blinking away. Geeks get creative. These days, a random family next door gets their business class FiOS paid for the entire year (with a good TV package) in return for a weekend or two a year of it being slammed. But that isn't keeping up with demand. I think sponsorship is in our future as far as bandwidth goes. Internally, the hotels charge for any ports. So if you need cross connects between rooms, it's pretty expensive. And it's managed by them so running tagged traffic is a no go an other things. So out comes miles of fiber and rolls of gaffers tape every year. And miles of cat5. The lan party is fairly concentrated, but other departments all have other network needs. HD video streams outbound, voip telephones, ARTNet, etc. It's crazy. But I guess it's a good way to keep skills sharp and learn new things. Also, Steam and others should make a caching server solution similar to what exists in Apple OSX server. - Ethan

On Mon, Aug 03, 2015 at 01:52:17PM -0700, alvin nanog wrote:

hi ethan

On 08/03/15 at 10:58am, Ethan wrote:

...

Getting bandwidth into the events is a pain. Huge venues are meant for large corporate events not lower budget cons and festivals. Venue pricing I believe is 750-1500$ per megabit. 100 megabit = $75,000 for the weekend. One year I rememeber there being a switch with 8 vlans on it sitting outside the back door with 8 clear modems spread out all blinking away.

for connectivity, does the hotels and convention centers still have wifi jammers so you cannot use your own 56Mbit wifi to get connection to the outside world ? if possible, stick a bunch of dark mirrored-glass covered vans outside the event for wifi access

In the US, the FCC has ruled that wifi jammers violate one or more parts of the FCC Rules and Regs. Marriott hotels paid a USD600K fine. A quick Google search on "FCC hotel jammer" pulls up a great many hits, of which these are the first seven: Jammer Enforcement | FCC.gov https://www.fcc.gov/.../jamme... U.S. Federal Communications Commission Federal law prohibits the operation, marketing, or sale of any type of jamming equipment, including devices that interfere with cellular and Personal ... Marriott to Pay $600K to Resolve WiFi-Blocking ... - FCC https://www.fcc.gov/.../marrio... U.S. Federal Communications Commission Oct 3, 2014 - Hotel Operator Admits Employees Improperly Used Wi-Fi Monitoring ... The complainant alleged that the Gaylord Opryland was “jamming ... WARNING: Wi-Fi Blocking is Prohibited | FCC.gov https://www.fcc.gov/.../warnin... U.S. Federal Communications Commission Jan 27, 2015 - which hotels and other commercial establishments block wireless ... into this kind of unlawful activity by the operator of a resort hotel and ... FCC warns hotels against blocking guests' wi-fi www.consumeraffairs.com/.../fcc-warns-hotels-against-blocking-guests-... Jan 28, 2015 - Hotels, miffed by guests who used their own wi-fi hotspots instead of paying ... It's illegal to jam legal radio transmissions of any kind, FCC vows tough enforcement ... Some had argued that jamming wi-fi and cellphone calls is ... Hotels ask FCC for permission to block guests' personal Wi ... www.pcworld.com/.../hotel-group-asks-fcc-for-permission-to-... PC World Dec 22, 2014 - Marriott argued some hotspot blocking may be justified, as long as the hotel isn't using illegal signal jammers. Unlicensed Wi-Fi hotspots ... FCC fines Marriott $600,000 for blocking guests' Wi-Fi ... www.cnn.com/2014/10/03/travel/marriott-fcc-wi-fi-fine/ CNN Oct 4, 2014 - It's the first time the FCC has investigated a hotel property for ... sense, where someone uses a jammer device to block wireless signals. Instead ... How This Hotel Made Sure Your Wi-Fi Hotspot Sucked ... readwrite.com/2014/.../marriott-nashville-opryland-jams-wifi-internet-wt... Oct 4, 2014 - Caught by FCC for Wi-Fi jamming, Marriott's still not sorry. -- Mike Andrews, W5EGO mikea@mikea.ath.cx Tired old sysadmin

The WiFi jammers have an interesting MO. They don't throw up static on the frequency, that would also block their own wifi. They spoof de-authentication packets. I've been looking for a way to detect this kind of jamming because my WiFi sucks and I live next to three hotels, what you get for living in downtown Atlanta. On Mon, Aug 3, 2015 at 5:09 PM, Roland Dobbins <rdobbins@arbor.net> wrote:

On 4 Aug 2015, at 4:03, mikea wrote:

In the US, the FCC has ruled that wifi jammers violate one or more parts

...

of the FCC Rules and Regs.

I travel quite a bit worldwide, and I've never run into this. I run my portable AP on 5GHz, FWIW.

----------------------------------- Roland Dobbins <rdobbins@arbor.net>

hi mr bugs :-) On 08/03/15 at 05:38pm, Mr Bugs wrote:

The WiFi jammers have an interesting MO. They don't throw up static on the frequency, that would also block their own wifi. They spoof de-authentication packets. I've been looking for a way to detect this kind of jamming because my WiFi sucks and I live next to three hotels, what you get for living in downtown Atlanta.

i forgot if kismet showed signal strengths of the wifi ap's ... "stronger" signal wins over weaker signal strengths might not be a jamming issue ?? kismet and tcpdump might be able to show you the packets you're looking for ? what happens if you put up a properly designed wire mess around the exterior windows of your house/condo/aptr?? i'd wag/blindly say the area is probably full of rogue wifi ap's floating around where evergbody is trying to wardrive each other and pick up un-suspecting traveling visitor's login and passwd info ... signals bouncing off steel/concrete is not ez to filter out what should be random background white noise if you're sitting next to the radiating source .. pixie dust alvin # DDoS-Mitigator.net # DDoS-Simulator.net

On Sun, 2 Aug 2015, Dave Pooser wrote:

I wonder if that would be a reason for the relatively anemic 1Gb Internet pipe-- making sure that a DDoS couldn't push enough packets through to inconvenience the LAN party.

I was involved in delivering 1GigE to Dreamhack in 2001 which at the time (if I remember correctly), 4500 computers that participants brought with them. Usually these events nowadays tend to use 5-20 gigabit/s for that amount of people, so 2x1GE is just not enough. Already in 2001 that GigE was fully loaded after 1-2 days. -- Mikael Abrahamsson email: swmike@swm.pp.se

On Sun, 2 Aug 2015, Niels Bakker wrote:

Also, 2 Gbps for 4,400 people? Pretty lackluster compared to European events. 30C3 had 100 Gbps to the conference building. And no NAT: every host got real IP addresses (IPv4 + IPv6).

Quakecon is essentially a giant LAN party. Bring Your Own Computer (BYOC), and there are big gaming rigs at Quakecon, and compete on the LAN. There isn't that much Internet traffic. There is only 100Mbps wired to each gaming station. I'm not a quake fanatic, I don't know what are the important network metrics for a good gaming experience. But I assume the important metrics are local, and they install a big central server complex in the center of the room. I'm assuming the critical lag is between the central server and the competitors; not the Internet. Otherwise they could have all stayed home and played in their basements across the Internet. Latency is probably more important than bulk bandwidth.