On Oct 10, 2011, at 7:46 PM, Nick Hilliard wrote:

if it's wifi that's causing the trouble, the usual causes are:

Don't forget RFI and various forms of spoofing used for MITM. ;> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> The basis of optimism is sheer terror. -- Oscar Wilde

...

...

if it's wifi that's causing the trouble, the usual causes are: is the complaint the hotel ROOM wireless? or the meeting-room?

meeting net, a-secure and a. really bad during the night, but still bouncing up until 08:30 when i turned laptop off to participate in breakfast. and conjecturbation as to what the problem was is amusing at best. i asked for actual diagnosis from whover is running the net. randy

Conjecturbation: They're using an old proteon router and someone is exhausting its arp and ndp caches.

i wanna picture!

On Oct 10, 2011, at 6:50 AM, Christopher Morrow wrote:

On Mon, Oct 10, 2011 at 9:01 AM, Dobbins, Roland <rdobbins@arbor.net> wrote:

...

On Oct 10, 2011, at 7:46 PM, Nick Hilliard wrote:

...

if it's wifi that's causing the trouble, the usual causes are:

is the complaint the hotel ROOM wireless? or the meeting-room? I noticed the nanog-a-secure bounce me 2x, so I moved back to ipsec-tunnel on nanog-a.. in the past nanog (plain) has been more 'stable' for me in general (and all you mac users can happily fight over -a!)

As to the hotel room wifi... apparently when you have 490 rooms in the hotel (full) and only provision your internal NAT space as a /23 ... things work 'fine' most days. When a networking conference comes to visit with 3+ devices requiring IP in each room... the whole hotel network stops :( Last night the display systems in the lobby and the hotel registration machines were all broken :( The hotel's network people (in NYC) are supposedly 'on a fix', who knows... (is expanding the nat subnet THAT hard?)

-chris

It would be wise for NANOG to approach future venues and specifically discuss these things with the hotel IT departments in question ahead of time so that they have some remote chance of being prepared. Owen

I would think that the contract with the hotel for the conference would include the specific requirements for the network. Is that not the case?

underlying problems o no hotel believe that we'll actually be significantly high use. they simply can not conceive of it. ietf, apricot, ... have seen this time and time again o the hotel does not manage the network, so you have two comms hops to anyone who can do anything. and anyway, they are not going to provision more bandwidth but the problems of which i spoke were the meeting network. which we do supposedly control. randy

On Mon, 10 Oct 2011 10:44:12 EDT, Randy Bush said:

o no hotel believe that we'll actually be significantly high use. they simply can not conceive of it. ietf, apricot, ... have seen this time and time again

To be fair, that's not a hotel-only problem. We've seen that problem within the IT industry. Actual discussion with a vendor who wanted to analyze our logs so they could size a solution: "Send us a day's worth of logs" "OK" ... "We said a *day's* worth, not a *week*". "That *was* a day" "Wow, that logfile was huge, we didn't think anybody actually did that much traffic a day..." The sad part was that the vendor in question *really* should have known better, we're pretty sure they targeted their solution at many sites bigger than us. Or maybe the other sites are bigger, but we pound the bejeebers out of stuff. I dunno.

In my historical knowledge of this: there are only so many venues that can have 500-650 people and fit. Jared Mauch On Oct 10, 2011, at 1:12 PM, "Frank Bulk" <frnkblk@iname.com> wrote:

Then the RFP for the meeting needs to be more specific with some basic SLAs that result in a smaller bill if not met.

Frank

-----Original Message----- From: Randy Bush [mailto:randy@psg.com] Sent: Monday, October 10, 2011 9:44 AM To: Randy Carpenter Cc: North American Network Operators' Group Subject: Re: meeting network

...

I would think that the contract with the hotel for the conference would include the specific requirements for the network. Is that not the case?

underlying problems

o no hotel believe that we'll actually be significantly high use. they simply can not conceive of it. ietf, apricot, ... have seen this time and time again

o the hotel does not manage the network, so you have two comms hops to anyone who can do anything. and anyway, they are not going to provision more bandwidth

but the problems of which i spoke were the meeting network. which we do supposedly control.

randy

On Mon, Oct 10, 2011 at 1:29 PM, Randy Carpenter <rcarpen@network1.net> wrote:

I have been at other conference that have triple or more participants, and it has never been anything close to the issues we are having at this hotel. Slightly slower performance is expected. Completely not working is not.

hotel or meeting ? (which network are we talking about now?)

+1 On Oct 10, 2011, at 10:29 AM, Randy Carpenter wrote:

I have been at other conference that have triple or more participants, and it has never been anything close to the issues we are having at this hotel. Slightly slower performance is expected. Completely not working is not.

-Randy

----- Original Message -----

...

In my historical knowledge of this: there are only so many venues that can have 500-650 people and fit.

Jared Mauch

On Oct 10, 2011, at 1:12 PM, "Frank Bulk" <frnkblk@iname.com> wrote:

...

Then the RFP for the meeting needs to be more specific with some basic SLAs that result in a smaller bill if not met.

Frank

-----Original Message----- From: Randy Bush [mailto:randy@psg.com] Sent: Monday, October 10, 2011 9:44 AM To: Randy Carpenter Cc: North American Network Operators' Group Subject: Re: meeting network

...

I would think that the contract with the hotel for the conference would include the specific requirements for the network. Is that not the case?

underlying problems

o no hotel believe that we'll actually be significantly high use. they simply can not conceive of it. ietf, apricot, ... have seen this time and time again

o the hotel does not manage the network, so you have two comms hops to anyone who can do anything. and anyway, they are not going to provision more bandwidth

but the problems of which i spoke were the meeting network. which we do supposedly control.

randy

On 10/10/11 10:20 AM, Randy Carpenter wrote:

I would think that the contract with the hotel for the conference would include the specific requirements for the network. Is that not the case?

My experiences planning and operating ARIN meeting networks taught me that it is difficult if not impossible to get a hotel make any changes to in room based wireless or wired networking. Often times they have contracts with third parties and don't actually have any control over how the network operates or issues such as capacity. This can also include an inability to disable access points or otherwise limit the ability of the contracted service provider to provide access to rooms. In room access is always something that we looked at when judging potential venues but the poor state of most in room Internet access infrastructure and the realities of existing business relationships made this a nice to have rather than a hard requirement.

"....Also note - the conference or hotel's IT people will always claim they have "done this before with no problems" even when they haven't....." Sounds like a true sales person :) - John Menerick ________________________________________ From: JC Dill [jcdill.lists@gmail.com] Sent: Monday, October 10, 2011 11:36 AM Cc: North American Network Operators' Group Subject: Re: meeting network On 10/10/11 7:00 AM, Owen DeLong wrote:

It would be wise for NANOG to approach future venues and specifically discuss these things with the hotel IT departments in question ahead of time so that they have some remote chance of being prepared.

I tried this approach many years ago, for a Blogher conference. The hotel's IT people were uncooperative, and incompetent, and they lied both about their network design and their equipment capabilities. I have since learned that this is par for the course. IMHO the only way to solve this problem is with big $$$ penalties in the contract, big enough that the incompetent IT people realize their jobs are on the line and relinquish control so experts can get access and set-up things properly. Also note - the conference or hotel's IT people will always claim they have "done this before with no problems" even when they haven't. jc NOTICE: This email and any attachments may contain confidential and proprietary information of NetSuite Inc. and is for the sole use of the intended recipient for the stated purpose. Any improper use or distribution is prohibited. If you are not the intended recipient, please notify the sender; do not review, copy or distribute; and promptly delete or destroy all transmitted information. Please note that all communications and information transmitted through this email system may be monitored by NetSuite or its agents and that all incoming email is automatically scanned by a third party spam and filtering service.

Years ago, on my own, when I used to attend, I used to call the venue about a month in advance and explain to them what was about to happen. Sort of a warning, per say. I explained, in detail, who NANOG was comprised of (I often would use the term "operators of the internet"). I explained even if they think they have seen this before, they haven't. Some listened. Some didn't. This would be something I would be willing to volunteer my time with - discussions with, and negotiations with, venues. It's all in the approach.

Holding the last 10% of the meeting room payment seems like a good start for any venue.

But as others have indicated, the market may be too small for free- market principles to be fully effective.

I tried this approach many years ago, for a Blogher conference. The hotel's IT people were uncooperative, and incompetent, and they lied both about their network design and their equipment capabilities. I have since learned that this is par for the course. IMHO the only way to solve this problem is with big $$$ penalties in the contract, big enough that the incompetent IT people realize their jobs are on the line and relinquish control so experts can get access and set-up things properly.

Also note - the conference or hotel's IT people will always claim they have "done this before with no problems" even when they haven't.

jc

On Mon, Oct 10, 2011 at 11:36 PM, Owen DeLong <owen@delong.com> wrote:

I don't think it is. I think that you can negotiate and I will point out that the hotel here has wanted our business enough that they have now scrambled to make life significantly better. You can also bet I'll be demanding that they credit my $54 that I put on the in-room access be credited to my bill even though ARIN would pay it.

I think the in-room wireless is actually supposed to already be creditted back at exit...

Just an FYI - even though you approved the wireless charge, it's actually free. They pull the per-diem/week charge off your bill. That applies to all NANOG attendees. Mike On Oct 10, 2011, at 11:36 PM, Owen DeLong wrote:

I don't think it is. I think that you can negotiate and I will point out that the hotel here has wanted our business enough that they have now scrambled to make life significantly better. You can also bet I'll be demanding that they credit my $54 that I put on the in-room access be credited to my bill even though ARIN would pay it.

I routinely do this when the conference network (or the in-room network) sucks and it's provided by the hotel. I have yet to have one refuse my refund request.

Owen

On Oct 10, 2011, at 3:41 PM, Frank Bulk wrote:

...

Holding the last 10% of the meeting room payment seems like a good start for any venue.

But as others have indicated, the market may be too small for free-market principles to be fully effective.

Frank

-----Original Message----- From: JC Dill [mailto:jcdill.lists@gmail.com] Sent: Monday, October 10, 2011 1:36 PM Cc: North American Network Operators' Group Subject: Re: meeting network

On 10/10/11 7:00 AM, Owen DeLong wrote:

...

It would be wise for NANOG to approach future venues and specifically discuss these things with the hotel IT departments in question ahead of time so that they have some remote chance of being prepared.

I tried this approach many years ago, for a Blogher conference. The hotel's IT people were uncooperative, and incompetent, and they lied both about their network design and their equipment capabilities. I have since learned that this is par for the course. IMHO the only way to solve this problem is with big $$$ penalties in the contract, big enough that the incompetent IT people realize their jobs are on the line and relinquish control so experts can get access and set-up things properly.

Also note - the conference or hotel's IT people will always claim they have "done this before with no problems" even when they haven't.

jc

-- Michael K. Smith - CISSP, GSEC, GISP Chief Technical Officer - Adhost Internet LLC mksmith@adhost.com w: +1 (206) 404-9500 f: +1 (206) 404-9050 PGP: B49A DDF5 8611 27F3 08B9 84BB E61E 38C0 (Key ID: 0x9A96777D)

The hotel will never refund at that level. The only thing that works is not to pay them in the first place. No hotel is that desperate enough to fill rooms that they're willing to return 50% of everything if the connectivity is poor or fails. They'll let their competitors have that business. Frank -----Original Message----- From: JC Dill [mailto:jcdill.lists@gmail.com] Sent: Tuesday, October 11, 2011 3:26 AM To: NANOG list Subject: Re: meeting network On 10/10/11 3:41 PM, Frank Bulk wrote:

Holding the last 10% of the meeting room payment seems like a good start for any venue. It's worthless. It's like being single-homed on a line with an SLA that refunds some small percent of your service provider fee for extended outages - fat lot of good that does you when your line Goes Down. The hotel's IT department will assure them (and you) that they have the situation covered, and then when it goes down you get a whole whopping 10% discount, but in the meantime you Have No Network.

To get their attention, to make sure they are really ready to provision the network capacity correctly (with adequate hardware, software, bandwidth, appropriate configs, etc.) the penalty needs to be something closer to "50% of all fees paid by the organization AND our attendees, for meeting rooms, food service, AND for lodging". Then when the network dies everyone gets 50% refunded. That will get the hotel management's attention and *possibly* help ensure that their IT department really DOES have the situation properly spec'd and provisioned to handle the traffic. jc

----- Original Message -----

From: "Owen DeLong" <owen@delong.com>

It would be wise for NANOG to approach future venues and specifically discuss these things with the hotel IT departments in question ahead of time so that they have some remote chance of being prepared.

It is nearly never "the hotel's IT department", in any sizable property; they've generally farmed it out to someone who Does That, and they often haven't even any Bog Red Switches to push when things break. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274

The hotel IT department is the guy who runs the as400 that gets reservations from corprate, and runs the POS terminals.

the room-net is by-in-large run by a third party such as lodgenet.

here at the lovely and reasonably priced loews, the dhcp disaster in the rooms killed the front desk randy

On Oct 11, 2011, at 8:22 AM, Owen DeLong wrote:

On Oct 10, 2011, at 10:32 PM, Joel jaeggli wrote:

...

On 10/10/11 07:00 , Owen DeLong wrote:

...

It would be wise for NANOG to approach future venues and specifically discuss these things with the hotel IT departments in question ahead of time so that they have some remote chance of being prepared.

The hotel IT department is the guy who runs the as400 that gets reservations from corprate, and runs the POS terminals.

the room-net is by-in-large run by a third party such as lodgenet.

In my experience, you start with the hotel IT department and they at least know who to talk to at LodgeNet/whoever in order to reach someone that can provide a useful response.

To be perfectly clear, the hotel IT department is a fine escalation point once you're close the actual event, and that they will bring in others as needed. This even works if you need to pull fiber into a facility for additional bandwidth, with the hotel IT/telecom team often getting involved months in advance. At the time of _contracting_ (more than 1 year in advance in many cases), the ability to pierce the sales veil of "Yes, we can do anything you need" and "It's no problem" can be quite difficult, even if one does an on-site visit and meets with the hotel IT team. They are trained to avoid raising any issues in the sales process, and prioritize any actual technical level engagement with their partners until well past contract. They often do not even have the ability to engage their partners except during an actual performance problem, so expecting them to get someone on the phone a year in advance of an event to commit to an unusual configuration may be quite limited (or even absent in the case of hotel chains whose wireless partner relationship is held by the hotel chain parent corporation.) I'm not saying that it is not worth trying; I just want folks to have a realistic understanding of how these arrangements are actually made. It is far better today then in the past, as there have been many conferences over the years where step 1 was pulling the coax or fiber through the hotel to establish their first-ever network infrastructure... :-) FYI, /John

Maybe instead of upgrading the network of cities, we could convince Google to practice by upgrading the networks of a variety of hotels in locations that NANOG might find appealing :) On Tue, Oct 11, 2011 at 10:48 AM, Nick Hilliard <nick@foobar.org> wrote:

On 11/10/2011 14:12, John Curran wrote:

...

is far better today then in the past, as there have been many conferences over the years where step 1 was pulling the coax or fiber through the hotel to establish their first-ever network infrastructure... :-)

There is nothing more dispiriting than "yeah sure, you can pull in that fibre cable, but only on condition that you remove it immediately after the [conference|meeting|whatever] is over. We already have internet Then they point at the 2Mb DSL wifi AP and expect you to be impressed at their technology.

Nick

Once upon a time, Nick Hilliard <nick@foobar.org> said:

There is nothing more dispiriting than "yeah sure, you can pull in that fibre cable, but only on condition that you remove it immediately after the [conference|meeting|whatever] is over. We already have the Internet".

I would say the situation depends on the hotel and which person you talk to. I volunteer for one of the largest science fiction conventions, and we take over 5 convention hotels for the con. I set up networking for our staff department's operations last year in one hotel, and initially we couldn't get anywhere because it was iBAHN and demanding an auth code on a captive web portal. When we got somebody from the hotel to look, he went into a closet around the corner and moved the wire, and we were then on the hotel's "direct" network. He then noticed I was running Linux, and we chatted about different distributions, and while I was setting up my (probably not allowed) wireless router, he showed back up with a box of cat5 and some ends (he was going to run some additional wires around the room for us, but saw I was running an AP and said "you're good, aren't you" and went on). We also have fiber pulled between the 5 hotels for our video feed, and that stays in place from year to year. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.

Problem for me at least has not been the MAC layer (either hotel room or meeting room), it was that the DHCP server was not responding. Ironically, I could still see everyone's Bonjour and SMB service advertisements. --Richard On Mon, Oct 10, 2011 at 8:46 AM, Nick Hilliard <nick@foobar.org> wrote:

On 10/10/2011 13:28, Randy Bush wrote:

...

perhaps as an educational exercise in network troubleshooting whoever is operating the meeting network could explain what the frack is wrong with the meeting network, how it is being debugged, and what they have learned about the cause of the suckage.

if it's wifi that's causing the trouble, the usual causes are:

- insufficient density of APs for the number of clients - APs configured with TX too high (should be set as low as possible) - APs configured to accept dot11b <= 9 megs - APs configured to use auto channel selection - stupid broken clients screaming at high volume across the room to APs which are impossibly far away

There is a more fundamental problem, though:  wifi was not designed with crazyass density in mind.

Bring back UTP?

Nick