On Sun, 26 Feb 2012, Randy Carpenter wrote:

I don't need that kind of HA, and understand that it is not going to be available. 15 minutes of downtime is fine. 6 hours is completely unacceptable, and it false advertising to say you have a "Cloud" service, and then have the realization that you could have *indefinite* downtime.

Um. You and I apparently work in different clouds. In my world, the SLAs I have agreed to state, roughly, that uptime is not guaranteed, nor is data recoverability. They suggest that that sort of thing is -my- problem to engineer and architect around. I don't use Rackspace's cloud solution - but I haven't seen anything to suggest that they advertise their service any differently. The "cloud" provides flexibility and rapid deployment at the expense of hands-on control and reliability (and SLAs). Perhaps you forgot to read the SLA? Or you can show us where someone defines "Cloud" as "highly available" and "without indefinite downtime" ? -- david raistrick http://www.netmeister.org/news/learn2quote.html drais@icantclick.org http://www.expita.com/nomime.html

-----Original Message----- From: Tony Patti [mailto:tony@swalter.com] Sent: 27 February 2012 02:42 To: 'david raistrick'; 'Randy Carpenter' Cc: 'Nanog' Subject: RE: Reliable Cloud host ?

...

-----Original Message----- From: david raistrick [mailto:drais@icantclick.org] Sent: Sunday, February 26, 2012 7:19 PM To: Randy Carpenter Cc: Nanog Subject: Re: Reliable Cloud host ?

On Sun, 26 Feb 2012, Randy Carpenter wrote:

...

I don't need that kind of HA, and understand that it is not going to be available. 15 minutes of downtime is fine. 6 hours is completely unacceptable, and it false advertising to say you have a "Cloud" service, and then have the realization that you could have *indefinite* downtime.

Um. You and I apparently work in different clouds.

Since it is the weekend, I can't resist writing down a little equation:

Marketing(cloud) <> Technology(cloud)

For some values of "cloud" perhaps?

Well indeed that is a valid point. All cloud to me means is that there is some abstracted instance of x and that it does not always relate to a particular physical device, indeed, it may well be spread around a few physical devices. I don't think there is any implied magic redundancy automatic failover move your instance to another bit of metal if something breaks in there unless that's specifically stated. caveat emptor -- Leigh ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

On Sun, Feb 26, 2012 at 7:02 PM, Randy Carpenter <rcarpen@network1.net> wrote:

...

On Feb 26, 2012, at 4:56 PM, Randy Carpenter wrote:

...

1. Full redundancy with instant failover to other hypervisor hosts upon hardware failure (I thought this was a given!)

This is actually a much harder problem to solve than it sounds, and gets progressively harder depending on what you mean by "failover".

At the very least, having two physical hosts capable of running your VM requires that your VM be stored on some kind of SAN (usually iSCSI based) storage system. Otherwise, two hosts have no way of accessing your VM's data if one were to die. This makes things an order of magnitude or higher more expensive.

This does not have to be true at all.  Even having a fully fault-tolerant SAN in addition to spare servers should not cost much more than having separate RAID arrays inside each of the server, when you are talking about 1,000s of server (which Rackspace certainly has)

Randy, You're kidding, right? SAN storage costs the better part of an order of magnitude more than server storage, which itself is several times more expensive than workstation storage. That's before you duplicate the SAN and set up the replication process so that cabinet and room level failures don't take you out. DR sites then create a ferocious (read: expensive) bandwidth challenge. Data can't flush from the primary SAN's write cache until the DR SAN acknowledges receipt. If you don't have enough bandwidth to keep up under the heaviest daily loads, the cache quickly fills and the writes block. I maintain 50ish VMs with about 30 different providers at the moment. Not one of them attempts to do anything like what you describe.

NetApp. HA heads. Done. Add a DR site with replication, and you can survive a site failure, and be back up and running in less than an hour. I would think that the big datacenter guys already have this type of thing set up.

That's expensive and VMs are sold primarily on price. You want high reliability, you start with the dedicated colo server. Customers who want DR in a VM environment buy two VMs and build data replication at the app layer. On Mon, Feb 27, 2012 at 9:31 AM, Max <perldork@webwizarddesign.com> wrote:

Linode.com is not cloud based but they offer IP failover between VPS instances at no additonal charge - their pricing is excellent, I have had no down time issues with them in 3+ years with 3 different customers using them and they have nice OOB and programmatic API access for controlling VPs instances as well.

Hi Max, I have had superb results from Linode and highly recommend them. However, they're facilitating application level failover not keeping your VM magically alive. And: http://library.linode.com/linux-ha/ip-failover-heartbeat-pacemaker-ubuntu-10... "Both Linodes must reside in the same datacenter for IP failover" So they don't support a full DR capability even if you're smart at the app level. On Mon, Feb 27, 2012 at 9:39 AM, Jared Mauch <jared@puck.nether.net> wrote:

Is the DNS service authoritative or recursive? If auth, you can solve this a few ways, either by giving the DNS name people point to multiple AAAA (and A) records pointing at a diverse set of instances. DNS is designed to work around a host being down. Same goes for MX and several other services. While it may make the service slightly slower, it's certainly not the end of the world.

Hi Jared, How DNS is designed to work and how it actually works is not the same. Look up "DNS Pinning" for example. For most kinds of DR you need IP level failover where the IP address is rerouted to the available site. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com  bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On Mon, Feb 27, 2012 at 12:09 PM, Jared Mauch <jared@puck.nether.net> wrote:

On Feb 27, 2012, at 10:28 AM, William Herrin wrote:

...

How DNS is designed to work and how it actually works is not the same. Look up "DNS Pinning" for example. For most kinds of DR you need IP level failover where the IP address is rerouted to the available site.

I'm never claimed your response would be perfect, but it will certainly work well enough to avoid major problems.

No, actually, it won't. In practice, most end user applications disregard the DNS TTL. In some cases this is because of carelessness: The application does a gethostbyname once when it starts, grabs the first IP address in the list and retains it indefinitely. The gethostbyname function doesn't even pass the TTL to the application. Ntpd is/used to be one of the notable offenders, continuing to poll the dead address for years after the server moved. In other cases disregarding the TTL was a deliberate design decision. Web browser DNS Pinning is an example of this. All modern web browsers implement a form of DNS Pinning where they refuse to try an alternate IP address for a web server on subsequent TCP connections after making the first successful contact. This plugs a javascript security leak where a client side application could be made to scan the interior of its user's firewall by switching the DNS back and forth between local and remote addresses. In some cases this stuck-address behavior can persist until the browser is completely closed and reopened, possibly when the PC is rebooted weeks later. The net result is that when you switch the IP address of your server, a percentage of your users (declining over time) will be unable to access it for hours, days, weeks or even years regardless of the DNS TTL setting. This isn't theoretical, by the way. I had to renumber a major web site once. 1 hour TTL at the beginning of the process. Three month overlap in which both addresses were online and the DNS pointed to the new one. At the end of the three months a fraction of a percent of the *real user traffic* was _still_ coming in the obsolete address. Using the correct name in the Host: header, so the user wasn't deliberately picking the IP address. If you want DR that *works*, reroute the IP address. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com  bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On Feb 27, 2012, at 2:53 PM, Valdis.Kletnieks@vt.edu wrote:

On Mon, 27 Feb 2012 14:02:04 EST, William Herrin said:

...

The net result is that when you switch the IP address of your server, a percentage of your users (declining over time) will be unable to access it for hours, days, weeks or even years regardless of the DNS TTL setting.

Amen brother.

So just for grins, after seeing William's I set up a listener on an address that had an NTP server on it many moons ago. As in the machine was shut down around 2002/06/30 22:49 and we didn't re-assign the IP address ever since *because* it kept getting hit with NTP packets.. Yes, a decade ago.

In the first 15 minutes, 234 different IP's have tried to NTP to that address.

I hereby reject the principle that one can not renumber a host/name and move it. Certainly some people will see breakage. This is because their software is defective, sometimes in a critical way, other times in a way that is non-obvious. But I reject the idea that you can't move a service, or have one MX, DNS, etc.. host be down and have it be fatal without something else being SERIOUSLY broken. If you are right, nobody could ever renumber anything ever, nor take a service down ever in the most absolute terms. I've been involved in large scale DNS server renumbering/moving/whatnot. It's harder these days than it was in the past, but its feasible. I know those resolver addresses that have been retired still get queries from *very* broken hosts. Just because they're getting queries, doesn't mean they are expecting an answer, or will properly handle it. Sometimes you have to break the service worse for people to repair it. Look at the DCWG.org site and try to get an idea if you're infected. At some point those will go away. Doesn't mean those people aren't broken/infected and REQUIRE remediation. - Jared

On Feb 28, 2012, at 10:22 AM, William Herrin wrote:

On Tue, Feb 28, 2012 at 9:02 AM, Jared Mauch <jared@puck.nether.net> wrote:

...

On Feb 27, 2012, at 2:53 PM, Valdis.Kletnieks@vt.edu wrote:

...

On Mon, 27 Feb 2012 14:02:04 EST, William Herrin said:

...

The net result is that when you switch the IP address of your server, a percentage of your users (declining over time) will be unable to access it for hours, days, weeks or even years regardless of the DNS TTL setting.

Amen brother.

So just for grins, after seeing William's I set up a listener on an address that had an NTP server on it many moons ago. As in the machine was shut down around 2002/06/30 22:49 and we didn't re-assign the IP address ever since *because* it kept getting hit with NTP packets.. Yes, a decade ago.

In the first 15 minutes, 234 different IP's have tried to NTP to that address.

I hereby reject the principle that one can not renumber a host/name and move it. I reject the idea that you can't move a service, or have one MX, DNS, etc.. host be down and have it be fatal without something else being SERIOUSLY broken. If you are right, nobody could ever renumber anything ever, nor take a service down ever in the most absolute terms.

Something else IS seriously broken. Several something elses actually:

1. DNS TTL at the application boundary, due in part to...

DNS TTL shouldn't make it to the application boundary...

2. Pushing the name to layer 3 address mapping process up from layer 4 to layer 7 where each application has to (incorrectly) reinvent the process, and...

But they don't have to... They can simply use getaddrinfo()/getnameinfo() and let the OS libraries do it. The fact that some applications choose to use their own resolvers instead of system libraries is what is broken.

3. A layer 4 protocol which overloads the layer 3 address as an inseverable component of its transport identifier.

Even stuff like SMTP which took care to respect the DNS TTL in its own standards gets busted at the back end: too many antispam process components rely on the source IP address, crushing large scale servers that suddenly appear, transmitting large amounts of email from a fresh IP address.

I think this is orthogonal to DNS TTL issues.

Shockingly enough we have a strongly functional network despite this brokenness. But, it's broken all the same and renumbering is majorly impaired as a consequence.

In my experience, the biggest hurdle to renumbering has nothing to do with DNS, DNS TTLs, respect or failure to respect them, etc. In my experience the biggest renumbering challenges come from the number of configuration files which contain your IP addresses yet are not under your control. VPNs (the configuration at the far side of the VPN) Firewalls (vendors, clients, etc. that have put your IP addresses into exceptions) Router configurations (vendors, clients, etc. that have special routing policy to reach you) etc. These are the things that make renumbering hard. The DNS stuff is usually fairly trivial to work around with a little time and planning.

Renumbering in light of these issues isn't impossible. An overlap period is required in which both old and new addresses are operable.

That's desirable even if you have a 5 second TTL and everyone did honor it.

The duration of that overlap period is not defined by the the protocol itself. Rather, it varies with the tolerable level or residual brokenness, literally how many nines of users should be operating on the new address before the old address can go away.

There is some truth to that. The combination of applications having their own (broken) resolver libraries and operating systems that provide even more broken resolvers (thanks, Redmond) has made this a bigger challenge than it should be. The ideal solution is to go back to using the OS resolver libraries and fix them. Best of luck actually achieving that. Owen

related to the topic: http://slashdot.org/story/12/02/29/153226/microsofts-azure-cloud-suffers-maj... -- -- ℱin del ℳensaje.

Check out Firehost. Just came back from RSA2012 and talked with them. VPS provider using VMWare ESX with Dell/Compellent (auto tiered with SSD) for storage. They offer DDoS mitigation (they use Arbor) out of the box along with managed firewall and web application firewall. More expensive than EC2, but their high touch features seems worthwhile. Live support is included. Rob -----Original Message----- From: Bobby Mac [mailto:bobbyjim@gmail.com] Sent: Wednesday, February 29, 2012 1:24 PM To: Tei Cc: Nanog Subject: Re: Reliable Cloud host ? HP has built an Openstack based cloud. I got a beta account and things are surprisingly stable. hpcloud dot com On Wed, Feb 29, 2012 at 1:12 PM, Tei <oscar.vives@gmail.com> wrote:

related to the topic:

http://slashdot.org/story/12/02/29/153226/microsofts-azure-cloud-suffers-maj...

-- -- ℱin del ℳensaje.

On Mon, 27 Feb 2012, William Herrin wrote:

In some cases this is because of carelessness: The application does a gethostbyname once when it starts, grabs the first IP address in the list and retains it indefinitely. The gethostbyname function doesn't even pass the TTL to the application. Ntpd is/used to be one of the notable offenders, continuing to poll the dead address for years after the server moved.

While yes it often is carelessness - it's been reported by hardcore development sorts that I trust that there is no standardized API to obtain the TTL... What needs to get fixed is get[hostbyname,addrinfo,etc] so programmers have better tools. -- david raistrick http://www.netmeister.org/news/learn2quote.html drais@icantclick.org http://www.expita.com/nomime.html

On Mon, Feb 27, 2012 at 3:43 PM, david raistrick <drais@icantclick.org> wrote:

On Mon, 27 Feb 2012, William Herrin wrote:

...

In some cases this is because of carelessness: The application does a gethostbyname once when it starts, grabs the first IP address in the list and retains it indefinitely. The gethostbyname function doesn't even pass the TTL to the application. Ntpd is/used to be one of the notable offenders, continuing to poll the dead address for years after the server moved.

While yes it often is carelessness - it's been reported by hardcore development sorts that I trust that there is no standardized API to obtain the TTL...  What needs to get fixed is get[hostbyname,addrinfo,etc] so programmers have better tools.

Meh. What should be fixed is that connect() should receive a name instead of an IP address. Having an application deal directly with the IP address should be the exception rather than the rule. Then, deal with the TTL issues once in the standard libraries instead of repeatedly in every single application. In theory, that'd even make the app code protocol agnostic so that it doesn't have to be rewritten yet again for IPv12. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com  bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On Mon, Feb 27, 2012 at 7:07 PM, Owen DeLong <owen@delong.com> wrote:

On Feb 27, 2012, at 3:50 PM, William Herrin wrote:

...

Meh. What should be fixed is that connect() should receive a name instead of an IP address. Having an application deal directly with the IP address should be the exception rather than the rule. Then, deal with the TTL issues once in the standard libraries instead of repeatedly in every single application.

In theory, that'd even make the app code protocol agnostic so that it doesn't have to be rewritten yet again for IPv12.

While I agree with the principle of what you are trying to say, I would argue that it should be dealt with in getnameinfo() / getaddrinfo() and not connect().

It is perfectly reasonable for connect() to deal with an address structure.

Yes, well, that's why we're still using a layer 4 protocol (TCP) that can't dynamically rebind to the protocol level below (IP). God help us when folks start overriding the ethernet MAC address to force machines to keep the same IPv6 address that's been hardcoded somewhere or is otherwise too much trouble to change. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com  bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On Feb 27, 2012, at 19:10, Owen DeLong <owen@delong.com> wrote:

On Feb 27, 2012, at 3:50 PM, William Herrin wrote:

...

On Mon, Feb 27, 2012 at 3:43 PM, david raistrick <drais@icantclick.org> wrote:

...

On Mon, 27 Feb 2012, William Herrin wrote:

...

In some cases this is because of carelessness: The application does a gethostbyname once when it starts, grabs the first IP address in the list and retains it indefinitely. The gethostbyname function doesn't even pass the TTL to the application. Ntpd is/used to be one of the notable offenders, continuing to poll the dead address for years after the server moved.

While yes it often is carelessness - it's been reported by hardcore development sorts that I trust that there is no standardized API to obtain the TTL... What needs to get fixed is get[hostbyname,addrinfo,etc] so programmers have better tools.

Meh. What should be fixed is that connect() should receive a name instead of an IP address. Having an application deal directly with the IP address should be the exception rather than the rule. Then, deal with the TTL issues once in the standard libraries instead of repeatedly in every single application.

In theory, that'd even make the app code protocol agnostic so that it doesn't have to be rewritten yet again for IPv12.

While I agree with the principle of what you are trying to say, I would argue that it should be dealt with in getnameinfo() / getaddrinfo() and not connect().

It is perfectly reasonable for connect() to deal with an address structure.

If people are not using getnameinfo()/getaddrinfo() from the standard libraries, then, I don't see any reason to believe that they would use connect() from the standard libraries if it incorporated their functionality.

gai/gni do not return TTL values on any platforms I'm aware of, the only way to get TTL currently is to use a non standard resolver (e.g. lwres). The issue is application developers not calling gai every time they connect (due to aforementioned security concerns, at least in the browser realm), instead opting to hold onto the original resolved address for unreasonable amounts of time. Modifying gai to provide TTL has been proposed in the past (dnsop '04) but afaik was shot down to prevent inconsistencies in the API. Maybe when happy eyeballs stabilizes someone will propose an API for inclusion in the standard library that implements HE style connections. Looks like there was already some talk on v6ops headed this way, but as always there's resistance to standardizing it. ~Matt

On Feb 27, 2012, at 9:45 PM, Mark Andrews wrote:

getaddrinfo was designed to be extensible as was struct addrinfo. Part of the problem with TTL is not data sources used by getaddrinfo have TTL information. Additionally for many uses you want to reconnect to the same server rather than the same name. Note there is nothing to prevent a getaddrinfo implementation maintaining its own cache though if I was implementing such a cache I would have a flag to to force a refresh.

Sorry if I wasn't clear... My point to Bill was that we should be using calls that don't have TTL information (GAI/GNI in their default forms). That we don't need to abuse connect() to achieve that. That if people use GAI/GNI(), then, any brokenness is system-wide brokenness in the system's resolver library and should be addressed there. Owen

In message <CAP-guGV09HF7in+vZbKpGk0RR1Q4gpMMo5jQREUZVEj+ewzmkg@mail.gmail.com>, William Herrin writes:

On Tue, Feb 28, 2012 at 12:45 AM, Mark Andrews <marka@isc.org> wrote:

...

getaddrinfo was designed to be extensible as was struct addrinfo. Part of the problem with TTL is not [all] dat=

a sources

...

used by getaddrinfo have TTL information.

Hi Mark,

By the time getaddrinfo replaced gethostbyname, NIS and similar systems were on their way out. It was reasonably well understood that many if not most of the calls would return information gained from the DNS. Depending on how you look at it, choosing not to propagate TTL knowledge was either a belligerent choice to continue disrespecting the DNS Time To Live or it was fatalistic acceptance that the DNS TTL isn't and would not become functional at the application level.

No. Propogating TTL is still a issue especially when you do not always have one. You can't just wave the problem away. As for DNS TTL addresses are about the only thing which have multiple sources. You also don't have to use getaddrinfo. It really is designed to be the first step in connecting to a host. If you need to reconnect you call it again.

Still works fine deeper in the query system, timing out which server holds the records though.

...

Additionally for many uses you want to reconnect to the same server rather than the same name.

The SRV record was designed to solve that whole class of problems without damaging the operation of the TTL. No one uses it.

You don't need to know the TTL to use SRV.

It's all really very unfortunate. The recipe for SOHO multihoming, the end of routing table bloat and IP roaming without pivoting off a home base all boils down to two technologies: (1) a layer 4 protocol that can dynamically rebind to the layer 3 IP address the same way IP uses ARP to rebind to a changing ethernet MAC and (2) a DNS TTL that actually works so that the DNS supports finding a connection's current IP address.

DNS TTL works. Applications that don't honour it arn't a indication that it doesn't work.

Regards, Bill Herrin

-- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004 -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

In message <CAP-guGXK3WQGPLpmnVsnM0xnnU8==4zONK=UWTLkYWuduA6T9Q@mail.gmail.com>, William Herrin writes:

On Tue, Feb 28, 2012 at 4:06 PM, Mark Andrews <marka@isc.org> wrote:

...

DNS TTL works. =A0Applications that don't honour it arn't a indication th= at it doesn't work.

Mark,

If three people died and the building burned down then the sprinkler system didn't work. It may have sprayed water, but it didn't *work*.

Not enough evidence to say if it worked or not. Sprinkler systems are designed to handle particular classes of fire, not every fire. A 0 TTL means use this information for this transaction. We don't tear down TCP sessions on DNS TTL going to zero. If one really want to deprecate addresses we need something a lot more complicated than A and AAAA records in the DNS. We need stuff like "use this address for new transactions", "this address is going away soon, don't use it unless no other works". One also has to use multiple addresses at the same time. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

On Wed, Feb 29, 2012 at 7:57 AM, Joe Greco <jgreco@ns.sol.net> wrote:

...

In message <CAP-guGXK3WQGPLpmnVsnM0xnnU8==4zONK=UWTLkYWuduA6T9Q@mail.gmail.com>,  William Herrin writes:

...

On Tue, Feb 28, 2012 at 4:06 PM, Mark Andrews <marka@isc.org> wrote:

...

DNS TTL works. =A0Applications that don't honour it arn't a indication th= at it doesn't work.

Mark,

If three people died and the building burned down then the sprinkler system didn't work. It may have sprayed water, but it didn't *work*.

Not enough evidence to say if it worked or not.  Sprinkler systems are designed to handle particular classes of fire, not every fire.

It is also worth noting that many fire systems are not intended to put out the fire, but to provide warning and then provide an extended window for people to exit the affected building through use of sprinklers and other measures to slow the spread of the fire.

Hi Joe, The sprinkler system is designed to delay the fire long enough for everyone to safely escape. As a secondary objective, it reduces the fire damage that occurs while waiting for firefighters to arrive and extinguish the fire. If "three people died" then the system failed. Perhaps the design was inadequate. Perhaps some age-related issue prevented the sprinkler heads from melting. Perhaps someone stacked boxes to the ceiling and it blocked the water. Perhaps the water was shut off and nobody knew it. Perhaps an initial explosion damaged the sprinkler system so it could no longer work effectively. Whatever the exact details, that sprinkler system failed. Whoever you want to blame, DNS TTL dysfunction at the application level is the same way. It's a failed system. With the TTL on an A record set to 60 seconds, you can't change the address attached to the A record and expect that 60 seconds later no one will continue to connect to the old address. Nor 600 seconds later nor 6000 seconds later. The "system" for renumbering a service of which the TTL setting is a part consistently fails to reliably function in that manner. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com  bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On 02/29/12 10:01, Owen DeLong wrote:

Further, DNS performance issues in the past have led developers of such applications to "take matters into their own hands" to try and improve the performance/behavior of their application in spite of DNS. This is one of the things that led to many of the TTL ignorant application-level DNS caches which you are complaining about.

I have found some carriers to run hacked nameservers. Several years ago I was moving a website and found that Cox was overriding the TTL for all "www" names. At least for their residential customers in Oklahoma. The TTL value our test subject was getting was larger than it had ever been set. -- Mr. Flibble King of the Potato People

On Wed, Feb 29, 2012 at 7:57 AM, Joe Greco <jgreco@ns.sol.net> wrote:

...

...

In message <CAP-guGXK3WQGPLpmnVsnM0xnnU8==4zONK=UWTLkYWuduA6T9Q@mail.gmail.com>, �William Herrin writes:

...

On Tue, Feb 28, 2012 at 4:06 PM, Mark Andrews <marka@isc.org> wrote:

...

DNS TTL works. =A0Applications that don't honour it arn't a indication th= at it doesn't work.

Mark,

If three people died and the building burned down then the sprinkler system didn't work. It may have sprayed water, but it didn't *work*.

Not enough evidence to say if it worked or not. �Sprinkler systems are designed to handle particular classes of fire, not every fire.

It is also worth noting that many fire systems are not intended to put out the fire, but to provide warning and then provide an extended window for people to exit the affected building through use of sprinklers and other measures to slow the spread of the fire.

Hi Joe,

The sprinkler system is designed to delay the fire long enough for everyone to safely escape.

Hi Bill, No, the sprinkler system is *intended* to delay the fire long enough for everyone to safely escape, however, in order to accomplish this, the designer chooses from some reasonable options to meet certain goals that are commonly accepted to allow that. For example, the suppression design applied to a multistory dwelling where people live, cook, and sleep is typically different from the single-story light office space. Neither design will be effective against all possible types of fire

As a secondary objective, it reduces the fire damage that occurs while waiting for firefighters to arrive and extinguish the fire. If "three people died" then the system failed.

That's silly. The system fails if the system *fails* or doesn't behave as designed. No system is capable of guaranteeing survival. Just yesterday, here in Milwaukee, we had a child killed at a railroad crossing. The crossing was well-marked, with signals and gates. Visibility of approaching trains for close to a mile in either direction. The crew on the train saw him crossing, blew their horn, laid on the emergency brakes. CP Rail inspected the gates and signals for any possible faults, but eyewitness accounts were that the gates and signals were working, and the train made every effort to make itself known. The 11 year old kid had his hood up and earbuds in, and apparently didn't see the signals or look up and down the track before crossing, and for whatever reason, didn't hear the train horn blaring at him. At a certain point, you just can't protect against every possible bad thing that can happen. I have a hard time seeing this as a failure of the railroad's fully functional railroad crossing and related safety mechanisms. The system doesn't guarantee survival.

Whoever you want to blame, DNS TTL dysfunction at the application level is the same way. It's a failed system. With the TTL on an A record set to 60 seconds, you can't change the address attached to the A record and expect that 60 seconds later no one will continue to connect to the old address. Nor 600 seconds later nor 6000 seconds later. The "system" for renumbering a service of which the TTL setting is a part consistently fails to reliably function in that manner.

It's a failure because people don't understand the intent of the system, and it is pretty safe to argue that it is a multifaceted failure, due to failures by client implementations, server implementations, sample code, attempts to use the system for things it wasn't meant for, etc. This is by no means limited to TTL; we've screwed up multiple addresses, IPv6 handling, negative caching, um, do I need to go on...? In the specific case of TTL, the problem is made much worse due to the way most client code has hidden this data from developers, so that many developers don't even have any idea that such a thing exists. I'm not sure how to see that a design failure of the TTL mechanism. I don't see developers ignoring DNS and hardcoding IP addresses into code as a failure of the DNS system. I see both as naive implementation errors. The difference with TTL is that the implementation errors are so widespread as to render any sane implementation relatively useless. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Wed, Feb 29, 2012 at 4:02 PM, Joe Greco <jgreco@ns.sol.net> wrote:

...

In the specific case of TTL, the problem is made much worse due to the way most client code has hidden this data from developers, so that many developers don't even have any idea that such a thing exists.

I'm not sure how to see that a design failure of the TTL mechanism.

Hi Joe,

You shouldn't see that as a design failure of the TTL mechanism. It isn't. It's a failure of the system of which DNS TTL is a component. The TTL component itself was reasonably designed.

Think that's pretty much what I said.

The failure is likened to installing a well designed sprinkler system (the DNS with a TTL) and then shutting off the water valve (gethostbyname/getaddrinfo).

No, the water still works as intended. I think your analogy starts to fail here. It's more like expecting a water suppression system to put out a grease fire. The TTL mechanism is completely suitable for what it was originally meant for, and in an environment where everyone has followed the rules, it works fine. If you take a light office space with sprinklers and remodel it into a short order grill, the fire inspector will require you to rework the fire suppression system to an appropriate system. Problem is, TTL is a relatively light-duty system that people have felt free to ignore, overload for other purposes, etc., but there's no fire inspector to come around and tell people how and why what they've done is broken. In the case of TTL, the system is even largely hidden from users, so that it is rarely thought about except now and then on NANOG, dns-operations, etc. ;-) No wonder it is even poorly understood.

...

I don't see developers ignoring DNS and hardcoding IP addresses into code as a failure of the DNS system.

It isn't. It's a failure of the sockets API design which calls on every application developer to (a) translate the name to a set of addresses with a mechanism that discards the TTL knowledge and (b) implement his own glue between name to address mapping and connect by address.

It would be like telling an app developer: here's the ARP function and the SEND function. When you Send to an IP address, make sure you attach the right destination MAC. Of course the app developer gets it wrong most of the time.

That's correct - and it doesn't imply that the system that was engineered is faulty. In all likelihood, the fault lies with what the app developer was told. You originally said: "If three people died and the building burned down then the sprinkler system didn't work. It may have sprayed water, but it didn't *work*." That's not true. If it sprayed water in the manner it was designed to, then it worked. If three people took sleeping pills and didn't wake up when the alarms blared, and an arsonist poured ten gallons of gas everywhere before lighting the fire, the system still worked. It failed to save those lives or protect the building from burning down, but I am aware of no fire suppression systems that realistically attempts to address that. It is an unreasonable expectation. I have a hard time seeing the many self-inflicted wounds of people who have attempted to abuse TTL for various purposes as a failure of the TTL design. The design is reasonable. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Thu, Mar 1, 2012 at 8:25 AM, Joe Greco <jgreco@ns.sol.net> wrote:

...

"If three people died and the building burned down then the sprinkler system didn't work. It may have sprayed water, but it didn't *work*."

That's not true. =A0If it sprayed water in the manner it was designed to, then it worked.

That's like the old crack about ICBM interceptors. Why yes, our system performed swimmingly in the latest test achieving nine out of the ten criteria for success. Which criteria didn't it achieve? It missed the target.

Difference: the fire suppression system worked as designed, the ICBM didn't. That's kind of the whole point here. If you have something like an automobile that's designed to protect you against certain kinds of accidents, it isn't a failure if it does not protect you against an accident that is not reasonably within the protection envelope. For example, cars these days are designed to protect against many different types of impacts and provide survivability. It is a failure if my car is designed to protect against a head-on crash at 30MPH by use of engineered crumple zones and deploying air bags, and I get into such an accident and am killed regardless. However, if I fly my car into a bridge abutment at 150MPH and am instantly pulverized, I am not prepared to consider that a failure of the car. Likewise, if a freeway overpass slab falls on my car and crushes me as I drive underneath it, I am not going to consider that a failure of the car. There's a definite distinction between a system that fails when it is deployed and used in the intended manner, and a system that doesn't work as you'd like it to when it is used in some incorrect manner, which is really not a failure as the word is normally used. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

GAI/GNI do not return TTL values, but this should not be a problem. If they were to return anything, it should not be a TTL, but a time() value, after which the result may no longer be used.

One way to achieve that would be for GAI to return an opaque structure that contained the IP and such a value, in a manner consumable by the sockets API, and adjust connect() to return an error if passed a structure containing a ' returned time + TTL' in the past.

AF_INET_TTL and AFINET6_TTL, with correspondingly expanded struct sockaddr_* ? Code that explictly requests AF_INET or AF_INET6 would get what it was expecting, code that requests AF_UNSPEC on a system with modified getaddrinfo() would get the expanded structs with the different ai_family set, and could pass them straight into a modified connect(). I'm sure I'm grossly oversimplifying somewhere though... Regards, Tim.

On Thu, Mar 1, 2012 at 7:20 AM, Owen DeLong <owen@delong.com> wrote:

The simpler approach and perfectly viable without mucking up what is already implemented and working:

Don't keep returns from GAI/GNI around longer than it takes to cycle through your connect() loop immediately after the GAI/GNI call.

The even simpler approach: create an AF_NAME with a sockaddr struct that contains a hostname instead of an IPvX address. Then let connect() figure out the details of caching, TTLs, protocol and address selection, etc. Such a connect() could even support a revised TCP stack which is able to retry with the other addresses at the first subsecond timeout rather than camping on each address in sequence for the typical system default of two minutes. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com  bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On 03/01/2012 06:26 AM, William Herrin wrote:

...

...

The simpler approach and perfectly viable without mucking up what is already implemented and working:

Don't keep returns from GAI/GNI around longer than it takes to cycle through your connect() loop immediately after the GAI/GNI call. The even simpler approach: create an AF_NAME with a sockaddr struct

On Thu, Mar 1, 2012 at 7:20 AM, Owen DeLong<owen@delong.com> wrote: that contains a hostname instead of an IPvX address. Then let connect() figure out the details of caching, TTLs, protocol and address selection, etc. Such a connect() could even support a revised TCP stack which is able to retry with the other addresses at the first subsecond timeout rather than camping on each address in sequence for the typical system default of two minutes.

The effect of what you're recommending is to move all of this into the kernel, and in the process greatly expand its scope. Also: even if you did this, you'd be saddled with the same problem because nothing existing would use an AF_NAME.

The real issue is that gethostbyxxx has been inadequate for a very long time. Moving it across the kernel boundary solves nothing and most likely causes even more trouble: what if I want, say, asynchronous name resolution? What if I want to use SRV records? What if a new DNS RR comes around -- do i have do recompile the kernel? It's for these reasons and probably a whole lot more that connect just confuses the actual issues.

When I was writing the first version of DKIM I used a library that I scraped off the net called ARES. It worked adequately for me, but the most notable thing was the very fact that I had to scrape it off the net at all. As far as I could tell, standard distos don't have libraries with lower level access to DNS (in my case, it needed to not block). Before positing a super-deluxe gethostbyxx that does addresses picking, etc, etc, it would be better to lobby all of the distos to settle on a decomposed resolver library from which that and more could be built.

It's deeper than just that, though. The whole paradigm is messy, from the point of view of someone who just wants to get stuff done. The examples are (almost?) all fatally flawed. The code that actually gets at least some of it right ends up being too complex and too hard for people to understand why things are done the way they are. Even in the "old days", before IPv6, geez, look at this: bcopy(host->h_addr_list[n], (char *)&addr->sin_addr.s_addr, sizeof(addr->sin_addr.s_addr)); That's real comprehensible - and it's essentially the data interface between the resolver library and the system's addressing structures for syscalls. On one hand, it's "great" that they wanted to abstract the dirty details of DNS away from users, but I'd say they failed pretty much even at that. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

Hi, On Mar 1, 2012, at 7:22 AM, Joe Greco wrote:

On Mar 1, 2012, at 7:01 AM, Michael Thomas wrote:

...

The effect of what you're recommending is to move all of this into the kernel, and in the process greatly expand its scope. Also: even if you did this, you'd be saddled with the same problem because nothing existing would use an AF_NAME.

I always thought the right way to deal with IPv6 would have been to use a 32-bit number from the class E space as a 'network handle' where the actual address (be it IPv4 or IPv6) was handled by the kernel. I suspect this would have allowed the majority of network-utilizing applications to magically just work, regardless of whether the name supplied by gethosbyname/getnameinfo/etc. was mapped to an address with A or AAAA. Probably would make stuff faster too since you'd only have to deal with an unsigned int instead of (worst case) 16 bytes that have to be copied back and forth. Instead, we have forced application developers to use a really odd mixture of old and new, e.g. 'struct sockaddr_in6' and GNI/GAI. Seems this is the worst of both worlds -- no backwards compatibility yet an adherence to a really broken model that requires applications to know useless details like the length of an address ("what do you mean a sizeof(struct sockaddr) isn't big enough to hold an IPv6 address?") and even its bit patterns.

...

Moving it across the kernel boundary solves nothing

Actually, it does. Right now, applications effectively cache the address in their data space, requiring the application developer to go to quite a bit of work to deal with the address changing (or, far more typically, just pretend addresses never change). This has a lot of unfortunate side effects.

...

and most likely causes even more trouble: what if I want, say, asynchronous name resolution?

Set non-blocking on the socket?

...

What if I want to use SRV records? What if a new DNS RR comes around -- do i have do recompile the kernel?

I believe with the exception of A/AAAA, RDATA is typically returned as either opaque (to the DNS) data blobs or names. This means the only stuff the kernel would need to deal with would be the A/AAAA lookups, everything else would be passed back as data, presumably via a new system call.

...

As far as I could tell, standard distos don't have libraries with lower level access to DNS (in my case, it needed to not block).

There have been lower-level resolver APIs since (at least) BSD 4.3 (man resolver(3)).

It's deeper than just that, though. The whole paradigm is messy, from the point of view of someone who just wants to get stuff done. The

examples are (almost?) all fatally flawed. The code that actually gets at least some of it right ends up being too complex and too hard for people to understand why things are done the way they are.

Exactly. Even before IPv6, it was icky. Now, it's just crazy. We had an opportunity to fix this with IPv6 since IPv6 required non-trivial kernel hackage. Unfortunately, we didn't take advantage of that opportunity. Regards, -drc

Jeroen, On Mar 1, 2012, at 9:25 AM, Jeroen Massar wrote:

...

I always thought the right way to deal with IPv6 would have been to use a 32-bit number from the class E space as a 'network handle' where the actual address (be it IPv4 or IPv6) was handled by the kernel.

This is the case when you pass in a sockaddr. Note, not a sockaddr_in or a sockaddr_in6, but just a sockaddr.

Sorry? On which system? As far as I'm aware, there are no libraries that make use of class E addresses to act as a layer of indirection similar to file handles. Would love to know such exists.

There is a nice 14 year old article about this: http://www.kame.net/newsletter/19980604/

Quoting from that article: "This way the network address and address family is will not live together, and leads to bunch of if/switch statement and mistakes in programming. " which is exactly the point. It has been 14 years and people are _STILL_ discussing this.

And actually those addresses are only 'copied' once: during accept() or connect(),

Assuming the application doesn't need to copy the address, ever.

Ever heard of sockaddr_storage?

Oddly, yes. It still astonishes me that sizeof(struct sockaddr) < sizeof(struct sockaddr_storage).

It was made to solve that little issue. See also, that article above.

Thus requiring people to go in and muck with code thereby increasing the cost of migration with obvious effect.

What you are talking about is an API wrapper. Depending on platform these have existed for years already. Quite a few do not expose addresses at all to the calling code.

And yet, look at the code Mark Andrews just referenced as his recommend way of dealing with initiating connections. How many applications actually do anything like that? More to the point, how many books/article/etc. exist that reference these APIs you're talking about vs. how many reference the traditional way one goes about dealing with networks? Rhetorical questions, no need to answer. Got tired of tilting at this windmill some time ago and I know nothing will change. I'm just amazed that people defend the abominable kludge that are the existing common sockets/resolver APIs. Regards, -drc

Michael, On Mar 1, 2012, at 10:00 AM, Michael Thomas wrote:

My rule of thumb is for this sort of thing "does it *require* kernel level access?" In this case, the answer is manifestly "no".

This is tilting at windmills since it's wildly unlikely anything will change, but... The idea is to add a level of indirection that does not currently exist, similar to the mapping of filename/file handle/inode in the filesystem. This layer of indirection allows the kernel to remap things as it sees fit without impacting the application. If such functionality existed, the kernel could manage the mapping between name and address to do things like honoring DNS TTL, transparently handling renumbering events, deal with protocol transitions even during a connection, etc. As things are now, it's like having to rewrite non-tivial sections of code for _all_ disk-aware applications because we've gone from a 32-bit file system to a 64-bit file system, even though the vast majority of those applications couldn't care less.

SRV records?

Do not have addresses in their RDATA, they have names. Regards, -drc

In message <CAC38B59-1F54-4788-87A2-A1A8BE453500@delong.com>, Owen DeLong write s:

...

=20 It's deeper than just that, though. The whole paradigm is messy, from the point of view of someone who just wants to get stuff done. The examples are (almost?) all fatally flawed. The code that actually = gets at least some of it right ends up being too complex and too hard for people to understand why things are done the way they are. =20 Even in the "old days", before IPv6, geez, look at this: =20 bcopy(host->h_addr_list[n], (char *)&addr->sin_addr.s_addr, = sizeof(addr->sin_addr.s_addr)); =20 That's real comprehensible - and it's essentially the data interface=20=

...

between the resolver library and the system's addressing structures for syscalls. =20 On one hand, it's "great" that they wanted to abstract the dirty = details of DNS away from users, but I'd say they failed pretty much even at = that. =20 ... JG --=20 Joe Greco - sol.net Network Services - Milwaukee, WI - = http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] = then I won't contact you again." - Direct Marketing Ass'n position on e-mail = spam(CNN) With 24 million small businesses in the US alone, that's way too many = apples.

I think that the modern set of getaddrinfo and connect is actually not = that complicated:

/* Hints for getaddrinfo() (tell it what we want) */ memset(&addrinfo, 0, sizeof(addrinfo)); /* Zero out the buffer = */ addrinfo.ai_family=3DPF_UNSPEC; /* Any and all = address families */ addrinfo.ai_socktype=3DSOCK_STREAM; /* Stream Socket */ addrinfo.ai_protocol=3DIPPROTO_TCP; /* TCP */ /* Ask the resolver library for the information. Exit on failure. */ /* argv[1] is the hostname passed in by the user. "demo" is the = service name */ if (rval =3D getaddrinfo(argv[1], "demo", &addrinfo, &res) !=3D 0) { fprintf(stderr, "%s: Failed to resolve address information.\n", = argv[0]); exit(2); }

/* Iterate through the results */ for (r=3Dres; r; r =3D r->ai_next) { /* Create a socket configured for the next candidate */ sockfd6 =3D socket(r->ai_family, r->ai_socktype, r->ai_protocol); /* Try to connect */ if (connect(sockfd6, r->ai_addr, r->ai_addrlen) < 0) { /* Failed to connect */ e_save =3D errno; /* Destroy socket */ (void) close(sockfd6); /* Recover the error information */ errno =3D e_save; /* Tell the user that this attempt failed */ fprintf(stderr, "%s: Failed attempt to %s.\n", argv[0],=20 get_ip_str((struct sockaddr *)r->ai_addr, buf, BUFLEN)); /* Give error details */ perror("Socket error"); } else { /* Success! */ /* Inform the user */ snprintf(s, BUFLEN, "%s: Succeeded to %s.", argv[0], get_ip_str((struct sockaddr *)r->ai_addr, buf, BUFLEN)); debug(5, argv[0], s); /* Flag our success */ success++; /* Stop iterating */ break; } } /* Out of the loop. Either we succeeded or ran out of possibilities */ if (success =3D=3D 0) /* If we ran out of possibilities... */ { /* Inform the user, free up the resources, and exit */ fprintf(stderr, "%s: Failed to connect to %s.\n", argv[0], argv[1]); freeaddrinfo(res); exit(5); } /* Succeeded. Inform the user and continue with the application */ printf("%s: Successfully connected to %s at %s on FD %d.\n", argv[0], = argv[1], get_ip_str((struct sockaddr *)r->ai_addr, buf, BUFLEN), sockfd6); /* Free up the memory held by the resolver results */ freeaddrinfo(res);

It's really hard to make a case that this is all that complex.

I put a lot of extra comments in there to make it clear what's happening = for people who may not be used to coding in C. It also contains a whole = lot of extra user notification and debugging instrumentation because it = is designed as an example people can use to learn with.=20

Yes, this was a lot messier and a lot stranger and harder to get right = with get*by{name,addr}, but, those days are long gone and anyone still = coding with those needs to move forward.

Owen

These days you want something more complicated as everyone is or will be soon multi-homed. The basic loop above has very bad error characteristics if the first machines are not reachable. I've got working select, poll and thread based examples here: http://www.isc.org/community/blog/201101/how-to-connect-to-a-multi-homed-ser....

From http://www.isc.org/files/imce/select-connect_0.c:

/* * Copyright (C) 2011 Internet Systems Consortium, Inc. ("ISC") * * Permission to use, copy, modify, and/or distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH * REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY * AND FITNESS. IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT, * INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM * LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE * OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR * PERFORMANCE OF THIS SOFTWARE. */ #define TIMEOUT 500 /* ms */ int connect_to_host(struct addrinfo *res0) { struct addrinfo *res; int fd = -1, n, i, j, flags, count, max = -1, *fds; struct timeval *timeout, timeout0 = { 0, TIMEOUT * 1000}; fd_set fdset, wrset; /* * Work out how many possible descriptors we could use. */ for (res = res0, count = 0; res; res = res->ai_next) count++; fds = calloc(count, sizeof(*fds)); if (fds == NULL) { perror("calloc"); goto cleanup; } FD_ZERO(&fdset); for (res = res0, i = 0, count = 0; res; res = res->ai_next) { fd = socket(res->ai_family, res->ai_socktype, res->ai_protocol); if (fd == -1) { /* * If AI_ADDRCONFIG is not supported we will get * EAFNOSUPPORT returned. Behave as if the address * was not there. */ if (errno != EAFNOSUPPORT) perror("socket"); else if (res->ai_next != NULL) continue; } else if (fd >= FD_SETSIZE) { close(fd); } else if ((flags = fcntl(fd, F_GETFL)) == -1) { perror("fcntl"); close(fd); } else if (fcntl(fd, F_SETFL, flags | O_NONBLOCK) == -1) { perror("fcntl"); close(fd); } else if (connect(fd, res->ai_addr, res->ai_addrlen) == -1) { if (errno != EINPROGRESS) { perror("connect"); close(fd); } else { /* * Record the information for this descriptor. */ fds[i] = fd; FD_SET(fd, &fdset); if (max == -1 || fd > max) max = fd; count++; i++; } } else { /* * We connected without blocking. */ goto done; } if (count == 0) continue; assert(max != -1); do { if (res->ai_next != NULL) timeout = &timeout0; else timeout = NULL; /* The write bit is set on both success and failure. */ wrset = fdset; n = select(max + 1, NULL, &wrset, NULL, timeout); if (n == 0) { timeout0.tv_usec >>= 1; break; } if (n < 0) { if (errno == EAGAIN || errno == EINTR) continue; perror("select"); fd = -1; goto done; } for (fd = 0; fd <= max; fd++) { if (FD_ISSET(fd, &wrset)) { socklen_t len; int err; for (j = 0; j < i; j++) if (fds[j] == fd) break; assert(j < i); /* * Test to see if the connect * succeeded. */ len = sizeof(err); n = getsockopt(fd, SOL_SOCKET, SO_ERROR, &err, &len); if (n != 0 || err != 0) { close(fd); FD_CLR(fd, &fdset); fds[j] = -1; count--; continue; } /* Connect succeeded. */ goto done; } } } while (timeout == NULL && count != 0); } /* We failed to connect. */ fd = -1; done: /* Close all other descriptors we have created. */ for (j = 0; j < i; j++) if (fds[j] != fd && fds[j] != -1) { close(fds[j]); } if (fd != -1) { /* Restore default blocking behaviour. */ if ((flags = fcntl(fd, F_GETFL)) != -1) { flags &= ~O_NONBLOCK; if (fcntl(fd, F_SETFL, flags) == -1) perror("fcntl"); } else perror("fcntl"); } cleanup: /* Free everything. */ if (fds) free(fds); return (fd); } -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

In message <CAP-guGXLpzai4LrxyJcNn06yQ1jAEu4QeRpVzGRah=+OGLy9Zw@mail.gmail.com> , William Herrin writes:

On Thu, Mar 1, 2012 at 4:07 PM, Owen DeLong <owen@delong.com> wrote:

...

I think that the modern set of getaddrinfo and connect is actually not th= at complicated:

Owen,

If took you 50 lines of code to do 'socket=connect("www.google.com",80,TCP);' and you still managed to produce a version which, due to the timeout on dead addresses, is worthless for any kind of interactive program like a web browser. And because that code isn't found in a system library, every single application programmer has to write it all over again.

And your 'socket=connect("www.google.com",80,TCP);' won't work for a web browser either unless you are using threads and are willing to have the thread stall. The existing connect() semantics actually work well for browsers but they need to be properly integrated into the system as a whole. Nameservers have similar connect() issues as web browsers with one advantage, most of the time we are connecting to a machine we have just connected to via UDP. That doesn't mean we don't do non-blocking connect however.

I'm a fan of Rube Goldberg machines but that was ridiculous.

Regards, Bill Herrin -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

On Thu, Mar 1, 2012 at 5:37 PM, Owen DeLong <owen@delong.com> wrote:

You don't have to reinvent what I've done. Neither does every or any other application programmer. You are welcome to use any of the many connection abstraction libraries that are available in open source. I suggest you make a trip through google code.

Which is what everybody basically does. And when it works during the decidedly non-rigorous testing, they move on to the next problem... with code that doesn't perform well in the corner cases. Such as when a host has just been renumbered or one of the host's addresses is unreachable. And because most everybody has made more or less the same errors, the DNS TTL fails to cause their applications to work as intended and loses its utility as a tool to facilitate renumbering.

If you want, program in Python where the libraries do provide the abstraction you seek. Of course, that means you have to cope with Python's other disgusting habits like spaces are meaningful and variables are indistinguishable from code, but, there's always a tradeoff.

::shudder:: I don't *want* to do anything in python. The occasional reality of a situation dictates that I do some work in python, but I most definitely don't *want* to. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com  bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On Mar 1, 2012, at 2:57 PM, William Herrin wrote:

On Thu, Mar 1, 2012 at 5:37 PM, Owen DeLong <owen@delong.com> wrote:

...

You don't have to reinvent what I've done. Neither does every or any other application programmer. You are welcome to use any of the many connection abstraction libraries that are available in open source. I suggest you make a trip through google code.

Which is what everybody basically does. And when it works during the decidedly non-rigorous testing, they move on to the next problem... with code that doesn't perform well in the corner cases. Such as when a host has just been renumbered or one of the host's addresses is unreachable.

Then push for better written abstraction libraries. There's no need to break the current functionality of the underlying system calls and libc functions which would be needed by any such library anyway.

And because most everybody has made more or less the same errors, the DNS TTL fails to cause their applications to work as intended and loses its utility as a tool to facilitate renumbering.

Since I don't write applications for a living, I will admit I haven't rigorously tested any of the libraries out there, but, I'm willing to bet that someone, somewhere has probably written a good one by now.

...

If you want, program in Python where the libraries do provide the abstraction you seek. Of course, that means you have to cope with Python's other disgusting habits like spaces are meaningful and variables are indistinguishable from code, but, there's always a tradeoff.

::shudder:: I don't *want* to do anything in python. The occasional reality of a situation dictates that I do some work in python, but I most definitely don't *want* to.

Believe me, I'm in the same boat on that one. However, it is the only language I know of that provides the kind of interface you are demanding. Perhaps this should tell you something about what you are asking for. ;-) Owen

On Mar 1, 2012, at 5:15 PM, William Herrin wrote:

On Thu, Mar 1, 2012 at 8:02 PM, Owen DeLong <owen@delong.com> wrote:

...

There's no need to break the current functionality of the underlying system calls and libc functions which would be needed by any such library anyway.

Owen,

Point to one sentence written by anybody in this entire thread in which breaking current functionality was proposed.

When you said that: connect(char *name, uint16_t port) should work That can't work without breaking the existing functionality of the connect() system call.

...

...

And because most everybody has made more or less the same errors, the DNS TTL fails to cause their applications to work as intended and loses its utility as a tool to facilitate renumbering.

Since I don't write applications for a living, I will admit I haven't rigorously tested any of the libraries out there, but, I'm willing to bet that someone, somewhere has probably written a good one by now.

Yeah, and if you give me a few weeks I can probably find it amidst all the others which aren't so hot.

I doubt it would take weeks, but, in any case, it's probably faster than writing and debugging your own. Owen

On Mar 1, 2012, at 9:34 PM, William Herrin wrote:

On Thu, Mar 1, 2012 at 8:47 PM, Owen DeLong <owen@delong.com> wrote:

...

On Mar 1, 2012, at 5:15 PM, William Herrin wrote:

...

On Thu, Mar 1, 2012 at 8:02 PM, Owen DeLong <owen@delong.com> wrote:

...

There's no need to break the current functionality of the underlying system calls and libc functions which would be needed by any such library anyway.

Owen,

Point to one sentence written by anybody in this entire thread in which breaking current functionality was proposed.

When you said that:

connect(char *name, uint16_t port) should work

That can't work without breaking the existing functionality of the connect() system call.

You know, when I wrote 'socket=connect("www.google.com",80,TCP);' I stopped and thought to myself, "I wonder if I should change that to 'connectbyname' instead just to make it clear that I'm not replacing the existing connect() call?" But then I thought, "No, there's a thousand ways someone determined to misunderstand what I'm saying will find to misunderstand it. To someone who wants to understand my point, this is crystal clear."

I'm all for additional library functionality built on top of what exists that does what you want. As I said, there are many such libraries out there to do that. If someone wants to add it to libc, more power to them. I'm not the libc maintainer. I just don't want conect() to stop working the way it does or for getaddrinfo() to stop working the way it does. Since you were hell bent on calling the existing mechanisms broken rather than conceding the point that the current process is not broken, but, could stand some improvements in the library (http://owend.corp.he.net/ipv6 I even say as much myself), it was not entirely clear that you did not intend to replace connect() rather than augment the current capabilities with additional more abstract functions with different names. Owen

On Mar 2, 2012, at 10:12 AM, William Herrin wrote:

On Fri, Mar 2, 2012 at 1:03 AM, Owen DeLong <owen@delong.com> wrote:

...

On Mar 1, 2012, at 9:34 PM, William Herrin wrote:

...

You know, when I wrote 'socket=connect("www.google.com",80,TCP);' I stopped and thought to myself, "I wonder if I should change that to 'connectbyname' instead just to make it clear that I'm not replacing the existing connect() call?" But then I thought, "No, there's a thousand ways someone determined to misunderstand what I'm saying will find to misunderstand it. To someone who wants to understand my point, this is crystal clear."

"Hyperbole." If I had remembered the word, I could have skipped the long description.

...

I'm all for additional library functionality I just don't want conect() to stop working the way it does or for getaddrinfo() to stop working the way it does.

Good. Let's move on.

First question: who actually maintains the standard for the C sockets API these days? Is it a POSIX standard?

Well, some of it seems to be documented in RFCs, but, I think what you're wanting doesn't require adds to the sockets library, per se. In fact, I think wanting to make it part of that is a mistake. As I said, this should be a higher level library. For example, in Perl, you have Socket (and Socket6), but, you also have several other abstraction libraries such as Net::HTTP. While there's no hierarchical naming scheme for the functions in libc, if you look at the source for any of the open source libc libraries out there, you'll find definite hierarchy. POSIX certainly controls one standard. The GNU libc maintainers control the standard for the libc that accompanies GCC to the best of my knowledge. I would suggest that is probably the best place to start since I think anything that gains acceptance there will probably filter to the others fairly quickly.

Next, we have a set of APIs which, with sufficient caution and skill (which is rarely the case) it's possible to string together a reasonable process which starts with a some kind of name in a text string and ends with established communication with a remote server for any sort of name and any sort of protocol. These APIs are complete but we repeatedly see certain kinds of error committed while using them.

Right... Since these are user-errors (at the developer level) I wouldn't try to fix them in the APIs. I would, instead, build more developer proof add-on APIs on top of them.

Is there a common set of activities an application programmer intends to perform 9 times out of 10 when using getaddrinfo+connect? I think there is, and it has the following functionality:

Create a [stream].to one of the hosts satisfying [name] + [service] within [timeout] and return a [socket].

Seems reasonable, but ignores UDP. If we're going to do this, I think we should target a more complete solution to include a broader range of probabilities than just the most common TCP connect scenario.

Does anybody disagree? Here's my reasoning:

Better than 9 times out of 10 a steam and usually a TCP stream at that. Connect also designates a receiver for a connectionless protocol like UDP, but its use for that has always been a little peculiar since the protocol doesn't actually connect. And indeed, sendto() can designate a different receiver for each packet sent through the socket.

Most applications using UDP that I have seen use sendto()/recvfrom() et. al. Netflow data would suggest that it's less than 9 out of ten times for TCP, but, yes, I would agree it is the most common scenario.

Name + Service. If TCP, a hostname and a port.

That would apply to UDP as well. Just the semantics of what you do once you have the filehandle are different. (and it's not really a stream, per se).

Sometimes you want to start multiple connection attempts in parallel or have some not-quire-threaded process implement its own scheduler for dealing with multiple connections at once, but that's the exception. Usually the only reason for dealing with the connect() in non-blocking mode is that you want to implement sensible error recover with timeouts.

Agreed.

And the timeout - the direction that control should be returned to the caller no later than X. If it would take more than X to complete, then fail instead.

Actually, this is one thing I would like to see added to connect() and that could be done without breaking the existing API.

Next item: how would this work under the hood?

Well, you have two tasks: find a list of candidate endpoints from the name, and establish a connection to one of them.

Find the candidates: ask all available name services in parallel (hosts, NIS, DNS, etc). Finished when:

1. All services have responded negative (failure)

2. You have a positive answer and all services which have not yet answered are at a lower priority (e.g. hosts answers, so you don't need to wait for NIS and DNS).

3. You have a positive answer from at least one name service and 1/2 of the requested time out has expired.

4. The full time out has expired (failure).

I think the existing getaddrinfo() does this pretty well already. I will note that the services you listed only apply to resolving the host name. Don't forget that you might also need to resolve the service to a port number. (An application should be looking up HTTP, not assuming it is 80, for example). Conveniently, getaddrinfo simultaneously handles both of these lookups.

Cache the knowledge somewhere along with TTLs (locally defined if the name service doesn't explicitly provide a TTL). This may well be the first of a series of connection requests for the same host. If cached and TTL valid knowledge was known for this name for a particular service, don't ask that service again.

I recommend against doing this above the level of getaddrinfo(). Just call getaddrinfo() again each time you need something. If it has cached data, it will return quickly and is cheap. If it doesn't return quickly, it will still work just as quickly as anything else most likely. If getaddrinfo() on a particular system is not well behaved, we should seek to fix that implementation of getaddrinfo(), not write yet another replacement.

Also need to let the app tell us to deprioritize a particular result later on. Why? Let's say I get an HTTP connection to a host but then that connection times out. If the app is managing the address list, it can try again to another address for the same name. We're now hiding that detail from the app, so we need a callback for the app to tell us, "when I try again, avoid giving me this answer because it didn't turn out to work."

I would suggest that instead of making this opaque and then complicating it with these hints when we return, that we return use a mecahism where we return a pointer to a dynamically allocated result (similar to getaddrinfo) and if we get called again with a pointer to that structure, we know to delete the previously connected host from the list we try next time. When the application is done with the struct, it should free it by calling an appropriate free function exported by this new API.

So, now we have a list of addresses with valid TTLs as of the start of our connection attempt. Next step: start the connection attempt.

Pick the "first" address (chosen by whatever the ordering rules are) and send the connection request packet and let the OS do its normal retry schedule. Wait one second (system or sysctl configurable) or until the previous connection request was either accepted or rejected, whichever is shorter. If not connected yet, background it, pick the next address and send a connection request. Repeat until a one connection request has been issued to all possible destination addresses for the name.

Finished when:

1. Any of the pending connection requests completes (others are aborted).

2. The time out is reached (all pending request aborted).

Once a connection is established, this should be cached alongside the address and its TTL so that next time around that address can be tried first.

Seems mostly reasonable. I would consider possibly having some form of inverse exponential backoff on the initial connection attempts. Maybe wait 5 seconds for the first one before trying the second one and waiting 2 seconds, then 1 second if the third one hasn't connected, then bottoming out somewhere around 500ms for the remainder.

...

Since you were hell bent on calling the existing mechanisms broken rather than conceding the point that the current process is not broken, but, could stand some improvements in the library

I hold that if an architecture encourages a certain implementation mistake largely to the exclusion of correct implementations then that architecture is in some way broken. That error may be in a particular

I don't believe that the architecture encourages the implementation mistake. Rather, I think human behavior and our tendency not to seek proper understanding of the theory of operation of various things prior to implementing things which depend on them is more at fault. I suppose that you can argue that the API should be built to avoid that, but, we'll have to agree to disagree on that point. I think that low-level APIs (and this is a low-level API) have to be able to rely on the engineers that use them making the effort to understand the theory of operation. I believe that the fault here is the lack of a standardized higher-level API in some languages.

component, but it could be that the components themselves are correct. There could be in a missing component or the components could strung together in a way that doesn't work right. Regardless of the exact cause, there is an architecture level mistake which is the root cause of the consistently broken implementations.

I suppose by your definition this constitutes a missing component. I don't see it that way. I see it as a complete and functional system for a low-level API. There are high-level APIs available. As you have noted, some better than others. A standardized well-written high-level API would, indeed, be useful. However, that does not make the low-level API broken just because it is common for poorly trained users to make improper use of it. It is common for people using hammers to hit their thumbs. This does not mean that hammers are architecturally broken or that they should be re-engineered to have elaborate thumb-protection mechanisms. The fact that you can electrocute yourself by sticking a fork into a toaster while it is operating is likewise, not an indication that toasters are architecturally broken. It is precisely this attitude that has significantly increased the overhead and unnecessary expense of many systems while making product liability lawyers quite wealthy. Owen

On Mar 1, 2012, at 17:10, William Herrin <bill@herrin.us> wrote:

If took you 50 lines of code to do 'socket=connect("www.google.com",80,TCP);' and you still managed to produce a version which, due to the timeout on dead addresses, is worthless for any kind of interactive program like a web browser. And because that code isn't found in a system library, every single application programmer has to write it all over again.

I'm a fan of Rube Goldberg machines but that was ridiculous.

I'm thinking for this to work it would have to be 2 separate calls: Call 1 being to the resolver (using lwres, system resolver, or whatever you want to use) and returning an array of struct addrinfo- same as gai does currently. If applications need TTL/SRV/$NEWRR awareness it would be implemented here. Call 2 would be a "happy eyeballs" connect syscall (mconnect? In the spirit of sendmmsg) which accepts an array of struct addrinfo and returns an fd. In the case of O_NONBLOCK it would return a dummy fd (as non-blocking connects do currently) then once one of the connections finishes handshake the kernel connects it to the FD and signals writable to trigger select/poll/epoll. This allows developers to keep using the same loops (and most of the APIs) they're already comfortable with, keeps DNS out of the kernel, but hopefully provides a better and easier to use connect() experience, for SOCK_STREAM at least. It's not as neat as a single connect() accepting a name, but seems to be a happy medium and provides a standardized/predictable connect() experience without breaking existing APIs. ~Matt

On Thu, Mar 1, 2012 at 10:01 AM, Michael Thomas <mike@mtcc.com> wrote:

On 03/01/2012 06:26 AM, William Herrin wrote:

...

The even simpler approach: create an AF_NAME with a sockaddr struct that contains a hostname instead of an IPvX address. Then let connect() figure out the details of caching, TTLs, protocol and address selection, etc.  Such a connect() could even support a revised TCP stack which is able to retry with the other addresses at the first subsecond timeout rather than camping on each address in sequence for the typical system default of two minutes.

The effect of what you're recommending is to move all of this into the kernel, and in the process greatly expand its scope.

Hi Michael, libc != kernel. I want to move the action into the standard libraries where it can be done once and done well. A little kernel action on top to parallelize connection attempts where there are multiple candidate addresses would be gravy, but not required.

even if you did this, you'd be saddled with the same problem because nothing existing would use an AF_NAME.

It won't instantly fix everything so we shouldn't do it at all?

what if I want, say, asynchronous name resolution? What if I want to use SRV records? What if a new DNS RR comes around

Then you do it the long way, same as you do now. But in the 99% of the time that you're initiating a connection the "normal" way, you don't have to (badly) reinvent the wheel.

As far as I could tell, standard distos don't have libraries with lower level access to DNS (in my case, it needed to not block). Before positing a super-deluxe gethostbyxx that does addresses picking, etc, etc it would be better to lobby all of the distos to settle on a decomposed resolver library from which that and more could be built.

(A) Revised standards are -how- multiple OSes from multiple vendors coordinate the deployment of an identical capability. (B) Application programmers generally DO want the abstraction from "DNS" to "Name resolution." If there's an /etc/hosts name or a NIS name or a Windows name available, you ordinarily want to use it. You don't want to build extra code to search each name service independently any more than you want to build extra code to cycle through candidate addresses. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com  bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On Thu, Mar 1, 2012 at 1:32 PM, Michael Thomas <mike@mtcc.com> wrote:

On 03/01/2012 08:58 AM, William Herrin wrote:

...

libc != kernel. I want to move the action into the standard libraries where [resolve and connect] can be done once and done well. A little kernel action on top to parallelize connection attempts where there are multiple candidate addresses would be gravy, but not required.

connect(2) is a kernel level call just like open(2), etc. It may have a thin wrapper, but that's OS dependent, IIRC.

man connect 2:

"The connect() system call connects the socket referred to by the file descriptor..."

Then name the new one something else and document it in man section 3. Next objection? -Bill -- William D. Herrin ................ herrin@dirtside.com  bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On Mar 1, 2012, at 6:26 AM, William Herrin wrote:

On Thu, Mar 1, 2012 at 7:20 AM, Owen DeLong <owen@delong.com> wrote:

...

The simpler approach and perfectly viable without mucking up what is already implemented and working:

Don't keep returns from GAI/GNI around longer than it takes to cycle through your connect() loop immediately after the GAI/GNI call.

The even simpler approach: create an AF_NAME with a sockaddr struct that contains a hostname instead of an IPvX address. Then let connect() figure out the details of caching, TTLs, protocol and address selection, etc. Such a connect() could even support a revised TCP stack which is able to retry with the other addresses at the first subsecond timeout rather than camping on each address in sequence for the typical system default of two minutes.

That's not simpler for the following reasons: 1. It takes away abilities to manage the connect() process that some applications want. 2. It requires a rewrite of a whole lot of software built on the current mechanisms. Most systems provide a mechanism for overriding the timeout for connect(). Further, there are lots of classes, libraries, etc. that you can already use if you want to abstract the gai/gni + connect functionality. What exists isn't broken at the API level. Please stop trying to fix what is not broken. Owen

On Mon, Feb 27, 2012 at 7:28 AM, William Herrin <bill@herrin.us> wrote:

On Sun, Feb 26, 2012 at 7:02 PM, Randy Carpenter <rcarpen@network1.net> wrote:

...

...

On Feb 26, 2012, at 4:56 PM, Randy Carpenter wrote:

...

1. Full redundancy with instant failover to other hypervisor hosts upon hardware failure (I thought this was a given!)

This is actually a much harder problem to solve than it sounds, and gets progressively harder depending on what you mean by "failover".

At the very least, having two physical hosts capable of running your VM requires that your VM be stored on some kind of SAN (usually iSCSI based) storage system. Otherwise, two hosts have no way of accessing your VM's data if one were to die. This makes things an order of magnitude or higher more expensive.

This does not have to be true at all.  Even having a fully fault-tolerant SAN in addition to spare servers should not cost much more than having separate RAID arrays inside each of the server, when you are talking about 1,000s of server (which Rackspace certainly has)

Randy,

You're kidding, right?

SAN storage costs the better part of an order of magnitude more than server storage, which itself is several times more expensive than workstation storage. That's before you duplicate the SAN and set up the replication process so that cabinet and room level failures don't take you out.

This is clearly becoming a not-NANOG-ish thread, however... Failing to have central shared storage (iSCSI, NAS, SAN, whatever you prefer) fails the smell test on a local enterprise-grade virtualization cluster, much less a shared cloud service. Some people have done tricks with distributing the data using one of the research-ish shared filesystems, rather than separate shared storage. That can be made to work if the host OS model and its available shared filesystems work for you. Doesn't work for Vmware Vcenter / Vmotion-ish stuff as far as I know. There are plenty of people doing non-enterprise-grade virtualization. There's no mandate that you have the ability to migrate a virtual to another node in realtime or restart it immediately on another node if the first node dies suddenly. But anyone saying "we have a cloud" and not providing that type of service, is in marketing not engineering.

From a systems architecture point of view, you can't do that.

-- -george william herbert george.herbert@gmail.com

On Mon, Feb 27, 2012 at 2:19 PM, George Herbert <george.herbert@gmail.com> wrote:

Failing to have central shared storage (iSCSI, NAS, SAN, whatever you prefer) fails the smell test on a local enterprise-grade virtualization cluster, much less a shared cloud service.

Hi George, Why would you imagine that a $30/month virtual private server is built on an enterprise-grade virtualization cluster? You know what it costs to builds fibre channel SANs and blade servers and DR. In what universe does $30/mo per customer recover that cost during the useful life of the equipment? A VPS is 2012's version of 2002's web server + CGI and a unix shell. Quite useful but don't expect magic from it. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com  bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On Mon, Feb 27, 2012 at 7:03 PM, George Herbert <george.herbert@gmail.com> wrote:

On Mon, Feb 27, 2012 at 3:45 PM, William Herrin <bill@herrin.us> wrote:

...

universe does $30/mo per customer recover that cost during the useful life of the equipment? As I stated, one can either do it with SANs or with alternate storage.

Should not assume Rackspace et. all provide any level of fault tolerance for extraordinary situations such as hardware failure beyond what they have promised or advertised. IaaS provided redundancy is not always necessary, and may be unwanted in various situations due to its cost; single redundancy means a minimum of twice the cost of non-redundant (plus the overhead of failover coordination). With various computing applications it may make a great deal of sense to handle in software -- should a node fail, the software can detect a node failed, eject it, reassign its unfinished work units later. Typical Enterprise level fault tolerant SAN manufacturer prices seen are ~ $12 to $15/GB usable storage, for ~50 IOPS/TB, data mostly at rest, and SAN equipment has a useful lifetime of 5-6 years; a typical 200gb server then exceeds $30/month in intrinsic FT SAN hardware cost. There IS a place for IaaS providers to sell such product, probably at four or five times the $/month for a typical server. Just like there is a place for Network service providers to sell transport and network access products that have redundancy built into the product, such as protected circuits, multiple-port, dual WAN , that can sustain any single router failure, etc. Those network products still can't reliably guarantee 100% uptime for the service. There is a place for IaaS providers to sell products where they do NOT promise a level of reliability/fault tolerance or performance guarantee that requires them to utilize an Enterprise FC SAN or similar solution. Just like there is a place for NSPs to sell transport and network access products that will fail if a single router, card, port fails, or if there is a single fiber break or erroneously unplugged cable. This way the end user can save on their network connectivity costs; a tradeoff based on the impact of the difference between their network with 1% downtime and their network with 0.001% downtime; VERSUS the impact of the cost difference between those two options. End users may also prefer to implement their redundancy through dual-homing via multiple providers. It is very important that the end user and the provider's sales/marketing know exactly which kind of product each offering is.

 Amazon hit those price points with a custom distributed filesystem that's more akin to the research distributed filesystems than anything

Amazon is quite unique; developing a custom distributed filesystem is a rather extraordinary measure, that provides them an advantage when selling certain services. But even EC2 instance storage is not guaranteed. The instance storage is scratch space If your instance becomes degraded, and you need to restart it, what you get is a "clean" instance, matching the original image. EBS and S3 are another matter. The same provider that offers some 'unprotected' services, may also offer some more expensive storage services that have greater protections. -- -JH

[...] For DNS, EasyDNS (https://web.easydns.com/DNS_hosting.php) are rather good and not too expensive, and you can get a 100% up-time guarantee if you want. A review of them regarding availability is at http://www.theregister.co.uk/2012/01/31/why_i_use_easydns/

I have been a very satisfied EasyDNS customer for about a decade and concur with the article. Nothing is perfect, but the rapid response and support I've received have always been top-notch.

Do let us know who you end up picking and how it goes.

Indeed. "Cloud" outside of references to mists and objects in the sky is a completely meaningless term for operators. In fact, it has made it harder to differentiate between services (which I'm sure is the point). As an operator (knowing how things can be subject to accelerated roll-out when $business feels they are missing out), I wonder if a lot of these "cloud" service bumps-in-the-road aren't just a symptom of not being fully baked in. ~JasonG

Pardon the weird question:

Is the DNS service authoritative or recursive? If auth, you can solve this a few ways, either by giving the DNS name people point to multiple AAAA (and A) records pointing at a diverse set of instances.

Authoritative. But, also not the only thing that we are running that needs some geographic and route diversity.

DNS is designed to work around a host being down. Same goes for MX and several other services. While it may make the service slightly slower, it's certainly not the end of the world.

Oh, how I wish this were true in practice. If I had a dollar for every time we had serious issues because one of a few authoritative DNS servers was not responding... OK, I wouldn't be rich, but this happens all the time. Caching servers out on the net get a "non-answer" because the server they chose to ask was down, and it caches that. They shouldn't do that, but they do, and there's nothing that can be done about it. -Randy