Cloud failure and the myth of the decentralized Web

Not every cloud has a silver lining…

When the history of the Internet is written, June 30 2012 may be remembered as The Day The Cloud Failed. An important Amazon Elastic Compute Cloud (EC2) data centre in Virginia was affected by a powerful storm which knocked out its power (prompting variations of the headline “Real clouds brought down virtual ones”, oh the wit!). This particular cloud centre hosted important content for various Internet services, including Netflix, Pinterest, and Instagram, bringing them offline for up to six hours. You can insert your own Obi-Wan Kenobi meme here: “I felt a great disturbance in the Force, as if millions of [hipsters/housewives/cat pictures] suddenly cried out in terror and were suddenly silenced.”

This is not really an isolated incident, this particular data centre had already crashed this month, and it seems that the web is starting to get used to something that seemed unthinkable some years ago: the ghost of extended downtime. Cloud outages are becoming so common that websites (and Twitter accounts) have sprung up to document each new occurrence. No provider is immune, from Amazon to Microsoft, cloud failure is now something to be expected.

Perhaps all of these cloud failures are the growing pains of an Internet that continues to expand at almost exponential rate, all of those cat videos have to be stored somewhere after all. The problem is that cloud computing was precisely sold as a means to avoid downtime of the type that we are experiencing. As online businesses thrive, cloud services allow enterprises to rent storage space, database processing and application usage from providers which have the infrastructure to manage large numbers of users. The idea behind the largest cloud services is that storage and processing are distributed amongst different data centres, which in theory should make outages more difficult. The whole principle of distribution rests on the assumption of network resilience, if you knock down a server, others can take up the load. This is the one of the founding pillars of the Internet as we know it.

However, something has been happening to our distributed and decentralized model of the Web, it no longer describes what is happening. From a distributed model we have been migrating our content to more and more centralized services. The first type of centralization is geographic, of the top 10 cloud providers listed by a specialised website, all are based in the United States; and while some offer localised services, most of the content is hosted in the US. Similarly, the market share of the top providers is growing, with Amazon alone holding an estimated 15% of the cloud market. This does not imply that Amazon now hosts 15% of the Internet, but it means that a large number of important services are hosted by them, as evidenced by yesterday’s failure.

This is problematic for all sorts of reasons. The obvious one is that an increasingly centralized Internet is not as resilient to random events as a distributed one, because the failure of important nodes can bring about cascading failures in downstream services reliant on that hub. Moreover, a more centralized Web is more likely to suffer from regulatory control, which is one of the reasons why the growing importance of new legislation from a global perspective. Jaeger et al commented about this in a 2009 article on cloud policy:

“When asked this question, a technologist will surely chuckle and reply something akin to, “The location of the cloud is irrelevant. Anyone will be able to tap into the power of the cloud from anywhere.” This answer, while technically accurate, misses an important set of issues. The main thesis of this article is that cloud computing represents centralization of information and computing resources, which can be easily controlled by corporations and governments.”

Everything we have been experiencing since this paragraph was written only helps to corroborate that we are increasingly putting all of our collective eggs in a few baskets. From Amazon removing Wikileaks content from its cloud services, to the seizing of Megaupload data, the continuing reliance on fewer providers has made control of the Internet an easier task. While many open Internet advocates (myself included) concentrate on fighting SOPA, PIPA, ACTA, and CISPA, we seem to be missing the true danger. The death of the open and distributed Web happens one cloud computing agreement at a time.

So, whatever happened to the dream of an open web?