The Cloud is the New Operating System

“Many years ago it became abundantly clear that some way had to be found to shield programmers from the complexity of the hardware… [T]his layer is the operating system.”—Andrew Tanenbaum, Operating Systems

The cloud is the new operating system for enterprises, and services are the new applications. The cloud provides the computing fabric upon which the next generation of services, from Pinterest to Instagram, foursquare to AirBnB, are being built. Just as Microsoft Windows and MacOS X have provided interfaces for the previous generation of desktop applications now on the decline, cloud providers like Amazon offer interfaces for the compute, storage, and networking these services require.

The cloud is a more fault-tolerant and flexible operating system than its predecessors. These two advantages derive from the cloud’s two hallmark features: it is both virtualized and distributed. Because it’s virtualized, failing hardware can be upgraded or swapped out, and virtual processes can be migrated to new machines with little end-user impact. Because it’s distributed across thousands of commodity boxes, services’ compute and bandwidth needs can be scaled up or down, and disk storage limitations are almost an anachronism.

In other ways the cloud raises new challenges, or rather returns us to the pre-PC era of client-server relations. If the cloud is a more powerful mainframe, then today’s clients are more intelligent than the dumb terminals of the past. The new clients are smart phones, tablets, and modern web browsers, whose local caching and compute capabilities are the essence of rich, responsive applications. But managing state between client and server, and across different classes of clients, introduces complexity for developers.

This new age of computing opens up new markets and opportunities. And the defining battle is: who will become the standard platform for cloud computing power?

The stakes are massive, and not simply because of the burgeoning spending on compute as a utility. Also at stake is control, as with past operating system battles, over the applications that run on cloud platforms.

Amazon Web Services:  The undisputed cloud champion

Amazon’s Web Services (AWS) has taken an early lead. AWS provides a growing set of peripherals beyond its core Elastic Compute Cloud offering, such as block storage, load balancing and content delivery networks. With almost 80 percent growth and estimated revenues of nearly $1 billion in 2011, AWS gives a glimpse of the market opportunity ahead. At a recent meeting of 22 startup chief technical officers, I asked how many were using AWS: every single hand went up.

But Amazon’s success also reveals concerns for its partners.  As Amazon’s expands its breadth of offerings up the stack, it threatens to cannibalize — as Microsoft did a generation before — many of the companies who have enriched its platform.

Microsoft Azure and the desktop’s decline

Few firms understand the threat and opportunity posed by this shift as well as Microsoft. So its late but commanding entrance into the cloud platform space, with Azure, should come as no surprise. Microsoft also has strategic leverage in offering a natural bridge from the desktop, where they currently dominate, to the new frontier of the cloud. Microsoft can help desktop developers, already comfortable with their programming APIs and runtime environment, extend their applications. Likewise the company’s current customer base can be conveyed smoothly into an embrace of cloud services.

Microsoft recognizes that the great desktop disruption is underway.  One-by-one, desktop applications are being displaced by cloud as-a-service equivalents in office productivity, financial accounting, and even photo editing. These services incorporate such features as seamless syncing across devices (Evernote) and social, real-time collaboration (Zendesk, Google Apps) that are naturally enabled by the cloud’s server-centric architecture.

Google’s App Engine: A stalling platform

Google has been the most visible and early leader in developing applications-as-services for consumers, through Gmail and the Google Apps suite. Yet Google has been less successful in developing its cloud platform, App Engine, for developers. Cloud platforms, like operating systems, succeed when they provide the right tools to application developers.

Compared with those of Microsoft Azure and Amazon Web Services, App Engine’s developer tools are higher-level, more Google-specific, and more tightly constrained.  Only select programming languages are supported and storage backends are limited. The result is that App Engine has seen relatively little adoption from serious developers.

Despite Google’s strength in creating its own cloud services, and its growing presence — via Android — in the client-side ecosystem, it must shift gears if it hopes to compete with a cloud platform offering.

EMC’s strength:  In data we trust

EMC is our final big contender for cloud dominance. Unlike Amazon, Microsoft, and Google, EMC sits far away from the consumer-side of IT and squarely faces the enterprise. EMC’s principle strength is in data. Its servers already play host to precious data for the world’s biggest enterprises. This is a powerful strategic advantage, because data is both at the core of the cloud’s value and the source of its vulnerability.

The cloud is fundamentally a data operating system (as Tim O’Reilly called the Internet).  The value of cloud services lies in being connected — via email, social media, business dashboards — to the pulsating digital streams of the planet.

But data’s increased connectivity also creates vulnerability; witness the spate of stories documenting consumer privacy breaches, more of which are sure to come. For enterprises, it is difficult to underestimate the sensitivity around data security. Data is a uniquely valuable and non-fungible corporate asset, more sensitive to theft and loss than even money.

EMC has already won the hard-earned trust of enterprises in handling their precious data. EMC cloud offerings play on this trust. In contrast to Amazon’s public cloud, EMC offers Virtual Private Cloud (VPC), smartly recognizing that the term “public cloud” has about as much appeal to corporate executives as “public toilet” and “public pool.”

The VPC offering highlights another one of EMC’s strengths: its depth in virtualization, the defining technology of the cloud. EMC was early to recognize the revolutionary impact of virtualizing data centers, acquiring VMWare in 2004, long before its value was widely understood.

The shape of clouds to come

The race to become the pre-eminent cloud platform is pivotal because, like other platform races, this one has winner-take-all market dynamics. Once a startup has its data on Amazon’s S3 storage service, spinning up its Hadoop clusters within Amazon is a cost-efficient choice (lower bandwidth costs), as is working with other service providers already on Amazon.

The perils of platform lock-in has spurred the growth of some open-source standards for cloud computing, most notably OpenStack, which has garnered the support of IBM, Rackspace, and as of last week’s announcement, HP.

Which cloud platform gains dominance and market share will be determined by a mixture of perceived momentum, migration costs, security, and developer friendliness (API design, standards, documentation). The winner stands to reap the massive financial rewards of being a foundational operating system for the next generation of native cloud applications and services.

Regardless of which platform ultimately prevails, forward-thinking firms will continue to enjoy the benefits of the storage, compute, and networking abstractions that a cloud operating system provides, while focusing their creative energies higher up on the stack. The result will be a blossoming of application as-a-service startups whose disruptive impact, on existing technology firms and on the economy as a whole, we are only just beginning to appreciate.

An earlier version of this post originally ran on the GigaOm network.