What the History of Linux Says About the Long Road to Decentralized Storage Adoption

The evolution of computing has been marked by a series of paradigm shifts, from mainframes to personal computers, and now to the cloud. Currently, the common path to deploying web infrastructure is to be a paying customer of a large company like Amazon or Google, and write infrastructure as code to spin up a constellation of interoperable services on their machines.

Claudia Richoux is the founder and CEO of Banyan, a decentralized file storage system built on top of the Filecoin network.

However, the current dominance of cloud hyperscalers, with their lion’s share of the world’s computation, is not a natural monopoly.

Instead, it’s a temporary state, as the open-source community works tirelessly to cook up a decentralized solution that will revolutionize the industry once again. A decentralized solution will allow users to freely move between service providers instead of being locked into one entity, which means more price competition and freedom for businesses to negotiate on terms with their cloud providers instead of the current state where they’re deeply entangled with and dependent on them.

The lessons of Linux

To understand the potential of a decentralized cloud, we need only look at the history of the Linux operating system. It took a decade for the Bells Labs developed software Unix to evolve into Minix, and another decade for Minix to become Linux. Finally, after yet another decade, Linux saw widespread enterprise adoption, becoming the obvious choice for developers.

Today, Linux open source software dominates the market, running on 80% of public servers and 100% of supercomputers. It’s what all devs inevitably choose when they spin up an EC2 (Elastic Compute Cloud), a service by Amazon that allows users to run applications in AWS’s public cloud.

Linux’s success was built on the power of compounding, shared, open development gains. Operating systems are monumental and intricate, requiring millions of hours of specialized labor, research and testing to create competitive ones from scratch. However, after years of government-funded and hobbyist-fueled incubation, Linux reached a usable state, and the benefits began to multiply.

As companies started using and depending on open-source operating systems, these massive operating system kernel codebases, or the the core of a computer’s operating system, became “the commons,” and people pitch in to improve it. Everyone who uses Linux kernel searches for bugs. Some even contribute patches.

In other words, open source software support can be cheaper than closed-source software support plans, just because the cost to create the product itself is so much smaller. The cost of Linux to a business is denominated in fractional developers lending a hand, or in support plans (for companies without in-house kernel expertise).

In contrast, the cost of building and maintaining a custom operating system is astronomical, requiring the controlling entity to cover the entire lifetime cost. This necessitates a high cost of consumption to make the endeavor economically viable. If multiple closed operating systems compete, each must fund its own development and find sufficient market share to recoup their substantial investments. This explains the state of the market today to a substantial extent.

The cloud is another operating system

The “cloud” as we use it today on hyperscalers is simply another operating system, a tool for abstracting away when and where jobs run, their resources and how they interact. Whether it’s a managed database, an open-source Kubernetes cluster or running a program on your computer, both clouds and operating systems are highly sophisticated and complex resource managers, schedulers and security providers.

The cloud operates at a much larger scale, making AWS, GCP, Azure and DigitalOcean the mainframe computers of our time, independently maintained within each company.

So, where is our open-source contender? OpenStack was developed by Rackspace and NASA in the 2010s as an open-source cloud stack intended to be deployed to any datacenter to turn it into your own AWS. It has been said to be dead (for reasons I’ll discuss later), but is still under active development by companies (mostly Chinese) that use it instead of AWS due to government infrastructure restrictions.

One thing we notice first is that open-source codebases are hard to kill because their knowledge and power are encoded in git repositories, not in fallible human processes. OpenStack will never actually be dead, even if the original business models based on it are long gone. This means that open-source contenders have a lot of chances to win, because they can just keep resurrecting.

Righting the wrongs from our last try

OpenStack “failed” because every single company wanted to have their own distribution of the software to control the development roadmap. This led to fragmentation and competition rather than collaboration. Most major vendors released their own OpenStack distributions, fracturing the ecosystem, and then fought for market supremacy (which prevented development and progress).

Lack of standardization across OpenStack distributions made it difficult for users to deploy and manage clouds consistently. Additionally, the complexity of OpenStack made it challenging to install and operate, especially compared to more polished public cloud offerings from Amazon, Microsoft and Google.

Without a strong central governing body, it was hard to drive a cohesive vision and rally the community around shared priorities. Many companies that initially supported OpenStack later reduced their investments or pulled out entirely.

How do we avoid failing again? This time, we have a secret weapon: nodes, consensus, block rewards, public goods and ecosystem alignment research. Node distribution is a mostly solved problem after 15 years of running blockchains, so the lack of standardization is something we can defeat.

By figuring out the right public goods mechanisms and how to align users, we can unite people around protocols instead of encouraging endless forking, competition and defection. Governance and shared roadmaps are a problem we’ve been continuously hacking on, and we’re getting there because the shared state and shared value of blockchains bind communities together and force them to collaborate (in ways that easily-forkable open-source repositories simply don’t).

We have the financial primitives and the collaboration tooling nearly ready to work. We already have promising starts, but we must keep working.

The road to the winning end-state

We’re closer than we realize to a decentralized cloud go-to-market, which could bring hundreds of billions of dollars to the ecosystem if we really capture the market the way Linux did. Cloud markets are also growing at 11% annually — and what can AI adoption do but accelerate that? It makes bitcoin ETFs look like lunch money.

There are some gaps holding us back from winning right now, though, and the place we’re the most sorely lacking is sensible product decisions. The current state of Web3 attempts at decentralized cloud services are questionable, with limited product-market fit. Most people use these protocols only to game their token incentives, which is unfortunate.

On the bright side, speculation can drive product improvement and growth cheaply, as long as we listen to user feedback and use reward mechanisms with care.

We may not need to reuse OpenStack directly, but we need to build something similarly usable (and similarly reliable, which is the difficult part). We will have to blend our home-grown incentive structures with Web2 and cloud hyperscaler monitoring, compliance and security technologies in order to succeed. We need to design hybrid centralized/decentralized SLAs (service level agreements — the contracts between you and a cloud storage provider). We need to secure it with a buffet of reliability incentives, security attestations, zero-knowledge proofs, fully homomorphic encryption, computation fraud proofs, governance protocols and more.

Most of us in this segment have prototypes and MVPs, shallowly cargo-culting Web2 cloud products’ successes, instead of thoroughly attempting to rival the incumbents by working with users and adapting to their needs. However, technically or theoretically doing storage or computing or CDN-speed delivery is not enough, which is where most of us are. This will not attract paying users, token inflation will run out, and you will die.

The thing that attracts paying users will be doing the product iteration cycle, evolving our offerings to meet their needs and going to market like a Web2 cloud company.

Go watch a YCombinator video or two. Choose to do the hard thing. Fortunately, the Web3 community is almost infinitely patient to test and give feedback, because we all want (need?) this to work. We’re competing with a mature and intimidatingly well-funded industry, and it’s going to be uphill for a few more years.

Look closely at projects like Banyan, Estuary and ArDrive, and ask yourself if you’d choose them over Dropbox, especially if your job depended on it. Now think about how much work needs to be done to get there. Still, decentralized cloud infrastructure isn’t a question of if, but when.