Decentralized Data Storage’s Only Competition Is the Public Cloud

This week, the Filecoin mainnet finally went live. This is huge news, not only because of the anticipation following Filecoin’s large token offering in 2017, but because creating a vibrant ecosystem of decentralized cloud offerings is an important step in the maturation of the cloud.

I suspect my feelings on this development may surprise a few people, given that it means a high-profile organization is entering into direct competition with my company, Storj Labs. Instead, I see it as the further evolution of a necessary industry, one that we have helped lead for several years now. 

Equally important is strengthening the cantilever against the extreme concentration in the public cloud market. The top four providers (Amazon, Microsoft, Google and Alibaba) control 80% of the public cloud market, and are all among the top 10 global companies by market cap. We want the decentralized cloud to have multiple strong players who take different approaches, drive better economics for users and create innovation.

Storj Labs, and a number of others (including Sia, Maidsafe and now Filecoin) are betting that a decentralized approach to storage can revolutionize public cloud in much the same way that a decentralized approach to networks (i.e. the internet) revolutionized communications over the past three decades. 

Rather than relying on expensive, capital-intensive data centers, we break data into small pieces, encrypt it and allow the data to be stored across 10s of thousands (and, hopefully soon, millions) of independently owned and operated drives. Since going into general availability in March, Storj has managed to create the equivalent of 26 data centers worth of capacity across a global network of thousands of independently run drives, spread across 85 countries, without taking a single plane trip, renting any space, buying any equipment, pouring any concrete, or, critically, consuming any extra power.

Decentralized storage and cloud must significantly decrease prices that end users pay.

Of course, the other advantage of having multiple players like Filecoin in a market is that it helps drive innovation and allows the market to see multiple approaches to solving a problem. Each of the decentralized storage players has its own unique areas of emphasis. Over the 16 months since Storj launched its first public beta of the V3 network, and in the six months that we’ve been in GA production, we’ve learned a few lessons.

We saw with Storj’s V2 network that it was relatively easy to build a large amount of storage capacity. But to ensure 100% file durability, 99.99% file availability and performance on par with the centralized cloud providers takes lots of hard work and iteration. 

For the decentralized cloud to be more than a science experiment, it has to support mainstream and enterprise use cases well. This meant becoming compatible with the Amazon Web Services out of the gate, since that is by far the most frequently used interface for mainstream cloud applications. It also meant supporting integrations with a wide range of databases, replication tools and backup tools. 

There is also increasing momentum for the use of decentralized storage for edge use cases, which aim to push data closer to where it is consumed. As an increasing amount of data is collected, analyzed and distributed at the edge, it makes sense to store that data across a broad, globally distributed set of nodes as well, which improves performance, durability and data integrity.

Balancing two markets

Imagine how long Lyft would last if it only focused on drivers but didn’t have passengers. Or, conversely, if it had lots of passengers but no drivers. Decentralized networks need to get the balance right as well. While many decentralized networks say a lot about miners and node operators, they don’t necessarily have a lot to say about real-world usage of the network. This is fine during a test or startup stage, but long-term viability requires end users get sufficient value from the network and that supply and demand stay in sync.

To this end, decentralized storage and cloud must significantly decrease prices that end users pay. To do so, however, it has to be easy and economically viable to be a node operator. We purposely designed our network so network operators wouldn’t have to buy specialized equipment, consume large incremental amounts of power or expend time and effort on computational tasks that aren’t directly related to providing uptime, storage and connectivity. 

We wanted to make it computationally expensive to cheat, but not computationally expensive to operate correctly. Having profitable, low-cost supply lets the network deliver storage at a fraction of the cost of the large, centralized providers.

The ability for a wide cross section of operators to provide storage profitably means we have a very broad cross section of people and organizations in 85 countries offering storage, with no large concentrations of nodes in any organization or geography. This is the dream of decentralization.

While we do have several corporate and university data centers bringing nodes to the network, the heart of the network remains the individual operator. In fact, the majority of the nodes on our network are operated by people who are operating two or fewer nodes. Over 80% of the pieces of data are stored on these nodes.

Part of the beauty of decentralized storage is there isn’t a single point of failure. Done correctly, no one should be able to see, mine or sell end user data. But that not only requires good decentralized design but also careful attention to things like meta data masking, end-to-end encryption and more. 

Once again, a hearty welcome to the latest entrant to the decentralized storage space. Exciting times lie ahead.