Golem Hive - building a cloud with the Task API (and then some more)

My name is Karol, and I’m building Golem Hive. I am working as a DevOps engineer at Golem Factory. This job requires to see through problems, analyze them and try to find optimal solutions that free up resources where possible. It’s also about simplification of things and eliminating bottlenecks. On this job, I need to look at things from many different abstract perspectives, know how the systems work, and how they interact with each other, on many different levels of abstraction. As a Golem DevOps, you also need to make sure that things run as efficiently as possible with the least amount of resources. At least, that's the ideal. Essentially you are trying to automate and optimize most of the inefficient processes that are occurring within Golem.

One of the perks in being at Golem is that you are allowed to dive into research, so I was able to experiment on the idea of a system built on top of the Golem Network.

My interest in Golem is within multiple areas, but one of the most promising is that Golem can be turned perhaps into the basis of a truly decentralized cloud. I wanted to explore what is possible currently with Golem and try out our new shiny TaskAPI. Many cool developments are coming to Golem in the near future, so I am really excited about what we can achieve.

Since Golem is Open Source, everyone can see what is possible with it right now.

I would like to tell you about my latest project: Golem Hive

Golem Hive can be thought of as the first layer of a decentralized cloud. It allows for creating a virtual network of Docker containers on top of the Golem Network. These are spread across multiple Golem Hive enabled nodes that fulfil resource requirements. Nodes are market price selected and settlements for usage will utilize Pay As You Go mechanism.

Golem Hive has some features of the traditional centralized cloud, but it is also something different. It is, of course, Open Source technology and built on the basis of slightly modified Golem and TaskAPI code. Hive has some parts that are not yet published because they are still too early in the development.

Currently, Golem Hive is a Proof of Concept of a meta use-case for Golem.

Golem Hive is an abstraction layer that enables traditional payloads built on centralized cloud and container management solutions (like e.g. Kubernetes or Docker Swarm) to be deployed in a truly decentralized fashion. The redundancy and availability are achieved by a selection of intelligent systems built into Golem Hive and replication of services in Kubernetes-like fashion.

Each selected Golem Provider (worker node - I like to call them Bees) is running a client-side Hive node. Which means that it forms a branch of topology in the star-topology configured network. The central point of this network is the Hive Coordination Node (Queen) which is run on Your Golem instance. This is a command and control node for a Hive. From here you manage all components of the system, your payloads, deployments, wallets, billings and can even access each Bee node via ssh in the web console.

Golem Hive gives you a REST API for controlling the management server and web GUI that is a convenient way to manage your system.

The way you can use Golem Hive is on-demand ephemeral Virtual Private Servers. They can run any kind of service chosen by you. It is automatically load-balanced and high-availability is achieved through service replication. There is no restriction on running only one node per service, but due to the nature of such a solution, reliability and data consistency cannot be guaranteed at all in such a scenario. The way high-availability of the service is achieved is through load-balancing via edge router running on the Coordination node.

As a developer, you get a private cloud environment that is at your disposal, fully connected, health-checked, load-balanced and ready to go in just a few clicks or through automated commands. Have thousands of nodes at your fingertips within a minute? No problem.

Or you can also run your web services in a decentralized backend, pointing your AWS Elastic Load-Balancer towards an Edge router that load-balances your decentralized web service running on Golem Hive. The possibilities are enormous.

What Golem Hive enables is a decentralized cloud-like pool of servers available on-demand, with specified resources required, fully auto-managed and connected, giving you a multi-node connected environment. You decide what you want to run on these servers and how much you want to pay for them. By enabling the "pay as you go" marketplace for TaskAPI it should be possible to rent out universal spare computing power on a minute basis.

So what is the current state of Golem Hive? The basics work, proof of concept has been verified and an internal demo was presented. It shows that essentially there are almost no technical limitations within Golem itself that would prevent further development of Golem Hive.

There's still lots of research, polishing, writing more code with a focus on the usability, ease of use and making familiar DevOps tools to work with Hive, but the concept works. There are still security, marketplace and risk-reward ratio analysis problems that need to be figured out before Golem Hive can be implemented as a mainstream part of Golem.

In its current state, due to the nature of Golem itself, it's unsecure for sensitive data, but there are ways for some payloads (like e.g. addressable static content) to be securely served.

Perhaps Golem Hive could also enable the development of decentralized applications and a whole new area of research on the decentralized computation from a perspective more related to traditional computing resources. Since you can run anything that runs under Docker, expose your services running on Golem Hive to the public network, who knows what exciting developments this might bring?