11+ after 11.11

Golem Project -

by Julian Zawistowski (CEO) and Piotr ‘Viggith’ Janiuk (CTO)

Anniversaries serve a social function. They allow us to pause, take stock and reflect on all that has come before us. To celebrate, or commiserate and to understand how our lives have changed.

Well, we just passed eleven months since our crowdfunding on 11.11.2016. And it might be a cliche to say it but what happened last year changed our lives forever. Yes, it’s a truism; every event changes our lives, and in today’s world, change is itself remorseless. However it’s a rare opportunity when you are finally afforded the chance to create the very thing you have been dreaming about your entire adult life.

In a few weeks we are about to hit one year, so what better moment to take stock, reflect and share with you not just a detailed assessment of the current state of Golem but also our re-worked roadmap for the next few months as we enter the homestretch to Brass.

Delivering Brass — milestones on the road to release

Let’s start with the best news. The research phase for Brass Golem is over, and all that remains is coding and testing. We now have a great understanding of what is left to be done. This is so important because when you are a pioneer in a given field knowing the shape of the road ahead and how that path is plotted is as important as knowing how far along you are on it. Research is the way we have figured out which path to take. Call it knowledge reconnaissance if you will. So for us to declare that our research stage for Brass is completed, is to have crossed a big milestone.

For a long while we’ve represented our progress with Github issue tracking which was far from ideal. Now as we near Brass we have decided to replace that with regularly updated Kanban boards in Trello. We are not giving completion dates yet, but Golem’s first production, our Brass rendering application which will sit atop our network, is very close. The beauty of the Kanban system is that you don’t have to take our word for it, you can see this yourself. And after our previous delays, we wanted to find a way to ensure the eyes of the community could peer right into our dev team’s work as practically as possible. To prove that we are seriously thinking about bringing this sense of transparency to another level, we would like to invite you to our AMA session (we will start answering your questions tomorrow, 18th October, 6 pm CEST) which should become regular.

.

As you can see from the graphic above, the first significant milestone is to improve Golem UX and create a version that can be really useful for early Blender users. It will still be on the testnet so no real economics will be introduced yet. UX will not be as refined we’d like it to be.

There are lots of elements to this first milestone so let’s break this down. We will have to slightly tweak our P2P network called P2P2. (The discussion about devp2p vs P2P2 is one for a later blogpost). And as part of our UX work we’ll have to improve our app installation & configuration process and make sure that tasks are “persistent”. We also need to collect more precise information from the network to understand much better why computations fail.

Afterwards we are going to focus on some other additional issues and implementing Concent. Concent is a new approach to add resilience to the Golem protocol and, despite being designed as a voluntary and additional part of the system, in the first release it will play an important role in securing parties’ interests in the network (see below).

Once we are ready to switch to the mainnet (after two iterations of Concent testnet), we are going to conduct an external audit and move to production. And that’s it. We’ll have reached Brass. Again, we are not giving precise dates yet, but we want you to see some major progress this year.

Trusting decentralised networks

During the course of Brass development we have also had to address quite a few security issues. Being able to trust a decentralised network in which real value is exchanged is always going to be key to it being successful.

We also faced similar types of challenges relating to the quality of the computation users would receive in a distributed, trustless network. To get technical here, our first production use-case is Monte Carlo rendering which is arguably one of the most prominent problems that can be solved in what’s known as a map-reduce framework.

But from a systems perspective, rendering is a problem which is inherently non-deterministic in its nature. To put that simply, when a user asks the network for their work to be rendered they themselves don’t know what the final output should look like. That’s a real challenge because in that situation, the user might well believe that the output they’ve been returned is the one they really should have gotten. That leaves a lot of room for people to cheat a system with no central authority or arbiter.

This major conundrum forced us to develop ways of trying to maximize the probability of obtaining the correct results for a task computed by the network. But as there are no efficient frameworks for trustless computation which could be used to run a photorealistic renderer, the “maximizing probability” approach only takes you so far before you need another solution altogether. We realised that adding components with higher trust guarantees would push this trust boundary much further.

Another absolutely central piece of Golem’s software is our transaction framework. It has to be cost-effective, reliable and relatively fast. During our research we devised a number of clever solutions and strategies to build this (we will publish more information about this in the very near future). Some of them might well be used in production, but a number of our fixes have to wait for Ethereum to scale up. We also believe that the transaction framework could benefit from an additional trusted component.

Last but not least, communicating with developers of proprietary, commercial rendering engines has deepened our insights into all types of licensed software and how they can be integrated into the Golem network.

Those lessons can be summarized like this:

  • Existing license server solutions are mostly centralized and guarantee quite high levels of trust.
  • Golem has to provide straightforward logic for interaction between 3rd party license infrastructure and the network itself.
  • In order to make Golem’s integrated solution more attractive, the final service should provide additional logic for verifying results.
  • There is no guarantee of 100% uptime, but, one can assume high availability of the service.

When considering commercial integrations with Golem, we concluded that integrating verification and transaction logic to the existing licensing solutions and then making that publicly available presented the best way of maintaining high levels of trust associated with providers of commercial software.

Welcome to Concent

We have named the corresponding infrastructure (for both open source and commercial software) Concent. The word was taken from “Anathem”, a novel by Neal Stephenson and refers to a community of highly-specialized individuals who, if you’re lucky enough to ask them at the right moment about the issue in their area of specialty, should be able to respond in a reasonable time with a precise and valid answer.

This description fits our requirements almost perfectly. Concent in Golem is implemented as an optional, special purpose service which increases the network reliability, albeit it is not guaranteed to answer every question nor to have 100% uptime. (Uptime and quality guarantees may be expected to be even better when Concent is implemented for commercial integrations e.g. commercial rendering engines).

We’ve been working on this idea for few months now and at this stage we have prepared a full specification for Brass and the development is underway. At the beginning, the service will be implemented in a partially centralized (but scalable) manner.

We believe that this is the right way to proceed because of the low entry threshold for the developers who would like to integrate with Golem in the future. And this applies to both commercial and open source integrations. But this is just the first step and our long-term goal is to prepare the framework so that it is not only scalable but also fully decentralized.

This issue can be approached from a number of angles; private blockchains with checkpointing in Ethereum; supernodes with higher trust guarantees; maintaining global reputation for Concent and so on. But all these ideas require a lot of research and testing and our gut feeling is that our production-ready solution should be both conceptually simpler and safer.

Our end goal is to provide a decentralised solution that is both safe and convenient to use by open source communities as well as proprietary software developers.

If we could add a trusted computation component to Concent and run it locally on an arbitrary machine, then most of the issues around centralisation would be resolved whilst also giving us a higher degree of trust and quality of the service. Amazingly such technology already exists and is gaining increasing recognition. For example our partners at Qubes OS recently suggested that we can use SGX. This neat piece of technology might well give us exactly what we need; a trusted computation component in an untrusted environment. However SGX is not developed in a way that we could just pull it off the shelf and make it service our needs.

Components like SGX can be used not only to benefit overall levels of security in the Golem network but also to unify the way Concent is implemented and accessed. (Interestingly nodes equipped with SGX ready processors could serve as either providers or verifiers or as special purpose nodes implementing a distributed version of Concent).

This is especially important because in such a setting, both proprietary and open source integrations would use exactly the same integration framework. In simpler terms, the underlying Concent code would not have to be reworked for every new application running on top of the Golem network. We hope our thinking around this satisfies those of you who are entirely vested in a decentralised solutions without any centralised components.

What can you do to support the project?

1. Stay with us!

We are late compared to what we’ve planned and we know that. But Golem is one of the most complex projects around. Other projects in the blockchain space are also not in the production yet. So, even if we are not up to our own and yours expectations, we believe our relative performance is pretty damn good.

It hasn’t come as quickly as we wanted that’s for sure, but it will be with us soon enough. And now you can monitor our progress here — so even if there are no dates, we hope this will be illustrative enough!

2. Join our test!

Simply running the node is a great help and detailed feedback on eg. installation problems is priceless. Here you can read how to do that.

3. Contribute to our open source repository!

Check those links:

If you’re a developer, you can help us resolve issues, but you can also help us by adding to our wiki and getting stuck in on Rocket. Big thanks for some github users who have already helped us, eg. matthewsedam, mitchellcash or bloose007.

4. Support cooperation in the blockchain ecosystem

Most projects in the cryptospace are late and struggle with similar problems. We believe that closer cooperation between projects will help us speed up development of the whole blockchain ecosystem. The old adage that a problem shared is a problem halved, has never been truer.

Finally, it goes without saying that for both of us and the rest of Golem’s founding team of developers, these last 11 months have been the most exciting and challenging any of us have known.