Golem Yagna 0.12 - A vast, new world of exciting possibilities

Get ready for a vast, new world of exciting possibilities.

Our New Network Driver is finally here and will bring groundbreaking changes to the Golem Network. With the new Yagna 0.12 Marble Castle, we’re set to deliver a new and improved Golem Network implementation. This update will include a wide range of dynamic features, as well as bug fixes, enhanced stability, and functionality upgrades.

Faster. Better. Stronger. More responsive. Thanks to the New Network Driver, you will enjoy faster file transfer speeds and reduced response times once you upgrade to our new release. The ability to transfer more data combined with lower latency will help unlock the potential of the Golem Network.

This is a significant step for Golem and one giant leap towards dApps.

Are you ready for the next generation? We have listened to your feedback and have been working hard behind the scenes to deliver a new and improved experience with multiple ‘quality of life’ benefits for users and a number of exciting experimental features.

Alongside Yagna 0.12, we’re also releasing a new version - 0.10 - of Yapapi, Golem’s Python API. The update introduces many improvements and essential features. The most significant is the addition of outbound network connectivity from VM runtimes, which will enable applications running on the provider nodes to access external web services.

For now, apps created by the community will only be able to access addresses included in a default whitelist shared by all the providers. The whitelist consists of Ethereum nodes, popular code and image repositories, and a select few public APIs. The main reason for this was security, in particular safeguarding Providers as much as possible. In the future, we’ll enable access to arbitrary addresses under certain conditions. This experimental feature is just the beginning. We will continue to explore and develop this further and have many additional ideas we want to experiment with in the future.

Another essential feature is the addition of a generic TCP socket proxy. Similarly to the already-available HTTP proxy, it now enables connections to all kinds of TCP services running on providers through a port exposed on the requestor’s machine without any intermediary tools.

On the one hand, it expands the range of applications that can be created for Golem, and on the other, it allows for more convenient development and debugging.

Last but not least, we’ve included several minor improvements and fixes that should make the development of applications on Golem faster and easier than ever before.

Now, let’s take a more in-depth look at the technical details of our new releases and implementation.

One Giant Leap: Welcome to the new Golem implementation

In a significant step toward the future of the Golem Network and to provide wide-ranging benefits for users, better scalability, and make possible experimental ideas and features, we have completely redesigned the current protocol. This has required us to migrate to a new implementation, which will bring a number of exciting and innovative improvements to your experience. There are also important changes for you to be aware of. Let’s take a look:

• First, once you have upgraded your Golem provider to our new and improved yagna v0.12, it will appear by default in the new “public” subnet and you'll no longer have the opportunity to give or receive tasks with users on older versions.
  
• The subnets - "public-beta" and "devnet-beta" - will remain only for continuity purposes with previous Yagna versions. However, we won’t be developing them any further. Instead, we’re building for the future.

• In a new ‘quality of life’ improvement based directly on your feedback, you will be automatically assigned to the new “public” network. As a result, don’t be surprised if, once you upgrade to the new version, your nodes are only visible on https://stats.golem.network/ on the “public” subnet.
• For your awareness, the new subnet “public” will be used both for the mainnet nodes and for the testnet ones.
• The main benefit is that you only have to use the right `payment network` in order to proceed with testnet/mainnet payments instead of using different subnets.
  

• We expect the old network to become smaller as Requestors gradually switch to the new version. Over time, there will be more tasks in the new network and fewer in the old. This is a natural progression as we move forward into a bold and dynamic future.

Port forwarding

To help the Golem Network grow, become more resilient, and increase its stability, it needs more nodes with public IP addresses. This is helpful whether you are running a Provider or Requestor node. Golem can't automatically configure your router to open port 11500. As a result, if you wish to contribute to this effort, you will need to manually set it up. For router-specific instructions on how to forward your ports, click here to go to our handbook and check for instructions. Don't worry though, if you are unable to get a public IP or open a port. This is completely optional and won’t prevent you from continuing to participate in the network.

Experimental outbound network access

Alongside Yagna 0.12, we’re releasing a new version - 0.10 - of Yapapi, Golem’s Python API. This new update comes with a set of both gradual improvements and important new features. These will expand the range of applications that can be developed on Golem and make the process easier by adding to the toolset developers have at their disposal.

For some time now, we’ve been aiming to give virtual machines access to the internet, knowing it would open a vast world of new possibilities. In particular, this will help transform the types of apps that can be created on Golem. As a result, one of the important enhancements that Yagna 0.12 and Yapapi 0.10 introduces is experimental support for outbound internet access from Golem VM runtimes.

We’re aware that giving access to all locations without any limits would make Golem a dream-come-true platform for botnets of all kinds. However, when we were designing this feature, we were faced by a real challenge. How could we balance the desire to give Requestors as many capabilities as possible while also safeguarding the security of Providers? Although enabling outbound internet access was not a problem from a technical perspective, there were many potential security issues to consider relating to this feature. For example, it could, in theory, allow Requestors to perpetrate malicious behavior, such as impersonating a Provider using their IP address. Ultimately, these security considerations guided the design decisions and initial limits we have placed on the current version.

The new implementation, at least in its current form, is the result of this need to find balance and is the first step to better outbound network support. We wanted to release an experimental version of this feature for now while we continue to explore and experiment our many ideas to advance this concept in the future. These additional ideas will open up even more possibilities while minimizing the risk even further.

For now, as a first step, the new release makes this feature experimentally available in two flavors. First, we’re adding a curated whitelist of locations that Providers will have set up by default, which can be accessed by any VM payload. The Requestors will also have another option - to reach out to us for a certificate that will allow their applications to be audited and explicitly cleared for outbound network access to specific network addresses. As well as restricting access to specific network addresses, this also restricts commands that Requestors can execute using this virtual image. The image can then be audited for security, and if restrictions were set, it could be considered safer to use. Alongside this feature, comes a new example - external-api-request - which showcases the latter type of usage.

The experimental outbound network feature is an exciting step forward that many of you have been asking about for a long time now and we’re pleased to be able to deliver this in our new release.

Generic TCP socket proxy

A few releases back, we made the local http proxy available to expose HTTP-based services through a local port. Since then, it has become clear that other types of services would also benefit from a similar mechanism.

As a result, in this release, we’re adding SocketProxy - a module that does exactly that. In other words, it exposes a local port on a Requestor’s machine and forwards all the traffic to and from a remote Provider node through it.

In effect, any type of service - be it an SSH server, a remote SQL database, or redis cache - can be accessed through a port on the Requestor’s machine without extra, intermediary tools. It just needs to be a TCP service running inside a VM.

We have also updated our SSH example to show the usage of this feature.

An important point to make is that what we’re adding here does not enable direct, inbound access to services running on the Provider’s own IP addresses. This will be a separate feature that we aim to make available in a future release.

Minor enhancements

IP address removal and recycling

While our initial support for Golem VPN allowed a user to define the IP addresses of launched services, it only allowed those addresses to be assigned once when a given service instance was started.

Even though it works well in a happy-path scenario when each of the commissioned services launches successfully the first time around, it becomes problematic when there is a need to restart an instance, either because we want to change it, or just because it failed to initialize properly, and we want to relaunch it on another provider node.

The new update enables the user to remove the IP address mapping for each of the nodes in the network, but it also automatically frees all the IP addresses when instances are terminated. Additionally, failing services with statically-assigned IPs are now able to be restarted correctly.

More convenient Service restart hooks

In the previous release, we added the Service.reset hook that allowed the application authors to define operations needed for a Service to be brought back to an initial state when an instance was about to be restarted.

That said, the engine didn’t give the user a convenient way to define the exact scenarios when instances should be restarted - the previous, respawn_condition hook was not easily available outside of the API itself.

As such, we have decided to define an explicit location to enable such customized behavior, namely the Service.restart_condition property.

The default condition is consistent with the previous, built-in behavior of the engine which triggers a restart only if the instance it had not been successfully started.  

Application authors can now override this default to provide their own, tailored criteria for restart.

Customized debit note and invoice acceptance criteria

Up until now, the Requestor agent always accepted the full amount specified in the invoices and debit notes coming from the Providers.

The new update adds the debit_note_accepted_amount and invoice_accepted_amount hooks to MarketStrategy. This enables the Requestor agent authors to define their own criteria and calculations that determine whether incoming debit notes and invoices are accepted, either in full or partially.

Automatic service activity health-check

Before, the Requestor agent had no way of knowing that a given service stopped being available, unless the launched Service instance interacted with the underlying activity on the Provider node.

In this release, we have added a background task to query the activity for its state. Thanks to that, the Requestor is notified of a given service’s failure as soon as it happens. This allows the application authors to program recovery behavior into their code more efficiently.

Thank you!

Thank you for reading our New Network Driver and Yagna 0.12 announcement. We hope you’re excited about updating to the new implementation after reading this blog. Remember that we couldn’t do any of this without you - our loyal and passionate community - and many of these exciting updates are a result of your feedback.

We’re also grateful to everyone who took part in the beta testing for our new release, and look forward to receiving your thoughts on the wealth of improvements and new features we have introduced. As we have explained in-depth above, this update will bring a huge number of exciting and important ‘quality of life’ enhancements and benefits to users of the Golem Network. More importantly, the possibilities it offers for the future are endless.

As always, for all the latest news and updates, and if you have any feedback or questions about the new release, please join us on Discord. Until next time, keep in touch and keep creating!