Golem Stats - A timeline of community input and collaboration

Mattias Nystrom

• 4 min read

Data collection can be a challenge to tackle for peer-to-peer networks. For a P2P marketplace such as Golem however, this information can be a useful tool for both providers and requestors in order to help individual participants build a pricing strategy and compare themselves to the rest of the market.

We’ve come a long way from Golem Brass and Clay, not just from the old stats page. In this blog, we’re going to go through what we’ve built and the power in community feedback.

Building on the above, we don't only want to highlight that we’ve made the Stats page better as a topic on it’s own. What’s been created is a product of how Yagna has been implemented, particularly in how metrics are reported, the work on the APIs (yapapi is primarily utilized for the Stats page), the growth and feedback of the community. These factors combined have had an influence in making the creation of such a page possible to become what it is now. With the help of the community and the dedication of Phillip for gathering the feedback and working to incorporate it into the new page, we’re excited to share a timeline of how it progressed.

A timeline of Golem Network Statistics

The new Stats page (now found at stats.golem.network) started out experimentally and quite simple, it gathered the provider nodes list on the by utilizing the yapapi low-level api list-offers.py example and displayed it on the previous URL golemstats.com (still up temporarily), which can be seen below:

It started out with adding improvements such as showing provider pricing in the online nodes table, average pricing per online node and cores were now shown by the threads values sent by the network.

Gradually, new features were added, giving the ability to click the provider node in the online table to get a detailed view of the node without much (yet!) exciting information included.

The performance of the page was very limited, 20 concurrent visitors on the page caused it to crawl to its knees. Optimisations were created by removing synchronous functions that caused a bottleneck in requests made to the site. This was replaced with asynchronous functions that allows to serve user requests concurrently, meeting the growing user needs. As for features, more pricing information was included on what the providers of the network offered and from this point on the community feedback gathered started to pile into the growth of the page (chronologically):

  • Graphed timelines on provider computational active history.
  • Yagna version adoption graphing, Network utilization in percentage and the usual network utilization.
  • Provider pages showing how much income a node has generated in the past 24h, 7d, 31d and 1y. Along with a provider page listing if a node is currently computing.
  • Node by operator tab.
  • Icons and total earnings for nodes in the online table on the homepage. It filters the table by highest earnings -> lowest earnings by default.
  • Node by operator button on the individual provider pages.
  • Top right corner on desktop including a GLM/Dollar icon you can click to toggle prices on the site between GLM/USD value. On click it will change it in the next API query which happens within 15s.
  • Price graphs over time.
  • Node by operator displaying the following:
    - Total offline + online nodes,
    - Mainnet wallet buttons,
    - Total Cores, Total Memory, Total Disk, Total Earnings.
  • Filter buttons on the market page for each GLM pair.
  • Addition of the requestors tab, displaying requestors and the amount of tasks requested on the network. By default it'll show the 15 largest requestors.

So much more along the way of the progress above that makes it difficult to place in chronological order; live graph progression, historical data, redesign of provider pages, lots of fixes in formatting and stability updates.

The introduction of the Golem Stats API

Not mentioned yet was the creation of the Golem Stats API and documentation. This has made it easier for the community to make gathering statistics for themselves easier. It can be used to create more automatic pricing strategies or even statistics sharing onto other platforms.

Community use of API

A community creation and example project on how stats.golem.network API can be utilized has been started already in the form of the Reddit golem-stat-bot. It scrapes Reddit posts, and replies if specific keywords/commands are found. Started by Figurestudios, it now has two community members collaborating with aldin joining in! For those interested to get involved, active discussion around the project can be found in the #project-collaboration channel on Discord.

This is where the Stats of the network are now

We now have so much information and resources for us and the community to help create a healthier ecosystem and P2P marketplace, the statistics page on stats.golem.network shows 6 active tabs; Network, Live Graphs, Historical Data, Node by operator, Requestors, Market Data, and links to the API Documentation.

At time of writing there are 732 Providers (reaching ~900 at a peak one week ago), 15195 cores, 20.76 TB Memory (RAM) and 147.80 TB Disk.

Compare the new Stats page with the previous page for Brass and Clay and we feel we’ve made some massive improvements. So wrapping up, this was all made possible by the new and improved Golem implementation (Yagna), Phillip working tirelessly to build the page (frontend, backend, Yagna metric collection) and the community. The Stats page is on course for 80 million API requests in the next few minutes.

Naturally, the Stats page and Stats API will continue to progress, we're just wanting to share some of the timeline so far. We're looking forward to the project continue and for your feedback.

A special thank you to the community feedback participants

Here we have a list of community members who participated in offering their feedback and suggestions on the page. There was A LOT of community feedback so please let us know if you've missed you so we can add it to the list: @ErikSouth @blue-notes-robot @MSD @Nebula @jaspervv99 @reza @mixmafritz @stelballe @bannanadjoe @Phazed @agnologiotto @bose25 @kermit @ChrisChrisChris @r4x @anshumillionaire