Golem Network ETH Solo Staking Tests: Summary

As stated in our June blog post, Golem Network has decided to stake its ETH to support the future growth and development of the project. With the initial milestones successfully achieved, we want to summarize the process and outline the forthcoming steps.

Objectives of the Staking Tests

The staking tests were conducted with the following objectives:

• Establish a secure and scalable method of depositing ETH into validators within our infrastructure
• Evaluate and optimize hardware and software components to ensure optimal performance and stability
• Identify and mitigate risks associated with Ethereum staking 
• Increase Ethereum Network clients' diversity

Securing Mainnet Tests

Given our decision to utilize a significant amount of ETH for testing within the Ethereum Mainnet, we prioritized high operational security. Our objective was to create a controlled environment where we could monitor the impact of various factors on the staking-related processes. To enhance the operational security of the tests conducted on the Ethereum Mainnet and to minimize the risk of spam transactions and interference with the process, we implemented the following measures:

• Utilized multiple, isolated staking pools with separate withdraw and deposit addresses
• Employed a combination of multisig contracts and regular Ethereum addresses to control the deposit and withdrawal addresses
• Routed transactions through centralized exchanges between Golem’s multisig and deposit addresses

Preparations 

• To deposit a substantial amount of ETH, we have implemented a solution based on the StakeFish BatchDeposit smart contract. (https://github.com/stakefish/eth2-batch-deposit)
  This solution underwent an internal audit, received approval, and was subsequently deployed on the blockchain with minor modifications.
• Multisig Contract: We have also decided to use the Gnosis multisig smart contract to manage the yield from a subset of withdrawal addresses (https://github.com/gnosis/MultiSigWallet) 
• We decided to compare the following clients:
  • Execution Layer: 
    • Geth
    • Nethermind
    • Erigon
  • Consensus Layer:
    • Lighthouse
    • Nimbus
• During the process of selecting the final clients and their proportions, we took into consideration the importance of maintaining Ethereum client diversity

Phases (in chronological order)

1. Holesky Testnet

We initiated the first round of staking tests on the Holesky Testnet and deployed a multisig wallet to facilitate staking deposits.

Deposit address: 0x899d2c70240a997394e55c95dd0a294d04db483d
Withdrawal address: 0xac7571242d095abea8f463da7b7b4351271346b6

Utilizing our version of the BatchDeposit smart contract, we initiated staking for multiple validators. The tests were successful, with the validators recognizing the deposits and actively participating in the Proof of Stake protocol. After running this configuration for several days, we have decided to proceed to the next phase.

2. Mainnet - “Sanity” Tests with 5 Validators

To verify our configuration on the Ethereum Mainnet, we conducted an additional small-scale test involving 5 validators.

Deposit address: 0x31fa755b6193da3Dfb53384a0C065A9bC268e0a3
Withdrawal address: 0x47d412E1b40cc3C73d05F4c070723d927971D6e3

The tests went smoothly, prompting us to move to the next phase - scaling up the number of validators.

3. Mainnet - Scaling Up Validators Count

Initially, we created three staking pools: one managed by a multisig wallet and two smaller pools controlled by regular ETH addresses. Subsequently, inspired by the Web3Pi project, we decided to evaluate the capability of a Raspberry Pi device in handling ETH staking and added an additional staking pool.

Staking Pool A - Tests at scale

Deposit address: 0x9b900Ed2086c1Bd97F70A264b631A2cb1A0aD943
Withdrawal address: 0x4297cc867B85B63297c34af4268ced9CaFC66F5a
Current validators count: 2000

Through this staking pool, we have been progressively adding more validators to assess the impact on the performance and stability of the associated validator infrastructure. During the process, we tested different hardware and software configurations and scenarios (e.g., validator keys transfer between machines, adding more validators to a node that is already running validators, with the goal of minimizing the downtime)

Staking Pool B - Software configuration tests

Deposit address: 0x527F63e97353fed5016E7C45FBbeE8365aBdAd91
Withdrawal address: 0x485aD74f70eA3ad814D1dfE7A9284085499DE682
Current validators count: 500

This staking pool is currently being utilized to test the impact of various software configurations on the performance and stability of staking.

Staking Pool C - Software configuration tests

Deposit address: 0x0bc8be639f9B1878a4669A4a3DA6a1623A5C9399
Withdrawal address: 0x3dD89506d79f53DaD1B252fb9301Bc95C9792354
Current validators count: 500

This staking pool is currently being utilized to test the impact of various software configurations on the performance and stability of staking.

Staking Pool D - Golem Ecosystem Fund

Deposit address: 0xFC6E18E22Ad11592494B3D541bA25Af63A807527
Withdrawal address: 0xFC6E18E22Ad11592494B3D541bA25Af63A807527
Current validators count: 1250

Having gained confidence in our ability to safely scale the number of concurrent providers within our infrastructure, we have decided to establish this staking pool through publicly visible transactions on the blockchain. Additionally, we have committed a significant portion of the yield from this staking activity to the Golem Ecosystem Fund.

Staking Pool E - Staking tests with Raspberry Pi device

Deposit address: [ will be revealed after tests are finished ]
Withdrawal address: [ will be revealed after tests are finished ]
Current validators count: 200

Inspired by the Web3Pi project, we have decided to evaluate the Raspberry Pi devices’ capability to handle ETH staking. Initially, we assumed the device would only be suitable for a small number of validators, but the results exceeded our expectations.

In the following section, we would like to share our insights. 

Raspberry Pi Test Setup

Before diving into the staking performance results, let's first discuss the test environment. We set up an isolated network with a 1Gbit/s symmetric link to the Internet. The setup included the following components:

• Staking node [ Raspberry Pi 5 ] 
  • Ethereum clients: 
    • Geth (1.14.7-stable-aa55f5ea) 
    • Nimbus (v24.7.0-99f657-stateofus)
  • This node was hosting all validators during our tests
    
• Helper node [ Raspberry Pi 5 ] 
  • Ethereum clients: Geth and Lighthouse
  • This node was responsible for querying Ethereum’s Beacon Chain to find the longest attestation gaps (minimizing the downtime resulting from consensus client resets - required during testing)
• Metrics server [ Raspberry Pi 4 ]
  • Grafana used to visualize the data
  • InfluxDB used to collect the data
  • This endpoint was responsible for collecting and displaying status of the Staking Node (i.e., sync status) and system resources used by the Staking Node
• Router [ Compute Module 4 ]
  • This unit was responsible for maintaining connection between Local Network and the Internet
• Switch [ TP-Link TL-SG1005D ]
  • It was responsible for connecting aforementioned devices in a single subnetwork

A more detailed specification of the Staking Setup (monitoring hardware includes a stock Raspberry Pi 5 with 8GB RAM, and a stock Raspberry Pi 4 with 4GB RAM) can be found below:

Raspberry Pi 5 Staking Node

Router

Switch

TP-Link TL-SG1008D - 1Gbit/s (5 port)

Performance Tests

The tests were conducted on the Ethereum Mainnet with MEV-Boost enabled, using Nimbus and Geth, both of which are well-suited to the capabilities of the Raspberry Pi 5.

Our initial goal was to gradually reach a total of 100 validators.

• The first benchmark was conducted with 60 validators running simultaneously, yielding strong results with an average attestation performance of 99.94%
• The next benchmark, with 75 active validators, resulted in an even better attestation performance of 99.98%
• Finally, we tested whether the Raspberry Pi would handle hosting 100 validators, and to our surprise, the device managed this with ease, achieving an attestation performance of 99.99%

Encouraged by the results, we extended the test setup by adding another 50 validators, without observing any decrease in performance. Furthermore, the addition of one more batch of 50 validators, likewise had no impact on staking performance.

In total, we’ve been running 200 validators on a single device for more than 10 consecutive days, with the node flawlessly producing blocks.

Further details can be found in the following summary:

Downtime Test, Temperatures & Throttling

To evaluate how well the Raspberry Pi 5 recovers after a period of Internet downtime, we disconnected the local network from the Internet for 5 hours while the staking node was running with 150 active validators.

After restoring the Internet connection:

• Geth resumed syncing and initiated the process of regenerating the snapshot
• Once the sync was completed, the node began attesting again, even during the snapshot regeneration, which took an additional 4 hours
• During the test, we noticed that the thermal pads had degraded, significantly reducing their contact area and therefore impairing cooling. Despite this, all attestations during the snapshot regeneration process were successful, even with the CPU being throttled, peaking at 300%, and temperatures reaching as high as 80°C.
• Once the snapshot regeneration was complete, the node resumed functioning as it had before losing the Internet connection

We were quite impressed by how smoothly Geth and Nimbus resumed their operations, and how well the Raspberry Pi handled the additional load caused by necessity to catch up with the blockchain and very high temperatures/throttling imposed by our test setup. 

Downtime Test - Hardware Usage Metrics

In our tests, with a properly functioning fan and correctly attached thermal pads, the Raspberry Pi 5 chipset never exceeded 70°C, even under a 400% CPU load (which is the case during initial blockchain synchronization which takes more than 10 hours). As a result, it never started throttling and was able to operate at full capacity during the entire synchronization process. Once synced, the temperature consistently remained below 60°C.

The SSD drive temperature remained below 50°C, regardless of whether the device was undergoing synchronization or operating in normal conditions.

Network Load

The Internet traffic, as measured by the Compute Module 4 serving as the router for the entire setup (i.e., monitoring devices included), peaked at 110 Mibit/s inbound and 103 Mibit/s outbound, while maintaining 1,183 UDP and 634 TCP connections.

Network Traffic

Opened Connections

Hardware Resources Usage

Readers interested in more information on CPU, storage, and memory usage can find additional statistics below (each image corresponds to the duration of each described test pass).

Raspberry Pi 5 for Solo Staking: Our Thoughts

While there is still a lot to explore about Raspberry Pi 5 and staking, we are confident that the device has proven to be a robust and efficient choice for solo staking. 

During our research, we have come across an article written by Adrian Sutton, where he states that “(...) the first 64 validators you add to a node are quite expensive in terms of CPU usage and network traffic, but if you can run 64 validators well, then you can likely run thousands of validators well because you’re already processing the bulk of the attestations.“ which is reflected in our findings as well. 

Although we won’t be adding more validators at this time, we believe that the limit is significantly higher than the 200 active validators achieved in our ongoing experiment.

Conclusion

During the process of setting up the architecture and infrastructure for our staking solution, we recognized fragmentation of knowledge regarding the Ethereum Proof of Stake protocol. This includes risks, the various available software components, hardware requirements, and the numerous possible configurations of the infrastructure.

We plan to fund an external project aimed at developing the 'Whitebook of Staking' - a comprehensive guide that will detail alternative approaches to staking ETH, along with their implications and step-by-step instructions.

Individuals or organizations interested in contributing to the creation of this guide are invited to apply here: https://www.golem.network/ecosystem.

We will keep you informed with updates on the progress of our tests and the 'Whitebook of Staking' initiative as they become available.

We would like to extend our gratitude to the Web3 Pi team for providing us a platform for tests and technical support.

Golem Team