Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add EIP: Hardware and Bandwidth Recommendations for Validators and Full Nodes #9270

Open
wants to merge 35 commits into
base: master
Choose a base branch
from

Conversation

kevaundray
Copy link
Contributor

@kevaundray kevaundray commented Jan 26, 2025

Original documents are here (for validators):

@github-actions github-actions bot added c-new Creates a brand new proposal s-draft This EIP is a Draft t-informational labels Jan 26, 2025
@eth-bot
Copy link
Collaborator

eth-bot commented Jan 26, 2025

File EIPS/eip-7870.md

Requires 1 more reviewers from @g11tech, @lightclient, @SamWilsn, @xinbenlv

@eth-bot eth-bot added the e-consensus Waiting on editor consensus label Jan 26, 2025
Comment on lines 97 to 103
- Statista states that as of January 2024:
- The global average download for broadband is 92 Mbps and the global average upload is 43 Mpbs.
- The global average download for mobile is 50 Mbps and 11 Mbps
- GSMA report showing the state of mobile internet connectivity in 2024 shows that:
- The mobile upload speeds in Higher Income Countries (HIC) is about 18 Mbps
- The global average mobile download is 48 Mbps.

Copy link
Contributor Author

@kevaundray kevaundray Jan 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me know how you would want these to be referred to in the document


## Backwards Compatibility

This EIP is informational and requires no protocol changes. We recommend that future EIPs include an assessment of their impact on these hardware requirements.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would the recommendation here require another EIP if this gets finalized?

In any case, I don't think it belongs here since this section is explicitly about whether the EIP breaks backwards compatibility

@github-actions github-actions bot added the w-ci Waiting on CI to pass label Jan 26, 2025

RAM/memory is dominated by state cache. As of January 2025, it is possible to run a full node with 16GB of RAM, however this has been known to not work with all combinations of EL and CL clients in the past.

On 32GB vs 64GB; 32GB works right now, however we recommend 64GB as [preliminary benchmarks](https://hackmd.io/@han/bench-hash-in-snark) have shown that zk-STARKS can consume a significant amount of memory and the difference in cost relative to the entire hardware setup for a validator is insignificant.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw that EIPs 2926, 2938, 3298, 3416 and 3607 also used hackmd links however the contents of them can easily be changed, so I wonder if there is a rule about using them?

@github-actions github-actions bot added w-ci Waiting on CI to pass and removed w-ci Waiting on CI to pass labels Jan 26, 2025
Copy link
Contributor

@smartprogrammer93 smartprogrammer93 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given EIPs are immutable once they become final, does it make sense to place these requirements in an EIP? I expect requirements will need to be updated in a couple of years. In that case, we will need to create a different EIP amending this one. Then, operators will need to check 2 different EIPs to know what the current requirements are. Seems impractical to me.

@restakeservice
Copy link

good idea!

@kevaundray
Copy link
Contributor Author

Given EIPs are immutable once they become final, does it make sense to place these requirements in an EIP? I expect requirements will need to be updated in a couple of years. In that case, we will need to create a different EIP amending this one. Then, operators will need to check 2 different EIPs to know what the current requirements are. Seems impractical to me.

In the call someone mentioned that, the status of this could be changed to "live" and we modify it in-place instead of creating a new EIP each time. I would defer to the EIP maintainers regarding that and whats possible there.

I agree that creating a new EIP whenever we change specs is undesirable.

@kevaundray kevaundray changed the title Add EIP: Hardware and Bandwidth Requirements for Full Nodes and Validators Add EIP: Hardware and Bandwidth Recommendations for Full Nodes and Validators Jan 27, 2025
@github-actions github-actions bot removed the w-ci Waiting on CI to pass label Jan 27, 2025
@github-actions github-actions bot added the w-ci Waiting on CI to pass label Jan 27, 2025
@kevaundray
Copy link
Contributor Author

@yorickdowne @potuz Thanks for the comments on hardware -- any comments on bandwidth?

@github-actions github-actions bot added the w-ci Waiting on CI to pass label Jan 29, 2025
Copy link

The commit fda25ca (as a parent of e2c2d40) contains errors.
Please inspect the Run Summary for details.

zbronstein added a commit to endaoment/endaoment-validator that referenced this pull request Jan 29, 2025
zbronstein added a commit to endaoment/endaoment-validator that referenced this pull request Jan 29, 2025
zbronstein added a commit to endaoment/endaoment-validator that referenced this pull request Jan 29, 2025
fredriksvantes and others added 2 commits January 31, 2025 15:01
Made adjustments to make it a bit easier to read.

Included information about how even if you use mev-boost you may need the same bandwidth as a local block builder under certain conditions.

Included some additional details for choosing a NVMe drive (such as avoiding QLC and DRAMless drives).

Added some information about what the PassMark CPU ratings translate into

Added a CPU section

Removed Statista as I don't believe it serves much of a purpose in the EIP, and I'm not sure how accurate these numbers really are when looking at the median.

Added Quick Reference for those who just want the details
kevaundray and others added 6 commits January 31, 2025 15:43
Co-authored-by: Justin Traglia <[email protected]>
Co-authored-by: Justin Traglia <[email protected]>
- format table
- "ram" - "memory"
- "Download upload speed" -> "Bandwidth Download / Upload"
Co-authored-by: Justin Traglia <[email protected]>
@kamilchodola
Copy link

General question to this one:

Should we better treat that as "Recommended setup" or "Minimal setup"?

Minimal would be then something we can refer to and adjust to it - having any users on our support lines which will be struggling with weaker setups could be then advised to think of any improvement and referred to this EIP.
Recommendation always gives quite a bit of room for complains.

Also having "Minimal supported setup" in place we can adjust any benchmarking tools to it and then make a proper assumptions.


Node operators typically run both an **Execution Layer (EL)** client and a **Consensus Layer (CL)** client on the same machine. The specifications below assume the combined resource usage of both.

| Node Type | Storage | Memory | CPU Cores / Threads | **PassMark CPU Rating** | Bandwidth Download / Upload |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are all those numbers applied based on some research made? Or based on experience from discussions with stakers?

From Nethermind perspective those are super comfortable.

Also shall we distinguish EL and CL requirements to be treated separately? I saw a lot of setups where actually both are running on different machines.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the average user would likely run these on the same machine?

The rationale for them is in the linked hackmd in the comment box, we could not include all of those links so the EIP has been somewhat watered down


This EIP is informational and requires no protocol changes. We recommend that future EIPs include an assessment of their impact on these hardware recommendations.

## Security Considerations

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing I'd suggest doing prior to applying that is to do any research among community either by pool on X or anything else which can have bigger outreach.

If it will appear that 51% of users have weaker than specified setup (which is super unlikely but still) we could potentially open a route for some unexpected attacks because we will test on too good hardware.

At least worth to know in case of noticed issue which affect weaker setups how much network (in terms of nodes) is put in danger.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could alternatively, test against the old NUCs for a year too though the current idea was that instead of us polling N validators, we would give them a tool to allow them to assess themselves and decide when they would want to update

@kevaundray
Copy link
Contributor Author

General question to this one:

Should we better treat that as "Recommended setup" or "Minimal setup"?

Minimal would be then something we can refer to and adjust to it - having any users on our support lines which will be struggling with weaker setups could be then advised to think of any improvement and referred to this EIP. Recommendation always gives quite a bit of room for complains.

Also having "Minimal supported setup" in place we can adjust any benchmarking tools to it and then make a proper assumptions.

This would be treated as "recommended" -- you can use weaker hardware but they would not be tested/benchmarked against is the thinking. The minimum is somewhat client specific, so I'd be hesitant to add that maintenance burden and it also would likely change more often per hardfork vs recommended due to the headroom.


| Node Type | Storage | Memory | CPU Cores / Threads | **PassMark CPU Rating** | Bandwidth Download / Upload |
| ----------------------- | --------- | ------ | ------------------- | ----------------------- | --------------------------- |
| **Full Node** | 4 TB NVMe | 32 GB | 4c / 8t | ~1000 ST / 3000 MT | 50 Mbps / 15 Mbps |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Considering history expiry, 4TB sounds like a lot to recommend just for a full node. I see it's elaborated on below but I would either provide an estimation how long it will sustain with 2 or 4tb.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also make it clear that these are recommended, not minimum viable specs

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same goes for 32GB, I believe 16 is enough for different client pairs I have been running. Maybe 24 could be more realistic than 32

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO nice to work on 16 or even 8gb but this EIP is some commitment of supported hardware and some guidelines for users on what to use - I'd prefer users to use better hardware especially cost difference (especially for RAM) is like 50-100 bucks to upgrade from 16 to 32 and it gives a lot and makes us a bit more secure from some heavy blocks especially if we consider an increase of their size.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Considering history expiry, 4TB sounds like a lot to recommend just for a full node. I see it's elaborated on below but I would either provide an estimation how long it will sustain with 2 or 4tb.

Yeah, we recognize that 2TB will be do-able with history expiry, but it's uncertain when that will actually ship. I would prefer the EIP's recommendations be on the safe side, 4TB.

I would also make it clear that these are recommended, not minimum viable specs

The word "recommendation" is used several times in this doc. It's even in the title:

title: Hardware and Bandwidth Recommendations for Validators and Full Nodes

Same goes for 32GB, I believe 16 is enough for different client pairs I have been running. Maybe 24 could be more realistic than 32

Yes, this 16GB is enough for today (and what my validator has) but might not be enough 3 years from now. Once again, this EIP does not state that you cannot run a full-node with less, just that 32GB is recommended.

Copy link

@kamilchodola kamilchodola Feb 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting minimum supported hardware for me is super beneficial and already mentioned that few comments above.

Will make it clear under which setups we can't put our signature and which ones we don't believe are good enough.

Copy link
Member

@jtraglia jtraglia Feb 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, minimum hardware requirements would be useful too, but it's a much more difficult problem. I wouldn't expect the minimum for all clients to be the same. In my opinion, minimum requirements are best left to client teams. For example, the big advantage of Nimbus is that it works very well with low-resource hardware. I believe the current recommendations will work for all clients for at least the next three years.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hardware will be much better in next 3 years and currently recommended will probably be much cheaper, so it will be even cheaper than some super cheap ones on which any client barely works now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand it's much harder to determine what are minimum specs and also harder to sustain so I don't believe it needs to part of the table. However settling now on this much room in recommended also makes me think devs might get too comfortable with the extra performance and it might become the new minimum. I believe we are building also for people with less hardware options.

@kamilchodola I wouldn't be so sure about significant change in the hardware getting cheaper. That assumes people need to upgrade to latest hardware, their existing nodes are not going to get stronger, only rustier. And I am not very bullish on hardware getting cheaper in general, last few years are proving otherwise and although inflation is slowing down, the progress in hardware is much smaller than it used to be, Moore's law is dead and we cannot count on getting twice as performance in couple of years. Current hardware generations offer rather incremental improvements for high prices

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we planned to prune pre-merge data in the first half of the year, which is roughly half a terrabyte.

I got this number by computing the total size of all of the files here: https://era1.ethportal.net/

Total size: 458,497,023 kB
= 447,751.00 MB
= 437.26 GB

If we use the history growth numbers from paradigm as an approximation, this buys us roughly a year: https://www.paradigm.xyz/2024/05/how-to-raise-the-gas-limit-2

For full history expiry, the storage occupied will be constant, though I think its still uncertain as to when that officially lands in most clients. If it lands in a few months, I think we can change this number for full nodes and validators -- most validators AFAIK have already updated to 4TB, ethstakers for example recommend this for new validators: https://ethstaker.org/staking-hardware


- **Why 64 GB for Validators (Attester / Local Block Builder)?**
- Running an Ethereum validator (both EL & CL clients) can be memory-intensive, with state cache dominating RAM usage.
- Certain memory intensive client combinations have historically failed with 16 GB.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's worth adding that although some pairs might have trouble with 16gb by default, it's possible to tune cache size in the client to make it work

@g11tech
Copy link
Contributor

g11tech commented Feb 25, 2025

hey @kevaundray , if you could get the linter/bots to pass, we can merge this PR and then you can continually update this basis discussion and feedback

@@ -0,0 +1,145 @@
---
eip: 7870
title: Hardware and Bandwidth Recommendations for Validators and Full Nodes
Copy link
Member

@jtraglia jtraglia Feb 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the EIP title, I think we should remove "for validators and full nodes" for the following reasons:

  1. The current name is too long. The max is 44 characters, this is 68. With that part it is 38 characters.
  2. It's sort of implied that we're talking about validators & full nodes. But I suppose we could add recommendations for light clients too. A somewhat ambiguous title would allow us to do that.
Suggested change
title: Hardware and Bandwidth Recommendations for Validators and Full Nodes
title: Hardware and Bandwidth Recommendations

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c-new Creates a brand new proposal e-consensus Waiting on editor consensus e-review Waiting on editor to review s-draft This EIP is a Draft t-informational w-ci Waiting on CI to pass
Projects
None yet
Development

Successfully merging this pull request may close these issues.