Reopen discussion - Validator Rewards

Governance
1

Hi everyone,

I would like to reopen the prior discussions around validator rewards and potentially re-submit the withdrawn CGP36

Reasoning

  • Complexity of Celo validator complete stack has not decreased, in fact, keeping on top of legal compliance on the attestation stack for three telephony providers is getting increasingly costly and fragmented
  • Resource footprint of mainnet increasing steadily (storage and memory primarily)
  • Increased awareness after mainnet network stall of operating enterprise monitoring solutions and timely human touch points
  • Documentation and available calculators all reference a now long out of date estimate of $75k cUSD per year
  • Macro market depression of CELO price means locked CELO rewards are no longer a meaningful offset against reduced cUSD rewards
  • Macro fiat inflation affecting all costs of operating a responsible validator entity
  • Finally, the validator groups and individuals who are the most engaged and responsive, are the smaller and/or independent groups where the bottom line is extremely sensitive to change. We should attempt to keep the cUSD rewards constant or at least offset the continued real-terms reduction

References:

June 2021 - Discussion on Celo Epoch rewards
June 2021 - Discussion on Validator Rewards
CGP Summary Slidedeck

Transparency:

  • I am founder and operator of an elected validator group and Foundation vote recipient (Vladiator Labs)
  • I am a steward of the Celo Community Fund version 2, called Prezenti
  • Vladiator Labs is a grant recipient from Celo Community Fund 1 for the celo-vido tool
  • Vladiator Labs is a member of the Staking Defense League founding charter, with the aim of taking action to protect decentralization principles of Proof of Stake networks
  • I have a minor personal long position on CELO

Edit notes: added transparency

3 Likes
2

To steelman the opposite view, here are some counterfactuals off the top of my head:

  • Celo validator seats are in high demand, low supply, and in a decentralized / open access system we could probably reduce rewards, or allow them to decay from target with no impact to the number or quality of elected validators
  • Celo validator rewards are already extremely unique and generous in that groups are primarily rewarded in cUSD, and do not have be net long the protocol token, or even perform any significant treasury management with CELO
  • Cloud provider compute costs have decreased significantly in the past three years (especially with custom processors like the AWS Graviton ARM-based series)
  • The Foundation primarily supports with votes entrenched validator groups with early ties to the ecosystem. More frequent reshuffling of elected seats on a profit margin basis is a good thing for intellectual and organisational diversity of the protocol.
3

There’s not really much activity onchain these days, Ubeswap daily volume at all time low of ~300k daily.

I feel like if the onchain activity is high (like what it was last year and Celo price was high), it made more sense to raise rewards, given that it was validation for a higher amount of daily $$$ volume. Kind of like “payment” for securing the network, but nowadays, it’s very stagnant onchain.

Is the current decreased validator rewards not break even anymore?

1 Like
4

It’s more than break-even, or else most ultimately rational economic actors would switch off their infrastructure.

At a protocol level I believe we are ahead of target on CELO issuance so the reward multiplier has reduced to around ~0.73, down from ~.87 when this discussion was first opened over a year ago.

The cost to validators to secure the network is not based on the dollar value throughput of transactions (although technically, signers would earn slightly more block gas fees denominated in fiat). The resource footprint of securing the network has actually increased over time, even as the price of CELO itself has gone to lows recently.

The main thrust of the argument is that the cost both in human time and technical resource footprint has increased, while the payments have (by design since CELO issuance is above target) and continue to decrease.

5

Are validators using AWS/GCP public cloud for the validator, or are people using Hetzner / OVH alternative cloud providers? The increase in technical resource footprint would be pretty cheap if running not on AWS / GCP.

For example, my OVH virginia box is $59/month and it has 64G ram and 1TB nvme. The same kind of box cost even less in Europe with Hetzner.

6

Yeah, the expense can definitely be optimized to smaller and simpler VPS providers like Hetzner. The big cloud providers are not cheap. It’s also true many are using them.

7

Hola! A few weeks ago I’ve helped a mate to run a side-project server at Hertznet in Finland with this specification,

AMD Ryzen™ 5 3600
CPU: 6 cores / 12 threads @ 3.6 GHz
Generation: Matisse (Zen 2)
RAM: 64 GB DDR4 RAM
Drives: 2 x 2 TB SATA Enterprise Hard Drive

for 38,44 euros per month. Unbelievable how this would cost thousands a few years ago, but, anyhow, it has the equivalent price of a simple quad-core shared-server with 8gb ram in South America… But even if you consider running 3 of this servers for Validator, Proxy and Attestation it still goes by less than $150/month.

It sounds a bit confusing for me because the validator reward of 85k cUSD per year is equivalent to years of work of many talented Linux hackers working full-time around here, in Colombia. (yup! ~$800/month, thought, I’ve met great devops without formal education having a ~$350 salary).

For a fixed $7k/month (44x times our minimum wage! or, 4 years of salary each month) “we” would definitely drop everything just to take care of servers! Even for $20k/year (~$1600/month) it sounds a remote job dream, specially since it is already so well documented that is gonna require just a few hours.

This should not be a tech-elite reserved attribution, and $7k is extremely overpaid.

Really, where do I sign to become one!? Wanna drop my shit job and become a validator with no less skilled dedication!

8

Why did you pick spinny rust??? Just stick with the NVME options.

And usually people just get a pair of these as primary secondary, and use docker containers to isolate the different workloads. Failover to the secondary host as needed for maintenance.

9

for that specific project was more import to have space than velocity. For most cases and the Validators, definitely a NVME would be the way to go…!

This is a fair point, but just checked back that a Intel Xeon E3-1275V6 with 4 cores and 2 x 512 GB SSD costs the same price.

My point is that this monster-machines are costing - per day - less than $1.5, while the validators are rewarded around $235. It is like a 13746% profit, not counting the 10k CELO you have to lock to participate.

10

Yes agreed if u compare hardware cost and maintenance hours, it is pretty high payout for individual solo validators.

There are a few bigger organizations that try to do staking as a service and their business employee overheads are higher so they might not like it as much.

For small fish like u and me, these opportunities are verrrrrrry attractive.

11

what we would like to have clarifications, please, is how the elections happens; just older validators judges?

12

No, all the holders of CELO who lock and vote with it are the ones that determine the elected validator seats.

13

So, clarifying, these top 4 elected validators got more than 10 million votes from unique holders? I feel am mistaking something with these big numbers.

Are there practical differences from a group or an individual validator? Here they are shown separated.

Do they run community managers or contract marketing advertisers to request votes? or it is more like a friends-of-friends network?

I’m serious dreaming about this. People in the north hemisphere have no idea how many talented people are in south america working hard for one year to earn what you get in a month.

We can’t talk about prosperity if the riches are getting richer, owning all the new web3 infrastructure.

And sorry for my naive radical honesty @Thylacine, but how could you become the Celo Community Fund “manager” and propose to reduce it’s fund in 20% ? It sounds like a Minister of Education willing to cut the public education funds, deteriorating it. More bizarre that the proposal suggests to move this fund to increase the validator rewards - being one of them! In Bogotá we call it corrupción.

14

Honesty and asking tough questions is never to be apologized for!

To clarify, the link you posted with a slide deck showing some potential ways these governance proposals could work together are not my ideas or suggestions, I’m just re-posting the links from the discussion threads that were opened last year on this topic. Perhaps they’re not relevant or recent enough and I should remove them to avoid confusion.

The purpose of this thread was simply to invite comments again on the discussion from last year.

Regarding the 10M votes on certain groups, yes these are truly 10M locked and voting CELO tokens on those validator groups. Many of the top groups own that volume of tokens themselves (as investors or early supporters of the project) and can simply vote for themselves. Most other smaller groups rely on the Foundation to vote them into election (minimum around 1.7M CELO).

Is there a practical difference between a group and a validator? Somewhat, a validator group can consist of validators from anywhere. Although in practise, most groups are populated by their own validators. Note that CELO holders vote for a group, not any individual validator. Hope that helps!

Regarding how the smaller groups actually receive the Foundation votes - there is an open grants program that has to be applied to periodically to get the support of the Foundation. I actually think the Foundation would be excited to have more South American validator representation and would be happy to support strong technical candidates. You can read more about the Foundation voting program here: Celo Foundation Voting Policy | Celo Documentation

15

Thank you for your gentle answer after my rude and wrong accusation, @Thylacine. Sorry.

Anyhow, it still sounds the Validators are getting such a big steak for the work they do. Truth be said that maintaining it, once you have all well configured in the first days, is being super overpaid in relation with the effort needed.

Correct if I’m wrong @diwu1989, if a few thousands transactions happens per hour on the blockchain, wouldn’t even a single new raspberry do the job? Or the proof-of-stake consensus is still so computing-consuming that makes sense paying more than one million per month ($7k x 150 validators) !? Sounds senseless, on my ignorance. $1 million would make much more sense to the Community Funds or to impactMarket, if willing to be fair and follow prosperity-for-all values.

I would like to re-purpose the original discussion on the Validators Reward, remembering what do we expect from the Validators. Should’t they also become a sort of access Providers? For example, playing around with web3@celo I just use the cLabs-run, (I guess) Forno gateway to access the web3, but they say:

Forno does not offer a terms of service and there are no guarantees about service uptime. For production applications, consider using {comercial companies}.

Why don’t the validators provide a wss://gateway to the blockchain as well? Of course we can run our own full nodes to access it, but they have it ready and I would never consider to pay for a reliable provider - as now Celo suggests. No other single reliable bridge not belonging to a company? wow.

Sounds a more important discussion how to decentralize and spread access to the blockchain and the contracts. The validators infrastructure are ready and paid, so, please, at least, run some more services onto it.

I wish to better understand on which datacenters are the validators running? Are they eco-friendly? Would’n t be a paradox to endorse and trade crypto carbon-offset tokens on a blockchain which 90% of servers are running on Amazon and Google infra?

With Ethereum finally moving to a new consensus, it is nice to imagine other future mechanisms, or,
how to decentralize the Validators? Like distributed webassembly-like droplets orchestrating IPFS clusters of decentralized octa-core personal computers, borrowed and paid in Celo for the needed computational amount of a fixed payment. One thousand people earning one thousand dollars for the job, randomly rotated every month among users with a $1k in stake and had already volunteered for months as a proof-of-provider. A growing queue of people volunteering their machines while waiting their chance. Scalable.

Tripping out, but we can have it much better than rewarding one million dollars per month to already-prosperous people!

16

No because EVM transaction logic is getting more and more complex / creative, so the amount of single-threaded performance required to validate a fully packed 30M gas block will crush a RPI. You should have a minimal geekbench5 score of 1000 single thread on the validator, and a few cores to ensure that mempool gossip and other compactions don’t overload the validator. RPI has single thread score of ~300 or so I think.

We recently had a stall on mainnet that was also due to enough validator running slower due to 50M packed block that the consensus rounds took enough extra rounds that bugs started appearing and stalled the network. Don’t cheap out on CPU, any Intel or AMD or Apple cpu from a $500 Costco/BestBuy/Amazon off the shelf computer provides more than enough compute for this.

Opening WSS on the validator becomes a denial of service attack vector. People can throw a LOT of expensive ETH_CALLs against it and drown them.

Wormhole bridge and Optics bridge I consider to be safe and reliable. They are run by company, but that to me is better than a decentralized Anon bridge that I cannot take to court if they screw up operations. I much prefer using services from service providers who can be tied to a real world known legal entity.

I don’t mind providing alternative for free WSS and RPC alternative to Forno in exchange for asking users to also stake Celo with my validator (if I end up running one). This is a good idea to provide public good service in exchange for crowdsourcing staked celo.
Once you add some throttling and abuse protection code into Geth or a proxy layer in front of Geth, it is not that hard to provide such service reliably without too much cost per month.

Many are using AWS and i think some of those data centers are low-carbon footprint. Hetzner Finland data center is actually certified 100% renewable, so that’s a better choice. https://thecelo.com has map of validator + IP addresses to check for location.

This is a bad idea because of how the Celo consensus algorithm works. You really don’t want Celo censensus to run networking across thousands of random residential machines with spotty networking… the mainnet will literally stale every other day to the point all users leave in frustration due to poor quality of service.

It’s better end user experience to just have ~100 machines in 100% certified renewable energy data centers that have reliable networking and strong CPU and can help us drive towards bigger blocks & faster block times (lower tx finality latency).

2 Likes
17

Such a lesson! thank you.

18

Alright I’m guessing there’s not a lot of appetite at the moment to adjust the validator rewards. I’m in the middle of an infrastructure clean up and re-costing myself, and its looking like a lot of savings from some simple optimizations and shopping around.

Thanks everyone!

19

I agree diwu1989.

IMO your comment highlights one of the two major existential problems the general public faces in trusting any crypto project: no recourse and; no easy way to buy groceries and gas.