What’s the problem?
Several validators reported an issue this morning (5/20/20) where they noticed they were not receiving rewards this epoch. The cLabs team spent much of the day reviewing this issue and we believe we have identified the cause. The issue stems from a rounding error of 1 wei – i.e. the function that checks that locked cGLD is >10k before disbursing rewards, sees 1 wei less and returns 0. See https://github.com/celo-org/celo-monorepo/blob/master/packages/protocol/contracts/governance/Election.sol#L561.
The reason this occurs is because an optimization was made to reduce the overhead of calculating individual voter rewards per group, by storing the ratio of the voter’s votes over the total number of votes the group has received. Eg. a 10k voter in a group with 50k votes is represented as 0.20. The downside here is that due to binary representation of decimal fractions, there are rounding errors that could cause the voter’s vote total to be represented as 1 wei lower than the actual amount when the denominator (the group’s total number of votes) changes.
Hence, when a validator or a group’s 10k ends up rounding down to 1 wei lower, it fails to pass the locked gold requirements and emits 0 rewards. Note that this does not affect your uptime score.
What’s the fix?
While this is inelegant, and an unfortunate issue given many voters are locking 10k cGLD exactly, there is a straightforward workaround in the short term. The cLabs team has also started working on a long term fix.
Short term workaround – any validator locking exactly 10k cGLD should lock up slightly more cGLD. Eg. 10,000.1 cGLD should be sufficient. There should be extra cGLD in all ReleaseGold contracts to fund signer keys, so we recommend transferring a small amount over. If you already have more than 10k cGLD locked up, there is no further action required at this time.
Long term fix – the Election smart contract will need to be upgraded to address this downward rounding error by adjusting the locked gold requirements to also be just below 10k cGLD. The cLabs team is working on smart contract upgrade processes to have clear governance proposals and provide backwards compatibility, but the estimated time frame for this to be ready is ~6 weeks. Once ready, cLabs team will update the Elections contract to address this issue.
Voter rewards % metrics – Some validators have expressed a concern that the voter rewards metrics reflected in the validator explorer (https://celo.org/validators/explore) are now inaccurate due to no fault of their own. cLabs team is aware of this, and are working hard to update the explorer so that validators’ scores do not reflect poorly. We’ll follow up with a timeline and how we’re addressing this shortly.
Thanks to all the validators who have reported this issue and relayed concern, as well as helped others. Please share if you have any feedback or any more questions about this issue. While this is an unfortunate hiccup, in other ways, it’s a reminder that we are all on Mainnet together now, and only have one direction we can go – forward :).