Activating the Block Throughput Increase Hotfix

Mainnet
1

Hi Validators,

Thank you to everyone who has already upgraded to v1.2.5 on Mainnet, and for your patience with the flurry of communications over the last week. If you haven’t already upgraded to v1.2.5, please do so now.

We’ve been hard at work load testing v1.2.5, and are delighted to report that it appears the block gas limit can be safely increased to 20MM!

The network performance appears to be more sensitive to the transaction pool size than the block gas limit, and as a result, we are asking that any validator who has not already upgraded to v1.2.5 include –txpool.globalslots=1024 in their flags when starting their celo-blockchain validator node. For validators that have already upgraded, this change is optional but encouraged.

Our load testing also showed that increased CPU seemed to help in periods of high congestion. For validators that can easily change their machine specs, we strongly recommend upgrading your validator node to at least 8 cores, and all other nodes to at least 4 cores. This will ensure that future throughput improvements remain unconstrained by compute.

We are also recommending that the target block density in Celo’s implementation of EIP-1559 be increased from 50% to 80%, and the adjustment speed from 0.5 to 1.25. This will make more of that 20MM capacity available for use without exponentially increasing gas prices, while ensuring that when the blockchain is at capacity, the gas price minimum rises at the same rate that it does currently.

In order to roll this increase out as quickly as possible, we are asking validators to approve a governance hotfix containing these changes.

The hotfix will be executable after the governance approvers and a quorum (i.e. ⅔) of validators have approved it. Note that any validators that are running <v1.2.5 after the hotfix is executed will risk having difficulty proposing blocks during periods of high load, degrading the performance of the network. As such, we are asking validators to begin approving the hotfix right away, but for the governance approvers to hold off until the overwhelming majority of validators have confirmed they’re upgraded to v1.2.5.

To approve the hotfix, please start by downloading this gist containing the JSON specification of the hotfix. Please read it carefully to confirm that it matches your expectations. Note that the target density and adjustment speed are both Fixidity values, which have a denominator of 10^24.

Then, calculate the hotfix hash with the following command:

celocli governance:hashhotfix --jsonTransactions PATH_TO_JSON --salt 0x853dff3851f957af96331bd0918899601a958825f48cb8da44fa6d385404062f

You should see the following hotfix hash:

hash: 0xfcfc98ec3db7c56f0866a7149e811bf7f9e30c9d40008b0def497fcc6fe90649

You can then approve the hotfix by running the following command. Note that you will need to sign with your validator signer or validator account. If you run multiple validators, please whitelist the hotfix with each of them. Please be extremely careful; as I’m sure you appreciate, these keys are incredibly sensitive.

celocli governance:whitelisthotfix --hash 0xfcfc98ec3db7c56f0866a7149e811bf7f9e30c9d40008b0def497fcc6fe90649 --from VALIDATOR_ADDRESS

You may need to add --useLedger if your key is held on a Ledger.

Finally, you can follow the hotfix’s progress by running:

celocli governance:viewhotfix --hash 0xfcfc98ec3db7c56f0866a7149e811bf7f9e30c9d40008b0def497fcc6fe90649

Tl;dr:

  1. Please upgrade to v1.2.5
  2. If you haven’t already upgraded, please include --txpool.globalslots=1024 in your validator flags.
  3. If you’re running with fewer than 4 CPU on any node, please upgrade your machines immediately. Optionally, if you’re running in the cloud, please ensure your validators are running with at least 8 CPU.
  4. Please approve the proposed hotfix, if you support it.

Thank you.

The cLabs Team

2

UPDATE:

Hi Validators,

This morning we discovered an issue with the hotfix that resulted in it being unexecutable. In short, the gas price minimum adjustment speed exceeded a limit put in place to prevent integer underflows. There was no risk to the network; the parameters in the hotfix would not result in an underflow, the contract handles underflows gracefully, and celo-blockchain handles contract failures gracefully.

Going forward, we recommend that we try only to modify the block gas limit with the hotfix, and leave any gas price minimum adjustments for future governance proposals.

A hotfix to just increase the block gas limit can be found here.

Using a salt of 0x853dff3851f957af96331bd0918899601a958825f48cb8da44fa6d385404062f results in a hotfix hash of 0xfcfc98ec3db7c56f0866a7149e811bf7f9e30c9d40008b0def497fcc6fe90649.

We’ve edited the forum post above to reflect these changes.

We apologize for the inconvenience and we ask that you redo the hotfix process above with the new updated information.

We’re looking forward to conducting and circulating a post-mortem covering the events of the last week, including this issue.

Thank you for your understanding and for keeping the Celo network secure.

The cLabs Team