# Validator consensus update plan **URL:** \ **Category:** Mainnet\ **Created:** [August 9, 2022, 11:26pm UTC](https://forum.celo.org/t/validator-consensus-update-plan/4113 "2022-08-09T23:26:23Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![pastoh](https://avatars.discourse-cdn.com/v4/letter/p/839c29/32.png) [@pastoh](https://forum.celo.org/u/pastoh)\ **Post date:** [August 9, 2022, 11:26pm UTC](https://forum.celo.org/t/validator-consensus-update-plan/4113/1 "2022-08-09T23:26:23Z")
To all validator-operators: The previous mainnet stall uncovered a defect in the celo-blockchain implementation of the Istanbul PBFT consensus, which was temporarily solved with the 1.5.8 patch and a hotfix on the gas limit per block. In order to increase the gas limit again, a change in the implementation of the Istanbul message types is needed. The technical TLDR; of the issue is the following: in the proposal message, a _RoundChangeCertificate_ is included if the validator is proposing for _round \> 0_. This certificate holds at least quorum amount of _RoundChange_ messages from validators. Said _RoundChange_ messages also may hold a _PreparedCertificate_ if they have one, which contains the previous proposal that may have passed prepare consensus. This leads to, in a worst case scenario, the _RoundChangeCertificate_ holding at least _[(validators \* ⅓) + 1]_ different blocks in the proposal message. The change we plan to implement is adding to the _RoundChange_ a signed slim certificate, referencing a block hash and not a full block via the _PreparedCertificate_, which can then be grouped in the new _RoundChangeCertificate_ (since only the highest round _PreparedCertificate_ block is used). Such change is backwards incompatible, and a careful approach must be taken to ensure the liveness guarantees of the blockchain. After some thought, we at cLabs have currently two different upgrade paths that lead to the fix being applied while maintaining the blockchain uptime: **1. Two phase rollout** A phase1 client where both message types are used, both for sending and receiving, increasing the bandwidth used. A phase2 client which only uses the new message types, reverting the increased bandwidth usage to normal, and the fix being completely applied at this point. It is important to note that phase2 can only be rolled out when at least quorum amount of validators are running phase1, but the more the safer. There is no particular rush to switch to phase2, other than the community need to have the issue fixed. Also, it is not necessary to have different version patches, since phase2 could easily be activated by a configuration flag or even an admin rpc call. **2. Consensus fork** Deciding a block number where the new message types will be used for Istanbul, marking a clear point in time where the change will be activated. This will mark a clear deadline, prompting operators to upgrade upon the risk of stalling the network if quorum is not reached. It’s important to note that this is not a chain Hard Fork, but only a consensus fork, and full nodes won’t be even aware that this is happening. * * * The reason why a mixed approach is not viable is that if we mix old and new _RoundChange_ messages, the still unupdated machines won’t be able to see or understand the new _RoundChange_ messages present in the _RoundChangeCertificate_. We would like to hear your input on what you think is the best approach for applying the update. ---
**Author:** ![Patrick](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.celo.org/patrick/32/6634_2.png) [@Patrick](https://forum.celo.org/u/Patrick)\ **Post date:** [August 10, 2022, 1:25am UTC](https://forum.celo.org/t/validator-consensus-update-plan/4113/2 "2022-08-10T01:25:22Z")
Option 1 sounds lower risk in terms of liveness because there is no risk of stalling the network. Whereas, option 2 risks a stall if a quorum is not reached before the deadline. Is that correct? ---
**Author:** ![pastoh](https://avatars.discourse-cdn.com/v4/letter/p/839c29/32.png) [@pastoh](https://forum.celo.org/u/pastoh)\ **Post date:** [August 10, 2022, 1:28am UTC](https://forum.celo.org/t/validator-consensus-update-plan/4113/3 "2022-08-10T01:28:41Z")
That is correct. And also option 2 gives a clear timeline on when the fix will be completely implemented, while option1 will depend heavily on when the phase1 update is fully delivered, and then waiting for the phase2 to be fully delivered. So I would probably risk saying that option1 will take longer than option2. So it will take longer to raise the gas limit. ---
**Author:** ![Patrick](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.celo.org/patrick/32/6634_2.png) [@Patrick](https://forum.celo.org/u/Patrick)\ **Post date:** [August 10, 2022, 1:52am UTC](https://forum.celo.org/t/validator-consensus-update-plan/4113/4 "2022-08-10T01:52:30Z")
What are the risks of taking longer to raise the gas limit? ---
**Author:** ![pastoh](https://avatars.discourse-cdn.com/v4/letter/p/839c29/32.png) [@pastoh](https://forum.celo.org/u/pastoh)\ **Post date:** [August 10, 2022, 2:06am UTC](https://forum.celo.org/t/validator-consensus-update-plan/4113/5 "2022-08-10T02:06:34Z")
Technically speaking, limited to L1 blockchain risks, none probably. ---
**Author:** ![Chainvibes](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.celo.org/chainvibes/32/11848_2.png) [@Chainvibes](https://forum.celo.org/u/Chainvibes)\ **Post date:** [August 10, 2022, 7:28am UTC](https://forum.celo.org/t/validator-consensus-update-plan/4113/6 "2022-08-10T07:28:34Z")
While option 1 sounds less risky, we have done option 2 on previous updates succesfully. If enough time is given it would not be a problem imo. (Also ask every validator to reply to the signal email when upgraded) Is it possible to techincally check if quorum is reached on Phase1? Are there any other possible side effects/risks in phase1 besides the increased bandwidth or could there be a negative side effect if Phase2 (admin rpc call) isn’t executed by all validators? ---
**Author:** ![diwu1989](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.celo.org/diwu1989/32/6551_2.png) [@diwu1989](https://forum.celo.org/u/diwu1989)\ **Post date:** [August 10, 2022, 4:40pm UTC](https://forum.celo.org/t/validator-consensus-update-plan/4113/7 "2022-08-10T16:40:43Z")
> [@Patrick](#): > > What are the risks of taking longer to raise the gas limit? I think the gas limit being high make this more likely to occur, but I have a hunch that with the right kind of “grinder” contract, even with current block gas limit, I can make enough validator machines stall to the point where they start missing rounds, and can hit the same stall. Think of the gas limit as this: - theres a budget for how many EVM bytecodes u can run for a given budget in the block - high gas limit mean i can run a LOT of operations that make validator slow down and miss round - if gas limit lower, I need to be more specific about which bytecode I pick and choose pathologically mis-priced EVM bytecodes like SSTORE and SSLOAD to randomized addresses to really grind the block storage on the validator So, i don’t necessarily think we’re fully safe right now on mainnet TBH. ---
**Author:** ![pastoh](https://avatars.discourse-cdn.com/v4/letter/p/839c29/32.png) [@pastoh](https://forum.celo.org/u/pastoh)\ **Post date:** [August 10, 2022, 4:50pm UTC](https://forum.celo.org/t/validator-consensus-update-plan/4113/8 "2022-08-10T16:50:36Z")
@diwu1989 that could be an attack to performance, not to liveness. if it takes longer to execute a block, then it might get to round 3 or 4 before it gets accepted, which is bad, but not a stall scenario. And it gets fixed by changing gas costs on for example cold storage reads (which has its specific gas cost) And a reminder that if you actually have instructions for a vulnerability that can be exploited to stall the network, maybe posting it in the public forum is not the best way to go 🙂 Of course discussing issues or possible problems with future configuration is very welcomed! But a more private approach is usually used for these kind of things, at least until fix or confirmation that it’s a non-issue. ---
**Author:** ![diwu1989](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.celo.org/diwu1989/32/6551_2.png) [@diwu1989](https://forum.celo.org/u/diwu1989)\ **Post date:** [August 10, 2022, 4:53pm UTC](https://forum.celo.org/t/validator-consensus-update-plan/4113/9 "2022-08-10T16:53:48Z")
Yeah ur right, when baklava is back healthy and ready for stress testing, I’ll stress test it with various workloads for us privately. Thanks. ---
**Author:** ![keefertaylor](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.celo.org/keefertaylor/32/8572_2.png) [@keefertaylor](https://forum.celo.org/u/keefertaylor)\ **Post date:** [August 10, 2022, 7:51pm UTC](https://forum.celo.org/t/validator-consensus-update-plan/4113/10 "2022-08-10T19:51:30Z")
Given that we’re in/coming up on vacation season for many parts of the world, Tessellated Geometry would rather rip the bandaid off and just do it in #2. I believe C-Labs or the Foundation circulated a validator contact form to entities, so it should be easy to coordinate quorum in a non-emergency situation. I realize we had trouble doing this on Baklava, but I’d be inclined to think that it has more to do with low stakes for whether or not that network is operating (as opposed to mainnet where there are real financial penalties if the network comes up and your node isn’t live). ---
**Author:** ![martinvol](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.celo.org/martinvol/32/7166_2.png) [@martinvol](https://forum.celo.org/u/martinvol)\ **Post date:** [August 11, 2022, 1:55pm UTC](https://forum.celo.org/t/validator-consensus-update-plan/4113/11 "2022-08-11T13:55:35Z")
I think I like more the Consensus fork (2), in the past hard-fork were implemented with no node staying behind in the desired block. It would probably make the implementation easier and preventing bugs (and testing) from supporting both messages at the same time. ---
**Author:** ![piersy](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.celo.org/piersy/32/7115_2.png) [@piersy](https://forum.celo.org/u/piersy)\ **Post date:** [August 14, 2022, 5:06pm UTC](https://forum.celo.org/t/validator-consensus-update-plan/4113/12 "2022-08-14T17:06:07Z")
What about making this a new p2p.Protocol? (istabul 68) It would be almost exactly the same as the current (istanbul 67) protocol except that it would encode the consensus messages in a more efficient manner. Then nodes would run both protocols and through the inbuilt protocol negotiation, each peer to peer connection would run the latest protocol supported by both sides. This is somewhat like the “Two phase rollout” suggestion except that it avoids increasing the bandwidth and it doesn’t require a second phase in order to enact the fix, instead the network will move towards a complete fix in a transparent manner as validators update. ---
**Author:** ![pastoh](https://avatars.discourse-cdn.com/v4/letter/p/839c29/32.png) [@pastoh](https://forum.celo.org/u/pastoh)\ **Post date:** [August 16, 2022, 4:02pm UTC](https://forum.celo.org/t/validator-consensus-update-plan/4113/13 "2022-08-16T16:02:08Z")
This would work for the Round change messages, but not for the Round Change certificate. Imagine the scenario where exactly 50% of the validators have upgraded. 1. The proposer of the current round does not propose (may be down, delayed, or whatever). 2. Everyone sends a round change message. 3. The new proposer is in the 68 (new) protocol. It will receive 50% of round change messages in the new format, and 50% in the old format. How does it create the Round Change Certificate for the old protocol? If it doesn’t have the old messages, it can’t, since the (old) RCC needs at least quorum Round Change (old) messages. This stalls the network since a new RCC can never be created. This is the reason for having to send both, to be able to craft both old and new RCC messages with quorum. (Granted, a more careful approach could only send the double message to the new ones, but it’s mostly the same). ---
**Author:** ![pastoh](https://avatars.discourse-cdn.com/v4/letter/p/839c29/32.png) [@pastoh](https://forum.celo.org/u/pastoh)\ **Post date:** [August 18, 2022, 6:54pm UTC](https://forum.celo.org/t/validator-consensus-update-plan/4113/14 "2022-08-18T18:54:24Z")
Given the replies we will go with the consensus fork approach. I’ll notify in discord when we start testing it, and then we’ll discuss dates. ---
**Author:** ![rootwarp](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.celo.org/rootwarp/32/6709_2.png) [@rootwarp](https://forum.celo.org/u/rootwarp)\ **Post date:** [October 4, 2022, 11:40am UTC](https://forum.celo.org/t/validator-consensus-update-plan/4113/15 "2022-10-04T11:40:23Z")
I think option 1 seems more safe and decentralized way.