Optics Recovery Mode

Announcements
1

I just posted this announcement in the Optics Discord server, and cross-posting here:

We recently discovered an issue with Optics beta and our team is actively working to resolve it. Here is what we know: Despite no known vulnerability, and without alerting the community, someone unilaterally activated the Optics recovery mode on the Ethereum GovernanceRouter contract. While Optics continues to function, recovery mode gives the recovery manager account full control of Optics, overriding the governance multisig. This event occurred less than 15 minutes after cLabs terminated James Prestwichā€™s employment for misconduct: a parting of ways that was not due to issues around Optics or Jamesā€™ technical work.

Here is the onchain and github record:

  • When it was deployed, the Optics recovery account was set to a single end user account, contrary to usersā€™ expectations of decentralized governance and best practices.
  • During the deployment, James Prestwich created a pull request for the config including the recovery address, requested confirmation of this address, and later claimed reimbursement for the funds used.
  • Optics was deployed with a recovery mode timelock of only 1 second, not 1 day as the checked-in configuration states, which gave users no time to react to the activation.
  • The contract was placed in recovery mode with two transactions just 6 blocks apart (0xbaf3ā€¦. and 0x8b1eā€¦).

What this means: The Optics bridge is working. We do not believe funds in the bridge are currently at risk. Neither the Ethereum nor Celo networks are impacted. But this type of takeover has no place in the community.

Since discovery we have pursued every avenue to engage James Prestwich to see if he has information that can help resolve this situation. We continue to do so. However we are publishing this statement now for full transparency and because those efforts so far have not been successful. Our priority is moving Optics out of recovery mode and restoring governance to a federation of known community members.

Weā€™ll be providing support to users on Optics Discord in #bridge-support. We also welcome feedback and technical assistance from builders in a new channel #community-owned-optics. We will also be hosting an AMA tomorrow for the Optics community to answer questions and work through a plan - stay tuned.

11/22/21: Edits in italics to clarify that we do not know if James can resolve the situation

5 Likes
2

Hey Tim!

As a third party with relationships with both Celo and Jamesā€¦ this sounds an awful lot like a public accusation rather than a measure of transparency :disappointed: Might I suggest a tactful edit that makes it clear whether or not you believe he was involved with any wrongdoing here?

7 Likes
3

Can someone ELI5 for me?

An ex-engineer who was in charge of deploying the Optics bridge contract (the one that holds everyoneā€™s ETH for bridging) checked into a github repo of some deployment configurations that looked correct.

But it turns out that when the deployment was actually made, a different set of configs were used instead?

The source of truth here is the configuration of the actual deployed contract, so only now we discover that the deployment has been poisoned with to begin with? How come people ā€œverifyā€ and approved the Github pull request without even checking that the actual deployment matched?

Can we have a public 5-whys on this so that thereā€™s improvements to the change-management / deployment process?

1 Like
4

Hello,

Can you share that pull request ?

1 Like
5

In the second transaction, the recovery manager was changed over
from 0x3d9330014952bf0a3863feb7a657bffa5c9d40b9 to 0xdcbf2088b7a6ef91f954be9ca658ea5b8e9b62d4 which was created by 0x2f4bea4cb44d0956ce4980e76a20a8928e00399a

So who owns Address 0x2f4bea4cb44d0956ce4980e76a20a8928e00399a | Etherscan
?

2 Likes
6

I would like to preface this message with the fact that I donā€™t mean this as an accusation of any party.
I am hoping to remain neutral, and to be honest I still feel that I donā€™t understand some of @timā€™s post above.

To answer your question, I believe thereā€™s reason to believe that the owner of this 0x2f4bā€¦399a address is involved with the $PARTY (PartyDAO) project. The wallet is 1 of only 177 total holders of $PARTY, but thereā€™s further evidence as well.

The only related Google results are block explorers, but it does show up in GitHub code search results (for users who are logged in with a GitHub account).

Line 6 of this specific README.md commit in the PartyDAO/partybid repository contains what I believe to be the first public instance of this address appearing on GitHub. Notably I say first public instance because itā€™s possible that this URL existed in private repositories beforehand.
It was committed by GitHub user anna-carroll on July 11 2021. I searched thoroughly and cannot find any older public instances of the address in this URL format, but if anyone else does find an older instance then please do correct me.

The README specifically provides the following URL for the $PARTY token contract:
https://etherscan.io/token/0x402eb84d9cb2d6cf66bde9b46d7277d3f4a16b54?a=0x2f4bea4cb44d0956ce4980e76a20a8928e00399a

If you follow that URL (or look at the last portion following the ā€˜?ā€™ character) you can see that it is specifically associated with the address 0x2f4bea4cb44d0956ce4980e76a20a8928e00399.
Etherscan generates such links when you click on a token contract link from within an address page.

A ā€œcorrectā€ URL for the $PARTY token contract would look like this:
https://etherscan.io/token/0x402eb84d9cb2d6cf66bde9b46d7277d3f4a16b54

This tells us that the original source of the above URL (not necessarily the first person to post it publicly on GitHub) was browsing the 0x2f4bā€¦399a page on Etherscan before they clicked on the token contract to retrieve its URL, and forgot to omit the final portion of the address. It has since proliferated to other repositories as well.

Please note that I do not mean to allege that the author of this commit is the owner of the address. Itā€™s possible that the commit author was either:

  1. directly provided with this URL, already containing the address, by someone else
    or
  2. was browsing the 0x2f4bā€¦0399 address page for an unrelated reason
    so this isnā€™t direct proof of them being the address owner.

I do wonder whether any PartyDAO members would know who the owner of that address is, or have any further insights about how to determine thisā€¦

2 Likes
7

Nice detective work!

To add to this, 0x2fā€¦99a was previously funded by 0x2a98b391aa26618b8ca7fe21ca75120f15dd4a42 (on Ethereum mainnet), which was itself funded via KuCoin. Iā€™m not up to date with the KYC requirements of KuCoin, but most likely his/her information is in their systems. There might be more links to CEXes that do have strict KYC requirements, I only had a quick look.

That being said, Iā€™d strongly advise to put out a warning that the bridge is potentially compromised. Literally stating that ā€œno funds are currently at riskā€ is rather misleading. The worst-case scenario - even when it is unlikely to manifest - is quite catastrophic.

image
image1141Ɨ174 40 KB

1 Like
8

Having known James for a while, I find it extremely hard to believe that he would be involved in wrongdoing. I have no idea why he was terminated or that he was terminated. But I know James to be of extremely high integrity. This seems like an unnecessarily public jump to conclusions before all the facts are in.

6 Likes
9

As a member of the Celo community, I am concerned that this report does not paint a holistic picture. Instead, it has some conjectures and makes a few unfounded statements that are not based on data.

I am requesting the team performing the investigation to provide us with additional information to clarify what exactly occurred.

You can find my exact requests in this public google document - Doc Link

Note:

  • This is not a formal review, i.e., in my comments here, I do not represent cLabs, Celo Foundation, or another third party. I am a member of the Celo community.
  • I am providing some comments here as a Celo community member and a cyber security Investigator with 10 years of experience. I have conducted over 50 investigations on state sponsored threat actors in various industries including banking, manufacturing, government, and crypto-currency amongst others.
  • While I am asking for a rigorous technical examination of this incident, my review itself is not rigorous as I do not have the bandwidth to do a rigorous analysis.
  • Also, as a community member, I am requesting additional information surrounding this incident to paint an unbiased investigative report regarding the incident.
  • While I was part of cLabsā€™ security team and an advisor previously, I do not have any insider information on the course of this investigation or the events surrounding the termination mentioned in this post.
7 Likes
10

Kucoin KYC is optional :frowning:

11

Hi Matt, thanks for the comment. Updated the post in an attempt to clarify that sentence.

1 Like
12

Would appear that Anna owns it: anna ā€” Mirror

She also made the commit you referenced and reported the 1s timelock bug 24 days ago.

13

It would appear that Anna (the person who made the commit you linked) is also the owner of the address: anna ā€” Mirror (used this account to also acquire $PARTY as you mentioned)

This makes sense when reading the Github issue Anna also created 24 days ago: recoveryTimelock mis-configured Ā· Issue #918 Ā· celo-org/optics-monorepo Ā· GitHub

1 Like
14

Thank you for sharing what you found. That does clarify things a lot. So Anna is also a core Celo contributor, and is the person who originally discovered and reported this 1-second misconfiguration with recoveryTimelock.

On a more broad note, while itā€™s nice to have an answer to that question, to be honest Iā€™m still confused about this entire situation. I donā€™t understand where any malicious action comes into play.

The nature of these transactions seems unclear, but so does their connection to James Prestwich in any negative way. I might be overlooking something; if that is the case then please correct me.

As a new user of Celo Iā€™m somewhat disappointed in the tone of the announcement. It comes off as a strong accusation and seems unwarranted by the surrounding context. I understand that Celo insiders likely have a more comprehensive understanding of this situation than any public observer. But I think itā€™s important to consider the potential negative impact of that tone when it comes to observersā€™ perceptions of Celo.

5 Likes
15

On what basis do you believe funds are not at risk, even though ownership of the bridge is now under unknown ownership outside of Celo/cLabs control? Unless Celo knows additional info about the bridge owner, it seems like the responsible course of action would be to evacuate funds from the current bridge and migrate to a new one.

3 Likes
17

Community Update #1 11/23/2021, Optics Recovery Mode

Hi all, Iā€™m Eric from the cLabs Dev Rel team. Yesterday we held a community call with the CEO of cLabs, Tim Moreton. In our conversation, the team committed to:

  • Record and publish the AMA video. The video has been published and is available here on YouTube.
  • Update the language on the Optics Bridge. The new language highlights that Optics Bridge is currently in Recovery mode. The bridge is operational. Users may bridge their tokens as they choose.
  • Share a contract audit with the community. We are working to release it and hope to do so shortly.

Also, here are a few notes to address some questions that have come up:

The cLabs Optics team is currently staffed by more than 10 people, including several engineers, a handful of folks from the early team, and people like me and others helping the community on Discord and forum. (Heads up the forum has a profanity bot that deletes posts automatically. If your post is removed, please revise and submit again.)

The bridge is operational and tokens bridged to and from Celo, Polygon and Ethereum are operational. The Optics Bridge process takes a minimum of 4 hours to complete. If your transaction has not been completed after this time, please visit the #bridge-support channel on Optics Discord.

For those asking what to do with your funds, please know that we are working to provide you with more guidance beyond what we said above, and will share additional thoughts in the next 24 hours at the latest.

Finally, we are putting together an incident report over the coming days as we continue to discover new facts

Thanks for your patience.

8 Likes
18

Optics v2 Announcement

After sharing yesterdayā€™s Optics v2 proposal in the #community-owned-optics channel on Discord and receiving feedback from the broader community in the AMA zoom call and Discord, we are seeing support for a new instance of Optics alongside a migration plan that will allow users to safely move their funds from the current Optics instance.

First and foremost, weā€™re focused on launching Optics v2 in a thoughtful and secure manner that maximizes trust and confidence in the new bridge. Given this goal, it will take a few days to stand up the new instance and properly audit it with the community.

Migration Plan

Since launching the new bridge will take some time, we are making recommendations for what people should do today to keep their funds safe until they are ready to migrate to the new bridge. We recommend that users perform the following actions as soon as possible:

  • For large holders not impacted by the cost of gas on Ethereum, we suggest using the current instance of Optics (v1) to move your funds back to Ethereum and then back to Celo using the newly deployed Optics v2 at a later time once it is launched.
  • For smaller holders not looking to be impacted by the price of gas on Ethereum, there are numerous non-optics assets that are not affected (e.g. cUSD, cEUR, cETH, cBTC) that we recommend trading into, and back to the Optics v2 assets once they are live.

Launch Plan

Based on the community feedback weā€™ve received so far, we are planning to move forward with the following:

Deploy Optics V2: Upon final testing and internal verification, we will publish the deploy config and work together with the community to explain it and have it checked and verified against the deployed onchain state. This config will include a 2 week timelock for entering recovery mode and an initial community-owned multisig controlling both the Governor and Recovery Manager.

Governance and Recovery Manager via Community Multisig: Before going live, we will assemble and agree as a community on a set of entities/projects to act as signers for the multisig. To get the deployment started, we have put together the following initial list but plan to amend this based on community feedback:

  • Moola Market (DeFi Dapp)
  • Mobius (DeFi Dapp)
  • Ubeswap (DeFi Dapp)
  • CeloWallet.app (Wallet)
  • cLabs (Core Development)
  • Censusworks (Validator)
  • Celo Foundation

Code Audit Released: An audit of the Optics code was completed previously and can now be viewed here.

Additional Planned Improvements

In addition to the above improvements around deployment and governance, Optics v2 will include a number of new features and enhancements which will be rolled out in the following weeks after launch:

Migration Page

We plan to release a migration page that will make it easier for people to move their assets between the two instances, but users do not need to wait for the migration GUIā€™s completion per the Migration Plan above.

Additionally, we are investigating creating additional pools, e.g. WETH (Optics v1) <> WETH (Optics v2) that would allow users to directly trade legacy optics assets for Optics v2 assets. We invite any parties interested in arbitraging these pools (and taking on any associated Optics v1 risk) to do so by sending assets over both bridges to maintain price parity.

Bridge Gas Refund

There is a CGP Proposal being discussed to automatically refund users for gas when sending funds towards Celo using a bridge. If passed, this proposal will make it easier and cheaper for users to bridge funds to Celo using future bridges.

Recovery Manager Timelock UI Notification

If the recovery manager activates recovery mode, it will take 2 weeks before it is enabled, and UIs will be modified to show a warning to users.

Faster and More Security

Weā€™re reducing the window before which funds are made available from 3 hrs to 30 mins. This means faster bridging. At the same time, we are working hard to enable the community to run Watcher Agents to further increase the security of the bridge (see the Optics design). We donā€™t have an exact timeline for this but stay tuned! Over time, there are other changes we plan to make with the community to strengthen the security of the system too.

Avalanche Support

Based on popular demand, Optics v2 will be deployed on the Avalanche C-Chain as well as Ethereum, Celo and Polygon, so that after testing, we can support bridging of AVAX and other assets.

Final Thoughts

We appreciate the communityā€™s feedback and participation in what have been some difficult conversations over the past few days. There are definitely lessons we have learned from this situation, and we will continue to look closely at what went wrong, and why. To that end, we plan on issuing a full incident report as soon as possible.

Now we are focused on this next chapter for Optics, and want to thank you for helping us get there. The team came up with a swift solution under difficult circumstances and I want to personally thank everyone involved for their efforts.

9 Likes
19

What about the current v1 bridge multi-sig?
Did Anna give back control of that multi-sig or is that still in limbo?

And as for ā€œWe invite all parties interested in arbitraging these pools to do so by sending assets over both bridges to maintain price parity.ā€, any funds sent over the v1 bridge is still vulnerable until control of the multi-sig is back in the orgā€™s control right? Not necessarily a zero-risk arb there.

20

Good point. Let me update the post to mention that risk.

1 Like
21

Is it possible to find a full list of affected tokens, so we can be totally sure weā€™re out of risk? Iā€™ve been reading about this topic on discord and here and I couldnā€™t find it.