Governance proposal to activate Celo stability protocol and enable cUSD transfers

Hi Everyone,

cLabs is planning to submit a governance proposal to activate the Celo Stability Protocol, which aims to enable a family of stable value assets, such as cUSD.

You may learn more about the details and background of this governance proposal on GitHub in the Celo Governance Proposal (CGP) folder here.

Anticipated timeline:

  • June 24: cLabs plans to submit the proposal; CELO holders will have 24h to upvote.
  • June 26: If dequeued, approvers review the proposals.
  • June 27: If approved, CELO holders have 48h to vote YES, NO, or ABSTAIN.
  • June 29: If the proposal passes, the Celo stability protocol will be activated to enable Celo Dollar transfers.

If you have any comments or questions we suggest replying here (vs. reaching out on Discord, Telegram, or email) so as to keep the conversation asynchronous and inclusive.

We look forward to hearing your thoughts,

The cLabs team

4 Likes

Afaik there hasn’t been any other legitimate exchanges that have started trading CELO other than Bittrex right?

Is the plan to go through with this proposal despite dependency on only a single exchange? Or should we expect some new announcements before June 24th?

1 Like

There might be an update in the near future.

2 Likes

Since we are now close to voting time, here are some more concerns from me:

  • Afaik, reporting oracle code is still not open source. This seems like a pretty big issue. If there is still some waiting for a security audit or something like that it would be useful to know at least. Or just some more information about what the hold up is to have the code open source would be useful. It is quite problematic that code for this very important part of the network is still not publicly visible.

  • Since Oracles are centralized for now, it would definitely help to know more details about oracle setup, and access restrictions. This article doesn’t have much detail around access control: https://medium.com/celoorg/an-introduction-to-celo-oracles-fd1a534669bb. Important questions in my mind would be:
    ** Who/how many people have access to Azure account where HSM keys are stored?
    ** Who/how many people have direct access to oracle machines?
    ** Who has access to deploy new code/image to oracle machines?
    ** What does internal or external auditing of this system look like? Is there an audit trail for all potential actions that might cause changes in Oracle operation?

  • There is still a concern on number and quality of exchanges that have picked up CELO, but this is less of a concern for now compared to the two above. Bittrex is a reasonable exchange but with medium to low volume overall. OkCoin is even less real volume, and also has history of fair bit of questionable events in its past. Its unfortunate that these are the only two options for now, it isn’t end of the world, but it will would definitely be concerning if this still remains the case 2-3 months from now.

2 Likes

Hey @thezviad!

Thanks for bringing up these concerns, I can add more information on the status of the oracles.

Afaik, reporting oracle code is still not open source.

I agree ideally the code should be open source— we’re currently waiting for a 3rd party security firm to audit the oracle code before it gets open sourced. It’s hard to know timelines exactly, but if I had to guess it will be a couple weeks before the code is released.

Who/how many people have access to Azure account where HSM keys are stored?
Who/how many people have direct access to oracle machines?
Who has access to deploy new code/image to oracle machines?

There’s a separate Azure subscription that only includes a small group of relevant engineers and is meant for production services (most of cLabs is on GCP, not Azure). The resource groups that house the mainnet oracle infrastructure are locked down such that only the relevant on-call engineers (6 people) have read access, and in order to change the infrastructure one of these on call engineers has to submit a request for more permissions (we do this using Privileged Identity Management to give just-in-time time-boxed permissions). Pushing a new Docker image to the container registry also requires a PIM request to get the sufficient permissions. Oracles are deployed in two AKS clusters that are split across 2 regions. In the coming days we plan to stand up a Kubernetes cluster in a third region with full nodes ready to go that we can fall back to in case of downtime in one region. AWS (who also has secp256k1 HSM support) support is also being worked on so we don’t depend too heavily on Azure.

What does internal or external auditing of this system look like?

An internal code review was done by a separate group of engineers who did not work on any of the oracle development. There was also an internal review of the security of the infrastructure prior to the mainnet deployment. Deployments are always made to Baklava as a staging environment prior to Mainnet.

From a monitoring point of view, there are dashboards with metrics exposed by the oracle clients themselves as well as relevant on-chain data. There are also a number of alerts for oracle client & on-chain issues (eg a client has an error, hasn’t reported, the number of on-chain reports is low, the on-chain rates are dramatically different, etc).

It’s also clear that the worst case isn’t necessarily if oracles stop reporting values (this will result in buckets not updating, which protects the reserve, and stops on chain exchanges from occurring if there is an expired report), but if the oracles report bad values that could deplete the reserve or result in a depeg. Given that, there are a number of “safegaurds” in the code mostly related to verifying exchange data is robust enough for use— these are things like only considering an exchange if the bid/ask spread is small enough, or refusing to report a price if less than N exchanges have provided exchange rates that are robust enough, or refusing to report if the price has changed a lot in a short period. Because this means there can be cases in which oracles not sending reports is actually intended, there’s been some talk on sharing some oracle metric charts (or a status page of some kind) to make it publicly clear what the state of the oracles are. Some ideas have been to share a stackdriver or grafana dashboard, but I’d love to hear any thoughts you have on this topic.

Is there an audit trail for all potential actions that might cause changes in Oracle operation?

On the Azure level, changes in permissions & resources are logged. On the K8s level, audit logging is recorded.

Let me know if you have any other questions. Thanks for bringing these points up!

4 Likes

Thanks Trevor for such a thorough answer. It might be useful if the high level points of how oracles are secured and how its access control is managed gets published somewhere too. Maybe just in docs.celo.org under protocol/stability sections, since it is pretty crucial part of the overall network and protocol security for stable tokens (i.e. cUSD).

As for stats, maybe stats.celo.org can expose those oracle reporter stats too? That would keep all core network/protocol stability and security type of stats all in one place.

2 Likes

Hey @thezviad! As part of the open sourcing of the Oracles, we’ll also release a set of blog posts and documentation around the architecture and security model. It’d be great to get your feedback once we release this. Thanks for raising the discussion.