Decentralized Oracles: Rewards Proposal

:wave: Celo Community!

As part of the decentralization efforts for Celo Oracles (See post), we would like to bring forth a Rewards Proposal for future Oracle partners.

Why
Partners will have costs related to Oracle uptime and maintenance. These rewards are meant to maintain a healthy Oracle operator community and continued support.

Without rewards, partners may not be able to cover costs or may not be incentivized to continue to keep supporting the network, even in times of distress. This also aligns with the current structure of funding validators to do the same.

Reward Payment Framework for Operators

  1. Initial Startup Payment - $3,000 / node
  • Will cover initial startup costs
  • Payed at time of activating oracle in CELO
  • E.x. Each oracle provider would support 3 nodes for all existing cStables (cUSD, cEUR, cREAL) for a total startup payment of $3,000 x 3 = $9,000
  1. On-going Monthly Costs - $750 / node
  • Will cover monthly expenses of Node, including support/servers
  • Payable in CELO using ReleaseGold Contract

With the existing cStables (cUSD, cEUR, cREAL), when a new partner is activated their costs will be:

  • $9,000 for initial startup setup costs
  • $2,250 / month for continued support

Rewards to be funded by the community forum in the same governance proposal where nodes are enabled. Total Celo amount is locked for a year upon activation, after which the CELO amount is recalculated and a new RG contract is deployed.

For reference, here is the RG Parameters:

releaseStartTime: target oracle activation date
releaseCliffTime: mainnet
numReleasePeriods: 12
releasePeriod: 1 month
amountReleasedPerPeriod: to be calculated on deploy time
revocable: true
beneficiary: Provided by oracle operator
releaseOwner: GovernanceProxy
refundAddress: GovernanceProxy
subjectToLiquidityProvision: false
initialDistributionRatio: 100%
canValidate: false
canVote: false

7 Likes

I think it’s good to introduce rewards for oracle operators. But how can we guarantee the uptime they will be getting rewarded for? I think it would be great to incorporate a way to slash these rewards based on their uptime or have of penalty for misbehaviour, similar to the validator penalties.

2 Likes

There’s currently no way to require a stake for Oracle providers and slash that in the protocol, though that would be nice in the future. For now, a governance proposal could revoke the remainder of the ReleaseGold contract, cancelling future payments. You could also delay receiving some of the “initial payment”, until the end of the year, effectively making that initial payment slashable?

2 Likes

Delaying a part of the initial payment makes sense to me. I’m not sure if this is already in place but do we have any requirements for an operator? How will we know an operator is actually even sending reports? What stops an operator from collecting the payments and not doing anything?

1 Like

I think we should put decentralized oracle operators into a “trial period” of say, a quarter, 3 months first, make sure that they get the system up and running and provide 99.99% uptime throughout the period before giving any kind of initial payment.

Ongoing, I feel like it’s expected for the oracle reporter to give a certain approximate number of reports per epoch, so that uptime SLA requirement should be required for any monthly reward to be distributed.

Btw, your proposed reward numbers are EXTREMELY generous. Thank you.

1 Like

I have always viewed cLabs oracles as the achilles heal of the Celo ecosystem, so I am excited to see progress being made on moving towards a more decentralized setup. Node Finance is interested in contributing to this effort. We will check back in once we are setup on Alfajores for a little bit.

2 Likes

You guys might wanna scale down the reward number per oracle because if a bunch more cStables get created, the total reward will get expensive. Maybe just cap it to a fixed amount per month, the same high availability server setup can easily report for N number of stables.

I doubt anyone is making a brand new kubernetes cluster per oracle, it’s more likely just additional pods and services in the same existing cluster.

1 Like

Thanks a lot for the interest!
Here’s the documentation on how to run them https://docs.celo.org/oracle/run
And also, cLabs security team is doing security assessments (similarly to the ones validators did), so please fill this form: Celo Oracle Questionnaire

Excited to see this initiative as well! Celo Tracker would love to run an Oracle and help with decentralization. We have infra in place using GCP HSMs, which might help cloud service diversification since the cLabs maintained ones run on AWS and Azure AFAIK.

I also agree the rewards are probably a bit high, especially since it seems like the cost doesn’t scale linearly with the number of stables as we can share the node and oracle instances for all of them like. @diwu1989 mentioned.

I can get a setup working on Alfajores and will reach out after :slight_smile:

4 Likes

I run my oracle report on GCP, and create oracle specific IAM service account that is assigned only to the oracle reporter machine. This oracle report machine has firewall block off all ports, the only way to remote access in is SSO into Google Cloud and use their authenticated Gcloud tool to get past the firewall. Machine has auto-restart and maintenance live-migration policy on it so GCP will move it around and keep it alive even if host machine has downtime.

From that machine service account, the oracle reporter docker instance uses the Google Secrets API to retrieve the actual private key. Keys are never in environment variable or on-disk, and loaded directly into memory of the oracle reporter nodejs process.

The original Celo Oracle reporter had integration for Azure and AWS key management, I added in the GCP Secret integration in my public fork (add gcp secret · diwu1989/celo-oracle@28328fb · GitHub)

Something like this can be cleaned up and upstreamed with unit tests and more validation to add official Google Secrets support.

1 Like

This is great feedback, thanks everyone! Would love to get some feedback/thoughts on a revised proposal, based on the community feedback above.

Revised Reward Payment Framework for Operators

  1. Initial Startup Payment - $3,000
  • Will cover initial startup costs
  • Payed at time of activating oracle in CELO
  1. On-going Monthly Reward - Dependent on number of cStables on network
  • Tier 1: Up to 5 cStables - $1000 / month
  • Tier 2: Up to 10 cStables - $1500 / month
  • Tier 3: 10+ cStables - $2500 / month
  1. Initial payment for new cStable launch - $300
  • Each new stablecoin launch will have a one-time payment of $300
  • This will cover costs to support a new cStable as we grow.

Other callouts

  • Will cover monthly expenses of node, including support/servers
  • Payable in CELO using ReleaseGold Contract
  • To provide validity/security/realiblity of partners, payments will locked for initial 3 months. This allows time for partners to prove reliability of their infrastructure.

With the existing cStables (cUSD, cEUR, cREAL), when a new partner is activated their costs will be:

  • $3,000 for initial startup setup costs
  • $1,000 / month for continued support
  • If we launch a new cStable (e.g. cCOP, cGBP, etc.) Oracle partners will receive $300 / one time payment per new coin launch.

I look forward to the day when celo has dozens of cStables with good peg and liquidity.

The current oracle reporter code is quite nice and can easily report on assets like Eth, btc, dai, usdc, etc…

Something to consider is I feel like chainlink honestly is so slow at deploying on celo that why don’t we just have decentralized oracles also report on official price feeds for the other non cStable assets.

Let’s say that we did have first class support for official sorted oracles on all of the big crypto assets, then we can market it as a feature of the network. Not that it excludes chainlink from coming to Celo but just that we been waiting for years for them and still no show…

3 Likes

Great Proposal! Virtual Hive would be very much interested to run an oracle and agrees with the updated Payment Framework. It is more reasonable now and should still cover the costs. We are a Mainnet Validator since August 2020 and will provide learnings and feedback after setting up an oracle test setup in Alfajores :slight_smile:

1 Like

We were wondering how the AWS HSM pricing is considered in the updated payment Framework. AWS charges 1.92 USD/h for CloudHSM usage in the EU, which amounts to approx. 1400 USD/month. This exceeds the Tier 1 monthly rewards. Thoughts on this?

GCP is cheaper, a lot cheaper.

Looks like the proposal to add my oracles passed and got executed.
Thanks everyone for the votes.

Happy to say that the 3 oracle reporters have been reliably updating every few minutes, with over 1000 reports on each already:

Seeing that the oracle actively report every 10 minutes, I’m going to set healthcheck alert to warn me whenever there hasn’t been a successful oracle tx from my reporter for that long.

6 Likes

Is there any natural way to integrate this into the validator client? Have the stake weight decide what the prices are?

Or would that be too difficult or have unintended consequences?

I think having decentralized oracles for non-stable assets is a great idea. Curious, any use cases you have on your side for non-price related feeds?

:wave:
Hopefully this will be the last update to this proposal. We’ve gotten some additional input from other users/validators who have global infrastructure and wanted to introduce V3 of this proposal (with some slight tweaks to the reward) .

This will apply to all existing and future Oracle partners that we onboard. We hope to bring this to a CGP in the next few weeks.

V3 Revised Reward Payment Framework for Operators

  1. Initial Startup Payment - $6,000
  • Will cover initial startup costs
  • Payed at time of activating oracle in CELO
  1. On-going Monthly Reward - Dependent on number of cStables on network
  • Tier 1: Up to 5 cStables - $1500 / month
  • Tier 2: Up to 10 cStables - $2000 / month
  • Tier 3: 10+ cStables - $2500 / month
  1. Initial payment for new cStable launch - $300
  • Each new stablecoin launch will have a one-time payment of $300
  • This will cover costs to support a new cStable as we grow.

Other callouts

  • Monthly rewards will cover monthly expenses of node, including support/servers
  • Payable in CELO using ReleaseGold Contract
  • To provide validity/security/realiblity of partners, payments will locked for initial 3 months. This allows time for partners to prove reliability of their infrastructure.
  • Based on actual support time, we expect to revisit these numbers a year from now

Example:
With the existing cStables (cUSD, cEUR, cREAL), when a new partner is activated their rewards will be:

  • $6,000 for initial startup setup costs
  • $1,500 / month for continued support
  • If we launch a new cStable (e.g. cCOP, cGBP, etc.) Oracle partners will receive $300 / one time payment per new coin launch.

Same reason why people need chainlink, need price feed for wETH wBTC etc… for any defi apps to price the loans.

Chainlink is not on celo and they drag their feet, why keep waiting for them?

1 Like