cLabs engineering and product updates
Sign up here to get the cLabs Tech Sync Newsletter in your inbox.
cLabs Tech Sync #2
We’re back. Welcome to the second cLabs engineering and product newsletter. This is the scoop on what’s on the minds and keyboards of the cLabs team. The most important part of any open source project is the people, and we hope to foster a little more of that connectedness.
Awesome interns
In the last two years cLabs has managed to find a significant number of truly remarkable interns, and even more remarkably managed to persuade a high percentage of them to join full time. Perhaps it’s because we do not shy away from giving them some of the most challenging work (with support, of course!). We’re highlighting some of those folks below. If you’re interested in joining cLabs for an internship, please get in touch!
Making protocol upgrades safe and easy
In order to deliver the most user-friendly cryptocurrency experience possible, Celo needs to be able to adapt quickly in response to user feedback and needs. Blockchain protocols, however, are notoriously difficult to upgrade quickly, safely, and in a decentralized manner. The Celo ecosystem has experienced these difficulties firsthand; since the launch of the Celo network back in April on Earth Day, many protocol features and bug fixes have been merged but remain unreleased. Fortunately, the cLabs Protocol team has been hard at work to ensure that these and future changes can be released safely, quickly, and in a way that gives everyone in the community a voice.
Upgrades to the Celo protocol can happen at two different layers, blockchain and contracts.
The blockchain layer consists of celo-blockchain, Celo’s fork of go-ethereum, and contains the logic around transaction processing, the EVM, and consensus. Shipping changes to celo-blockchain looks a lot like shipping changes to Ethereum. Some changes, like performance improvements, are fully backwards compatible, and relatively easy to ship. Others, like changes to the state machine, are “forks” that are not backwards compatible, and need to be shipped in a coordinated way such that every node changes the state machine that they’re running at exactly the same time. Mariano Cortesi, cLabs blockchain lead, is assembling changes and the team will be proposing the first hard fork next month — stay tuned!
The smart contract layer consists of the Celo core smart contracts, which encode the stability protocol, lightweight identity protocol, validator elections and proof-of-stake, and on-chain governance. Asa Oines and Martin Chrzanowski (who joined as cLabs’ first protocol intern in 2018 — and never left!) have been working on adding semantic versioning to contracts, adding checks in CI ensure storage layout compatibility, and building tooling and a release process so contract upgrades can be proposed via on-chain governance and verified by voters as soon as audits are completed.
Celo Core Contracts Release 1
The first upgrade to Celo Core Contracts, Release 1, is on its way. It will include a new approach to downtime slashing for validators, a buy with limit operation on the decentralized exchange (CIP-14), and support for transferring attestations which will support fee-less attestations.
Yorke Rhodes (another intern turned full-timer) and Emily Williams (recently joined from Summa) have been working with OpenZeppelin to audit and prepare for this release.
Learn more and see the timetable for the governance process on Claire Belmont’s forum post and join the community call on Oct 7.
Celo Blockchain Release 1.1.0
celo-blockchain 1.1.0 is out! This release fixes issues that can cause crashes to leave on-chain data in an inconsistent state — which a number of us saw in the production incident last week. It also adds flags to automatically use the genesis blocks and static nodes for each network, new RPCs to encrypt and decrypt, many new metrics, removes the build time dependency on Rust, and also adds upstream changes from go-ethereum up to 1.9.10. If you operate a node, please try it out on baklava first!
Making it Easier to be Available
There are more features in the pipe that node operators will be interested in.
We have long heard the woes of validators wanting to be able to make changes without risk of downtime. So to make it easier to confidently deploy (or rollback!) upgrades in the future, the team has been working on enabling replica validators to be hot-swapped using RPCs. Joshua Gutow, another of those awesome former interns, has been driving this. This will also enable a set of proxies to back more than one validator. This is a precursor to a longer-term feature where a set of validators sign together using threshold signatures.
While many of Celo’s parameters can be set by on-chain governance, an important one that can’t right now is the uptime lookback window — the number of consecutive blocks for which a validator can miss signatures before its uptime score is affected. Andres di Giovanni, who’s interning with us in Buenos Aires, has made this configurable on-chain, paving the way to raise it by governance proposal in the future.
Trevor Porter and Kevin Jue’s work on multi-proxy support is in final testing. It’s been quite a journey! I checked in with Kevin: “This PR ended being one of the largest in the history of the celo-blockchain repo. Not only did we want to add in multiproxy support, but we also did a major refactor of the original proxy code by creating two new golang subpackages within the istanbul package. A big driver for that was the fact that the cLabs blockchain team is quickly expanding, and we wanted to make sure that the code itself wouldn’t be a bottleneck by making it much more modular. Because of the size of the PR (and the fact that it modified part of the announce protocol implementation), we wanted to thoroughly test it by making sure it contained good unit test coverage and stress testing it by manually verifying it could handle ‘blue moon’ scenarios, such as a full validator set change on an epoch transition.”
Blockchain Throughput
The blockchain team kicked off a long-term roadmap of work on scalability and transaction throughput, to allow the network to absorb substantial user growth — for Valora, other wallets, and other services built on Celo — in the months to come. As anyone who has done performance optimization work will tell you, there’s often a bunch of low-hanging fruit, but it’s hard to predict exactly where you’ll get wins.
The first step was for Javi Cortejoso to build a repeatable load testing environment of 100+ validators into which we can inject latency, node and network failures. Javi used this to benchmark the current version, 1.0.1, and found that it could comfortably sustain load amounting to a block gas limit of 13M, 30% higher than the current level of 10M. Fruit lying literally on the ground! In fact, this is fortunate, because some of the Celo Core contracts are so large that they cannot be deployed at a lower limit (and there’s no easy way of splitting them up). So Celo Governance Proposal CGP 11 proposes to raise the limit accordingly, ahead of the upgrade to Core Contracts Release 1.
Tong Wang, who is interning with us in San Francisco, has been working on reducing validator CPU usage from assembling and verifying blocks. It turns out each validator was doing this work three times: when building a block to propose (even though only one out of 100 validators are likely to use it — a legacy from a codebase built around proof of work), when verifying a proposed block, and finally when that committed block was eventually gossiped out. Tong’s changes could improve throughput by up to 3x.
Expect more updates about scalability in the months ahead, but this is a promising start.
Researching Account Recovery
The Valora team is considering options for making account recovery easier and safer. To many users, understanding and safely storing mnemonics can be a challenge.
Maggie Valentine, who’s just completed a Product Design Internship, delved into the world of social recovery, a process where a user designates a trusted individual who can recover the account together with the user. Her research goal was to gain an understanding of user feelings and behaviors around social recovery. She created an interactive prototype that prompted users to choose their designated “recoverers” and go through the recovery flow.
I chatted with Maggie about her findings: “I heard many users who started under the assumption that their designated ‘recoverer’ would be able to see their financial transactions, so felt it was a feature designed for parents/children. As they learnt more, people developed a strong preference for having a tech-savvy contact, and strongly preferred partners to parents. Most users were comfortable with the idea of social recovery being an ‘add-on’ feature, but were wary when presented with it as the only recovery option. Some were concerned about the burden of responsibility it places, and others were uncomfortable with having to depend on someone else for recovery. Overall, there’s a lot of promise, but a number of usability and education challenges to consider.”
Meet the team! Yaz Khoury
We’re going to introduce you to one member of the team in each newsletter. In this edition, Yaz Khoury, who has just joined as Developer Relations Engineer with a focus on validators and node operators. He’s yaz#4327 on Discord. In his own words:
“I have been a fan of cutting-edge technology and social impact projects for a long time so my involvement in crypto communities in the last few years is based on that. I was Developer Relations director at the Ethereum Classic Cooperative where I led a lot of initiatives for onboarding developers. Before that, my career was in software engineering at social media analytic startups and working as an electronic engineer on several robotic projects, including an award-winning Mars habitat-building robot for NASA.
I am super excited about joining cLabs to help grow the Celo community! Celo’s vision and values is the main thing that originally attracted me to the crypto space. I was always on the lookout for how Bitcoin would help marginalized communities suffering extreme inflation by centralized governments. Bitcoin however is harder to use for the average person, and this is where Celo really improves upon the concept and truly helps banking the unbanked without anything except the phone in their pockets.
I look forward to helping bring Celo’s vision forward and to help every validator staking on the network to protect this vision. I am currently based in Montreal, Canada, having recently moved there from Manhattan, New York where I spent the last five years building my career.”
Events you shouldn’t miss
cLabs Hackathon demos — Oct 6
- The cLabs hackathon is underway and the community is taking part! To see what’s being built, check out #hackathon on Discord.
- Join the final demos on this Hangout on Oct 6 8am PT, 1500 UTC
Core Contracts Release 1 Community Call — Oct 7
- Learn about and discuss governance proposals to enable the first core contracts upgrade on this community call 8am PT, 1500 UTC.
- Keep an eye on #governance on Discord for more details.
Validator Happy Hour — Oct 8
- Bring coffee or beer and hang out virtually with other validators, node operators and developers. This time it’ll be Thursday Oct 8, 6pm PT / 9am CST / 10am KST.
- Keep an eye on #validators-announce on Discord for more details.
Celo Tech Talk — Oct 9
- Interested in contributing code to Celo Blockchain but don’t know where to start? In Celo Blockchain Internals Deepdive, Kevin Jue will explain how everything fits together
Developing on Celo Office Hours — every Tuesday
- Hangout with cLabs developers about building on top of Celo and using the SDK.
- 10/13 at 4pm PT, 10/20 at 8am PT. Calendar of future events here.
Get involved
If you’re interested in following the progress more closely, please take a look at the celo-org Github projects, and join the Celo Discord server. If you’d like to contribute to the Celo ecosystem, please don’t hesitate to reach out!
Happy hacking, building and operating!
Tim
Sign up here to get the cLabs Tech Sync Newsletter in your inbox.
