Chaos Labs, a cloud security and testing platform for smart contract applications, joined the SES incubation program in November 2021. Since joining, Chaos Labs has been building an agent- and scenario-based simulation platform that enables robust testing of protocol performance under volatile market conditions, adversarial agents and heavy network traffic. Specifically, we’ve built a cloud environment that allows Maker engineers to test liquidations and auction mechanisms under different on-chain conditions.
You can read our previous update to the community here.
Chaos Labs Motivation and Work to Date
There have been previous attempts to create a production-grade testing platform for the Maker protocol, but the undertaking has been outside of the scope of the existing core units with complexity beyond their capacity for commitment. With protocols securing tens of billions in value across the ecosystem, testing and simulation tooling have never been more critical.
Chaos Labs focuses on developer tooling and cloud infrastructure, allowing protocol developers to move fast on stable infrastructure. Our aim is to abstract away the complications of development and scale to provide a simple to use, custom tool for MakerDAO developers with a future-proof, cutting-edge development experience. The platform is currently live for protocols on the Ethereum, Polygon & Avalanche networks with expected expansion to all major Layer-2 protocols.
While a part of the SES Program, we have been focused on a few major priorities:
- Deeply understand the Maker protocol to understand the specific complexities within both the codebase and community processes to build sophisticated, useful simulations
- Build up the infrastructure to support Maker Core Units at scale, reliably (platform flexibility, simulation scale, user flows, dashboards, etc.)
- Design simulation outputs to more clearly translate what is happening under the hood to the community
The mission of the Chaos Labs Core Unit is to provide a future-proof blockchain testing platform and services to enable MakerDAO to continue building products with best-in-class and easy to display security as the protocol enters a new era of technological and economic complexity. Although high-quality simulation environments are critical for developers, it isn’t our only focus. We will also be targeting verifiability as a high priority item moving forward. It should be easy for the Maker community to verify the results of simulations before participating in governance.
Why is economic security important?
MakerDAO already has a number of technical Core Units with one specifically focused on the areas of Oracles, Smart Contracts, Risk, and others — why does it need another to focus on “economic security”?
Since the rise of “DeFi summer”, we’ve seen nefarious actors managing to manipulate core protocols in increasingly creative manners, no longer are they just looking for flaws in the code, but they are manipulating the market around the target protocol to gain entrance and exploit it.
What Chaos Labs is building will provide each of these other specialty Core Units a new dimension by which they can use their expertise to test and secure the protocol, such as:
- Adding multi-block, on-chain simulations to the Risk Core Unit’s dashboard
- Test new auction/liquidation mechanisms and process flows for new assets
- Run new assets through black swan event simulations prior to onboarding
- Oracle manipulation testing
- Additional layer of testing for all new protocol updates to understand long-duration impact, even if indirect
- Run each governance proposal (spell) through a set of sanity simulations to ensure the core functionality of Maker persists
Challenges Unique to Maker
In working with the SES Incubation Program, the team has pushed us regularly on the concept of “why should Chaos Labs not just be a vendor to MakerDAO?” The reason is simple: Maker is not a simple fork of another protocol with 200 lines of code, but a massively complex system with its own structure, terminology, and ethos. A vendor will always be playing catch up to new changes and will entirely rely on stakeholders for broader context whereas a Core Unit will be deeply ingrained in the protocol, able to not just support, but also partner with other Core Units, developers, and community members to better test new assets, protocol changes, and economic mechanisms via robust simulations with well-understood edge-cases.
In contrast to other DeFi protocols that deploy completely new versions and deprecate older ones (think AAVE v1, v2, or Uniswap, etc), Maker has elected to continuously expand the existing offering and smart contract capabilities. This has led to it becoming one of the larger (and more unique) DeFi codebases that does not have a lot of conceptual ties to other, similar protocols. This large surface area needs to be covered end-to-end with automated run-time verification built by experts on each of the smart contract calls (bark, flip, dog, kick, etc.) and Maker-specific terminology.
To demonstrate the value and capabilities of the Chaos Lab Core Unit we decided to focus on the following themes:
- Oracle configuration - specialized for the Maker Oracle Security Model
- Auction & liquidation simulations
- Collateral onboarding
- Third party contagion testing (i.e. liquidation cascades)
- Auctions & liquidations are one of the most critical flows for Maker. In times of high market, volatility auctions are a line of defense that protects the protocol from insolvency. To this end, we partnered with the independent MakerDAO core unit, Sidestream.
Sidestream works on auction services for Maker which relies on the liquidation process to ensure DAI maintains its peg. The team needs to simulate and work with a variety of liquidation scenarios that may not have happened yet or that are very hard to recreate. Imagine the following scenarios:
- Massive Collateralized Debt Positions going underwater
- Understanding Keeper incentives as a function of on-chain conditions. Is it economically beneficial to liquidate a vault?
- Adding new assets and understanding their relative risk and liquidation mechanisms (i.e. how much debt can be liquidated with minimal slippage)
This challenge is perfectly suited for the Chaos Platform. Using the Chaos Platform we easily built a set of simulations that could support a wide variety of liquidations scenarios within the Maker protocol. Each simulation instance triggers an on-demand Ethereum fork allowing the simulation to run based on the latest on-chain data related to account balances, pool depth, and mainnet code deployed on downstream protocols. Any team using our platform could spawn multiple customized scenarios and replay them when needed: based on new code they want to test or on a recurring, scheduled basis to understand changing market dynamics.
Ultimately, the Chaos Platform now supports a series Maker liquidations simulations such as:
- Dropping prices of OSM oracles by n% of the original price per given collateral type
- This can be dropped immediately and over a certain number of blocks, or
- Via a generic capability to control over the price drop (drop by X every Y block starting from Z initial value) so it can be easily configured to Black Thursday scenario where the price dropped by 1.2% every 100 blocks over 3710 blocks total
- Configuring keepers to watch for given vault IDs or ilk type as they try to trigger liquidations for eligible vaults
- Triggering the full liquidation flow for: specific vault IDs and top eligible vaults per collateral type
- New asset additions and the auction/liquidation impact
- Surplus auctions & selling DAI
- Flips and Flaps have been integrated into the simulation engine
- Flops is on the near-term roadmap to be included
Before Chaos, there was no mechanism by which these teams could accurately understand what impact their proposed changes would have on the broader MakerDAO protocol using onchain, real data. The Sidestream team recently got access to the hosted Chaos simulation environment and has been utilizing simulations run via API and CI in their development environment. These tools allow for a significantly greater testing surface area at each stage of development to iteration on options for mechanism design and parameter setting to bring forth the best potential option for integration.
Below is an early iteration output for the Flap surplus auction 1 that we will walk through in our update call on Friday.
This simulation is manipulating Maker’s System Stabilizer Module (Flapper, Vow, Vat) to test the auction process of selling excess DAI for MKR - Surplus Auctions. The Fill Observer is tracking the amount of DAI being auctioned. The kick Event Counter Observer is counting Flapper kick events per block, each Kick event represents a new surplus auction starting
Collateral Onboarding and MakerDAO
Most recently we’ve been focused on helping the Sidestream team in understanding risk vectors and concerns around assets that the community wants to support. After the approval of rETH, they are working to prepare for its full integration into the protocol.
To properly simulate a new asset, we would need to have coordination across multiple Core Units to integrate the full functionality of the asset’s integration into the protocol and understand it’s potential negative impact, such as:
join- Token Adapter contract. specific for each collateral type.- Median contract - responsible for taking the pricing information provided by the Relayers, and verifying that the information is signed by the authorized list of Feeds
pip(OSM) - the contract which holds the current price of a given ilkclip- Auction manager contractclac- Auction price discovery curve. decided by the Auction Price Function gov proposal parameter
In partnership with Sidestream, we aim to build lasting infrastructure that will allow them to not only solve the problems at hand today, but provide tooling for future situations, as well. Instead of optimizing for just rETH, we want to optimize for a system that allows the user to test the risk spectrum for any asset listed in the future, even providing simulation outputs during the voting process instead of just after the community has approved a potential asset without full consideration of the risks involved.
With each new functionality within the MakerDAO protocol we are getting a step closer to full protocol simulation functionality robust enough to truly streamline testing for all Core Units and the community at large.
What's Next
On the MakerDAO SES side - Chaos Labs is nearing graduation of the SES Incubation Program and will be releasing its MIP set for Core Unit consideration in the coming weeks for community review. Over the coming weeks, alongside the Core Unit proposal process, we will continue work with the Sidestream team while engaging other Core Units & protocol developers to understand expanded simulation scope and feature needs.