Chaos Labs, a cloud security and testing platform for smart contract applications, joined the SES incubation program in November 2021. Since joining, Chaos Labs has been building an agent- and scenario-based simulation platform that enables robust testing of protocol performance under volatile market conditions, adversarial agents and heavy network traffic. Specifically, we've built a cloud environment that allows Maker engineers to test liquidations and auction mechanisms under different on-chain conditions.
Historical Context
There have been previous attempts to create a production-grade testing platform for the Maker protcol. The importance of this cannot be understated. With protocols securing tens of billions in value across the ecosystem, testing and simulation tooling has never been more critical. However, previous attempts at solving this were unsuccessful due to a subpar developer experience which hindered adoption. Some of the larger challenges were:
- Latency of simulations - could take up to 20+ minutes to run a simulation
- Lack of simulation tooling - creating a single simulation was a challenging and time-consuming process
- Lack of analytics - simulation results were difficult to understand for engineers working across the stack
Chaos Labs focuses on developer tooling and cloud infrastructure, allowing protocol developers to move fast on stable infrastructure. Our aim is to abstract the challenges above and provide developers with a future proof, cutting edge development experience. The platform is currently live for protocols on the Ethereum network and later expanding to additional blockchains such as Terra, Polygon and more.
Challenges Unique to Maker
Maker is one of the true blue-chip DeFi protocols. In contrast to other leaders, who deploy completely new versions and deprecate older ones (think AAVE v1, v2 or Uniswap etc), Maker has elected to continuously expand the existing offering and smart contract capabilities. This is excellent for a lot of reasons. Users don't need to migrate liquidity and can continue deriving value from a familiar protocol. However, on the engineering side this has led to one of the larger DeFi codebases. This large surface area needs to be covered end-to-end with automated run-time verification. To demonstrate the value and capabilities of Chaos we decided to focus on the following themes:
- Oracle configuration - specialized for the Maker Oracle Security Model
- Auction simulations
- Liquidation simulations
Auctions are one of the most critical flows for Maker. In times of high market volatility auctions are a line of defense that protects the protocol from insolvency. To this end, we partnered with the independent MakerDAO core unit, Sidestream.
Sidestream Collaboration
Sidestream works on auction services for Maker. Auction services rely on the liquidation process. The team needs to simulate and work with a variety of liquidations scenarios that may not have happened yet or that are very hard to recreate. Imagine the following scenarios:
- Black Thursday
- Massive Collateralized Debt Positions going underwater
- Understanding Keeper invcentives as a function of on-chain conditions. Is it econoimically benefical to liquidate a vault?
This challenge is perfectly suited for the Chaos Cloud. Using the Chaos Cloud we easily built a set of simulations that could support a wide variety of liquidations scenarios within the Maker protocol. As each simulation instance is triggered an on-demand Ethereum fork is spawned. Sidestream could spawn multiple customized scenarios and replay them when needed. Previously, Sidestream worked with a predefined set of auctions and collaterals to test their system.
As we worked closely with Sidestream we used their feedback to build tools and features to improve the experience of working with our cloud. These include:
- Secure API with programmatic access to create, initialize and interact with the simulations running on Chaos cloud
- RPC access to any simulation forks for debugging and interacting with the blockchain with a specified created scenario
- A CLI that automates the flow of triggering and connecting to a specified simulation
- Exposing a reverse proxy for the simulation node - enabling working with the nodes from localhost.
Ultimately, the Chaos cloud now supports a series Maker liquidations simulations such as:
- Dropping prices of OSM oracles by
n%of the original price per givenilktype - Configuring keepers to watch for given vault IDs or
ilktype as they try to trigger liquidations for eligible vaults - Triggering the full liquidation flow for: specific vault IDs and top eligible vaults per collateral type
What's Next
MakerDAO continues to grow, and so does its protocol and developer ecosystem. As new products are shipped and new teams are welcomed into the developer ecosystem, testing and security infrastructure are critical to ensuring that Maker continues to produce software of the highest quality. Although high-quality simulation environments are critical for developers, it isn't our only focus. We will also be targeting verifiability as a high priority item moving forward. It should be easy for the Maker community to verify the results of simulations before participating in governance.
On the MakerSES side - Chaos Labs is nearing graduation of the SES Incubation Program and beginning to draft our MIP set. Development work on the platform will continue, but the focus of the team will be dedicated to preparing our campaign to become a Core Unit. The MakerDAO community is encouraged to explore our Core Unit concept in greater detail by watching Chaos Labs' recorded Core Unit Overview and engaging with us directly :)