Welcome back to the MakerDAO Simulation Series, where we highlight how the Chaos Labs platform can be used to enhance the testing of key protocol infrastructure in stateful, orchestrated environments. Though this series specifically focuses on some of our work during the MakerDAO SES program, the Chaos Labs platform can help you iterate on enhancements and optimize for a variety of potential parameters.
If you haven’t read the introduction to the series, you can check it out before continuing.
If you prefer video, check out this tutorial:
In this installment of the series, we simulate Black Thursday & Flip Auctions for the Maker Protocol. Through simulating Ether’s 43% drop on Black Thursday, we were able to provide insights for MakerDAO on the risk implications of collateral auctions in volatile markets and a baseline for future functionality testing.
Simulation Overview
In MakerDAO, collateral Auctions (“Flip Auctions”) are used to sell assets from Vaults that have become undercollateralized in order to preserve the overall collateralization of the system. Since this simulation works directly with MakerDAO’s Auction functions, it’s important to have a baseline understanding of the syntax for auctions.
While MakerDAO’s collateral Auctions function similar to other similar protocols, the taxonomy used by MakerDAO is a bit unique. For the purpose of understanding this simulation, here are some terms you need to know:
cat: the system's liquidation agent, it enables keepers to mark positions as unsafe and sends them to auction.bite: bite can be called at anytime but will only succeed if the Vault is unsafe. A Vault is unsafe when its locked collateral (ink) times its collateral's liquidation price (spot) is less than its debt (art times the fee for the collateral rate). Liquidation price is the oracle-reported price scaled by the collateral's liquidation ratio.- The Cat sends bitten collateral to the Flip module to be auctioned off to keepers. The collateral auction has two phases: tend and dent.
- Flip (Collateral) Auction is the primary functionality that ensures the stability of Maker and DAI, therefore it is considered a critical functionality that must be thoroughly tested and verified.
In short, Maker’s liquidation agent (cat) lets users mark a vault as unsafe when the value (spot) of the collateral (ink) falls below the liquidation ratio (art x rate). If the vault is unsafe, cat will identify (bite) the collateral, triggering a collateral auction (flip auction).
Check out Maker Protocol’s technical docs for more information on Maker’s Auction frameworks.
Simulation Set Up
With a basic understanding of how Flip Auctions work, we are ready to create a (simulated) black swan event for MakerDAO.
Using the Chaos Labs platform, we can configure specific environments for user testing, including oracle return value manipulation and wallet balance adjustments. For this simulation, we are going to manipulate the MakerDAO Oracle to trigger the expected smart contracts. To simulate the pain of Black Thursday (ETH -43% in 14hrs), the simulation drops the price feed provided by the MakerDAO oracles by 1.2% every 100 blocks over 2500 blocks.
As a infamous historic event in Ethereum history, Black Thursday significantly impacted the MakerDAO ecosystem, with nearly 4,000 liquidation transactions happening on the platform. As a result, the Sidestream team (a MakerDAO core unit) tests MakerDAO’s auction framework against this scenario to determine whether the protocol can handle the auction phase and scale.
With the simulated event set up, we can use the Chaos Labs platform to monitor key outputs throughout the simulation. In this event, we were primarily concerned with two outputs: ETH price and active auctions data. For active auctions data, we analyzed the number of active auctions, the total amount of ETH-A available for purchase, and the total amount of DAI needed for auctions to cover the debt.
Simulation Flow
With everything set up, we let the on-chain transactions flow unabated to see how agents interact with the protocol. Specifically, we are seeing the following play out:
- Price of ETH drops 43% in 14hrs
- Several vaults become under-collateralized
- Keepers bark and kick the vaults preparing them for auction
- Flip Auctions are triggered
- Vaults get liquidated
To see the simulation in action, you can watch the product demo alongside a walk-through of how the Chaos Labs platform:
- Interacts with the data;
- Sets up agents & observable; and
- Creates observable data types throughout the simulation.
Simulation Takeaways
Following this simulation, there were a number of takeaways for both the Chaos Labs platform and for the MakerDAO team. For Chaos Labs, the demo demonstrated how the platform can be impactful for a wide array of protocols and developers looking for more sophisticated testing environments.
Through the Black Thursday simulation, we can see how testing environments can be easily controlled, including:
- Protocol state: Users have the ability to create a fork with any desired state, even before it exists on the Mainnet. Composability: Chaos Labs can build on top of existing simulations or building blocks (agent/scenarios) to quickly spin up a new simulation. Any code deployed to the Chaos Labs platform can be directly transferred to the mainnet so you don’t need to make changes between testing and deployment.
- Integrations: How our simulations can be connected with an RPC interface: wallets, code, or any utility that is used to talk to Ethereum Mainnet. In this case, Sidestream can interface with our simulation forks exactly how they interface with production.
- Observability: The ability to track what happens in the simulation through easy-to-build & understand visualizations. Observers that track the system auction and events can be reused for multiple simulations and purposes.
For MakerDAO, the demo gave Sidestream Auction Services the opportunity to test their system with scale-intensive simulation states - high throughput of auctions. By creating a seamless testing environment that is 100% compatible with mainnet production, the team could make adjustments that could be directly deployed to their mainnet.
In addition to the specific benefits to Sidestream, Chaos Labs provided automated and reproducible tests on-demand to verify collateral offboarding and ensure that altering clipper configurations yielded the desired result.
In turn, the Chaos Labs platform and Black Thursday simulation helped increase MakerDAO’s security and protect the platform through:
- Automated integration testing of auctions
- Severe risk implications as collateral auctions are not closed and Maker is left with bad debt in volatile markets.
- Chaos provides a test environment for them, other CUs, and the community to feel confident in the execution in volatile markets.
About Chaos Labs
Chaos Labs is a software company building a simulation platform to allow teams to efficiently test protocols and understand how they will react to adversarial market environments. The backbone of our technology is a cloud-based, agent- and scenario-based simulation platform that allows users to orchestrate blockchain state, test new features, and optimize risk parameter selection.
Our technology allows users to:
- Orchestrate protocol/blockchain state
- Generate wallets with behavioral attributes
- Test protocol performance in chaotic market conditions
- Optimize risk parameters
Our mission is to secure and optimize protocols through verifiable agent- and scenario-based simulations.
The Chaos Labs simulation platform is built to emulate a production environment. Each simulation runs on a mainnet fork with the chain's current state so that your simulations include up-to-date account balances and the latest contracts and code deployed across DeFi. You cannot look at your protocol in a silo when you're testing adversarial environments. You must ensure you understand how external factors such as cascading liquidations, oracle failures, variable gas fees, liquidity crises, and more will impact your protocol in various situations.