Earlier today the senior staff from the Chia Network technical team went to Zoom and YouTube for a post mortem of the Chia Dust Storm that caused a number of issues, especially for low powered farming hardware, on Halloween weekend.
Overall the presentation was good, they clearly outlined what happened and even produced a couple of nifty slides for us to check out. They outlined the nature of the performance patches they put out, which is nice, and explained how they are testing in testnet with much larger loads to ensure this doesn’t happen again. They also have prepared a post-mortem document on GitHub.
I don’t think that’s enough. I don’t think any mission critical production system should be run without at least 4 separate environments – Development, Testing, User Acceptance Testing/Pre-Production and Production. I think the Chia team is doing Dev-Test-Prod deployment which isn’t good enough for mission critical applications. Now that they are branching into real enterprise applications for their technology they are going to have to treat this seriously. These kind of unknowns can’t just happen to a system running the world’s carbon markets.
I have said this before, and I will reiterate, that I think they need to do use one, or multiple, fork networks as a testbed for stuff. There are real people with a variety of hardware and network connections, exactly the problems they are attempting to solve synthetically. Or they could set up their own UAT network. A real mainnet coin that runs a week ahead of the main Chia blockchain and is used to test DeFi and stuff but with coins that people can trade and keep. The CentOS to the Chia-Blockchain RHEL.
The post mortem part of the Q&A is pretty interesting if you are into the technical details behind Chia and the consensus mechanism but if you are just a farmer updating when a new version is released a lot of the details don’t matter here. Justin and Mariano both did very clear explanations of their pieces of the puzzle, and the exact changes and optimizations made to deal with transaction storms make sense to me now.
I really would like to talk to the Dust Stormer again, and get their thoughts on the optimizations Chia has made as well as the changes to the ecosystem in the last couple of weeks, with the World Bank announcement, the Costa Rica partnership and the release of CATs and the light wallet. So if you are reading this, and you want a follow up, please reach out and we can do our Satoshi dance again.