Jesus Christ, Chia Network. Jesus Christ. This is a bad one. Chia Network has, in a very rapid turn of events, eliminated an entire asset class from their blockchain (CAT1) and replaced it an entirely new asset class (CAT2), removed the original from the client and caused a ton of chaos with a short notice change to their Chia Asset Token standard. All from an extremely centralized position. Projects had about 24 hours on a Monday morning to reissue all their tokens or they would be useless.
But if that’s all it was, then cool. But the real problem here is a development attitude that prioritizes cool features and rapid deployment over actually testing to make sure stuff works properly. The issue at play here is that offer files between a CAT1 token and XCH were exploitable. And have been the whole time. Chia Network even decided to use this exploit in order to close offers and return transactions to people causing everyone with an open offer a tax nightmare. As well, because they didn’t do so instantly but set a “final block” for everyone, some offers were just plain accepted for ridiculous prices to get XCH out of people not paying attention, knowing that overspending the USDS after the cutoff block would be inconsequential.
Because of the way they did this, they guaranteed that people could be robbed because of it. Guaranteed it. They also guaranteed that any projects using CAT1s would be scrambling to change over their infrastructure and update code and that projects like Space Marmots in the process of using offer files to accept SM1 tokens now have to do a complete audit and will likely face user complaints about missing tokens. There is no possible way this is acceptable under Stably’s terms of service, and I don’t see how they aren’t going to suffer either a loss of confidence here or an actual loss of funds. If I have USDS in CAT1 form that i got after the cutoff block what legal recourse does Stably have to refuse me my return? I think the user above should take his 100 USDS in CAT1 and make Stably exchange it, then let them sue Chia Network for the difference. Regardless if his claims are true or not.
I don’t know what clogged brain came up with the idea of giving people 24 hours notice on this, just enough time to for malicious actors to create a plan of action but not enough time for anybody else to come up with a way to handle this exchange properly. There are, of course, idiots online who are defending Chia Network and this clusterfuck of a decision saying that “wow doing all this in 24 hours is amazing, what an amazing company”. No. Stop. This was a stupid decision made to fix a stupid mistake caused by stupidly not going through the rigorous testing process that financial exchange protocols should go through. Over and over and over again I have railed at this company for not following financial industry best practices when developing and releasing software. Did they get an audit? Yeah. After they had rolled the software into production and real businesses were using it for real work.
And to use the exploit they discovered against their own users? Wow. I understand the technical reasoning behind it, closing out open offers before they can be used against people in the way described above. But its a bad look. And they didn’t do that!!! They started after the cutoff block passed and took their sweet time on it. There was a window of opportunity there that malicious actors could exploit open offers of XCH for CATs. Unless there were only a few open offers total and they could do them all in one block then there was no possibility of doing it all with no exploitable window. The right way to do this, if you were going to force it down on everyone, would be to do it all as a “surprise, motherfuckers!” moment and cut off offers, freeze the chain and use the exploit to close every open offer in the system all at once. And announce it at the same time. The way they “planned” this the exploit didn’t even need to leak for people to use the chaos in order to steal from others.
Furthermore, they also lifted the thin veil of decentralization away from their company and network. This may be the most long term harmful result of this decision. First the decision to release a brand new financial exchange protocol into production pre-audit was made top down from Chia Network. (Edit: it was audited, just not enough apparently) Then the decision to just flip the table and break everything was also made top down by Chia Network. Nobody has a choice, there is no option. There is only “do what we say because we said it” both times. Nobody but Chia Network has any visibility or insight into the blockchain. Despite being open source and freely licensed it might as well be a Microsoft project for all the say the community has in the direction of the project. They parcel out some crumbs here or there, but at the end of the day Chia Network Inc is a private company and chia-blockchain is a software package designed explicitly to meet the goals and needs of that private company, and nobody else. This must change.
My instinct on how to start fixing that is that the Chia Blockchain needs a CAB (Change Advisory Board) to review all major updates to the chain absent the business needs of Chia Network. It is obvious in hindsight that the decision to launch offer files and CATs was made too soon in order to hit release windows rather than properly based on best practice Release Management for such critical software. A CAB filled up with people both internal to Chia Network and external would have a more diverse set of incentives and would not be so quick to approve changes without proper audits complete. Chia Network would be required to defend their procedures and those defenses would be recorded for post-mortem review. In theory there would be someone like me on that board whose first instinct for every release is “No. Why do you think this is a good idea, and do you think you have done your due diligence on it yet?”. That would definitely slow down development but at this point its clear that would be beneficial to everyone, including Chia Network.
If Chia Network really does want to set itself apart from Ethereum as a decentralized smart platform, they need to start soon. We have an XCH Foundation (although completely dissimilar to the Ethereum Foundation) and I think it is perfectly reasonable to ask that Chia Network strongly consider bringing in them or an outside board of stakeholders to tell them “No” when they need to be told no without a paycheck hanging over their heads. Its hard to tell your boss he’s making a mistake, even after the fact during a review. I strongly recommend that Chia Network approach the people at XCH Foundation, or someone else who isn’t directly associated with Chia Network, to assist with making these decisions. Or create an adversarial change management system internal to the company like other enterprises servicing the financial services sector do it, but that option is much more expensive.
Now, I don’t know what kind of liability Chia Network has created for themselves by bragging about how secure offer files are and how secure Chialisp is and advertising themselves on that fact. But the next time they market something as “secure” they should make goddamn sure it is first.
Oh yeah, go download Chia Blockchain 1.5.0 I guess.
Harsh, fair. Thanks for covering this in more detail.
What the reddit OP states in their thread makes no sense. You can’t have a debit for an offer, then two days later receive the credit side.
Just because that person is lying, it’s still a viable problem.
Sure, but using them as an example to illustrate seems like an odd choice.
They outlined the exact issue very well.
That user doesn’t even exist on Reddit…
Space Marmots = “real businesses were using it for real work.” ….. give me a break.
If there was a CAB in place the same exact thing would have played out. You said yourself an audit was done. Yeh, at this point it is easy to see the audit missed something. But at the time the due diligence was done and the update would have been approved (CAB or not). As with all of your articles for quite a while now, this one is totally biased and obviously shaded by the work you need to do to fix your system after their sudden change.
I used to enjoy your views on the Chia ecosystem but ever since you started work on the Marmot crap this blog has been very limited on scope and very biased. I hope you see this soon. It’s getting old.
Agreed, my visits to this site are much less frequent now and when I do visit, I’m usually disappointed (i.e. this article). It’s a shame, when this site first started it was a great resource for the Chia community.
That’s it, I stop reading your blog.
It was good last year, but now there are too many wrong information.
1. The cat issuers didn’t have 24 hours to migrate from Monday morning as you say, they had to reissue the coins after the 1.5.0 release (tuesday), whenever they can. No 24h hurry or whatever
2. This issue wasn’t due to a lack of testing, seriously. This is a security issue, that even the first audit didn’t manage to find. All the chia code is covered by auto tests now…
3. Chia made one of the best crisis management I saw for a long time. It’s a big crisis, as it happen in every projects. They handled it as good as it’s possible, providing update/scripts/documentation/video and the cat1 website to make this transition as smooth as it’s humanly possible. Maybe you’d have preferred a Solana solution : we stop the blockchain few hours/days?
When I see the horrible technical solution of the Space Marmots, I understand were all your indulgence went…
erbarr from keybase here. Yup agree, I smell the salt: https://twitter.com/hoffmang/status/1550893928718557185
The security audits sound (rightfully) ongoing, bugs exist in software and have for all time. It’s not fair to hold that against them as something that should have been “perfected” beforehand. It sounds like Chia team’s fast action here was just to ensure that they got rid of the exploit before it was exploited. Like you said, there is a lot of money on the line, and their actions could have been performed by someone malicious instead of themselves, and we’d be having an entirely different conversation.
Also Chia’s blockchain IS open source, it’s not the community’s problem that we need to wait for them to explain to us what the fix or exploit is. If we as a community were “smarter” (or there were even more money on the line) there’d be another conversation happening here. Maybe you could argue in this context, Chia team did not need to act as fast as they did. However, as an employee at a third party auditing firm, information about the exploit has a monetary value attached to it, and is really a ticking time bomb no matter how you look at it.
The lesser of the two evils
Your article is too emotional. We aren’t getting things up to speed if we keep on throwing mud at each other. Yes you can disagree that’s your right but don’t see what we prove with slinging mud to the XCH team.
what an absolute piece of garbage blog, but this is what to expect come from you, with your “write blog without the proper research, then backpedal later when you are wrong” style
We were told that chialisp would prevent this kind of shit.
Fucking cunts