Thought this was a good read exploring some how the “how and why” including several apparent sock puppet accounts that convinced the original dev (Lasse Collin) to hand over the baton.
Thought this was a good read exploring some how the “how and why” including several apparent sock puppet accounts that convinced the original dev (Lasse Collin) to hand over the baton.
The first 3 seem incredibly far-fetched.
I think it’s likey that, of all the mainstream compression formats, lzma was the least audited (after all, it was being maintained by one overworked person). Zstd has lots of eyes on it from Google and Facebook, all of the most talented experts in the world on data compression contributing to it, and lots of contributors. Zlib has lots of forks and overall probably more attention than lzma. Bz2 is rarely used anymore. So that leaves lzma
Cloudflare deploys Zstd, and many web servers and CDNs use it. Endless possibilities for Facebook and US gov. They can put Yann Collet out of the way or gag order him.
LZMA is the highest compression algorithm outside of PAQ and SuperRep+LOLZ, while being magnitudes faster than both. Zstd compression ratio is a joke and is only good for webpage asset loading times.
Facebook may be evil but I don’t think they’re anywhere near “inject malware into global supply chains to push adoption of a public engineering side project that they don’t directly profit from and most executives don’t care about” level of evil. Is it possible? Sure anything is possible, but that is wildly beyond many many more plausible explanations and there’s zero evidence leading us down this path. And why would they go through the trouble of backdooring zstd, which has a highly observed codebase, when they just successfully backdoored lzma because it didn’t have a lot of maintainers?
While it’s true that zstd is commonly favored for having “good” compression at blazingly fast speeds, which is useful on the web and on servers, Zstd 's max compression setting (
zstd --long -19) is actually within about 5% of LZMA’s but faster, so it replaces most use cases of LZMA except when that extra 5% (and that’s not even constant; some inputs are even better on zstd) really does matter at all speed costI have extensively benchmarked Zstd and it is a joke compared to LZMA2 when it comes to compression ratio. And not even that, the lack of features Zstd has, that 7Z does have, makes it a far bigger joke. 7Z is a feature complete archival solution unlike Zstd, with possible options for archive repair. RAR is far superior for that bitrot resistance.
The amount of possibilities Facebook and US gov get with backdooring XZ are endless, since it could destroy trust in it if uncaught, and Zstd adoption meant web malware deployment could become a matter of when, because Facebook already does it right now with actual malware JS scripts through fbcdn domain.