In February, Twitter user Brodan, an engineer at Giphy, noticed something odd about Bored Ape Yatch Club (BAYC), the premiere ape-based non-fungible token collection. A record intended to cryptographically prove the trustworthiness of the bored apes contained 31 identical entries, a situation that was supposed to be impossible. “There’s something super-suspicious about some of your apes,” Brodan wrote.
Six months later, when the newsletter Garbage Day brought it to wider attention, Brodan’s query still hadn’t been answered. The situation is all too common in the crypto industry and the wider open-source community, and raises the question of whether there’s something fundamentally wrong with the idea that a crowd of amateurs can effectively hold large projects to account.
The issue lies with an obscure record called the “provenance hash”. This is a record, published by BAYC’s creators Yuga Labs, that is intended to prove there was no monkey business (sorry) in the initial allocation of the apes. The problem the team had to solve is that some apes are rarer – and more valuable – than others. But in the initial “mint”, they were allocated randomly across the first 10,000 to apply. To prove they were distributed randomly, rather than a few valuable ones distributed to insiders, they published a provenance hash: a list of cryptographically generated signatures for each of the 10,000 apes, showing that the apes had been pre-generated and pre-assigned, without revealing what their characteristics were.
So far so good, except that 31 of those signatures were identical. Since the 31 apes they were assigned to were distinct, that means the provenance record for those apes was broken – and they could, theoretically, have been changed to match the desires of the person who bought them.
Earlier this summer I asked Yuga Labs about the duplicates, and the company initially pointed to the circumstantial evidence that it hadn’t pulled a fast one: none of the 31 apes had gone to anyone with insider connections, nor had they been generated with particularly desirable traits. Which is true – but also unsatisfactory. If you learned that your burglar alarm had never been wired up by the company that installed it, “Nothing’s gone missing, has it?” would only be a partial answer.
When pushed, the company examined the problem further, and found the cause of the problem: when it was preparing the provenance hashes, it triggered a rate-limiting error from the server storing the images of the apes. That error meant that, 31 times, rather than generating a cryptographic signature of a picture of a monkey, the company unknowingly generated a cryptographic signature of the error message “429 Too Many Requests”. Oops.
I asked Yuga Labs co-founder Kerem Atalay, who works under the handle Emperor Tomato Ketchup, whether he felt the multi-year gap between the problem and its resolution undercut the justification for provenance hashes. If no one is checking these things, what’s the point? “I think in this case, perhaps the reason it went unnoticed for so long is that this is such a heavily scrutinised project to begin with,” Atalay said. “The provenance hash became a less important feature of this whole project the moment it exploded. If a single pixel had changed in the entire collection after that point, it would have been extremely glaringly obvious.”
In that telling, provenance hashes are useful to rebut accusations of favouritism – but if there are no accusations, it’s not surprising that no one checks the hashes. Yuga Labs made a similar defence for another years-long oversight, spotted a few months ago: the company had kept control of the ability to create new apes whenever it wanted, despite promising to destroy it. Unlike the provenance hash, that ability was noticed rapidly: in June 2021, Yuga Labs said they would be fixing the oversight “in the next day or two”.
In fact, it took over a year. “While we’d been meaning to do this for a long time, we hadn’t out of an abundance of caution,” Atalay tweeted. “Felt comfortable doing it now. All done.”
Such issues are by no means confined to Yuga Labs, or the crypto sector at large. Last week, Google’s cybersecurity team, Project Zero, announced the discovery of a new security vulnerability in Android. Well, it was new to them: the exploit had already been used by hackers “since at least November 2020”. But the root cause of the bug was older still, and had been reported to the open-source development team in August 2016 – and a proposed fix had been rejected a month later.
That is years of meaningful security weaknesses for almost every Android phone on the market, despite the problem being visible in the public record for anyone to see.
It’s unclear how long that vulnerability had been present in the code, but in other situations that time can be the source of major problems. In April, a flaw was discovered in a command-line tool called Curl that had been present for 20 years.
And last December, a weakness in a logging tool called Log4j was discovered that was, the National Cyber Security Centre said, “potentially the most severe computer vulnerability in years”. The bug was hardly complex, and an attacker would barely have had to try before potentially taking control of “millions of computers worldwide”. But it had sat, undiscovered, in the source software for eight years. That oversight was not only embarrassing for people who believed in the security model of open-source software, but also catastrophic: it meant that affected versions of the software were everywhere, and the ongoing cleanup process might never be completed.
Tiny bugs, big problems
Open-source software such as Log4j underpins much of the modern world. But over time, the basic assumptions of the model have started to show their weaknesses. A small piece of software, used and reused by thousands of programs to end up installed on millions of computers, should have all the eyes in the world scanning it for problems. Instead, it seems, the more ubiquitous and functional a piece of software, the more people are willing to rely on it without checking. (There is, as ever, a relevant cartoon from the web comic XKCD).
In a perverse way, crypto has solved some of these problems, by putting a tangible economic benefit on finding bugs. The idea of a “bug bounty” is nothing new: a large software developer like Apple or Microsoft will pay people who report security vulnerabilities. The idea is to provide an incentive to report a flaw, rather than build malware that abuses it, and to fund the sort of crowdsourced investigation that open-source software is supposed to encourage.
With crypto projects, there’s effectively an in-built bug bounty running 24/7 from the moment they’re turned on: if you are the clever person who finds a bug in the right crypto project, your bug bounty can be … all the money that project holds. So when hackers from North Korea found a hole in video game Axie Infinity, they made off with more than half a billion dollars. The downside of such an approach, of course, is that while bugs are discovered quickly, the project tends not to survive the experience.
For Yuga Labs, the saving grace is that the only people who could abuse the oversights were Yuga Labs employees themselves, who rapidly came to be seen as trustworthy enough to not worry about. But investors in the broader crypto ecosystem would be good to be wary: even if someone says they’ve published proof they are trustworthy, experience shows that there’s no reason to believe anyone has checked it.
The wider TechScape
