Orbifolds and Other Games

Staying Safe with Open Source

Thu 24 January 2019 by Moshe Zadka

A couple of months ago, a successful attack against the Node ecosystem resulted in stealing an undisclosed amount of bitcoins from CoPay wallets.

The technical flow of the attack is well-summarized by the NPM blog post. Quick summary:

  1. nodemon, a popular way to run Node applications, depends on event-stream.
  2. The event-stream maintainer has not had time to maintain it.
  3. right9control asked event-stream maintainer for commit privileges to help maintain it.
  4. right9control added a dependency on a new library, flatmap-stream.
  5. flatmap-stream contained malicious code to steal wallets.

Obfuscation

A number of methods were done to disguise the attack.

The dependency was an added in a minor version, and a new version was immediately released. This meant that most projects, which pin to minor, would get the updates, while it stayed invisible on the main GitHub landing page, or the main npm landing page.

The malicious code was only in the minified version of the library that was uploaded to npm.org. The non-minified source code on both GitHub and npm.org, as well as the minified code on GitHub, did not contain the malicious code.

The malicious code was encrypted with a key that used the description of other packages in the dependency tree. That made it impossible to understand the attack without guessing which package decrypts it.

The combination of all those methods meant that the problem remained undetected for two months. It was only luck that detected it: the decryption code was using a deprecated function, and investigating the deprecation message led to the issue being figured out.

This bears thinking about: if the code had been written slightly better, the problem would have still be happening now, and nobody would be the wiser. We should not discount the possibility that currently, someone who followed the same playbook but managed to use AES correctly is still attacking some package, and we have no idea.

Solutions and Non-Solutions

I want to discuss some non-solutions in trying to understand how this problem came about.

Better Vetting of Maintainers

It is true, the person who made this commit had an obviously-auto-generated username (<word>-<digit>-<word>) and made few contributions before getting control. But short of meeting people in person, I do not think this would work.

Attackers adapt. Ask for better usernames, they will generate "<firstname>-<lastname>" names. Are you going to disallow my GitHub username, moshez? Ask for more contributions, you will get some trivial-code-that's-uploaded-to-npm, autogenerated a bit to disguise it. Ask for longer commit history, they'll send fixes to trivial issues.

Remember that this is a distributed problem, with each lead maintainer having to come up with a vetting procedure. Otherwise, you get usernames through the vetting process, and then you use those to spam maintainers, who now are sure they can trust those "vetted".

In short, this is one of the classical defenses that fails to take into considerations that attackers adapt.

Any Solution That Depends on JavaScript-specific Things

This attack could easily have been executed against PyPI or RubyGems. Any solution that relies on JavaScript's ability to have a least-access-based solution only helps make sure that these attacks go elsewhere.

It's not bad to do it. It just does not solve the root cause.

This also means that "stop relying on minified code" is a non-solution in the world where we encourage Python engineers to upload wheels.

Any Solution That Depends on "Audit Code"

A typical medium-sized JavaScript client app depends on some 2000 packages. Auditing each one, on each update, would make using third-packages untenable. This means that start-ups playing fast and loose with these rules would gain an advantage over those who do not. Few companies can afford that pay that much for security.

Hell, we knew this was a possibility a few months before the attack was initiated and still nobody did code auditing. Starting now would mostly mean availability bias, which means it would be over as soon as another couple of months go by without a documented attack.

Partial Solution -- Open Source Sustainability

If we could just pay maintainers, they would be slightly more comfortable maintaining packages and less desperate for help. This means that it would become inherently slightly harder to quickly become a new maintainer.

However, it is worthwhile to consider that this still would not solve the subtler "adding a new dependency" attack described earlier: just making a "good" library and getting other libraries to depend on it.

Summary

I do not know how to prevent the "next" attack. Hillel makes the point that a lot of "root causes" will only prevent almost-exact repeats, while failing to address trivial variations. Remember that one trivial variation, avoiding deprecation warnings, would have made this attack much more successful.

I am concerned that, as an industry, we are not discussing this attack a mere two months after discovery and mitigation. We are vulnerable. We will be attacked again. We need to be prepared.