Yesterday I jumped the gun committing some patches to LibreSSL. We receive advance copies of the advisory and patches so that when the new OpenSSL ships, we’re ready to ship as well. Between the time we receive advance notice and the public release, we’re supposed to keep this information confidential. This is the embargo. During the embargo time we get patches lined up and a source tree for each cvs branch in a precommit state. Then we wait with our fingers on the trigger.
What happened yesterday was I woke up to a couple OpenBSD developers talking about the EBCDIC CVE. Oh, it’s public already? Check the OpenSSL git repo and sure enough, there are a bunch of commits for embargoed issues. Pull the trigger! Pull the trigger! Launch the missiles! Alas, we didn’t look closely enough at the exact issues fixed and had missed the fact that only low severity issues had been made public. The high severity issues were still secret. We were too hasty.
Before moving on to some other thoughts, the screw up yesterday was our fault. It wasn’t intentional, but we could have done better. Like some other OpenBSD developers, I have a somewhat dim view of embargoes (more on that soon enough), but if it’s not our bug to disclose, we shouldn’t disclose it. I don’t happen to like the rules, but there’s no satisfaction in breaking them like there is with jaywalking. Next time we wait for the official signal.
Premature release is a common failure mode of embargoed vulnerabilities. Infamously, long long ago, Dan Kaminsky was going to reveal a major vuln in DNS on stage in Vegas, complete with fire breathing white tigers. To ensure people came to Blackhat to see the show, he preannounced the vuln, minus details, one month in advance. As you may guess, things didn’t go according to plan. Dan did get to go on stage, and the FBI didn’t shred his handouts, but the big reveal was spoiled by some idle speculation, some less than idle speculation, and some accidentally on purpose confirmation and clarification. (Thomas says the publication really was an accident, though I note the blog post was certainly written and queued for eventual publication on purpose. Another example of the difficulty in simultaneously maintaining a secret while preparing to discuss it.)
There are several difficulties maintaining embargoes. Keeping secrets is against human nature. I don’t want to be the one who leaks, but if I see something that looks like the secret is out, it’s a relief to be able to speak freely. There is a bias towards recognizing such signs where they may not really exist. (Exacerbated by broad embargoes where some parts leak but other parts don’t. It’s actually very hard to tell what’s not publicly known when you know everything.)
The most thorough embargo and release timeline reconstruction is the heartbleed timeline. It’s another great case study. Who exactly decided who were the haves and have nots? Was it determined by who needed to know or who you needed to know? Eventually the dam started to crack.
The theory behind the embargo is it allows all the good guys to patch first before the bad guys develop exploits. But does it? I don’t have the science numbers at hand, but exploited systems aren’t necessarily broken within hours of patch announcement. You get owned for failing to apply last week’s patch, not this morning’s patch. (One can hope. Some vulns are so trivial to exploit by the time you’ve read the announcement exploits are already in the wild. Does the embargo still help in that case?)
It’s an unresolved question whether the time getting a patch from upstream to a vendor is more important than getting a patch from vendor to user, but I will argue we should be looking at the end to end time, from discovery to end user patch. Embargoes and coordination, by necessity, prolong the process.
The embargo clearly doesn’t include all affected parties. Did your access point auto update today? Your thermostat? Your bathroom scale? Hopefully they all update at some point, but in the mean time, the race is on.
When Cloudflare brags that they get advance notice of vulnerabilities, attracting more customers, and therefore requiring even more early access, how are smaller players to compete? What happens if you’re not big enough to prenotify?
Sometimes vulnerabilities are announced unplanned. Zero day cyber missiles are part of our reality, which means end users don’t really have the luxury of only patching on Tuesday. They need to apply patches when they appear. If applying patches at inconvenient times is a problem, make it not a problem. Not really a gripe about embargoes per se, but the scheduled timing of coordinated release at the end of the embargo is catering to a problem that shouldn’t exist.
There’s also the kind of but not really embargo. As it happened, yesterday there was a Medium post mentioning some potential shell injection vulns in ImageMagick, but this was quickly replaced with a web site that mostly talked about magic bytes and mitigations. Discerning that the real problem was filenames containing shell metacharacters was left as an exercise. Of course, it’s pretty hard to undisclose a vuln, and even dropping mitigation hints will lead anyone who cares to know more right to the bug. But disclosing only workarounds, and only partial workarounds at that, leads people to believe they are safe when they aren’t. Advice to edit a particular config file only works if nobody is using a patched installation that prioritizes a different config file. What, linux distros patch software to use non default configs? Now you tell me!
This results in a situation where the good guys think they’re secure, but don’t know how to check, while the bad guys quickly develop working exploits. And soon, perhaps not today, but later this week, the bad guys will be offering free testing services to check if your mitigations have been applied correctly. If a vulnerability is public, people shouldn’t need to dig through forum posts and comments to discover its true nature.
Solutions? Shorter embargoes. Or no embargoes. Find a bug, fix a bug. Preferably one at a time.