Some thoughts on what it means to be POSIX compliant and why OpenBSD deliberately fails to comply in some cases. For background, I think the POSIX standard is a bit ridiculous, but that’s more about the absurdities one encounters when super strictly interpreting a standards document. This is more about refusing to follow along with even a more even handed interpretation.

two steps forward, two steps back

The new MySQL hotness, MariaDB, uses a function called setcontext. This function was part of POSIX, until it wasn’t. New code, but written against an old standard. Even if one climbs on the latest, greatest standard treadmill, there’s no moving forward. Some developer somewhere will expect you to support all of the standards, all of the time.

shm_open

Among the new functions all the cool kids are dying to use is shm_open. Read all about it. Short version: it’s open with a different name. Among the interesting notes from the rationale section: “the shm_open() function may be implemented using a macro that invokes open()” and “A future version might require the shm_open() and shm_unlink() functions to have semantics similar to normal file system operations.” In other words, an implementer can, at least for today, have shm_open do something rather un-open-like, but since most implementations are going to return a normal file descriptor and therefore software will inevitably come to depend on that behavior, we reserve the right to make that part of the standard. Not today, but maybe tomorrow.

Personally, something like shm_open may have been useful if it at least did some of the ftruncate/mmap dance for you. But it doesn’t. It does nothing for you except open a file.

party like it’s 1999

When this first came up in the OpenBSD project, we took a look at what programs were doing with shm_open. Maybe it would be easier to patch a handful of them, instead of adding another interface. Maybe we could even convince upstream to fight the good fight against ever expanding standards. har har.

What did these programs do, more or less universally? They used shm_open to reinvent /tmp races like it’s 1999. With hand-rolled random filename generators that are pathetic, even by mktemp standards (worth noting that mktemp itself was finally removed from POSIX and replaced by mkstemp). None of the software examined appeared to want to coordinate between multiple users, except inadvertently (insecurely). Most programs, like webkit, did make a half hearted, but ultimately ineffective, effort to keep its shared memory private to the user.

Adding shm_open to OpenBSD would mean exposing our users to these bugs. On the one hand, it’s not really our job to exclusively protect users from stupid software. And library functions are just tools, it’s not their fault they’re misused. But at the same time, I didn’t want to be the guy who builds something knowing it’s going to be misused.

sharing is not caring

After much debate about how to proceed a compromise was reached. As with any good compromise, nobody really liked the outcome. We would implement shm_open, but only a crippled version that didn’t fully support shared memory. Different processes of the same user could use it to share memory, but sharing memory between users is strictly forbidden. The permission mask is reset to forbid anyone but owner access, and if by chance you open an existing file, we check its owner. Nobody is allowed to “gift” you a shared memory file. (The downside is now we have this oddball function which will eventually cause some future program to fail in mysterious ways.)

shm_open was committed at t2k13. I didn’t mention it in my hackathon writeup because the code was technically written sometime before, we just happened to have several interested parties all in the room together to break the deadlock of apathy. It will be part of OpenBSD 5.4.

Yes, /tmp races and insecure file permissions exist in tons of existing programs. And yes, many people have devised per-user temp directory solutions that OpenBSD has resisted. This just seemed like one straw too many. If anybody had used shm_open to meaningfully share memory between users, maybe we would have implemented the full version (which, ironically, would have been less work). But the existing usage is basically a shibboleth for crappy code.

the standard after tomorrow

The sad thing is mkstemp would have been the ideal function for these programs to call. It gives you back a nice, clean, safe file descriptor which you can mmap to your heart’s content. But it’s not like the newest, hottest POSIX function. Why use a secure function introduced in last year’s standard when you can prove how hip you are by using an insecure function introduced in this year’s standard?

Every time a new interface is added, it takes time for cultural/institutional knowledge about its sharp corners to develop. The faster new interfaces are added, the less we know about any of them. And then we repeat the mistakes of yesterday. And by yesterday, I mean last century.