A few thoughts after reading The History of a Security Hole about a series of bugs in the OpenBSD kernel. It’s a good explanation of an instance of a problem I’ll call hard state soft state confusion, which can lead to some serious bugs, occurs with some regularity, but doesn’t seem to be often discussed.

Hard state is what the hardware sees. Most simply, it’s the contents of hardware registers, CPU and other devices, but also includes memory which those registers may point to. Soft state is what the kernel sees, and uses to program and initialize the hardware. Bad things happen when the hard state and soft state don’t agree.

As an extreme example, every process has a bunch of state, like user id, open files, etc., including virtual address space, for which there is a page table. If the kernel switches processes, but forgets to set the CR3 register to point to the correct page table, flaming death will ensue.

As a trivial example, hardware frequently resets itself back to default values following suspend and resume (or hibernate). You may have turned the volume all the way up, but after resume discover it’s very quiet again. The kernel forgot to reprogram the hardware. My ThinkPad suffers a similar affliction where the LED on the mute button becomes inverted, and turning mute off silences the speakers while turning mute on plays music. Such is life.

One technique to simplify code and hopefully prevent such problems is to reuse the hard state as the soft state. In cases where the hard state is a region of memory, we can simply define the soft state to be a pointer to this memory. This doesn’t work in all cases, because it’s not feasible to represent all hardware in C, but defining a struct that looks like memory is a common solution. Now every update to the soft state is guaranteed to propagate to the hard state. Problem solved.

Except now we have the potential for a new problem. If somebody modifies the soft state, or its definition, without realizing that it’s actually hard state, we are back to bad things happening. For instance, the order of fields shouldn’t matter much to C code, but it matters very much to hardware expecting fixed sizes and offsets. See above.

Here’s another fun bug. The hard state includes the current CPU registers. The soft state includes the value of those registers for all the processes not currently running. When task switching from one process to another, the current values are saved, and the saved values from the about to run process are restored. There is the potential for some hard state soft state confusion here if the CPU and the kernel do not agree about how many registers there are. This sounds kind of unlikely, but with the evergrowing set of XMM/SSE/AVX extensions and registers, Intel added an XSAVE instruction to save all those registers, and some care must be taken to only save registers the kernel has allocated soft state space for.

A similar class of bug can occur in networking code. It’s very convenient to reuse the wire format for internal data to reduce translation, right up until somebody starts modifying its layout.