There are a number of network protocols that we might refer to as text protocols, where we send a line of text to the server and it sends back a response. Perhaps I should call them CRLF protocols, to distinguish from protocols where we’re blasting JSON back and forth. To speed things up, it can be tempting to have the client send multiple requests without waiting for individual responses.

This is called pipelining. Support varies by protocol. NNTP requires that servers support it. HTTP 1.1 defines it, but support is effectively dead because. SMTP offers it as an option, but otherwise requires that clients not pipeline due to server bugs. I thought this was surprising. How can this happen? In my mind, a simple server, unaware of pipelining, will read one request at a time, process it, and send the reply. The pending pipelined requests will simply wait to be read. What can go wrong?

What I came up with is that the server might be relying on implicit state to drive its state machine. A server of the time would likely be using select to multiplex IO. I’m imagining a server might assume that each connection can only create one wakeup for one reason at a time. We wake up to read from the net connection. We read the MAIL FROM, then start some operation to check that, and wait for a response. If we wake up again because the net connection has RCPT TO pending, we might reply to that before the from address has been checked. This is quite some time before SPF, but we can imagine something like it.

So the intended sequence goes like this:

fd 3 ready
read MAIL FROM alice
send DNS request
fd 4 ready
read TXT record
write 250 OK
fd 3 ready
read RCPT TO bobby
write 550 nobody home
fd 3 ready
read RCPT TO carol
write 250 OK

But if fd 3 is ready too soon, we might get mixed up.

fd 3 ready
read MAIL FROM alice
send DNS request
fd 3 ready
read RCPT TO bobby
write 550 nobody home
fd 3 ready
read RCPT TO carol
write 250 OK
fd 4 ready
read TXT record
write 250 OK

So now we rejected the MAIL FROM command and accepted the incorrectly addressed RCPT TO.

It would of course be better, and correct, to remember more specifically where we are in the connection, but I can imagine assuming that if commands arrive one at a time, we can save a byte and rely on the ready descriptor to guide us through the state machine. If there’s a command pending, it must be because the client received our previous replay. If there’s a DNS answer pending, it’s the answer the client is waiting for. Etc.

This is all speculation, but RFC 2920 offers some concrete examples. Mostly related to stdio buffers and fork/exec. If you read ahead on the stream, then exec a helper process, the eaten data disappears. This makes a lot more sense, given the time frame. I haven’t thought about stdio buffering bugs in a while, either.

There’s another note in the RFC about the dangers of deadlocking while pipelining, which was interesting. You can’t forget to read while you’re sending all these requests, or the server might get stuck with all the responses that don’t fit in the pipe coming back.