reversing an openbsd kernel syspatch
OpenBSD has provided binary patches for a select few architectures for a while now, to save users from the daunting task of running make on their own. Alas, this means you might now apply a patch without first reviewing it. In the olden times, you had a source patch, so obviously you meticulously studied every line before application, just like you advised new users on IRC to do. But now, who will believe you do this when the binary syspatch is right there, so easy, so tempting.
Ah ha, you say, I will simply tell everyone that I inspect the binary files. Excellent idea. The binary syspatch should match the changes in the source patch; after all, it comes from the same place and you’re already trusting them with just about everything else, but you get extra internet points every time you say trust but verify. Let’s dig just deep enough that we might plausibly brag to our friends, that oh yes, we totally do this for every syspatch.
syspatch
First, we’ll need the syspatch. These aren’t actually linked anywhere on the website, since you aren’t supposed to need them, but it’s not particularly difficult to discover where they live. For this example, I’m using syspatch71-001_wifi.tgz for the 7.1 001 wifi erratum. Download that, then check its certificate of authenticity. Accept no substitutes.
$ signify -C -x SHA256.sig syspatch71-001_wifi.tgz
Signature Verified
syspatch71-001_wifi.tgz: OKtar tzf reveals it’s just a bunch of files, located somewhere in /usr. There’s no special structure to a syspatch. It’s literally just files that get dropped into the file system. (There’s some backup and sanity checks in the syspatch script, of course, but the file doesn’t require much in the way of deep inspection.)
zxf, preferably in a directory other than /, and we’ve got some kernel objects to inspect.
blowfish tartare
Our inspection will probably go a bit smoother if we have something to compare with. On a live system, you’d just look at the files in /usr. But if we’re offline, or whatever, we need another copy. The files should be in the base71.tgz set, but I failed to find them on my first attempt. Had to a look a little closer to realize that the kernel object files are distributed in their own kernel.tgz file. Tarception!
I was all set to explain how to extract files from an inner tar without extracting the whole thing, but this does not appear possible with OpenBSD tar, even with fancy fd redirection. Side quest failed. If you install the gtar DLC, however, there is a -O option.
Anyway, you run tar twice, and we’re ready to compare files.
vers.o
I’ll start with a very simple file, and a very simple analysis tool. There was a funny bug in the first version of this syspatch where it would update the system into a nonupdateable state. A reasonable thing for us to double check. The binary syspatch should match the source patch, but sometimes it doesn’t. Trust, but verify.
I just ran hexdump -C and diffed the output.
--- v0 Tue May 24 11:01:07 2022
+++ v1 Tue May 24 11:01:44 2022
@@ -3,21 +3,21 @@
00000020 00 00 00 00 00 00 00 00 f8 07 00 00 00 00 00 00 |................|
00000030 00 00 00 00 40 00 00 00 00 00 40 00 0c 00 01 00 |....@.....@.....|
00000040 4f 70 65 6e 42 53 44 00 37 2e 31 00 47 45 4e 45 |OpenBSD.7.1.GENE|
-00000050 52 49 43 2e 4d 50 23 34 36 35 00 00 00 00 00 00 |RIC.MP#465......|
+00000050 52 49 43 2e 4d 50 23 30 00 00 00 00 00 00 00 00 |RIC.MP#0........|
00000060 20 20 20 20 40 28 23 29 4f 70 65 6e 42 53 44 20 | @(#)OpenBSD |
00000070 37 2e 31 20 28 47 45 4e 45 52 49 43 2e 4d 50 29 |7.1 (GENERIC.MP)|
-00000080 20 23 34 36 35 3a 20 4d 6f 6e 20 41 70 72 20 31 | #465: Mon Apr 1|
-00000090 31 20 31 38 3a 30 33 3a 35 37 20 4d 44 54 20 32 |1 18:03:57 MDT 2|
-000000a0 30 32 32 0a 00 00 00 00 00 00 00 00 00 00 00 00 |022.............|
+00000080 20 23 30 3a 20 53 75 6e 20 41 70 72 20 32 34 20 | #0: Sun Apr 24 |
+00000090 30 39 3a 33 30 3a 34 33 20 4d 44 54 20 32 30 32 |09:30:43 MDT 202|
+000000a0 32 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |2...............|
000000b0 4f 70 65 6e 42 53 44 20 37 2e 31 20 28 47 45 4e |OpenBSD 7.1 (GEN|Sure enough, the build string has changed, but no changes to the status, like -stable. After that there’s a bit more.
@@ -34,9 +34,9 @@
00000360 00 00 00 01 01 09 09 03 00 00 00 00 00 00 00 00 |................|
00000370 03 40 00 00 00 04 4c 00 00 00 04 00 02 00 00 00 |.@....L.........|
00000380 00 8b 00 00 00 01 01 0a 09 03 00 00 00 00 00 00 |................|
-00000390 00 00 03 40 00 00 00 04 4c 00 00 00 0f 00 02 00 |...@....L.......|
+00000390 00 00 03 40 00 00 00 04 4c 00 00 00 0d 00 02 00 |...@....L.......|
000003a0 00 00 00 ad 00 00 00 01 01 0b 09 03 00 00 00 00 |................|
-000003b0 00 00 00 00 03 40 00 00 00 04 4c 00 00 00 45 00 |.....@....L...E.|
+000003b0 00 00 00 00 03 40 00 00 00 04 4c 00 00 00 43 00 |.....@....L...C.|
000003c0 02 00 00 00 00 cf 00 00 00 01 01 0d 09 03 00 00 |................|
000003d0 00 00 00 00 00 00 03 40 00 00 00 08 4c 00 00 00 |.......@....L...|
000003e0 00 02 00 00 4f 70 65 6e 42 53 44 20 63 6c 61 6e |....OpenBSD clan|Hmm, a few more bytes have changed. Even a tiny change like switching a je to jne instruction could have catastrophic effects. In this case, I’m pretty sure the difference relates to the two byte difference in the previous string, but we may have exhausted the potential of hexdump here.
what
Moving on to a real file, an.o, diffing hexdumps is a disaster. We need something a little more intelligent like objdump. The regular output of objdump isn’t very amenable to diffing, either, because even tiny changes propagate throughout the file, affecting offsets everywhere. Nothing we can’t solve with a bit of code, though.
This seemed like an easy bit of awk to write, then I read the manual. Hexadecimal numbers convert to 0. This is even mentioned as a feature! Or a bug fix. Woe to those who’d like to process hexadecimal inputs.
Anyway, this isn’t hard. So I wrote a little utility called what. Where hides a trap? Basically...
re_symbol := regexp.MustCompile(`^([[:xdigit:]]+) <(\w+)>:\n$`)
re_insn := regexp.MustCompile(`\s+([[:xdigit:]]+):\s*((?:[[:xdigit:]] ?)+)\s*(.*)\n$`)
for {
line, err := r.ReadString('\n')
if msym := re_symbol.FindStringSubmatch(line); len(msym) > 0 {
symstart, _ := strconv.ParseInt(msym[1], 16, 0)
for {
line, err := r.ReadString('\n')
if minsn := re_insn.FindStringSubmatch(line); len(minsn) > 0 {
addr, _ := strconv.ParseInt(minsn[1], 16, 0)
offset := addr - symstart
decode := minsn[3]
fmt.Printf("insn 0x%x:\t%s\n", offset, decode)And now the output is mostly cleaned up. There’s a few cases where the offsets have changed, but not by much.
-insn 0xe2: je 1de6 <an_txeof+0x106>
+insn 0xe2: je 1df6 <an_txeof+0x106>It’s funny that objdump knows we’re jumping to the same offset within the function, but still prints the fixed file offset, breaking our diff. Whatever.
re_insnfixup := regexp.MustCompile(`(callq?|jmpq?|ja|jb|jn?e|jge?|jn?s)\s+([[:xdigit:]]+)`)
if re_insnfixup.MatchString(decode) {
decode = re_insnfixup.ReplaceAllString(decode, "$1 xxx")
}More of the same.
-insn 0x0: mov 0(%rip),%r11 # 2fe7 <an_set_nwkey+0x7>
+insn 0x0: mov 0(%rip),%r11 # 2ff7 <an_set_nwkey+0x7>re_commentfixup := regexp.MustCompile(`# ([[:xdigit:]]+)`)
if re_commentfixup.MatchString(decode) {
decode = re_commentfixup.ReplaceAllString(decode, "# xxx")
}Okay, finally, getting some good results.
-insn 0x36a: mov 0x1(%rsi),%al
-insn 0x36d: test $0x40,%al
-insn 0x36f: je xxx <an_rxeof+0x37d>
-insn 0x371: and $0xbf,%al
-insn 0x373: mov %al,0x1(%rsi)
-insn 0x376: orb $0x1,0xfffffffffffffe80(%rbp)
-insn 0x37d: mov %r14,%rdi
-insn 0x380: callq xxx <an_rxeof+0x385>
-insn 0x385: mov %rax,%r12
-insn 0x388: movzbl 0xffffffffffffff89(%rbp),%eax
-insn 0x38c: mov %eax,0xfffffffffffffe88(%rbp)
-insn 0x392: mov 0xffffffffffffff80(%rbp),%eax
-insn 0x395: mov %eax,0xfffffffffffffe84(%rbp)
-insn 0x39b: lea 0xfffffffffffffe80(%rbp),%rcx
+insn 0x36b: movq $0x0,0xfffffffffffffe80(%rbp)
+insn 0x372:
+insn 0x376: mov 0x1(%rsi),%al
+insn 0x379: test $0x40,%al
+insn 0x37b: je xxx <an_rxeof+0x389>
+insn 0x37d: and $0xbf,%al
+insn 0x37f: mov %al,0x1(%rsi)
+insn 0x382: orb $0x1,0xfffffffffffffe80(%rbp)
+insn 0x389: mov %r14,%rdi
+insn 0x38c: callq xxx <an_rxeof+0x391>
+insn 0x391: mov %rax,%r12
+insn 0x394: movzbl 0xffffffffffffff89(%rbp),%eax
+insn 0x398: mov %eax,0xfffffffffffffe88(%rbp)
+insn 0x39e: mov 0xffffffffffffff80(%rbp),%eax
+insn 0x3a1: mov %eax,0xfffffffffffffe84(%rbp)
+insn 0x3a7: lea 0xfffffffffffffe80(%rbp),%rcxThere’s the heart of the change. But still a fair bit noisy. It’s obvious most of the instructions are the same, but the offset change is still causing trouble. Smash it!
offset = 0And presto!
insn 0x0: callq xxx <an_rxeof+0x358>
insn 0x0: add $0x48,%r14
insn 0x0: mov 0x10(%r15),%rsi
-insn 0x0: movl $0x0,0xfffffffffffffe80(%rbp)
+insn 0x0: movq $0x0,0xfffffffffffffe88(%rbp)
insn 0x0:
+insn 0x0: movq $0x0,0xfffffffffffffe80(%rbp)
+insn 0x0:
insn 0x0: mov 0x1(%rsi),%al
insn 0x0: test $0x40,%alThat looks like a pretty reasonable assembly change for the source patch.
--- sys/dev/ic/an.c 25 Feb 2021 02:48:20 -0000 1.78
+++ sys/dev/ic/an.c 21 Apr 2022 22:24:17 -0000
@@ -462,7 +462,7 @@ an_rxeof(struct an_softc *sc)
#endif /* NBPFILTER > 0 */
wh = mtod(m, struct ieee80211_frame *);
- rxi.rxi_flags = 0;
+ memset(&rxi, 0, sizeof(rxi));
if (wh->i_fc[1] & IEEE80211_FC1_WEP) {
/*
* WEP is decrypted by hardware. Clear WEP bitWhew! Now just need to repeat that for the 64 remaining files.
wrap up
If I thought about it, preserving the offset was never necessary for this approach. I was imagining we’d need some more sophisticated processing. And maybe we would for a more complicated change. There’s a reason I selected a very simple patch, although it’s a nice showcase of how kernel patches are distributed. Maybe next time we’ll try using a tool that’s actually designed for binary analysis and forensics instead of banging magnetic rocks together.
Having realized how simple the task is, I leave you with this sed liner.
function sedit { sed -E -e 's/^[[:xdigit:]]+ </</' -e 's/^ *[[:xdigit:]]+:([[:space:]][[:xdigit:]]{2})* *//' -e 's/(callq?|jmpq?|ja|jb|jn?e|jge?|jn?s) +([[:xdigit:]]+)/$1/' -e 's/# ([[:xdigit:]]+)//'; }; objdump -d base/usr/share/relink/kernel/GENERIC.MP/an.o | sedit | (exec 3< /dev/stdin; objdump -d 001/usr/share/relink/kernel/GENERIC.MP/an.o | sedit | diff -u /dev/fd/3 -;)You know it’s good because it even ends with smiley.
Tagged: openbsd