For the past few months, my iPhone has had a peculiar bug. Apple services didn’t work in my house. I could listen Amazon music, but not Apple music. I could update my Facebook status, but not the Facebook app itself. I could read Apple’s website and learn about security updates in the latest version of iOS, but not download them.

If I disabled wifi, all of these things became possible. Of course, that meant burning through cellular data. And important iOS updates can’t be downloaded except via wifi. If I walked down to a Starbucks and used their wifi, I could download everything. By process of elimination, I concluded the problem was not the phone, Apple’s services, the wifi radio, etc. The problem was local to my home network. (Or perhaps somewhere one step beyond, in the ISP network, but that seemed a somewhat less likely suspect. Also, multiple other local wifi networks worked.)

Knowing nothing about how Apple update works, but speculating about technologies I doubt work in my house, I figure maybe they use some peer to peer sharing service? Or something other than TCP or UDP and it’s not getting NATed correctly? How to identify the problem? In tcpdump we trust.

Running tcpdump on my router and attempting to check for updates a few times was sufficient to spot a suspicious packet. There was a fair bit of noise at first because every app is constantly chattering away, but eventually it settles down and every update attempt ends with the same outbound packets before failing.

18:45:20.904650 10.3.3.53.55928 > 10.3.3.3.53: 45472+ A? mesu.apple.com. (32)
18:45:21.913779 10.3.3.53.55928 > 10.3.3.3.53: 45472+ A? mesu.apple.com. (32)
18:45:24.925548 10.3.3.53.55928 > 10.3.3.3.53: 45472+ A? mesu.apple.com. (32)

Not such a strange packet, actually. Just a DNS request. But sure enough, trying to lookup this host on my laptop fails as well.

host mesu.apple.com
;; connection timed out; no servers could be reached

The problem was definitely only (or at least, mostly only) mesu.apple.com. Other Apple hosts worked fine.

host www.apple.com  
www.apple.com is an alias for www.apple.com.edgekey.net.
www.apple.com.edgekey.net is an alias for www.apple.com.edgekey.net.globalredir.akadns.net.
www.apple.com.edgekey.net.globalredir.akadns.net is an alias for e6858.dscc.akamaiedge.net.
e6858.dscc.akamaiedge.net has address 104.106.255.68
e6858.dscc.akamaiedge.net has IPv6 address 2001:559:1b:18b::1aca
e6858.dscc.akamaiedge.net has IPv6 address 2001:559:1b:186::1aca
e6858.dscc.akamaiedge.net has IPv6 address 2001:559:1b:19a::1aca

I’m running unbound on my router, so the next thing was to try bypassing it and using a public DNS server like 8.8.8.8. Instead of mucking about with the settings on both my phone and my laptop, and then toggling both back as the experiment progressed, let’s use pf.

pass in on cnmac1 proto { udp , tcp } from any to any port 53 rdr-to 8.8.8.8 port 53

This redirects DNS requests around the local server directly to 8.8.8.8. And sure enough, mesu started working.

host mesu.apple.com
mesu.apple.com is an alias for mesu.apple.com.akadns.net.
mesu.apple.com.akadns.net is an alias for mesu.g.aaplimg.com.
mesu.g.aaplimg.com has address 17.253.15.206
mesu.g.aaplimg.com has address 17.253.15.208

And the iPhone commenced downloading its long awaited updates.

Curious as to the cause of the problem, since unbound itself was being very quiet, I switched the pf rule off, reconfirmed the problem, then restarted with verbose debugging. And the problem immediately resolved itself.

What actually happened? No idea really. That particular query somehow found itself trapped in a bad spot in unbound’s cache. Maybe? Oh well, until next time.

Another report with more information.