Despite their wonky customer service, I have generally been happy with Comcast’s technical service. Occasionally though, I have to question what in the world is going on with their network. Recently I noticed that my internet connection would alternate between working and not on roughly a two minute interval. One minute things work fine, the next minute nothing works at all, the next minute everything is fine. During the blackout minutes, making a new connection would timeout, but established ssh connections would remain up, but nothing would happen until the blackout was over. Here are a few pings I sent out (2014-02-13).

64 bytes from 8.8.8.8: icmp_seq=0 ttl=43 time=98089.012 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=43 time=96073.905 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=43 time=97083.976 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=43 time=95064.000 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=43 time=94053.992 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=43 time=93044.984 ms
64 bytes from 8.8.8.8: icmp_seq=7 ttl=43 time=91025.183 ms
64 bytes from 8.8.8.8: icmp_seq=6 ttl=43 time=92035.340 ms
64 bytes from 8.8.8.8: icmp_seq=9 ttl=43 time=89005.105 ms
64 bytes from 8.8.8.8: icmp_seq=10 ttl=43 time=87995.872 ms
64 bytes from 8.8.8.8: icmp_seq=8 ttl=43 time=90020.101 ms

Woah, that’s some latency. A few observations. We didn’t lose any packets. They didn’t quite arrive in sequence, but they’re not totally out of order (in fact, 8 is the only packet to be more than one out of place). And they all arrived at the same time. It’s like they were taped to the leg of a carrier pigeon and then when enough packets were queued up somebody opened the cage door.

Anything that works asynchronously, like email, is reasonable, but the blackout minutes makes web browsing abominably painful. I’ve experienced this twice so far in a period of a few months. It lasts for about 24-36 hours. The more recent occurrence ended just as I was setting up some better testing and logging, so I can’t yet determine whether it’s malice or incompetence.

Guessing, it looks like a misconfigured redundant router pair, only one of which can send packets. The broken router simply queues packets until it times out and fails over, then pushes everything down the working pipe. Then for some reason, the system fails back to the broken router.

Another instance from 2015-06-25:

64 bytes from 8.8.8.8: icmp_seq=12 ttl=55 time=9109.139 ms
64 bytes from 8.8.8.8: icmp_seq=13 ttl=55 time=8099.379 ms
64 bytes from 8.8.8.8: icmp_seq=14 ttl=55 time=7089.483 ms
64 bytes from 8.8.8.8: icmp_seq=15 ttl=55 time=6079.572 ms
64 bytes from 8.8.8.8: icmp_seq=16 ttl=55 time=5069.646 ms
64 bytes from 8.8.8.8: icmp_seq=17 ttl=55 time=4059.737 ms
64 bytes from 8.8.8.8: icmp_seq=18 ttl=55 time=3048.371 ms
64 bytes from 8.8.8.8: icmp_seq=19 ttl=55 time=2039.933 ms
64 bytes from 8.8.8.8: icmp_seq=20 ttl=55 time=1029.060 ms

64 bytes from 8.8.8.8: icmp_seq=21 ttl=55 time=9378.332 ms
64 bytes from 8.8.8.8: icmp_seq=22 ttl=55 time=8366.536 ms
64 bytes from 8.8.8.8: icmp_seq=23 ttl=55 time=7356.391 ms
64 bytes from 8.8.8.8: icmp_seq=24 ttl=55 time=6346.711 ms
64 bytes from 8.8.8.8: icmp_seq=25 ttl=55 time=5336.931 ms
64 bytes from 8.8.8.8: icmp_seq=26 ttl=55 time=4327.367 ms
64 bytes from 8.8.8.8: icmp_seq=27 ttl=55 time=3316.542 ms
64 bytes from 8.8.8.8: icmp_seq=28 ttl=55 time=2306.854 ms
64 bytes from 8.8.8.8: icmp_seq=29 ttl=55 time=1296.952 ms
64 bytes from 8.8.8.8: icmp_seq=30 ttl=55 time=289.789 ms

Now the internet is on a ten second burst cycle.