Erik Kline
Mark Townsley
Recently, we had occasion to debug a strange IPv6 timeout. It was peculiar to one Windows Vista machine located on a home network for which Google over IPv6 was enabled (which is what made this problem apparent). Typing www.google.com in the address bar resulted in a minimum 30 seconds delay, after which the main page loaded quickly. The cache links on the search results page contained IPv4 addresses, indicating that the site had loaded over IPv4 rather than IPv6.
Debugging with ping -6 www.google.com initially revealed no difficulties (ICMPv6 echo replies were returned). MTU issues were suspected, though quickly dispelled (pings with large packets appeared to behave as expected). All other machines on the network in question worked just fine; access to Google services over IPv6 and other IPv6-accessible sites worked exactly as expected. To complicate matters these other working machines were Apple Macbook Pros running MacOS X 10.4.x and 10.5.x, so some effort was spent looking at platform dependencies and differences.
Failure to understand the subtle differences between IPv4 and IPv6 can lead to mysterious, difficult to debug brokenness. This can cause unnecessary harm to the reputation of a site or application which manifests the problem, the platform on which the problem occurs, the network or provider thereof involved, and even to IPv6 as a whole. This can frequently lead to such specious advice as "just disable IPv6", when there is in fact nothing wrong with IPv6 nor even any other IPv6 component in the system of elements exhibiting the difficulty.
In this particular case an unrecorded version of Norton Internet Security was disabling all inbound ICMPv6 traffic not part of an existing session (i.e. traffic that was not part of a response to outbound traffic initiated by the host). This caused Neighbor Unreachability Detection (NUD) to fail. While trivially obvious now, the ways in which IPv6 appeared to work did not make determining the root cause simple (and it certainly would have been beyond the ability of someone not versed in networking). For this reason we have decided to document below our observations and lessons learned.
ICMPv6 firewalling recommendations are already documented in RFC 4890, Recommendations for Filtering ICMPv6 Messages in Firewalls. If they had been followed, specifically Appendix A.6, this difficult user experience might have been avoided.
Though many debugging strategies were tried the one that prevailed involved generating IPv6 traffic from the troubled host toward an IPv6 target and using tcpdump to watch the traffic at both ends. Rather than document all our tests and findings only this one approach is examined in some depth. The text outputs of tcpdump from different networks are attached below. The files labeled "vista" document the broken behaviour. (The files labeled "mac" document a working Mac laptop performing the same tests, for comparison.) The traffic captured and discussed below is from one continuous ping from the Vista host to the target host on a second network (so that the echo request sequence numbers can be easily matched up).
At the outset, a simple ping -6 2001:db8:y:a3f0:y:y:y:6061 from a command line window on the Vista host worked perfectly. First the Vista can be seen issuing a Neighbor Solicitation (NS) to confirm the MAC address of the default gateway, and the home gateway responds with a Neighbor Advertisement (NA) reply:
15:26:46.613754 IP6 (hlim 255, next-header: ICMPv6 (58), length: 32) 2001:db8:x:bee0:x:x:x:e82b > ff02::1:ffb8:49b0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::x:x:x:49b0
source link-address option (1), length 8 (1): x:x:x:x:7e:76
0x0000: xxxx xxxx 7e76
15:26:46.614726 IP6 (hlim 255, next-header: ICMPv6 (58), length: 32) fe80::x:x:x:49b0 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is fe80::x:x:x:49b0, Flags [router, solicited, override]
destination link-address option (2), length 8 (1): x:x:x:x:49:b0
0x0000: xxxx xxxx 49b0
There follow the default 4 echo request/reply attempts, all successful:
15:26:46.615021 IP6 (hlim 128, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 794
15:26:46.713586 IP6 (hlim 62, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 794
15:26:47.615287 IP6 (hlim 128, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 795
15:26:47.694747 IP6 (hlim 62, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 795
15:26:48.616397 IP6 (hlim 128, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 796
15:26:48.695768 IP6 (hlim 62, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 796
15:26:49.617422 IP6 (hlim 128, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 797
15:26:49.696553 IP6 (hlim 62, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 797
After momentary rumination on the success of trials like this, we iterated over attempts to send packets of various sizes into to examine MTU behaviour. These generally provided no new information.
It was observed eventually that back-to-back ping attempts failed. What's more, they failed deterministically! With this observation we tried ping -t -6 2001:db8:y:a3f0:y:y:y:6061 (-t for continuous ping). After roughly 3-5 seconds of successful echo requests and replies we observed timeout errors reported for each subsequent echo request. Naturally we needed to confirm that the echo requests were reaching the target host.
A tcpdump on the target host showed some curious behaviour. Initially, echo requests were received and replies were successfully sent:
15:26:46.640814 IP6 (hlim 126, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 794
15:26:46.640929 IP6 (hlim 64, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 794
15:26:47.623918 IP6 (hlim 126, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 795
15:26:47.624009 IP6 (hlim 64, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 795
15:26:48.625856 IP6 (hlim 126, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 796
15:26:48.625947 IP6 (hlim 64, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 796
15:26:49.626353 IP6 (hlim 126, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 797
15:26:49.626443 IP6 (hlim 64, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 797
Later, at the same time that the Vista source host was emitting timeout messages for unreceived ICMPv6 echo replies, the target host continued to see each of the echo requests but began to receive errors for each reply it sent:
15:26:54.634550 IP6 (hlim 126, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 802
15:26:54.634640 IP6 (hlim 64, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 802
15:26:57.717243 IP6 (hlim 63, next-header: ICMPv6 (58), length: 88) 2001:db8:x:bee0::1 > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, destination unreachable, length 88, unreachable address 2001:db8:x:bee0:x:x:x:e82b
15:26:59.147489 IP6 (hlim 126, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 803
15:26:59.147582 IP6 (hlim 64, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 803
15:27:02.229050 IP6 (hlim 63, next-header: ICMPv6 (58), length: 88) 2001:db8:x:bee0::1 > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, destination unreachable, length 88, unreachable address 2001:db8:x:bee0:x:x:x:e82b
Here we see that the target host replied and was then informed the destination was unreachable by the home gateway in the source network. Then another echo request arrives from the previously unreachable destination, the target host replies, and the home gateway returns "destination unreachable". This repeats ad nauseum. Time to see what's happening on the source network again!
Finally, a tcpdump on the source network catches the beginning of the trouble in action:
15:26:50.618485 IP6 (hlim 128, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 798
15:26:50.705719 IP6 (hlim 62, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 798
15:26:51.611219 IP6 (hlim 255, next-header: ICMPv6 (58), length: 32) fe80::x:x:x:49b0 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:x:bee0:x:x:x:e82b
source link-address option (1), length 8 (1): x:x:x:x:49:b0
0x0000: xxxx xxxx 49b0
15:26:51.619552 IP6 (hlim 128, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 799
15:26:51.731206 IP6 (hlim 62, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 799
15:26:52.611250 IP6 (hlim 255, next-header: ICMPv6 (58), length: 32) fe80::x:x:x:49b0 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:x:bee0:x:x:x:e82b
source link-address option (1), length 8 (1): x:x:x:x:49:b0
0x0000: xxxx xxxx 49b0
What we see here is that the home gateway, after 5 seconds from the initial NS/NA exchange and the start of the IPv6 echo request/replies, begins NUD by issuing a NS for the Vista host. The gateway continues to forward the echo replies while Neighbor Solicitations are sent. Finally, the home gateway determines the Vista host is not receiving its traffic, while continuing to issue Neighbor Solicitations (because the home gateway is still forwarding the echo requests it receives from the Vista host and is trying to deliver the echo replies):
15:26:53.623623 IP6 (hlim 128, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 801
15:26:53.704645 IP6 (hlim 62, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 801
15:26:54.624739 IP6 (hlim 128, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 802
15:26:54.707857 IP6 (hlim 255, next-header: ICMPv6 (58), length: 32) fe80::x:x:x:49b0 > ff02::1:ff68:e82b: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:x:bee0:x:x:x:e82b
source link-address option (1), length 8 (1): x:x:x:x:49:b0
0x0000: xxxx xxxx 49b0
15:26:55.707992 IP6 (hlim 255, next-header: ICMPv6 (58), length: 32) fe80::x:x:x:49b0 > ff02::1:ff68:e82b: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:x:bee0:x:x:x:e82b
source link-address option (1), length 8 (1): x:x:x:x:49:b0
0x0000: xxxx xxxx 49b0
15:26:56.707867 IP6 (hlim 255, next-header: ICMPv6 (58), length: 32) fe80::x:x:x:49b0 > ff02::1:ff68:e82b: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:x:bee0:x:x:x:e82b
source link-address option (1), length 8 (1): x:x:x:x:49:b0
0x0000: xxxx xxxx 49b0
15:26:59.138046 IP6 (hlim 128, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 803
15:26:59.219653 IP6 (hlim 255, next-header: ICMPv6 (58), length: 32) fe80::x:x:x:49b0 > ff02::1:ff68:e82b: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:x:bee0:x:x:x:e82b
source link-address option (1), length 8 (1): x:x:x:x:49:b0
0x0000: xxxx xxxx 49b0
15:27:00.219627 IP6 (hlim 255, next-header: ICMPv6 (58), length: 32) fe80::x:x:x:49b0 > ff02::1:ff68:e82b: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:x:bee0:x:x:x:e82b
source link-address option (1), length 8 (1): x:x:x:x:49:b0
0x0000: xxxx xxxx 49b0
To summarize the order of events:
In the end this was simply a matter of a poorly implemented firewall. The firewall apparently does a reasonable job of identifying IPv6 packets that are part of a host-initiated flow. But the implementation is not fully correct and the entire process reveals some interesting points worth discussing.
Historically there have been attack vectors via ICMPv4. At first glance it seems reasonable to assume that there might also be attack vectors via ICMPv6. However, failure to properly appreciate the ways in which IPv6 is different from IPv4 led to this bizarre, broken behaviour. Implementations need to understand such differences as:
Neighbor Unreachability Detection is part of how nodes talk to one another on a link. It's a mechanism whereby nodes can detect unidirectional traffic failures. Denying all "inbound" or non-host-initiated ICMPv6 is sort of akin to denying ARP in IPv4. Problematic, at best.
Again, implementors should read and follow IETF guidelines: RFC 4890, Recommendations for Filtering ICMPv6 Messages in Firewalls.
One other thing that came to light during the debugging process was the timing of changes in the neighbor cache state machine in the router. The router appears to have been using the default timers from RFC 4861. After 5 seconds NUD began, since there were no other hints (like TCP ACKs) to indicate that forward progress was being made. After 3 attempts roughly 1 second apart the router was forced to conclude that a unidirectional traffic problem existed. Had these timers been longer, especially if they had been considerably longer, then the problem would have been even more difficult to debug. While the trade-off is that IPv6 can be "more chatty", the value of failing fast makes it easier to identify the failure point of origin.