ICMPv6 is Non-Optional

Erik Kline

Mark Townsley

Recently, we had occasion to debug a strange IPv6 timeout. It was peculiar to one Windows Vista machine located on a home network for which Google over IPv6 was enabled (which is what made this problem apparent). Typing www.google.com in the address bar resulted in a minimum 30 seconds delay, after which the main page loaded quickly. The cache links on the search results page contained IPv4 addresses, indicating that the site had loaded over IPv4 rather than IPv6.

Debugging with ping -6 www.google.com initially revealed no difficulties (ICMPv6 echo replies were returned). MTU issues were suspected, though quickly dispelled (pings with large packets appeared to behave as expected). All other machines on the network in question worked just fine; access to Google services over IPv6 and other IPv6-accessible sites worked exactly as expected. To complicate matters these other working machines were Apple Macbook Pros running MacOS X 10.4.x and 10.5.x, so some effort was spent looking at platform dependencies and differences.

Summary, or IPv6 is not just IPv4 with bigger addresses

Failure to understand the subtle differences between IPv4 and IPv6 can lead to mysterious, difficult to debug brokenness. This can cause unnecessary harm to the reputation of a site or application which manifests the problem, the platform on which the problem occurs, the network or provider thereof involved, and even to IPv6 as a whole. This can frequently lead to such specious advice as "just disable IPv6", when there is in fact nothing wrong with IPv6 nor even any other IPv6 component in the system of elements exhibiting the difficulty.

In this particular case an unrecorded version of Norton Internet Security was disabling all inbound ICMPv6 traffic not part of an existing session (i.e. traffic that was not part of a response to outbound traffic initiated by the host). This caused Neighbor Unreachability Detection (NUD) to fail. While trivially obvious now, the ways in which IPv6 appeared to work did not make determining the root cause simple (and it certainly would have been beyond the ability of someone not versed in networking). For this reason we have decided to document below our observations and lessons learned.

ICMPv6 firewalling recommendations are already documented in RFC 4890, Recommendations for Filtering ICMPv6 Messages in Firewalls. If they had been followed, specifically Appendix A.6, this difficult user experience might have been avoided.

Debugging in Detail

Though many debugging strategies were tried the one that prevailed involved generating IPv6 traffic from the troubled host toward an IPv6 target and using tcpdump to watch the traffic at both ends. Rather than document all our tests and findings only this one approach is examined in some depth. The text outputs of tcpdump from different networks are attached below. The files labeled "vista" document the broken behaviour. (The files labeled "mac" document a working Mac laptop performing the same tests, for comparison.) The traffic captured and discussed below is from one continuous ping from the Vista host to the target host on a second network (so that the echo request sequence numbers can be easily matched up).

Dramatis Personae

IPv6 ping Appears to Work

At the outset, a simple ping -6 2001:db8:y:a3f0:y:y:y:6061 from a command line window on the Vista host worked perfectly. First the Vista can be seen issuing a Neighbor Solicitation (NS) to confirm the MAC address of the default gateway, and the home gateway responds with a Neighbor Advertisement (NA) reply:

15:26:46.613754 IP6 (hlim 255, next-header: ICMPv6 (58), length: 32) 2001:db8:x:bee0:x:x:x:e82b > ff02::1:ffb8:49b0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::x:x:x:49b0

source link-address option (1), length 8 (1): x:x:x:x:7e:76

0x0000: xxxx xxxx 7e76

15:26:46.614726 IP6 (hlim 255, next-header: ICMPv6 (58), length: 32) fe80::x:x:x:49b0 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is fe80::x:x:x:49b0, Flags [router, solicited, override]

destination link-address option (2), length 8 (1): x:x:x:x:49:b0

0x0000: xxxx xxxx 49b0

There follow the default 4 echo request/reply attempts, all successful:

15:26:46.615021 IP6 (hlim 128, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 794

15:26:46.713586 IP6 (hlim 62, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 794

15:26:47.615287 IP6 (hlim 128, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 795

15:26:47.694747 IP6 (hlim 62, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 795

15:26:48.616397 IP6 (hlim 128, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 796

15:26:48.695768 IP6 (hlim 62, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 796

15:26:49.617422 IP6 (hlim 128, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 797

15:26:49.696553 IP6 (hlim 62, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 797

After momentary rumination on the success of trials like this, we iterated over attempts to send packets of various sizes into to examine MTU behaviour. These generally provided no new information.

Extended pings Fail

It was observed eventually that back-to-back ping attempts failed. What's more, they failed deterministically! With this observation we tried ping -t -6 2001:db8:y:a3f0:y:y:y:6061 (-t for continuous ping). After roughly 3-5 seconds of successful echo requests and replies we observed timeout errors reported for each subsequent echo request. Naturally we needed to confirm that the echo requests were reaching the target host.

The Unreachable Destination That's Originating Traffic

A tcpdump on the target host showed some curious behaviour. Initially, echo requests were received and replies were successfully sent:

15:26:46.640814 IP6 (hlim 126, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 794

15:26:46.640929 IP6 (hlim 64, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 794

15:26:47.623918 IP6 (hlim 126, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 795

15:26:47.624009 IP6 (hlim 64, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 795

15:26:48.625856 IP6 (hlim 126, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 796

15:26:48.625947 IP6 (hlim 64, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 796

15:26:49.626353 IP6 (hlim 126, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 797

15:26:49.626443 IP6 (hlim 64, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 797

Later, at the same time that the Vista source host was emitting timeout messages for unreceived ICMPv6 echo replies, the target host continued to see each of the echo requests but began to receive errors for each reply it sent:

15:26:54.634550 IP6 (hlim 126, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 802

15:26:54.634640 IP6 (hlim 64, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 802

15:26:57.717243 IP6 (hlim 63, next-header: ICMPv6 (58), length: 88) 2001:db8:x:bee0::1 > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, destination unreachable, length 88, unreachable address 2001:db8:x:bee0:x:x:x:e82b

15:26:59.147489 IP6 (hlim 126, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 803

15:26:59.147582 IP6 (hlim 64, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 803

15:27:02.229050 IP6 (hlim 63, next-header: ICMPv6 (58), length: 88) 2001:db8:x:bee0::1 > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, destination unreachable, length 88, unreachable address 2001:db8:x:bee0:x:x:x:e82b

Here we see that the target host replied and was then informed the destination was unreachable by the home gateway in the source network. Then another echo request arrives from the previously unreachable destination, the target host replies, and the home gateway returns "destination unreachable". This repeats ad nauseum. Time to see what's happening on the source network again!

A Tale of Two Neighbors

Finally, a tcpdump on the source network catches the beginning of the trouble in action:

15:26:50.618485 IP6 (hlim 128, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 798

15:26:50.705719 IP6 (hlim 62, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 798

15:26:51.611219 IP6 (hlim 255, next-header: ICMPv6 (58), length: 32) fe80::x:x:x:49b0 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:x:bee0:x:x:x:e82b

source link-address option (1), length 8 (1): x:x:x:x:49:b0

0x0000: xxxx xxxx 49b0

15:26:51.619552 IP6 (hlim 128, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 799

15:26:51.731206 IP6 (hlim 62, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 799

15:26:52.611250 IP6 (hlim 255, next-header: ICMPv6 (58), length: 32) fe80::x:x:x:49b0 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:x:bee0:x:x:x:e82b

source link-address option (1), length 8 (1): x:x:x:x:49:b0

0x0000: xxxx xxxx 49b0

What we see here is that the home gateway, after 5 seconds from the initial NS/NA exchange and the start of the IPv6 echo request/replies, begins NUD by issuing a NS for the Vista host. The gateway continues to forward the echo replies while Neighbor Solicitations are sent. Finally, the home gateway determines the Vista host is not receiving its traffic, while continuing to issue Neighbor Solicitations (because the home gateway is still forwarding the echo requests it receives from the Vista host and is trying to deliver the echo replies):

15:26:53.623623 IP6 (hlim 128, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 801

15:26:53.704645 IP6 (hlim 62, next-header: ICMPv6 (58), length: 40) 2001:db8:y:a3f0:y:y:y:6061 > 2001:db8:x:bee0:x:x:x:e82b: [icmp6 sum ok] ICMP6, echo reply, length 40, seq 801

15:26:54.624739 IP6 (hlim 128, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 802

15:26:54.707857 IP6 (hlim 255, next-header: ICMPv6 (58), length: 32) fe80::x:x:x:49b0 > ff02::1:ff68:e82b: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:x:bee0:x:x:x:e82b

source link-address option (1), length 8 (1): x:x:x:x:49:b0

0x0000: xxxx xxxx 49b0

15:26:55.707992 IP6 (hlim 255, next-header: ICMPv6 (58), length: 32) fe80::x:x:x:49b0 > ff02::1:ff68:e82b: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:x:bee0:x:x:x:e82b

source link-address option (1), length 8 (1): x:x:x:x:49:b0

0x0000: xxxx xxxx 49b0

15:26:56.707867 IP6 (hlim 255, next-header: ICMPv6 (58), length: 32) fe80::x:x:x:49b0 > ff02::1:ff68:e82b: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:x:bee0:x:x:x:e82b

source link-address option (1), length 8 (1): x:x:x:x:49:b0

0x0000: xxxx xxxx 49b0

15:26:59.138046 IP6 (hlim 128, next-header: ICMPv6 (58), length: 40) 2001:db8:x:bee0:x:x:x:e82b > 2001:db8:y:a3f0:y:y:y:6061: [icmp6 sum ok] ICMP6, echo request, length 40, seq 803

15:26:59.219653 IP6 (hlim 255, next-header: ICMPv6 (58), length: 32) fe80::x:x:x:49b0 > ff02::1:ff68:e82b: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:x:bee0:x:x:x:e82b

source link-address option (1), length 8 (1): x:x:x:x:49:b0

0x0000: xxxx xxxx 49b0

15:27:00.219627 IP6 (hlim 255, next-header: ICMPv6 (58), length: 32) fe80::x:x:x:49b0 > ff02::1:ff68:e82b: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:x:bee0:x:x:x:e82b

source link-address option (1), length 8 (1): x:x:x:x:49:b0

0x0000: xxxx xxxx 49b0

To summarize the order of events:

  1. The host issues a NS to identify/confirm the router's Layer 2 reachability.
  2. The router responds with a solicited NA, while correspondingly adding the host's neighbor information into its own neighbor cache.
  3. IPv6 traffic begins to flow freely.
  4. After 5 seconds, the router begins NUD, sending Neighbor Solicitations to the multicast group for the host.
  5. After 3 seconds of not receiving a solicited NA from the host the router concludes that forward progress is not occurring. Note that during this time it is receiving packets from the host, but is not able to confirm that it can forward packets to the host.
  6. The router stops attempting to forward packets to the host and notifies others that the destination has become unreachable.

Conclusions

In the end this was simply a matter of a poorly implemented firewall. The firewall apparently does a reasonable job of identifying IPv6 packets that are part of a host-initiated flow. But the implementation is not fully correct and the entire process reveals some interesting points worth discussing.

You Wouldn't Firewall ARP, Would You?

Historically there have been attack vectors via ICMPv4. At first glance it seems reasonable to assume that there might also be attack vectors via ICMPv6. However, failure to properly appreciate the ways in which IPv6 is different from IPv4 led to this bizarre, broken behaviour. Implementations need to understand such differences as:

  1. IGMP functions have been folded in to ICMPv6,
  2. PathMTU (via ICMPv6 "packet too big" messages) is effectively required to be allowed to succeed for links that have MTUs greater than the minimum (1280), and
  3. some ICMPv6 messages are mandatory for basic network operation.

Neighbor Unreachability Detection is part of how nodes talk to one another on a link. It's a mechanism whereby nodes can detect unidirectional traffic failures. Denying all "inbound" or non-host-initiated ICMPv6 is sort of akin to denying ARP in IPv4. Problematic, at best.

Again, implementors should read and follow IETF guidelines: RFC 4890, Recommendations for Filtering ICMPv6 Messages in Firewalls.

Fail Fast

One other thing that came to light during the debugging process was the timing of changes in the neighbor cache state machine in the router. The router appears to have been using the default timers from RFC 4861. After 5 seconds NUD began, since there were no other hints (like TCP ACKs) to indicate that forward progress was being made. After 3 attempts roughly 1 second apart the router was forced to conclude that a unidirectional traffic problem existed. Had these timers been longer, especially if they had been considerably longer, then the problem would have been even more difficult to debug. While the trade-off is that IPv6 can be "more chatty", the value of failing fast makes it easier to identify the failure point of origin.