Yeti DNS Project Phase-2
--A Live IPv6-only Root DNS Server System Testbed

The Yeti project is a live testbed. This page serves as an all-in-one page to collect all information of Yeti work including Yeti documents, various results that we have gotten from experiments, software tools, and observations during the operation of the Yeti testbed.

1. Project & Testbed Description

2. Yeti operational findings

During the setup and operation of the Yeti project, a number of issues were discovered and documented that are unrelated to specific experiments, but were nevertheless interesting.

It is noteworthy that all of the name servers used in the project had fixes or changes based on the work. That includes BIND 9, NSD, Knot, PowerDNS, and Microsoft DNS.

IPv6 Fragmentation Issues

IPv6 fragmentation is a concern due to DNS large responses. The Yeti experiment produces responses larger than 1900 bytes, so is affected.

Via some tests and discussion inside Yeti mailing list, we have identifed two issues regarding IPv6 fragmentation, not only for the DNS root but for DNS in general.

An experiment regarding the IPv6 framentation issue in Yeti testbed :

1.Scoring the Yeti DNS Root Server System

Root Server Glue and BIND 9

It was discovered that BIND 9 does not return glue for the root zone, unless also configured as authoritative for the zone servers themselves. This is true for the IANA root, since the root servers are authoritative for the root and the ROOT-SERVERS.NET domain. For Yeti, this is not true since there is no Yeti zone: all of the Yeti name servers are managed from an independent delegation.

This issue was discussed on the Yeti discuss list:

http://lists.yeti-dns.org/pipermail/discuss/2015-May/000013.html

The BIND 9 team rejected the idea of changing the BIND 9 behavior:

https://lists.isc.org/pipermail/bind-workers/2015-May/003317.html

However the Yeti project produced a patch for BIND 9 which some operators now use:

http://lists.yeti-dns.org/pipermail/discuss/2015-June/000089.html

The discussion on root naming issue are introduce in the Yet experience I-D:

https://tools.ietf.org/html/draft-song-yeti-testbed-experience-03#section-4.1

The delay observed on Yeti root zone update

It was observed one server on Yeti testbed have some bugs on SOA update with more than 10 hours delay which is running on Bundy 1.2.0 on FreeBSD 10.2-RELEASE. To better understand the issue a monitoring test was done on the behavior of zone update and DM-selection of each Yeti root serve. The findings is that more than half of Yeti root servers suffer from more than 20 min delay on the testbed. One possible reason may be that the server failed to pull the Zone on one DM and turn to another DM which introduces the delay due to IPv6 fragmenation. (more investigation is needed to shoot the problem)

More information for this issue : http://yeti-dns.org/yeti/blog/2017/03/26/Monitoring-on-Yeti-Root-Zone-Update.html

dnscap Losing Packets

One of the Yeti participants noticed that dnscap, a tool written to capture DNS packets on the wire, was dropping packets:

http://lists.yeti-dns.org/pipermail/discuss/2015-June/000046.html

Workarounds were found for Yeti, although the dnscap developers continued to research and eventually discovered a fix included in DNSCAP release 1.1.0:

https://lists.dns-oarc.net/pipermail/dnscap-users/2016-October/000014.html

- Use helper library `pcap-thread` when capturing to solve
  missing packets during very low traffic"

Later another losing-packets bug was found in the dnscap v1.3. It is resolved in later version of DNSCAP.

https://github.com/DNS-OARC/dnscap/issues/580

libpcap Corrupting pcap output

One of our systems had a full disk and ended up with corrupted pcap files from dnscap. We tracked that down to an issue in libpcap, where the underlying writes were not being properly checked. A fix was made and submitted to the upstream library. While not a perfect solution, it is the best that can be done with the underlying I/O API as well as the published API within libpcap:

https://github.com/the-tcpdump-group/libpcap/pull/494

RFC 5011 Hold-Down Timer in BIND 9 and Unbound

The first KSK roll on the Yeti project was not carefully planned, but rather handled as the KSK roll for any zone. Because of this we encountered problems with DNS resolvers configured to use RFC 5011. RFC 5011 automatically updates the trust anchors for a zone, but requires that the new KSK be in place for 30 days. What ended up happening is that BIND 9 continued to function, because it does not use the 30 day hold-down timer, but that Unbound stopped working, because it does (as per the recommendation in the RFC).

http://lists.yeti-dns.org/pipermail/discuss/2015-July/000127.html

Corner Case: Knot Name Compression & Go dns Library

A Yeti participant discovered problems in both a popular Go DNS library and a popular authoritative DNS server, Knot when using Knot to server the Yeti root zone and querying the server with a Go program:

https://github.com/miekg/dns/issues/234

The problem was fixed by the developers of these open source projects. There were no reports of this affecting end-users.

NSD Glue Truncation

NSD by default is configured to send minimal responses, which needs to be re-compiled in order to send a complete glue:

http://lists.yeti-dns.org/pipermail/discuss/2016-May/000522.html

3. Yeti experiment and findings

Yeti root testbed is design for experiments and findings are expected. We summarized all experiment and findings in section 4 of draft-song-yeti-testbed-experience. Some important experiments related with Root Key management are introduced here with detailed report:

Multiple ZSK Experiment

The Multi-ZSK (MZSK) experiment was designed to test operating the Yeti root using more than a single ZSK. The goal was to have each distribution master (DM) have a separate ZSK, signed by a single KSK. This allows each DM to operate independently, each maintaining their own key secret material.

A description of the MZSK experiment and the results can be found in Yeti Project Report: Multi-ZSK Experiment (MZSK).

Of particular interest, the experiment triggered a problem with IXFR for some software, the results of which are documented in An IXFR Fallback to AXFR Case.

Big ZSK Experiment

The Big ZSK experiment was designed to test operating the Yeti root with a 2048-bit ZSK. This was based on Verisign’s announcement that they were going to change the ZSK size of the IANA root to 2014-bits.

A description of the BGZSK experiment and the results can be found in Yeti DNS Project Github repository.

KSK Roll Experiment

Since ICANN is going to start KSK rollover on September 19, 2017, the Yeti KSK roll experiment was designed to perform a KSK roll for the Yeti root and observe the effects. One major goal is to deliver some useful feedback before the IANA KSK roll. A significant result was that DNSSEC failures are reported if a new view is added to a BIND server after the KSK roll has started.

For more information:

  1. The Yeti KSK rollover experiment plan is documented in:
    https://github.com/BII-Lab/Yeti-Project/blob/master/doc/Experiment-KROLL.md

  2. A detailed report is published in:
    https://github.com/BII-Lab/Yeti-Project/blob/master/doc/Report-KROLL.md

4. Yeti Data Analysis

Now we only have some preliminary analysis on Yeti traffic collected from Yeti server which brief introduced on the presentation Davey Song gave in 2016 Seoul workshop. It is expected that more information will be dug up in 2017 from the traffic, awaiting for more resources spent on this work.

5. Software Tools

BII has written a number of programs during the course of running the Yeti project.

PcapParser: Easy Handling of Large DNS Captured Packets

In Yeti, as with many DNS operations, we use pcap format to store captured DNS traffic. Because Yeti is concerned with large packets, we often have DNS messages that are fragmented or sent via TCP messages. These are hard to analyze, so we wrote a tool to convert these into defragmented, UDP pcap files.

http://dnsv6lab.net/2016/09/06/DNS-pcap-fragments/

As part of this work we are pushing the IPv6 IP defragmention code back into the underlying gopacket library.

ymmv: Easy and Safe Comparison of IANA and Yeti Traffic

Yeti needs as much traffic as possible, ideally with real user query data. Because it is often unacceptable to use Yeti in production environments, the ymmv program was created, which will send the same queries to Yeti servers as a resolver sends to the IANA servers, and also compare the replies and note any differences.

http://dnsv6lab.net/2016/10/13/ymmv/

6. Other Resources