Time: 2020/03/29
Participants: Hugo Salgado-Hernánde (via video, Chile); Praveen (India); Pavel Павел Храмцов (Russia); Li Zhen, Kevin Gong, Davey Song, Polo, Zhang Yuancheng, Shine (China)
Moderator: Polo

1. Sessions

• Opening and Introduction
  Because of the Zoom login issue, Kato-san can’t attend the meeting.
  Due to the impact of COVID-2019, Yeti meeting had been delayed.
• COVID-19 defense suggestions:
  • Prepare enough stuffs (food, mask, hand sanitizers, rubbing alcohol and so on)
  • Stay at home, do not go outside, unless it’s emerengcy.
  • When going outside, you should wear a mask.
  • Wash your hands often and carefully.
• Decentralized Domain Name System (Li Zhen)
  • P2P network for Fault-tolerant Distribution Master Architecture (FDMA)
• Distributed HSM (Hugo/Video)
  • new feature: Threshold signing with ECDSA support
  • golang implement for all components
• Distributed HSM Lab Test (Kevin)
  • Study the technology for FDMA
  • HSM-tools zone sign issue
• Free discussion
  • What type of P2P network and the details will be discussed in Yeti mailing list

2. Yeti2 Plan for 2020

Yeti phase-2 Fault-tolerant Distribution Master Architecture
a) 2020/4-2020/6 Request for comments and detailed discussion on technology adoption
b) 2020/6-2020/12 Prototype development, test in lab, technical specifications update
c) 2021/1-2021/12 Run on test bed

3. slides share

Yeti-meeting-slides-2020

4. Next Step

1) Comments for Distributed HSM Lab Test
the HSM-tools issue to be confirmed
2) Comments for Yeti phase-2 Decentralized Domain Name System
P2P technology discussion
3) Comments for Distributed HSM
4) Website updates for Yeti-Phase II

Yeti phase-2 is an open technical research with practice project. We look forward to welcome more organizations and individuals to the Yeti phase-2. Please contact coordinators@lists.yeti-dns.org.

Last Monday (22nd July) during the IETF 105 meeting, Yeti Phase-2 kick-off meeting was held in Montreal. We had 7 people including three coordinators on site and two participating remotely. The meeting covered topics on the result of second KSK rollover lab experiment, the vision and plan of the Yeti phase-2, its new charter, the technical proposal of Fault-tolerant root system and the decentralized distributed multi-DM system etc.(Slides are available here). Suggestions and comments are received from Participants. Everyone agreed on making innovative attempts in decentralization and multi-signature as a practice. Some experts also suggested that the Yeti phase-2 project should continue to push innovation in DNSEC on Root level.

Some participants encouraged experiments to expand from the root zone to more zones and domains, and different domains operate independently. It is expected that Yeti phase-2 will have more innovations and positively be the first to be applied as best practice for domains beyond root zone. Finally, some experts reminded the Yeti phase-2 work to focus on technical stuff and avoid sensitive political topics.

According to the project plan, we need to complete the draft of structural systems design before the end of this year. It is proposed that the next Yeti phase-2 workshop be held in Russia in December, and the relevant schedule and plan will be discussed and confirmed as soon as possible.

Yeti phase-2 is an open technical research with practice project. We look forward to welcome more organizations and individuals to the Yeti phase-2. Please contact coordinators@lists.yeti-dns.org.

Introduction

In this April we called participation of second algorithm rollover in a controlled lab environment. We try rolled the algorithm rollover in root level with four approaches and record the monitoring data every 10 minutes. The results help us better understand the protocol as well as the gap between protocol and implementation. In this post we skip a full introduction of algorithm rollover background and the testing proposal of the Second algorithm rollover test. Because that information was well documented in previous posts : Call for participation and Algorithm test and monitoring page. Instead, this post only demonstrates the interesting results and findings which give us a second thought on algorithm rollover for root.

Note that in this test we use BIND9.11.5-P1, UNBOUND1.8.3, PowerDNS recursor 4.2.0 alpha1 as our testing resolvers.

Experience and Findings

Use of dnssec-signzone

We use BIND dnssec-signzone tool to sign the zone, generate NSEC and RRSIG records and produces a signed version of the zone. In schedule of the test (described in test and monitoring page), it is important to manipulate the keys to sign the zone and DNSKEY Records. This is achieved by manipulate the key repository of dnsse-signzone with -K option).

For example, in the test we use dnssec-signzone this way :

$ dnssec-signzone -K ${ROOT_KEY} -P -o . -O full -S -x ${ZONE_DATA}/root.zone

in which, -K option specifies a directory to search for DNSSEC keys (private keys in the directory are used to sign). And -x option is used to only sign the DNSKEY RRset with key-signing keys. Note that we use -P to disable post sign verification tests in Case 1 which will be introduce later. (More information of options of dnssec-signzone).

Everything seems perfect using dnssec-signzone to roll the algorithm by placing different keys in ROOT_KEY directory. However, it was found that when old RSA KSK is revoked or deleted, the RSA ZSK will be used to sign both the zone and DNSKEY records, even when -x option is set. -x option means “only sign the DNSKEY RRset with key-signing keys”.

There is a guess that smart signing (-S option) of BIND will ignore key flag when there is only a private key available for one algorithm. As a result BIND signs DNSKEY records with both New ECDSA KSK and old RSA ZSK. We corrected this with special script manually by stripping the DNSKEY RRSIG by RSA ZSK when the algorithm was really rolled from RSA to ECDSA. Operators who are going to implement algorithm rollover using dnssec-signzoe should notice and take care of it.

Double-DS for Algorithm Rollover in Case 1

Both RFC4035 and RFC6781 specify that “There MUST be an RRSIG for each RRset using at least one DNSKEY of each algorithm in the zone apex DNSKEY RRset.” It is inferred that it is the reason of excluding the Double-DS approach for algorithm roll because “when adding a new algorithm, the signatures should be added first”

Case 1 is designed to test against this specification in which the new algorithm (new ECDSA KSK) is re-published without corresponding RRSIG. Firstly we use dnssec-signzone without -P option, and the post sign verification tests failed. With -P option, a signed zone was generated and published.

On the resolver side, it is seems that resolver have no sense of that specification at all (requirement). The introduction of pre-published ECDSA KSK transitioned the state of resolver from Start to AddPend and start the Add Hold-Down Time (RFC5011 State table). And the state transitioned to Valid state after the timer expired. In addition no validation failure was reported during the Algorithm rollover in case 1.

It is known that Double-DS has the benefit of generating smaller size of DNS response. In the context of RFC5011, it requires no exchange with parent in root level. Exchanging with parent is viewed as the major drawback of Double-DS approach. So the question comes: is it worth of a second thought on Double-DS for algorithm rollover in future ? Any risk ?

Difference behaviors between BIND and Unbound NO.1 (To be confirmed)

It was reported in our first algorithm rollover test that BIND and Unbound responsed differently when they had validation failures.

In the first test, an misconfiguration of old RSA KSK with an earlier inactive time which results no active signing key in the middle of the rollover and causes validation failure. We reset the old RSA KSK to be active but it still had impact on resolvers. As a response to this failure, it was observed BIND resolver restarts the Add Hold-Down Timer of the new key/algorithm for another 30 days when the old RSA KSK became active.

In the contrast, Unbound’s timer did not stoped and trusted the KSK/Algorithm after the timer expired. It is inferred that BIND treats the failure as KeyRem event, but UNBOUND treat it as a non-rfc5011 event like network failure.

Difference behaviors between BIND and Unbound NO.2

RFC5011 specifies that a resolver MUST query that trust point for an automatic update of keys from a particular trust point. e.g., do a lookup for the DNSKEY RRSet and related RRSIG records. This operation is called Active Refresh.

According to the formula suggest in RFC5011, queryInterval = MAX(1 hr, MIN (15 days, 1/2OrigTTL, 1/2RRSigExpirationInterval)). Because the TTL of DNSKEY RRSET is set to 600 second to remove the impact of cache in this test. So the query interval is supposed to set as 1 hour for both BIND and Unbound resolvers. It is found that BIND update the keys strictly every one hour, however unbound are more sensitive to the change of keys in DNSKEY. It is inferred that UNBOUND update the key via alternative approach by listening to the normal DNSKEY query and response which are generated by monitoring purpose (query every 10 minutes).

Algorithm rollover without change on ZSK

We rolled the algorithm twice, we found the algorithm rollover with only KSK works for the resolver in Case 1 (with -P option) and Case 2. The BIND and Unbound can work with it without change on ZSK.

Configuration error in old version of PowerDNS

PowerDNS (pdns-recursor 4.0.0~alpha2-2ubuntu0.1) resolver failed when configuring multiple algorithm KSK.

Stand-by Key

In the second algorithm rollover test, we deployed stand-by key by introducing the algorithm with two Key Signing Keys. The stand-by Key is not used to roll algorithm but used to sign in the absence of “incumbent” key in case of failure. The switch to stand-by key is smooth and swift without any error after the “incumbent” key is being silenced.

Conclusion and suggestion for Future Yeti Algorithm Rollover

After two tests on algorithm rollover in lab environment, we have following conclusions:

• The dnssec-signzone of BIND is a handy tool to deploy DNSSEC for most cases. But for algorithm rollover, we should take care of the case when old ZSK is used to sign the DNSKEY record.

• Via Case 1 we learn the issue of Double-DS approach in algorithm rollover. And if anyone insist to roll the algorithm using Double-DNS approach, please use “

” option of dnssec-signzone. The risk is unknown so far.

• Case 2 prove the capability of rolling algorithm without ZSK. But the tests lack diversity of DNS resolver to provide a convincing result. It is worth to dig in depth.

• Case 3 and Case 4 are expected to pass the tests with liberal and conservative approaches respectively. The tests just confirmed it.

• We also found that BIND and Unbound have some different behaviors.

As to the suggestions for future algorithm rollover in Yeti, we should invite more DNS resolvers. If you expected more findings please try case 2 or Case 1 with “-P” option using dnssec-signzone.

First Algorithm Rollover in lab environment

ECDSA and Algorithm rollover is listed in current agenda of Yeti experiment. ECC(Elliptic Curve Cryptography) is widely considered as a promising DNSSEC algorithm after RSA due to its property: ECC has much smaller keys and signatures with equivalent or better key strength. ECDSA P-256 and P-384 are standardized for use in DNSSEC in RFC 6605 (2012).

However, There are some uncertainties of ECC for DNSSEC. The major one is the uncertainty of impact of resolvers’ switching to ECC. ECC readiness of resolvers (due to Large installed base) and rollover to ECC are uncertain. There are some experience of algorithm rollover in second level domain and TLD level. RIPE NCC suggest to roll both ZSK and KSK (2015) and .SE Algo Roll experiment shown that liberal approach is widely acceptable (only 6 out of 10,000 fail). However, there is little experience for algorithm rollover in root level, the automated Algorithm rollover for trust anchors compliant to RFC5011.

Before proposed and tested in Yeti Root, Davey and Kevin built a lab testbed in BII lab which is introduce in a document in Yeti’s GitHub repo. The idea is to test potential rollover approaches as many as possible: 1) Conservative Vs. liberal approaches; 2)rolling KSK without ZSK Vs. rolling KSK&ZSK at the same time.Accordingly, we setup 4 tests with 4 root servers, with different rolling approaches and time lines:

• Test 1: Republish KSK without signature as we rolled the key (like ICANN KSK rollover).
• Test 2: Similar with Test 1 but republish KSK and its signature without rolling ZSK;
• Test 3: Roll both ZSK and KSK in liberal approach;
• Test 4: Roll both ZSk and KSK in conservative approach.

Test 1 is expected to fail because it is designed with intentional violation of RFC6781 with requires Key and signature should be published at the same time in zone or in zone & cache; Test 2 is a fix to Test 1 but it is uncertain because only KSK is rolled. Test 3 and Test 4 is expected to pass because it is compliant to RFC6781 with liberal and conservative respectively.

The finding is interesting and surprised that four approaches proceeded successfully for BIND (9.11.5-P1) and UNBOUND(1.8.3) resolver. Maybe the samples of resolver is too limited. It is proposed to call for more resolvers to join. It is worth mentioning that in test 2,3 and 4 there is an accidental mistake in configuring the ZSK’s inactive time which results no active signing key in the middle of the rollover and causes validation failure(we recovered it with a new ZSK but it still had impact on resolver). As a response to this failure, it is observed BIND restarts the Add Hold-Down Time of new key/algorithm for another 30 days when new valid signing key is available but Unbound continue the timer and trusted the KSK/Algorithm after the rfc5011-timer expired. It is inferred that BIND treat the failure as a signal to quite the existing rfc5011 timer, but UNBOUND treat it as a un-rfc5011 signal(like network failure). UNBOUND stops the timer and continues it after failure resolved.

Note the first Algo roll test was presented in Yeti DNS Workshop last March, the slides are here.

Second Algorithm Rollover and How to join

The Second Algorithm rollover in lab test is planned in this April. We are going to outreach more interested people and call for participation. Participants are invited to join part of tests or all of them(Test 1,2,3,4). The timeline and schedule will follow the document in Yeti’s GitHub repo as the first rollover.

Participates joining the test are required to configured their resolvers manually with a new Hint file (containing name and address of testing roots) and Trust anchor (initial RSA KSK ). The timeline and more detailed information will be delivered to Yeti discuss mailing list.

If you are reading this article and interested to contribute by participating this test, please send mail to ljsong@biigroup.cn. You will be kept post during the experiment and receive immediate reply if you have any questions.

Last Saturday (2nd March) Yeti DNS Workshop was held in Tokyo. It is the timing between the end of first phase of Yeti DNS project and preparation of the second. People gathered with the goal of revisiting what have been done, discussing the current work and restarting Yeti project with recent technical attention and focus different from 4 years ago. Besides Yeti business, we have some invited talks on Internet governance and decentralized infrastructure over root. (Slides are available here, Talk audios are updated in YouTube: morning session, afternoon session (the first part), and Hugo’s record video).

The morning keynote was given by Dr. Paul Vixie. He revisited the background and the history related to the “third rail” of Internet governance and pressures on current DNS operation on Name Space(s). As a response to the controversy of alternative root, he made clarity on the difference the function of editing, signing and publication of DNS(Root) zone. It is a emphasis that Yeti from the beginning keep boundaries for One Unique Internet Identifier and not amending the IANA name space.

After Paul’s presentation, Davey Song on behalf of Yeti’s operational team gave a report reviewing Yeti DNS Project and Proposed Future AgendaYeti DNS Project. As usual, like every status-quo report given in previous Yeti meetings, it covers the background of Yeti DNS project including the motivation, principles and Yeti architecture (described in RFC8483). In addition he concluded and proposed two notable directions which deserve our effort in future. One is about Large DNS response issue and another is Yeti redundancy model with multiple signers. The two directions are discussed in the afternoon sessions.

In the afternoon, there were two sessions. The first one was about building a fault-tolerant root with decentralized Internet functions. Davey introduced the background, requirement and potentials. As a design principle, Yeti adds redundancy into root system to avoid single point of failure, both in root zone signing and publication function. So we have 3 Signers (DMs) and 25 servers which is different from the IANA root system. But it was found that such horizontal scaling will increase the size of DNS response, for example by containing more keys in the DNSKEY response. It is a undesirable side effect which also presents in previous proposal by John Ihren & Bill manning and Brenden Kuerbis & Milton Mueller.

Is it possible to introduce multiple independent Signers without changing the DNSSEC validation interface? And without augmenting the size of DNS response as well? In the end of Davey’s slides, threshold Signatures(Signing) was borrowed from blockchain field and introduced as a desirable property in the context of DNSSEC signing.

Presentation from Wei Xinpeng and Hugo Salgado-Hernández also gave us some potentials. In Wei’s presentation an ambitious decentralized Internet infrastructure was proposed covering DNS, RPKI and PKI functions. Enabled by blockchain technology (Permissioned Chain), all Internet resources like IP, domain name and certificate can be managed in both registration and resolution. As to DNS root, the good news is that it is technical possible to remove the necessity of centralized roles in editing, signing and distribution of root zone. During the discussion of participants, it was pointed out that if it targets the existing IP and DNS users, the radical change to current NIC&DNS system will make it hard to prevail (lacks of economic incentive for stakeholders to join)，but they are planning to explore some directions that would be more compatible to the existing system. There is also performance concerns if DNS hierarchy is replaced by a flat chain.

Hugo Salgado-Hernández who remotely participated the meeting gave us a record presentation on “Distributed Threshold cryptography: a novel way to distribute keys and signing for DNSSEC”. He showed us how the threshold cryptography possibly works for DNSSEC purpose in which M (or more) out of N signer can guarantee the robustness of the system. The amazing part of this proposal is the aggregatable property of Keys and Signatures in which the signatures signed by multiple signers can combined into one signature fit DNSSEC purpose. And the public key used for validated can be aggregated by multiple public keys published by those signers.

Although there are more details should be explored in how to generate and distributed partial keys, it is definitely a promising path to our desirable fault-tolerant root. It was proposed by Hugo to test this threshold cryptography in Yeti DNS. Note that in Hugo’s proposal there is still a centralized roll called SD (Signing Dealer) in charge of distributing zone file and collecting & aggregating the signatures. Compared with the pure decentralized approach with blockchain and a certain consensus algorithm. Hugo’s presentation provide a practical approach but with limitations in decentralized feature.

Another session in the afternoon is about ECDSA and Algorithm rollover which is listed in current agenda of Yeti experiment. ECC(Elliptic Curve Cryptography) is widely considered as a promising DNSSEC algorithm after RSA due to its property: ECC has much smaller keys and signatures with equivalent or better key strength. There are some experience of algorithm rollover in second level domain and TLD level, but there is little experience for algorithm rollover in root level. Before proposed and tested in Yeti Root, Davey and Kevin built a lab testbed in BII lab which is introduce in a document in Yeti’s GitHub repo.

Basically, we rolled the algorithm in four approaches with different configuration and time lines. The finding is interesting that four approaches proceeded successfully for BIND (9.11.5-P1) and UNBOUND(1.8.3) resolver. Note that there is an accidental mistake in configuring the ZSK’s inactive time which results no active signing key in the middle of the rollover and causes validation failure(we recovered it with a new ZSK but it still had impact on resolver). As a response to this failure, it is observed BIND restarts the Add Hold-Down Time of new key/algorithm for another 30 days when new valid signing key is available but Unbound continue the timer and trusted the KSK/Algorithm after the rfc5011-timer expired. It is planned that more lab test for rollover should be done before roll the algorithm of Yeti. We will call for more resolvers to join this test.

After all presentations and prepared topic, we have one hour open and wrap-up discussion. In the end of the meeting, a consensus was achieved among Yeti coordinator and Yeti participants that there are concrete and interesting items we are going to explore. We decided to move to Yeti phase-2 after 2019 after fully preparation with clear goal and a technical proposal document.

Special thanks for our local host Professor Akira Kato from Keio University providing the venue and recommending fancy restaurant.

Introduction

The Internet’s Domain Name System(DNS) is a vital component of Internet infrastructure mapping human-readable names into Internet Protocol (IP) addresses. DNS is designed and built on a single root, known as the Root Server System with the functions root zone generation and root zone distribution etc. As part of IANA function, DNS Root zone management has a long history(SAC067) and experienced technical changes (IPv6, DNSSEC, anycast etc) as well as policy changes (IANA transition, new gTLD). None of previous changes violates the assumption that a centralized entity responsible for one particular function in the Root zone management. Take DNSSEC at root level for example, VeriSign is the only Root Zone Maintainer(RZM) who maintain the ZSK, signs the zone taken from ICANN and distribute the zone to 13 root servers.

In order to introduce redundancy and avoid single point of failure, Yeti DNS testbed introduce Multi-DM model which uses 3 internal master servers doing DNSSEC signing (the VeriSign role) and root zone distribution. Each of them acts autonomously and equivalently to its siblings. They are referred to as Distribution Masters (DMs) in the context of root server system.

Multi-DM practice in Yeti testbed received expected benefit due to its property of redundancy. However, root operation based on Multi-DM model exposed some issues as well. One issue is reported on the observation of delay of root zone distribution which is introduced in section 5.2 of draft-song-yeti-testbed-experience. Another issue is recently observed during PINZ experiment which will be introduced in this post.

In this post, the Yeti’s Multi-DM Architecture is revisited with recently issues exposed during PINZ experiment. It is inferred that a fault-tolerant property is desired for future Yeti DM architecture. Most specifically, a consensus should be achieved before any signed zone be published among a group of entities. It is proposed in conclusion that a Fault-tolerant Distribtion Master Architecuture (FDMA) is worth of doing as an experiment and important property in Yeti’s future agenda.

Brief report of PINZ and the problem statement

PINZ experiment was performed in the May 2018. The goal of PINZ is to minimize the difference between Yeti root zone and the zone retrieved from IANA. It is done by preserving IANA NSEC Chain and ZSK RRSIGs in Yeti root zone. It is firstly proposed as a experiment in Yeti mailing list and passed the test in lab environment. As expected, when it was performed in Yeti testbed, more than 50 validating resolvers were under monitoring which can validate the signed RR with signatures from IANA. And there was no failure reports on other Yeti resolvers due to tailoring root zone with Yeti and IANA DNSSEC stuff.

As we did during KSK rollover in Yeti testbed, we deployed Atlas probes to observed the size of response for DNSKEY query. It was increased upto 1693 octets when IANA’s ZSK was added into Yeti root zone overlapping with Yeti’s ZSK rollover. Unsurprisingly, the failure rate for one probe under the large response increase to nearly 7% , then same as we reported previously in KSK rollover. In addition If KSK rollover is performed in the same time when PINZ is adopted, it will definitely produce larger response than we observed in KSK rollover. So adding more ZSKs into root zone is not a good approach which is one reason lead me to revisit the Multi-DM model.

Another reason is that we experienced operational event during PINZ experiment due to the power failure in one of DMs which produced a zone without any signature!! which will cause validating resolver to fail. It is not the flaw in design of PINZ but the inherit nature of Multi-DM model. In current Multi-DM model, each DM implements the loop of zone retrieval, DNSSEC signing and zone distribution independently. They only synchronize some meta data in advance but without confirmation on the zone to be published. The approach adding more DMs in to the system do add more redundancy but also add possibility of failure points. By simple calculation, if the failure probability of one DM is p, then the failure probability of the whole system with n DM is 1-(1-p)^n which is larger than p (suppose p=0.01% and n=3, then 1-(1-p)^n=0.02997% which is three times larger). It is inferred that the redundancy property of Multi-DM is at the risk of the increase of failure probability, especially when n is large. So adding more DM into root system is not a good choice in practice.

In addition, the role of DM as root zone master server is a sort of special. From the Internet governance point of view, it is a authority on the root zone , the top-level name space of Internet! Adding another 2 DMs may be a little better than 1 but do not change the game with centralized role. People may still ask the same question why 3? Who should take the role of DM? How we trust the 3 DM if we are not one of them? So it returns to the beginning.

We do add some pre-publish checking to enhance the service guarantee by comparing the zone with other DMs’ after PINZ experiment, but the issues mentioned above still exist. We may need a new look on Yeti DM architecture.

Conception of Fault-tolerant Distribution Master Architecture (FDMA)

To answer and resolve the questions proposed above, we may need to think about a fault-tolerant property in design for Yeti DM Architecture. More concepts and technologies of modern fault-tolerant distributed system are worth of learning from. We may think about the DNSSEC of root zone in the background of classic BFT(Byzantine fault tolerance) context in which we can borrow tools from distributed ledge (So called Blockchain).

Those thought and consideration will lead us to propose a Fault-tolerant Distribution Master Architecture (FDMA) for Yeti root testbed. The intuition is simple that a group of DM operators can work together to form a BFT-like system, in which each DM dependently generate the zone, signing the zone with their own key and publish to the BFT-like system just as we did in Yeti multi-DM. However, before the root zone published into DNS, a certain consensus is required every time among all DMs to confirm the To-Be-Published root zone for each SOA serial.

The properties of this conception of FDMA are as follows:

• Most notably, the system is fault-tolerant both on operation failures or malicious attack given no trust is required for each node in BFT-like system as long as the majority of nodes agree on each transaction (generating a new root zone in Yeti DM case).
• The number of DM operator is not limited. We may require permission to join to simplify the scalability issue (similar with permission chain).
• Only one publish key (ZSK or KSK) is to be added to root zone if some kind of threshold signature scheme is used to support DNSSEC.
• Diversity of source for root zone data. DM can independently retrieved root zone from different sources, like ICANN, root servers, TLD authoritative server, or some real time DNSDB.
• Merit on Internet Governance. Root zone management (without registration) will be come a truly self-organized system without centralized entity.

To achieve the goal, FDMA client should have functions listed below:

• Each DM client can generate root zone according to a specified and to-be-defined routine or guidance which guarantee the data consistency.
• Each DM client can sign the data and share the data with others which is basic building block of P2P network before achieving consensus on the to-be-published root zone.
• A consensus algorithm should be run among each DM client so that majority of them can confirm a to-be-published root zone (may be added to a distributed ledge).
• Most vital functions of FDMA is a sort of group signing scheme to fit DNSSEC context. It is required to provide a same interface for DNSSEC validating resolver. Threshold signature scheme provides basis primitives for that purpose producing three tuple for one message <one Public key, {secret shares}, one signature of the msg>. In proposed FDMA, each DM client is expected to sign the zone with its secret share and broadcast to other clients. m of n signatures can be transformed to the to-be-publish signature for each resource record according to the basic primitive of Threshold signature scheme.

More detailed design is needed such as initial trust setup, key generation, the choice of crypto curve, KSK rollover in FDMA etc. But again it is worth of time and effort to introduce more fault-tolerant property into Yeti Root DM Architecture.

It is a statement and decision of one-year extension for Yeti DNS Project, to the end of 2019.

Yeti DNS Project was started on 8th of May 2015. It was planned as a three-year project with a list of research agenda. However, It spent nearly one year to build the stable DNS root testbed. At that time it was expected the project would be extended a little bit , to the end of 2018. In the past 3 years, several planned Yeti experiments were performed, the system was shown to work, interesting findings were found and Yeti mailing list became a platform to discuss DNS root and IPv6 DNS. (More information was recorded in http://yeti-dns.org/documents.html and I-D: draft-song-yeti-testbed-experience )

It seems that we can safely end Yeti project by the end of 2018 gracefully. But after several round of discussions with Yeti coordinators and sponsors, we think one year extension is needed due to following reasons:

1) There is an important Yeti experiments which need time to be performed: KSK algorithm rollover. The KSK algorithm rollover will be a little challenging, because it is not done in real operational root system. And it may be more complex than KSK rollover due to unknown readiness for various implementations for rolling to ECDSA , especially considering the readiness monitoring and backward compatibility of rollover which IMHO are lessons from ICANN’s KSK rollover.

2) Some Yeti sponsors hope Yeti DNS project be continued with some major changes. So we need time to “Re-charter” Yeti project with clear goal. It was proposed to change the project in a significant way. One is for operational use, another is something more experimental breaking the rule of keeping root zone intact. After resolving the distractions and ambiguity of Yeti in 2017, all coordinators had reached the consensus that the best value of Yeti is being a testbed for research purpose. So we need time to carefully consider what can we do in Yeti DNS testbed next, other than to build a system in a rash.

One candidate input is to try Blockchain technology in Yeti testbed to change how DNS zone organized nowadays. It is inspired by recent discussions in DNSOARC mailing list on “blockchain DNS” and seciton 3.12 in RFC8324. We can introduce a new testing TLD for that purpose without breaking current Yeti root services. It is still open for other proposal candidates of how to change Yeti root significantly.

3) In the meanwhile of Yeti Re-chartering, we need time to look for more sponsors and resource for future Yeti. We need stable funding, full time employers and commitment from our partner.

If both 2# and 3# are successful, then we can proceed more confidently.

Thanks for all sponsors, operators and participants of Yeti project in past years. I hope we can work together in future. Please let us your thoughts and suggestions.

Cheers,

Davey Song on behalf of Yeti coordinators

Introductions

Yeti DNS Project takes the IANA root zone, and performs minimal changes needed to serve the zone from the Yeti root servers instead of the IANA root servers. The Yeti-DM-Setup document describes what’s the minimal changes is necessary and how it is done in Yeti DM.

One change of DNSSEC is that the current Yeti root zone is signed by Yeti’s key(ZSK and KSK) replacing all keys and signatures from IANA in the zone. However, it is proved that this change is not that “minimal” and the difference (by diff) between IANA root zone and Yeti root zone can be further reduced. It is proposed as a Yeti experiment which preserves IANA NSEC Chain and ZSK RRSIGs in Yeti root zone. The purpose of this experiment is to verify the capability of root zone including record-by-record signatures and respective keys based on current DNS protocol and implementation, and to verify whether it is workable without breaking DNSSEC validation in validating resolver.

This document describes in detail the operational steps to accomplish a Yeti experiment called PINZ (Preserving IANA NSEC Chain and ZSK RRSIGs). It is first introduced as an experiment proposal in a Yeti blog post and a lab test was done for feasibility study. The steps to be performed are based on the experiment proposal, meetings of Yeti coordinator, and comments collected from Yeti participants.

Preparation and considerations

There are mainly three parts for preparation and consideration for this experiment plan.

The Changes in PINZ

To simply understand the changes made in PINZ, there is a table provided to compare the difference between current Yeti root zone with the zone in PINZ experiment. It is helpful for DM operators to prepare the system to generate new zone in PINZ.

Table 1. The changes made in PINZ Compared with current Yeti zone

Before the experiment, DM operators are required to develop a new routine (Python or Perl script) to apply the changes when generating a new root zone according to the experiment proposal.

The example of diff result between IANA’s zone and BII-DM’s zone at SOA Serial 2018032601 is shown : http://yeti-dns.org/diff_iana_bii.txt

The example of diff result between BII-DM’s zone and WIDE-DM’s zone at SOA Serial 2018032601 is shown: http://yeti-dns.org/diff_bii_wide.txt

PINZ Transition

Similar to ZKS rollover process, the transition to PINZ requires administrator to consider the fact that data published in previous versions of Yeti zone still lives in caches. For example the signatures of NSEC Chain and DS records saved from IANA root zone will meet the old DNSKEY RRSet which is still cached in Yeti’s resolvers.

As a result, it is important to design a transition plan for PINZ experiment, pretty similar to the Yeti ZSK rollover plan which follows the Pre-Publish approach defined in RFC6781. The only difference is that the PINZ transition does not require a DNSKEY removal process suggested in RFC6781, because the Yeti ZSK will continue to sign the . SOA and . NS RRs. It means all Yeti keys and IANA ZSK will remain in the Yeti root zone unless any potential serious problem happens.

Note that In case of serious failure taking down the system, a fall back mechanism will be triggered to roll the system back. The Fallback Plan is introduced later in this document.

Note that it is desired, according to Yeti DM operators, that the PINZ transition period avoids IANA-ZSK rollover and ZSK rollover of each DM. The ZSK rollover information is given as below:

• Next ZSK rollover time of IANA: June 20th

• Next ZSK rollover time of BII: pre-publish at 2018042100 (and next rollover at 2018050500), activate in 3 days

• Next ZSK rollover time of WIDE: pre-publish at 2018041700 (and next rollover at 2018051700), activate in 3 days

Experiment schedule

PINZ transition can be simply divided into two phases:

Phase A : Publication

According to The table 1, the first change made by PINZ is to publish IANA ZSK of the time into Root DNSKEY RRset. This phase is called Publication for Phase A. And it will last 10 days according to table 2.

Phase A is successful when the new key is successfully configured in both server and resolver as a valid ZSK in DNSKEY RRset. And there is no identifiable systemic failure impacting DNSSEC validating servers and resolvers.

Phase B : New RRSIGs

After Phase A, it is considered that all Yeti caches are updated with new key. It is time to replace Yeti’s RRSIGs for TLD’s DS and NSEC with RRSIGs signed by the new key. It is called New RRSIGs for Phase B. The end of Phase B currently is not defined. It could be a normal configuration of Yeti root testbed if there is no significant failure taking down the system.

Phase B is successful when no operational issues remain after transition from current Yeti root zone to PINZ zone.

Schedule

There is a timeline for PINZ experiment in which one “slot” is 10 days long:

The tentative time:

• Starting Slot 2 on May 1st 2018.

Note that WIDE and BII will make the DM system ready at 0700 UTC on May 1st waiting for next IANA serial to publish IANA ZSK.

• Starting Slot 4 on May 21st 2018.

Note that WIDE and BII will make DM system ready at 0700 UTC on May 21st waiting for next IANA serial to publish IANA RRSIG.

Fallback Plan

Even though Yeti testbed does not have large-scale network, but for research purpose, it is worth considering and preparing a fallback plan, in case any unforeseen circumstances occur which cause very serious failures and can not be solved by (temporarily) disabling DNSSEC validation for small group of validating resolvers.

According to the milestones in the experiment schedule, the beginning of Phase A (slot 2) and Phase B (slot 4) are the two failure points where fallback may happen.

• If a problem arises during the slot 2 when IANA ZSK is Pre-publised in the Yeti root zone, the fallback plan is to simply un-publish that IANA ZSK from Yeti root zone, and continue signing with the current Yeti ZSK (one DM’s ZSk). The experiment will be postponed for looking into what’s wrong.
• If a problem arises during the slot 4 when IANA’s RRSIGs (DS/NSEC) are included replacing the RRSIG signed by Yeti ZSK, the fallback plan is simply to revert to signing NSEC RR and TLD’s DS with Yeti ZSK. The IANA Key is still published in the root zone until some decision is made to remove it.

Note that there may be other corner cases which are not obvious right now. So it is worth doing the monitoring at high alert and report any issue ASAP.

What to Measure

As a Yeti experiment, generally we should capture weird and failing events with sufficient information to diagnose them. Logs and captured packets will be helpful. Specific to PINZ, there are mainly two expected events worth of measuring:

###1) DNSSEC validation failure

In lab test, we tested as a black box to check the RCODE of DNS response to query with DO bit. We also recorded the DNSSEC log to demonstrate whether it worked or not. So Yeti resolver operators can monitor in that way. If anything weird happens they can report the event to Yeti discuss mailing list with descriptions of the event and DNSSEC log.

31-May-2017 15:46:49.278 dnssec: debug 3: validating net/DS: starting
31-May-2017 15:46:49.278 dnssec: debug 3: validating net/DS: attempting positive response validation
31-May-2017 15:46:49.278 dnssec: debug 3: validating net/DS: keyset with trust secure
31-May-2017 15:46:49.278 dnssec: debug 3: validating net/DS: verify rdataset
(keyid=14796): success
31-May-2017 15:46:49.278 dnssec: debug 3: validating net/DS: marking as secure, noqname proof not needed

In the above DNSSEC log, the keyid 14796 is IANA ZSK. It’s successful because the IANA ZSK is recognized and used for validation.

Note that it is also proposed that a small piece of script running on volunteer resolver and recording the event in a formal and unified format may help for analysis and statistic.

###2) Response size increase for DNSKEY query.

Similar to the monitoring on Yeti KSK rollover, a monitoring on response size should be performed during the experiment. Although it can be foreseen and calculated by adding additional key, the monitoring is useful for providing extra information on the impact of PINZ experiment.

Notification to Yeti resolvers

Currently Yeti has hundreds of resolvers who send queries to Yeti root servers. Although we fully expect the change to occur without incidents. However, unforeseen problems may be beyond our control which may cause DNSSEC validation fail for some validating resolvers. So it needs more publicity and awareness of these changes in PINZ among Yeti community members before the kick off of the experiment.

One-month notify and warning in advanced is necessary for Yeti resolvers with mails and blog post introducing how the experiment will be performed as well as the impact.

One simple way to fix unforeseen problems on resolver side is to disable DNSSEC if any validation failure is noticed during the transition period (new RRSIG phase).

One topic in Yeti DNS research agenda is about the DNS root scaling issues. However, when different people talking about the root scaling issue they may have different context in their mind. One is about the root zone augmentation and impact[1][2], which is closely related to ICANN’s New gTLD program. Another one is about how to pervasively replicated and distribute the root zone file[4][5]. Yeti is considered the later scenario, but combined with the consideration for the issues of external dependency and surveillance risk.

Different from related discussion and proposal, Yeti adopt a straightforward approach by setting up 25 IPv6-only root name servers beyond the limitation of magic number of “13”. So far, no notable system outage and failures reported by our monitoring system and end users who made conscious decision to join the experimental, non-production system.

This article revisits “magic 13” briefly for audiences without the background, explores what a appropriate number is, and introduces what is the case in Yeti.

The Brief Background of Magic Number “13”

People have questions on the limitation of the number of DNS name server. One typical case is 13 root letters,the name server of the root (“.”). I remember 5 years ago, I was the guy who public asked the very questions: “why there are only 13 root letters”. Today in ICANN RSSAC public meeting and related mailing list, the question still bounces from time to time. From my 5 years experience on DNS Root issues, I can see there are at least 2 aspects in which people are prudent on scaling the DNS name server system, especially the Root. (More reading on that topicThe Root of the DNS and RSSAC028)

1) Historical choice and operational inertia

RFC1035 is frequently referred as the “initiator” for the limited space for DNS message over UDP. Because at that time, 30 years ago, the poor network condition did not support large datagrams or fragments, partially due to limited host’s buffer. IPv4 MTU is set 576 octets which requires a proper size limit on DNS message. RFC1035 gives a way out that if the length of DNS message is greater than 512 octets, it should be truncated asking for DNS query over TCP.

From the operational point of view, UDP is preferred to TCP as the transmission protocol for DNS due to performance consideration. So not surpassing the size limit(512 octets) become a practical choice at that time. In addition, it is a operational inertia that the response to the NS query (like priming query) is always expected to carry complete information of NS server, both the NS records and the addresses. It in another way puts a constrain on the number of NS records.

After IPv6 and DNSSEC introduced, the normal response containing all IPv4/IPv6 addresses and DNSSEC signature already surpassed 512 octets. The size limit of 512 octets has long gone. However, the “Magic 13” as an operational choice and inertia has been observed not be loosened even after EDNS0 introduced and widely deployed during all these years.

Note that when Internet governance became popular these years, “more root” become a political sensitive topic and the inertia become harder.

2) IPv6 Fragmentation issue in DNS

IPv6 has a fragmentation model that is different from IPv4 – in particular, fragmentation always takes place on the sending host, and not on an intermediate router. Fragmentation may cause serious issues; if a single fragment is lost, it results in the loss of the entire datagram of which the fragment was a part, and in the DNS frequently triggers a timeout. It is known at this moment that only a limited number of security middle-box implementations support IPv6 fragments.

Some public measurements and reports [draft-taylor-v6ops-fragdrop],[RFC7872] shows that there is notable packets drop rate due to the mistreatment of middle-box on IPv6 fragment. One APNIC study [IPv6-frag-DNS] reported that 37% of endpoints using IPv6-capable DNS resolver cannot receive a fragmented IPv6 response over UDP.

Some root servers (A, B, G and J) truncate the response once the large IPv6 packet surpasses 1280 octets to avoid IPv6 fragmentation issue during ICANN KSK rollover process. Another APNIC study show that to avoid fragmentation surpassing the maximum Ethernet packets size of 1500 octets, 1500 octets will be a new size limit for DNS packets, given EDNS0 is widely deployed.

Limit of 13 NS records still applies?

After above analysis, instead of 512 octets, 1500 octets seems a practical and safe size limit to avoid fragmentation of outbound packets for both IPv4 and IPv6. Another calculation is necessary to find how many NS records (N) can be accommodated in this size limit.

     --Header: 12 octets

     --Question: 5 octets

     --Answer: 31+15*(N-1) octets [calculated base on current naming scheme : x.root-server.net, which can be compressed]

     --Additional: 16*N +28*N [each A record is 16 bytes, and each AAAA record is 28 bytes]

If 512 octets still applies, than we should got N in the formula below:

    12+5+31+15*(N-1)+16*N +28*N <512, then Maximum N = 8

If we put 1452 octets (1500-40-8) as the size limit, we got N=24. If the roots are IPv6 only, we got:

    12+5+31+15*(N-1)+28*N <1452, then Maximum N =33

If consider the DNSSEC signature (RSA/SHA-256)in the response, when DO bit is set in NS query, then extra 286 octets(RRSIG) and 11 octets(OPT RR) is required in the DNS packet. Then we got N = 19 for dual-stack, N=26 for IPv6-only.

Worthwhile to mention that all calculation above is based on the assumption that the response to NS query contains all NS records and address information. According to the section 4.2 of RFC8109, if the additional section of DNS response may omit certain root server address information, the recursive resolver can to issue direct queries for A and AAAA RRsets for the remaining names. So if the address information is no required or partially required in the response, there are more space for NS records to be carried. There are cases in Yeti DNS testbed where the server only return NS records, or return small part of address information.

So as to the question “Does the limit of 13 NS records still applies?”. We can confidently say that the technical limit of 13 NS records has long gone but the operation inertia in people’s mind exists.

A performance test for many NS servers

Since the DNS size limit takes UDP as the default transmission protocol. People may think using TCP or DOH(DNS over HTTPs) to break the limit. To understand the performance difference of UDP and TCP with many NS servers, we conduct a simple lab test for that purpose.

The configuration is simple that a client directly connected with a server (Dell R320, 8G DDR, CentOS6.4, BIND9.9.5). The client using querytcp and queryperf to launch queries via TCP and UDP respectively. tcp_client is set 10000 in named.conf. The name in NS record is constructed in a format “x.root-servers.net” in which x is natural number starting from 1. When more names add to the NS RR, we record the qps as the performance metrics. Since EDNS0 support 4096 octets, more names (more than 270 names) will cause truncation and fallback to TCP. So we only test from 13 names to 200 names to explore the trend. The diagram below shows that the packets size increase and the qps of UDP DNS is rapidly decrease.

Regardless of fragmentation issue, Using either UDP or TCP with many NS records will introduce impact on the QPS of authoritative server. In addition there is a concern that too many records may require extra resource on recursive server to keep track the state of each server, doing RTT-sorting and recording EDNS0 support etc.

For modern resolver CPU and RAM related resource is not the bottle neck, compared to timeout and latency due to network failures and switching from one name server to another. So the practical and safer choice is continue to keep a fixed number of NS record (but not 13) and return as much address information as possible if UDP is chosen.

I do recommend choosing TCP or DOH for large DNS response. It is deserve a special treatment for IPv6 DNS especially. An technical article once introduced Ali-Cloud using a elastic security network to defend DDOS attack. The key idea is that its DNS-liked system can return thousands dynamic NS and A records for their services.

Yeti case with 25 name servers

As introduced, Yeti has 25 root servers. People may ask also why 25 ? It seems explained by previous calculation for IPv6-only root, less and equal than 26 within a packets size limit of 1500 octets containing necessary signature and opt RR. In fact Yeti developed to 25 root servers without such design behind.

Many of the Yeti experimental design choices were expected to trigger larger responses:

1) Choose a root naming scheme (free-form) can not be compressed.

2) Introduce more NS records for root zone enlarge priming response containing all NS records and their glue.

3) Yeti DNSSEC experiment like multiple ZSKs, KSK rollover, and reusing IANA Key and RRSIG, greatly enlarges the apex DNSKEY RRSet.

In the beginning we did not consider the size limit of any form. 25 is the initial target of Yeti testbed development allow an fair distribution of root in 5 Continents with qualified and responsible operators. And the number 25 is also viewed a fair number as a challenge case for “Magic 13”.

When Yeti developed with 25 servers, the size of priming response is up to 1754 octets with RRSIG and all address information (With Shane’s patch), surpassing the size limit of 1500 octets. It is under a risk of IPv6 fragmentation observed in Yeti testbed, which has around 7% packet drop rate if the packet size is larger than 1500 octets. But thanks to the diversity of Yeti testbed, Yeti servers response differently according to the local configuration and DNS software.

Yeti DNS uses free-form names for each root server which can not be compressed. According to the protocol (BIND9 implementation) not all glue record is necessary be included in the additional section. Some root servers adopted a patch (developed by Shane Kerr) to contain all glue records but the response will be fragmented. Some root server set 1280 octets as a IPv6 packets size limit (1222 octets response), so they will contain as much address information as possible until the response reach the limit. I wondering why nobody use 1500 as a size limit which will fully use the room to contain more IPv6 address.

As to the servers response with no glue, it is in a questions that because all names is under the root. There should be at least 2 glue records contain in the additional section. As far as I know BIND9/Unbound work because they take the addresses listed in the hint file as trusted glue records. After checking rfc1034#section-5.3.3, it is proved compliant to the DND specification that “If the search for NS RRs fails, then the resolver initializes SLIST from the safety belt SBELT”. (The safety belt SBELT is a local a configuration file, the hint file)

Conclusion

In conclusion the limit of 13 is not technical applied. But people still has a inertia that 13 is the fix number of NS records. Now IPv6 fragmentation issue introduces new size limit that exert constraint on the size of response. Now the practical and safe choice is to keep the limit 19 for dualstack, 26 for IPv6-only taking 1500 octets as a packets limit. All glue records can be contained. If you want to add more NS records, you will lose some availability (less glue records).

Yeti is taken as an case study in this context. Yeti testbed now has 25 root name servers which already surpassed the limit. The response behaviors of different root servers vary. Although the consistency of DNS response is lost, it is good to have diversity for testbed purpose.

Thanks for suggestions and editorial work from co-authors and volunteer reviewers. The 06-version of Yeti Testbed experience draft is ready for public review.(Github repo of this draft) Now it is in the queue of IETF Independent Submission Editor(ISE). The comments from reviewer are welcome and to be sent to the ISE.

It is worth mentioning that in this draft, a conclusion section is added for the convenience of audience which includes a brief list of findings and experience. In addition it suggests several topics worthy of future study based on experience gained during the operation of Yeti DNS testbed.

o Priming Truncation and TCP-only Yeti-Root servers: observe and measure the worst-possible case for priming truncation by responding with TC=1 to all priming queries received over UDP transport, forcing clients to retry using TCP. This should also give some insight into the usefulness of TCP-only DNS in general.

o KSK ECDSA Rollover: one possible way to reduce DNSKEY response sizes is to change to an elliptic curve signing algorithm. While in principle this can be done separately for the KSK and the ZSK, the RIPE NCC has done research recently and discovered that some resolvers require that both KSK and ZSK use the same algorithm. This means that an algorithm roll also involves a KSK roll. Performing an algorithm roll at the root would be an interesting challenge.

o Sticky Notify for zone transfer: the non-applicability of IXFR as a zone transfer mechanism in the Yeti DNS testbed could be mitigated by the implementation of a sticky preference for master server for each slave, such that an initial AXFR response could be followed up with IXFR requests without compromising zone integrity in the case (as with Yeti) that equivalent but incongruent versions of a zone are served by different masters.

o Key distribution for zone transfer credentials: the use of a shared secret between slave and master requires key distribution and management whose scaling properties are not ideally suited to systems with large numbers of transfer clients. Other approaches for key distribution and authentication could be considered.

Recently there were confusions on BII IPv6 public DNS and Yeti DNS within Yeti technical community, and here on behalf of BII would like to reaffirm BII’s perspective on the Yeti DNS Project and BII IPv6 public DNS.

As it is stated on Yeti website, Yeti DNS Project is a parallel experimental live IPv6 DNS root system, aiming for network science and experiment. It is an IPv6-only root server testbed which tests and implements new ideas. There is no connection either in infrastructure, software, operation, or mission, between the Yeti DNS Project, and the BII IPv6 public DNS system, which aims to provide a safe and stable DNS service with low latency and support the transition from IPv4 to IPv6. Meanwhile BII does mirror our IPv6 public DNS system’s traffic to the Yeti DNS Project to support the research activities.

We believe that experience and technics from Yeti is important, and we will keep discovering the limits of DNS root name service and delivering useful technical output with the belief of “One World, One Internet, One namepace”.

Introduction

DNS Root Server system is an early form of cloud model to replicate the root zone file and enhance high availability. To expand the capacity of the root server system, the root anycast instances were deployed widely given that the list of root servers are the proof of the integrity of root zone content. People trust the data by trusting the servers where the data is pulled from.

Entering DNSSEC, data can be authenticated with crypto-signature. People gain confidence on data integrity by self-validating data. The place where the root zone file is replicated and stored becomes irrelevant. RFC7706 is one example providing a mechanism to give the same answer back from the “local” root server as from the “real” root server on the same level of authentication. Although they may bring some operation concerns and risks, it does gain in terms of performance (reducing the latency). So it becomes popular among some public DNS operators.

By contrast, Yeti provides a different zone file distribution model which introduces a new set of authoritative servers and their instances for the root zone. It is an efficient and large-scale approach to replicate and distribute the root zone file in a straightforward manner compared to RFC7706. Imagine that you have more than 1000 resolvers in your network, you may prefer one root instance located in your network to the management of 1000 “local” root servers.

However, compared to RFC7706, Yeti model cannot provide an unmolested DNSKEY RRset or an unmolested apex RRset or RRSIG. It is questioned as being an alternative root, with the ability to change the TLD meta data and sign with alternative keys. It is regarded as violating the political correctness of keeping a unique identifier system. It is not a technical issue, but an “ethic” risk which Yeti model advocates are undertaking.

To relieve the risk as much as possible, an proposal is formed to build a new root zone containing both Yeti and IANA Keys and RRSIGs. The new root zone are divided into two parts: one is apex records : SOA, NS , DNSKEY including their RRSIG signed by Yeti keys(ZSK & KSK). The other part is TLD meta data like DS, NSEC, NS, A/AAAA glue as well as their RRSIG signed by IANA ZSK(glue and NS has no RRSIG). Note that DNSKEY RRset includes Yeti ZKS(s), KSK(s) and IANA ZSK(s).

Lab test and result

To validate this proposal, a lab environment test was done. In normal DM operation loop, the IANA DNSKEY RRset and DNSSEC signature is removed after downloading the root zone. In the test, we keep IANA ZSK and non-Apex RRSIG (signature for DS and NSEC) and NSEC in the Yeti root zone.

Authority server setup

The routine for generation of a new Yeti root zone is as follows:

• Download the latest IANA root zone.(using AXFR from F.ROOT-SERVERS.NET for instance);
• Strip apex SOA, NS and DNSKEY as well as their RRSIGs; Keep the NSEC ,TLD DNS and Glue, RRSIG of non-apex RR unchanged;
• Put a new DNSKEY RRset into the zone with Yeti ZSK and KSK as well as IANA ZSK.
• Put a new SOA RR, copying IANA SOA SERIAL, setting MNAME to www.yeti-dns.org. and setting RNAME to <dm-operator>.yeti-dns.org. where <dm-operator> is currently one of “wide”, “bii” or “tisf”.
• Put a new NS RRset with existing 25 servers in Yeti testbed.
• Sign the apex SOA, NS RR with Yeti ZSK and sign the DNSKEY RRset with Yeti KSK in the zone.
• Publish the root zone to testing server A.

There is a diagram of the constructed new Yeti root zone. At left is the IANA root zone, at right Yeti’s one. Only one delegation is put, only to show the mechanism. The (I) suffix means “IANA” and “Y” Yeti:

SOA(I)                          SOA(Y)
RRSIG(. SOA(I)(ZSK(I))          RRSIG(. SOA(Y)(ZSK(Y))

DNSKEY(I)                       DNSKEY(Y)
    KSK(I)                           KSK(Y)
    ZSK(I)                           ZSK(I)
                                     ZSK(Y)
RRSIG(. DNSKEY(I))(KSK(I))      RRSIG(. DNSKEY(Y))(KSK(Y))

. NS IANA_NS1                   . NS Yeti_NS1
. NS IANA_NS2                   . NS Yeti_NS2
. NS IANA_NS3                   . NS Yeti_NS3
RRSIG(. NS)(ZSK(I))             RRSIG(. NS)(ZSK(Y))

NSEC1(.->net.)                  NSEC1(.->net.)
RRSIG(NSEC1)(ZSK(I))            RRSIG(NSEC1)(ZSK(I))

net. NS NS7                     net. NS NS7
net. NS NS8                     net. NS NS8
net. DS                         net. DS
RRSIG(net. DS)(ZSK(I))          RRSIG(net. DS)(ZSK(I))

NSEC2(net.->.)                  NSEC2(net.->.)
RRSIG(NSEC2)(ZSK(I))            RRSIG(NSEC2)(ZSK(I))

Resolver setup

There was a DNSSEC validating resolver B set up (BIND). Resolver B used Yeti KSK as its trust anchor. The glues record of Yeti_NS1, Yeti_NS2, Yeti_NS3 are configured with the address of server A. So that all queries will be sent to server A. The rollover of KSK and ZSK of the new Yeti root zone is expected to be fine because Yeti Multiple-ZSK experiment had been tested and workable with many keys in DNSKEY RRset. The only impact is the size of DNSKEY response.

Result analyses

By analyzing the log, it was found that server B validate the response using the RRSIG signed by IANA zsk (keyid=14796).

Resolver B’s DNSSEC log：

31-May-2017 15:46:49.278 dnssec: debug 3: validating net/DS: starting 
31-May-2017 15:46:49.278 dnssec: debug 3: validating net/DS: attempting positive response validation 
31-May-2017 15:46:49.278 dnssec: debug 3: validating net/DS: keyset with trust secure 
31-May-2017 15:46:49.278 dnssec: debug 3: validating net/DS: verify rdataset (keyid=14796): success 
31-May-2017 15:46:49.278 dnssec: debug 3: validating net/DS: marking as secure, noqname proof not needed

The current dnskey rrset of IANA root zone is:

dig @f.root-servers.net . dnskey +multi

.                       172800 IN DNSKEY 257 3 8 (
                                AwEAAagAIKlVZrpC6Ia7gEzahOR+9W29euxhJhVVLOyQ
                                bSEW0O8gcCjFFVQUTf6v58fLjwBd0YI0EzrAcQqBGCzh
                                /RStIoO8g0NfnfL2MTJRkxoXbfDaUeVPQuYEhg37NZWA
                                JQ9VnMVDxP/VHL496M/QZxkjf5/Efucp2gaDX6RS6CXp
                                oY68LsvPVjR0ZSwzz1apAzvN9dlzEheX7ICJBBtuA6G3
                                LQpzW5hOA2hzCTMjJPJ8LbqF6dsV6DoBQzgul0sGIcGO
                                Yl7OyQdXfZ57relSQageu+ipAdTTJ25AsRTAoub8ONGc
                                LmqrAmRLKBP1dfwhYB4N7knNnulqQxA+Uk1ihz0=
                                ) ; KSK; alg = RSASHA256; key id = 19036
                                
.                       172800 IN DNSKEY 256 3 8 (
                                AwEAAZLsvvPf/WDYE8L8ZUOiXEN9PSg3Nqyqu8WWKVu4
                                LIvPW77Y0W1mBgmroFEHxzA9zM2edQbYIFhj536sx3fj
                                1bx4n8yKwQ0y1MYkXvgU3aKWf0jnm8qz5RPVa4iEUgP5
                                og/70muAXmo38Ru10asc3SNtN2OlAT1Tivp+U4JUlVA8
                                ICvierUPg2Kgf2gtNFhuqMTx/IHrE5DdV+Z/rmHGqESe
                                /mnaDzHdFnKy7Ma5zcPTekmwZ86CnG+vNWMT6wiO56YS
                                AaiHX0jSCjmkL7JzT3xpp/vtoWLK1rir70i6NbsU48sP
                                ZOHLfhwzbownewAsng0QqMOcVcmhl6ifE3C/vjE=
                                ) ; ZSK; alg = RSASHA256; key id = 14796

Conclusion and future work

There is an effort to make and demonstrate a diff file between IANA root zone and Yeti root zone in the monitoring website. By reusing IANA DNSSEC stuff, this proposal generates a Yeti root zone including IANA ZSK and signatures, consequently reducing the size of the diff file (smaller diff means less change). If IANA ZSK and RRSIG in Yeti root zone can be viewed in a way as a proof integrity, then it “provides”:

1) Guarantee no edit or changes to existing TLD meta data(DS RR);

2) Guarantee no existing TLD is removed or new TLD is added into the root zone;

3) Yeti keys only sign apex will avoid multiple RRSIGs and keep the response size small.

Background

Yeti has successfully performed a KSK roll in 2016. During the first KSK rollover, it was discovered that a view added to a BIND 9 resolver did not use the same RFC 5011 timings as other views, including the default view. An administrator may expect that a BIND 9 resolver will handle RFC 5011 KSK rollover for all zones identically. The newly added views will fail to resolve when the KSK roll happens.It was documented in a posts in Yeti blog.

In order to add more information to the earlier roll, we performed another KSK roll before the ICANN KSK roll begins (2017-10-11). We used similar phases to the IANA roll, although with different timings so that we can finish the roll faster. Our “slots” will be 10 days long, and look like this:

As with the first KSK roll, the Yeti root KSK uses the RSA/SHA-256 algorithm with a 2048 bit key, the same as the IANA root KSK. Note that Yeti root ZSK uses 2048 bit key (RSA/SHA-256) as well.

In the design of Multi-ZSK, each Yeti DM used their own ZSK to sign the zone. That is to say at least 3 ZSKs are in the DNSKEY RRset, or more than 3 ZSKs (up to 6) due to ZSK rollover which may cause some overlap. Take the KSK rollover into consideration, it is possible that there are 8 keys (6 ZSK and 2 KSK) in DNSKEY RRset which will produce a very large response to the DNSEKY query with DO bit set. It is cool for yeti experiment looking for impacts of large DNS response.

Monitoring setup

Like we did in the first KSK roll, monitoring was set up on Yeti validating resolvers to capture validation failures. Besides DNSSEC validation, fragmentation and packet loss due to large DNS response are in question which Yeti was started to investigate. We set up RIPE Atlas measurements which are able to spot failures like timeout for DNSKEY queries via UDP.

For each Yeti server, a measurement was setup which ask for 100 IPv6-enabled probes from 5 regions, every 2 hours sending DNS query for DNSKEY via UDP with DO bit set.

The measurement API compatible specification is as follows:

curl --dump-header - -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d '{
 "definitions": [
  {
   "target": "240c:f:1:22::6",
   "af": 6,
   "query_class": "IN",
   "query_type": "DNSKEY",
   "query_argument": ".",
   "use_macros": false,
   "description": "DNS measurement to 240c:f:1:22::6",
   "interval": 7200,
   "use_probe_resolver": false,
   "resolve_on_probe": false,
   "set_nsid_bit": false,
   "protocol": "UDP",
   "udp_payload_size": 4096,
   "retry": 3,
   "skip_dns_check": false,
   "include_qbuf": false,
   "include_abuf": true,
   "prepend_probe_id": false,
   "set_rd_bit": false,
   "set_do_bit": true,
   "set_cd_bit": false,
   "timeout": 5000,
   "type": "dns"
  }
 ],
 "probes": [
  {
   "tags": {
    "include": [
     "system-ipv6-works"
    ],
    "exclude": []
   },
   "type": "area",
   "value": "West",
   "requested": 20
  },
  {
   "tags": {
    "include": [
     "system-ipv6-works"
    ],
    "exclude": []
   },
   "type": "area",
   "value": "North-Central",
   "requested": 20
  },
  {
   "tags": {
    "include": [
     "system-ipv6-works"
    ],
    "exclude": []
   },
   "type": "area",
   "value": "South-Central",
   "requested": 20
  },
  {
   "tags": {
    "include": [
     "system-ipv6-works"
    ],
    "exclude": []
   },
   "type": "area",
   "value": "North-East",
   "requested": 20
  },
  {
   "tags": {
    "include": [
     "system-ipv6-works"
    ],
    "exclude": []
   },
   "type": "area",
   "value": "South-East",
   "requested": 20
  }
 ],
 "is_oneoff": false,
 "stop_time": 1501422358
}' https://atlas.ripe.net/api/v2/measurements//?key=YOUR_KEY_HERE

Given there are 25 servers, 25 RIPE Atlas measurements were set up during the rollover(from 2017-05-08 to 2017-05-25). Result files in JSON format were collected.

Note although each measurements ask for 100 IPv6 enabled probes to do the job, there are may lack of qualified probes in some regions. In most of case there are around 66 probes allocated for each measurement, which is acceptable for this experiment.

Data analysis

The basic plan is to simply count the number of failures under different DNS response sizes during the experiment. Failure rate of each response size is the metric to measure if there is a correlation between failure rate and the DNS response size.

Simple Data pre-processing

Before the analysis is proceeded, some observation were made to filter bad samples and result.

• There are always a few probes found in each measurement failed all the time (>80% failure rate). They are labeled as a weak probes which are excluded for that measurement.
• There are two probes (prb_id: 17901, 12786) found weird in behavior, who reported timeout for small response(1139 octets) but works for large response (1975). They are excluded for all measurement.
• The are two measurements (msm_id: 8552432,8552458) reported far more failures and weak probes than other measurement. May be there are some routing problems on the path to these Yeti servers. So They are excluded for further analysis.

DNS response size

Although it is possible that there are 8 keys (6 ZSK and 2 KSK) in DNSKEY RRset and many possible combinations, during KSK rollover experiment we actually observed 4 cases:

• 3 ZSK and 2 KSK with 2 DNSKEY signature which produces DNSKEY response of 1975 octets; (overlaped with one ZKS rollover)
• 2 ZSK and 2 KSK with 1 DNSKEY signature which produces DNSKEY response of 1414 octets;
• 2 ZSK and 1 KSK which produces DNSKEY response of 1139 octets;

WIDE rolls its ZSK every month. BII rolls its ZSK every two weeks. And TISF does not use its own Key but reuse WIDE ZSK instead. It is proved that during the measurements only BII rolled its key which overlapped with KSK rollover.

Note there are also other DNS response size were observed like 1419,1430,1143,1982 octets on other servers.

Counting Failures （timeout）

For every tome slot(2 hours) around 66 probes sent queries to Yeti root servers. We counted the number of failures and number of total queries which are categorized according to the packets size. The statistic is shown as following table.

The following figure shows vividly the failure rate change during 1 KSK rollover and 2 ZSK rollovers (Note the monitoring was setup when two KSK were published). It experienced four periods: 2KSK+3ZSK, 1KSK+3ZSK, 1KSK+2ZSK, 1KSK+3ZSK.

Figure 1 Failure rate change during Yeti KSK rollover

In ICANN’s KSK rollover plan, the packet size will exceed 1280 Octets limit up to 1414 octets on 2017-Sept-19 and 1424 octets on 2018-Jan-11. It means around 2% IPv6 resolvers(or IPv6 DNSKEY queries with DO bit set) will experience timeout. One study shows that 17% of resolvers cannot ask a query in TCP. So probably in extreme case there are 0.34% of all resolvers around the world will fail to validate the answers. 0.34% of millions, It is not a trivial number.

Figure 2 Impact on the KSK Rollover Process (ICANN)

It should also be noted that during the KSK rollover and other experience in Yeti testbed, no error report was spotted directly due to packet size problem. So it should be further observed and evaluated on the impact of large packets issue.

Conclusion

The monitoring result shows that statistically large packets will trigger higher failure rate (up to 7%) due to IPv6 fragmentation issues, which accordingly increase probability of retries and TCP fallback. Even within 1500 bytes, when response size reaches 1414 bytes, the failure rate reaches around 2%. Note that ICANN KSK rollover will produce packets exceeding 1414 Bytes.

To avoid less than 10% anomaly inside DNS transaction, asking for TCP support seems a light-weighted solution currently. If you want to avoid TCP fallback, you can hold a certain size as a boundary for response, like 512-octets was hold for a long time in DNS specification. In IPv6 era, a new boundary 1220 octets (for IPv6 TCP MSS value) was proposed in APNIC blog posts. I would like to ask is it worthwhile to take any heavier measures to this issue? Does it sound like a plan to use stateful connection in the first place to transmit DNS like TCP or HTTP for queries causing large response, or fragment the packets in the DNS layer?

Background

Regarding the timing of Root Zone fetch and soa update, Each Yeti DM checks the root zone serial hourly to see if the IANA root zone has changed , on the following schedule:

A new version of the Yeti root zone is generated if the IANA root zone has changed. In this model, root servers will pull the zone from one DM consistently for each new update, because 20 min is expected to be enough for root zone update for all root severs in ideal environment.

There is a finding in the past one Yeti root servers have long delay to update the root zone which is first reported in the Yeti experience I-D:

It is observed one server on Yeti testbed have some bugs on SOA update with more than 10 hours delay. It is running on Bundy 1.2.0 on FreeBSD 10.2-RELEASE. A workaround is to check DM’s SOA status in regular base. But it still need some work to find the bug in code path to improve the software.

To better understand the issue, we design a monitoring test and trace the behavior of zone update and DM selection of each Yeti root server. Now there is some preliminary result.

Methodology

To setup this monitoring, there are mainly three steps:

1.In the loop of DM, we ask each DM operator to use a special rname .yeti-dns.org. in SOA record of Yeti root zone, such as wide.yeti-dns.org. for WIDE DM. By this setting, we marked the root zone with the original DM where each root server pull the zone.

2.In the loop of monitoring, it is simple to query soa record against each root server every minute. The information such as rname, the soa serial, and timestamp of the response is record.

3.Based on the data we collected, it is easy to figure out when the root zone update and where the zone is pulled, for each server and each soa serial.

Note that the time we observer a new zone update on a particular server includes the time of rtt and time error introduced by the interval(every minute) we measured. It is acceptable and does not impact the result and conclusions we made.

Preliminary result

There are there figures in the below to present the preliminary result for three continuous soa serial of Yeti. In the bar chart, the x-axis represent each root server with a bar. The value of bar in the y-axis is the delay in minute. We calculated the delay （d_i） of server i use the simple formula:

d_i=(T_i-T_min)/60+10

• T_i is the timestamp for root server i to update the zone
• T_min is the smallest timestamp of all server under a same soa serial

Note that we add 10 min to each value to make the bar chart more visible to figure out where the server pull the zone. So the actual delay is the value minus 10.

Figure 1 The latency of SOA 2017032102

Figure 2 The latency of SOA 2017032200

Figure 3 The latency of SOA 2017032201

Intuitively there are some findings from the results:

• Two servers from dnsworkshop.org still endure high latency of zone update, as we reported before. This issue is not resolved.
• Besides the dnsworkshop.org servers, half of Yeti servers has more than 20 min delay, some even with 40 min delay. One possible reason may be that the server failed to pull the Zone on one DM and turn to another DM which introduces the delay.
• In figure 2, it is observed that server dahu2.yeti.eu.org. has two bars in the chart which means for the soa serial 2017032200 dahu2.yeti.eu.org. pull twice first from TISF DM secondly BII DM. It’s weird.
• Also in figure 2, it is observed that even in the same 20-minutes time frame, not all servers pull from a single DM. I guess may be some servers not use FCFS strategy to pull the zone after they receive the notify. They may pull the zone based on other metrics like the rtt , or manual preference.

Another findings: Double SOA RR

During the test, we happened to find that dahu2.yeti.eu.org has two soa record due to the multiple-DM settings in Yeti testbed. (We once reported the IXFR fallback issue due to Multiple-DM model.)

$dig @dahu2.yeti.eu.org . soa +short 
www.yeti-dns.org. tisf.yeti-dns.org. 2017032002 1800 900 604800 86400 
www.yeti-dns.org. wide.yeti-dns.org. 2017032001 1800 900 604800 86400

As far as I know, dahu2.yeti.eu.org uses Knot 2.1.0 at that time. We will check if everything goes well after it update to latest version.

About EDNS Fallback

EDNS fallback is briefly defined in RFC6891 that a requester can detects and caches the information of remote end whether it support ENDS(0) or not. This behavior avoids fallback delays in the future. According to one of ISC’s document, BIND EDNS fallback has a process describe in below:

1) Query with EDNS, advertising size 4096, DO (DNSSEC OK) bit set
2) If no response, retry with EDNS, size 512, DO bit set
3) If no response, retry without EDNS (no DNSSEC, and buffer size maximum 512)
4) If no response, retry the query over TCP

The merit of EDNS fallback is to identify the capacity of remote server and shorten the delay with less retries. But if the intermittent network causes packet losses or DNS manipulation, it can result in SERVFAILs due to servers that should support EDNS being marked as EDNS-incapable.

A failure case observed in Yeti resolver

All yeti resolvers are required to be DNSSEC-aware. It is reported one of resolver using BIND 9.11.0-P2 (call R1) in China received many SERVFAIL response due to EDNS fallback. With the debug information, we found this resolver has such experience:

1) A client try to resolve www.facebook.com via R1; 2) R1 got a response (modified) and start doing DNSSEC validation for facebook.com; 3) R1 query the DS record of www.facebook.com via one NS of .com but got a modified response : www.facebook.com in A x.x.x.x.; 4) R1 tried all the other NS of .com, and got modified answer too; 5) R1 fallback to query the DS record with ENDS0 buffer size 512 bytes but still got the modified response; 6) R1 fallback again to query the DS record of www.facebook.com without EDNS0 option and receive the same modified response. 7) R1 can not validate the www.facebook.com. And the client got SERVFAIL.

But when any client try to resolve other normal domains. R1 got the right response for both A or AAAA record, but when it do the DNSSEC validation process, R1 sent the DS query without EDNS0 option, then the validation process failed. Finally, the client got SERVFAIL. There is a log for that process by querying dnsv6lab.net after www.facebook.com.

15-Feb-2017 13:24:01.203 dnssec: debug 3: validating dnsv6lab.net/DS: starting
15-Feb-2017 13:24:01.203 dnssec: debug 3: validating dnsv6lab.net/DS: attempting negative response validation
15-Feb-2017 13:24:01.203 dnssec: debug 3:   validating net/SOA: starting
15-Feb-2017 13:24:01.203 dnssec: debug 3:   validating net/SOA: attempting insecurity proof
15-Feb-2017 13:24:01.203 dnssec: debug 3:   validating net/SOA: checking existence of DS at 'net'
15-Feb-2017 13:24:01.203 dnssec: debug 3:   validating net/SOA: insecurity proof failed
15-Feb-2017 13:24:01.203 dnssec: info:   validating net/SOA: got insecure response; parent indicates it should be secure
15-Feb-2017 13:24:01.203 dnssec: debug 3: validating dnsv6lab.net/DS: in authvalidated
15-Feb-2017 13:24:01.203 dnssec: debug 3: validating dnsv6lab.net/DS: authvalidated: got insecurity proof failed
15-Feb-2017 13:24:01.203 dnssec: debug 3: validating dnsv6lab.net/DS: resuming nsecvalidate
15-Feb-2017 13:24:01.203 dnssec: debug 3: validating dnsv6lab.net/DS: nonexistence proof(s) not found
15-Feb-2017 13:24:01.203 dnssec: debug 3: validating dnsv6lab.net/DS: checking existence of DS at 'net'
15-Feb-2017 13:24:01.203 dnssec: debug 3: validating dnsv6lab.net/DS: checking existence of DS at 'dnsv6lab.net'
15-Feb-2017 13:24:01.203 dnssec: debug 3: validating dnsv6lab.net/DS: continuing validation would lead to deadlock: aborting validation
15-Feb-2017 13:24:01.203 dnssec: debug 3: validating dnsv6lab.net/DS: deadlock found (create_fetch)

It is obvious that BIND 9 gets confused about EDNS support and this breaks later DNSSEC lookups. The intuitive thinking in author’s mind is that all BIND 9 deployed in China may affected by this issue. It explains the low penetration of DNSSEC and complains on DNSSEC in that region. In general this bug may cause BIND 9 vulnerable to the on-path DOS attack against the DNSSEC-aware resolver.

Patch to this issue

After locating this problem we contact ISC people and got a following patch to fix it.

diff --git a/lib/dns/resolver.c b/lib/dns/resolver.c
index f935a67..5ca9c47 100644
--- a/lib/dns/resolver.c
+++ b/lib/dns/resolver.c
@@ -8145,6 +8145,7 @@ resquery_response(isc_task_t *task, isc_event_t *event) {
 		dns_adb_changeflags(fctx->adb, query->addrinfo,
 				    DNS_FETCHOPT_NOEDNS0,
 				    DNS_FETCHOPT_NOEDNS0);
+#if 0
 	} else if (opt == NULL && (message->flags & DNS_MESSAGEFLAG_TC) == 0 &&
 		   !EDNSOK(query->addrinfo) &&
 		   (message->rcode == dns_rcode_noerror ||
@@ -8169,6 +8170,7 @@ resquery_response(isc_task_t *task, isc_event_t *event) {
 		dns_adb_changeflags(fctx->adb, query->addrinfo,
 				    DNS_FETCHOPT_NOEDNS0,
 				    DNS_FETCHOPT_NOEDNS0);
+#endif
 	}
 
 	/*

Conclusion

EDNS fallback is proposed for good but it may introduce false positives and collateral impacts due to temporary network failure or malicious manipulations. When the name server of certain TLD like .com and .net are marked EDNS-incapable , it will become a disaster for validating resolvers.

One intuitive idea is to stop marking TLD’s NS server as EDNS-incapable, given the fact that 7040 of 7060 (99.72%) of name servers support EDNS. Or we can turn off the fallback function when it comes to DS record(the query to the parent).

A newly generated KSK will be published into the Yeti root zone for experiment today. Volunteer resolvers are welcome to join this test. There are some notes for your information:

1) Two actions:

• A new key(59302 ) will be published today at the serial 2017030200

• The document and KSK.pub file on the Github repo and yeti website will be update to contain two keys 10 days later(2017-03-12), leaving 10 days to welcome new resolver to join this experiment.

2) About the timeline:

Slot 1: 2017-02-20 to 2017-03-01   change the RRSIG validity period
Slot 2: 2017-03-02 to 2017-03-11   publish the new KSK
Slot 3: 2017-03-12 to 2017-03-23   publish the new KSK
Slot 4: 2017-03-24 to 2017-04-03   publish the new KSK
Slot 5: 2017-04-03 to 2017-04-13   publish the new KSK
Slot 6: 2017-04-14 to 2017-04-23   sign with the new KSK
Slot 7: 2017-04-24 to 2017-05-03   sign with the new KSK
Slot 8: 2017-05-04 to 2017-05-13   revoke the old KSK
Slot 9: 2017-05-14 to 2017-05-23   no longer publish the old KSK

3) For BIND users:

In the last KSK rollover experiment, we found multiple views of BIND may cause problem during the rollover period. Recently ISC published a post to explain it and ask BIND users to aware the change during the KSK rollover.

https://www.isc.org/blogs/2017-root-key-rollover-what-does-it-mean-for-bind-users/

4) For new resolver

If you would like to join the experiment, please follow the instructions in http://yeti-dns.org/join.html and set it up before 2017-03-12, because the page will be changed containing the two keys for new comer to start with.

Please let us know, if you found something weird during the experiment.

5) Reference

Second KSK rollover experiment in Yeti testbed

source: https://www.afnic.fr/en/resources/blog/yeti-dns-over-tls-public-resolver.html

There is a new DNS-over-TLS public DNS resolver, and it uses the Yeti root. You want explanations? You’re right.

First, about DNS-over-TLS. The DNS (Domain Name System) protocol is a critical part of the Internet infrastructure. It is used for almost every transaction on the Internet. By default, it does not provide any privacy (see RFC 7626 for a complete discussion of DNS privacy considerations). Among its weaknesses is the fact that, today, DNS requests and responses are sent in the clear so any sniffer can learn that you are interested in www.aa.org or jane-smith-server.accounting.company.example. To address this specific problem, a standard for encryption of DNS requests and responses, using the well-known protocol TLS (Transport Layer Security), has been developed. The standard is in RFC 7858.

As of today, there are very few DNS resolvers that accept DNS-over-TLS. The typical ISP resolver, or the big public resolvers, don’t use it. Sadly, this is also the case of resolvers pretending to provide a service for people who do not trust the other resolvers. (See an up-to-date list of existing public resolvers.)

And Yeti, what is it? It is an alternative DNS root which focus, not on creating “dummy” TLDs and selling them, but on technical experimentations about the DNS root service, experimentation which cannot be done on the “real” root, which is way too sensitive. Note there was no public Yeti resolver. To use the Yeti root, the only way was to configure your resolver to forward to the Yeti root.

But, first, a warning: Yeti is a technical experimentation, not a political one. Be aware that DNS queries to the Yeti root name servers are stored, and studied by researchers. (This is the same with the “real” root, by the way, not to mention the unofficial uses such as MoreCowBell.)

Since there are few DNS-over-TLS resolvers, and in order to gather more information from experience, we have set up a public DNS-over-TLS resolver using the Yeti root. It answers on the standard DNS-over-TLS port, 853, at dns-resolver.yeti.eu.org. It is IPv6-only, which makes sense for Yeti, whose name servers use only IPv6.

Two warnings: it is an experimental service, managed only on a “best effort” basis, and since it sends requests to the Yeti root, the user’s data is captured and analyzed. So, it is to test technically privacy-enhancing techniques, not to provide actual privacy. (We would be glad to see a real privacy-enabled public DNS resolver, with DNS-over-TLS and other features.)

Usage

Today, most DNS clients cannot speak DNS-over-TLS. If you want to use it and don’t know DNS-over-TLS clients, you can find some listed at the DNS privacy portal.

A way to use this service as a forwarder for a local resolver. The Unbound server can do that with a setup like:

server:
  ...
  auto-trust-anchor-file: "autokey/yeti-key.key"
  ssl-upstream: yes

forward-zone:
  name: "."
  #forward-host: "dns-resolver.yeti.eu.org" # Or the IP address:
  forward-addr: 2001:4b98:dc2:43:216:3eff:fea9:41a@853
  forward-first: no

If you have the getdns utilities installed (for instance via the Debian package getdns-utils), you can test the resolver with the command getdns_query:

% getdns_query  @2001:4b98:dc2:43:216:3eff:fea9:41a -s -l L www.eff.org AAAA
...
  "just_address_answers":
  [
    {
      "address_data": <bindata for 2a04:4e42:1b::201>,
      "address_type": <bindata of "IPv6">
    }
  ],
...

If you use the proxy Stubby, you can run it with:

% stubby  @2001:4b98:dc2:43:216:3eff:fea9:41a -L

(Or similar arguments from Stubby configuration file.)

Good luck with this service and, if there is a problem, do not hesitate to ask details and/or help on the Yeti mailing lists.

Implementation

The public resolver itself is implemented with Unbound. Here is its configuration:

server:
  use-syslog: yes
  root-hints: "yeti-hints"
  auto-trust-anchor-file: autokey/yeti-key.key
  interface: 2001:4b98:dc2:43:216:3eff:fea9:41a@853go
  qname-minimisation: yes
  harden-below-nxdomain: yes
  harden-referral-path: yes
  harden-glue: yes
  ssl-service-key: "/etc/unbound/tls-server.key"
  ssl-service-pem: "/etc/unbound/tls-server.pem"
  ssl-port: 853
  access-control: ::0/0 allow
  log-queries: yes

As you see, the requests (query name and source IP address) are logged locally (see above the warning about privacy) but not transmitted. The query name is sent to the Yeti coordinators.

(You can also see that, today, QNAME minimisation (RFC 7816 is activated on this resolver, thus limiting the amount of data sent to the Yeti root name servers.)

The Yeti project is a live testbed. This page serves as an all-in-one page to collect all information of Yeti work including Yeti documents, various results that we have gotten from experiments, software tools, and observations during the operation of the Yeti testbed.

1. Project & Testbed Description

• Problem statement of Yeti DNS Project
  The problem statement describes why the Yeti project was started and the questions that it aims to answer.

• ID: draft-song-yeti-testbed-experience
  The Yeti Testbed Experience document briefly covers why the project was started, and also describes the setup, and some of the experiments. It has been submitted to the IETF for publication as an RFC.

• How to become a Yeti operator
  This document explains how new Yeti root servers are added.

• How Yeti DMs work: Yeti-DM-Setup

• How Yeti DMs work: Yeti-DM-Sync
  These documents cover the setup of the Yeti Distribution Masters (DM), which are the parts of Yeti which convert the IANA root zone to the Yeti root zone and then distribute that to the Yeti root servers.

• Data sharing document
  This describes the rules for protecting privacy in the Yeti project.

• Yeti Health monitoring
  The monitoring framework of Yeti is described here.

• DSC/Data server backup plan
  We have redundancy of the server that collects Yeti data, as described in this document.

2. Yeti operational findings

During the setup and operation of the Yeti project, a number of issues were discovered and documented that are unrelated to specific experiments, but were nevertheless interesting.

It is noteworthy that all of the name servers used in the project had fixes or changes based on the work. That includes BIND 9, NSD, Knot, PowerDNS, and Microsoft DNS.

IPv6 Fragmentation Issues

IPv6 fragmentation is a concern due to DNS large responses. The Yeti experiment produces responses larger than 1600 bytes, so is affected.

Via some tests and discussion inside Yeti mailing list, we have identifed two issues regarding IPv6 fragmentation, not only for the DNS root but for DNS in general.

• One issue is that the stateless model of UDP-based applications like DNS makes it difficult to use ICMP/ICMPv6 signaling. More information: APNIC article on IPv6 fragmentation.

• Another issue regarding IPv6 fragmentation is related to the coordination between TCP MSS and IPV6_USE_MIN_MTU option. One TCP segment is fragmented into two IP packets and one of them may be dropped in the middle. Please see TCP and MTU in IPv6 presented by Akira Kato at the 2016 Yeti workshop.

Root Server Glue and BIND 9

It was discovered that BIND 9 does not return glue for the root zone, unless also configured as authoritative for the zone servers themselves. This is true for the IANA root, since the root servers are authoritative for the root and the ROOT-SERVERS.NET domain. For Yeti, this is not true since there is no Yeti zone: all of the Yeti name servers are managed from an independent delegation.

This issue was discussed on the Yeti discuss list:

http://lists.yeti-dns.org/pipermail/discuss/2015-May/000013.html

The BIND 9 team rejected the idea of changing the BIND 9 behavior:

https://lists.isc.org/pipermail/bind-workers/2015-May/003317.html

However the Yeti project produced a patch for BIND 9 which some operators now use:

http://lists.yeti-dns.org/pipermail/discuss/2015-June/000089.html

The discussion on root naming issue are introduce in the Yet experience I-D:

https://tools.ietf.org/html/draft-song-yeti-testbed-experience-03#section-4.1

dnscap Losing Packets

One of the Yeti participants noticed that dnscap, a tool written to capture DNS packets on the wire, was dropping packets:

http://lists.yeti-dns.org/pipermail/discuss/2015-June/000046.html

Workarounds were found for Yeti, although the dnscap developers continued to research and eventually discovered a fix:

https://lists.dns-oarc.net/pipermail/dnscap-users/2016-October/000014.html

- Use helper library `pcap-thread` when capturing to solve
  missing packets during very low traffic"

libpcap Corrupting pcap output

One of our systems had a full disk and ended up with corrupted pcap files from dnscap. We tracked that down to an issue in libpcap, where the underlying writes were not being properly checked. A fix was made and submitted to the upstream library. While not a perfect solution, it is the best that can be done with the underlying I/O API as well as the published API within libpcap:

https://github.com/the-tcpdump-group/libpcap/pull/494

RFC 5011 Hold-Down Timer in BIND 9 and Unbound

The first KSK roll on the Yeti project was not carefully planned, but rather handled as the KSK roll for any zone. Because of this we encountered problems with DNS resolvers configured to use RFC 5011. RFC 5011 automatically updates the trust anchors for a zone, but requires that the new KSK be in place for 30 days. What ended up happening is that BIND 9 continued to function, because it does not use the 30 day hold-down timer, but that Unbound stopped working, because it does (as per the recommendation in the RFC).

http://lists.yeti-dns.org/pipermail/discuss/2015-July/000127.html

Corner Case: Knot Name Compression & Go dns Library

A Yeti participant discovered problems in both a popular Go DNS library and a popular authoritative DNS server, Knot when using Knot to server the Yeti root zone and querying the server with a Go program:

https://github.com/miekg/dns/issues/234

The problem was fixed by the developers of these open source projects. There were no reports of this affecting end-users.

NSD Glue Truncation

NSD by default is configured to send minimal responses, which needs to be re-compiled in order to send a complete glue:

http://lists.yeti-dns.org/pipermail/discuss/2016-May/000522.html

3. Yeti experiment and findings

Yeti root testbed is design for experiments and findings are expected. We summarized all experiment and findings in section 4 of draft-song-yeti-testbed-experience. Some important experiments related with Root Key management are introduced here with detailed report:

Multiple ZSK Experiment

The Multi-ZSK (MZSK) experiment was designed to test operating the Yeti root using more than a single ZSK. The goal was to have each distribution master (DM) have a separate ZSK, signed by a single KSK. This allows each DM to operate independently, each maintaining their own key secret material.

A description of the MZSK experiment and the results can be found in Yeti Project Report: Multi-ZSK Experiment (MZSK).

Of particular interest, the experiment triggered a problem with IXFR for some software, the results of which are documented in An IXFR Fallback to AXFR Case.

Big ZSK Experiment

The Big ZSK experiment was designed to test operating the Yeti root with a 2048-bit ZSK. This was based on Verisign’s announcement that they were going to change the ZSK size of the IANA root to 2014-bits.

A description of the BGZSK experiment and the results can be found in Yeti DNS Project Github repository.

KSK Roll Experiment

Since ICANN is going to start KSK rollover on September 19, 2017, the Yeti KSK roll experiment was designed to perform a KSK roll for the Yeti root and observe the effects. One major goal is to deliver some useful feedback before the IANA KSK roll. A significant result was that DNSSEC failures are reported if a new view is added to a BIND server after the KSK roll has started.

For more information:

1. The Yeti KSK rollover experiment plan is documented in:
   https://github.com/BII-Lab/Yeti-Project/blob/master/doc/Experiment-KROLL.md

2. A detailed report is published in:
   https://github.com/BII-Lab/Yeti-Project/blob/master/doc/Report-KROLL.md

4. Yeti Data Analysis

Now we only have some preliminary analysis on Yeti traffic collected from Yeti server which brief introduced on the presentation Davey Song gave in 2016 Seoul workshop. It is expected that more information will be dug up in 2017 from the traffic, awaiting for more resources spent on this work.

5. Software Tools

BII has written a number of programs during the course of running the Yeti project.

PcapParser: Easy Handling of Large DNS Captured Packets

In Yeti, as with many DNS operations, we use pcap format to store captured DNS traffic. Because Yeti is concerned with large packets, we often have DNS messages that are fragmented or sent via TCP messages. These are hard to analyze, so we wrote a tool to convert these into defragmented, UDP pcap files.

http://dnsv6lab.net/2016/09/06/DNS-pcap-fragments/

As part of this work we are pushing the IPv6 IP defragmention code back into the underlying gopacket library.

ymmv: Easy and Safe Comparison of IANA and Yeti Traffic

Yeti needs as much traffic as possible, ideally with real user query data. Because it is often unacceptable to use Yeti in production environments, the ymmv program was created, which will send the same queries to Yeti servers as a resolver sends to the IANA servers, and also compare the replies and note any differences.

http://dnsv6lab.net/2016/10/13/ymmv/

6. Other Resources

• 2016 Seoul Yeti DNS Workshop materials(slides, pic, audio)

• Yeti experiment and findings at 2016 DNS-OARC Workshop

• The meeting note of 2015 Yeti DNS workshop in Yokohama

• “One name space, Many circles” by Paul Vixie, 2015

• “Potential Root services futures” by David Conrad, 2015

• A summary for the Workshop on DNS Future Root Service Architecture, 2014

  • Video: Day 1
  • Video: Day 2

1. Introduction

Yeti DNS Project ^[1] is a live DNS root server system testbed which is designed and operated for experiments and testing. Many people are curious how it works under an environment which is pure IPv6, with more than 13 NS servers, with multiple signers, and testing ZSK/KSK rolling. One key issue here is large DNS responses, because many Yeti experiments lead to large DNS responses - even more than 1600 octets.

Why large DNS responses are relevant and how DNS handles large responses is introduced comprehensively by Geoff’s articles published in the APNIC blog Evaluating IPv4 and IPv6 packet fragmentation ^[2] and Fragmenting IPv6 ^[3] (thanks Geoff!). More recently, Geoff published a pair of articles examining the root servers system behavior, Scoring the Root Server System ^[4] and Scoring the DNS Root Server System, Pt 2 - A Sixth Star? ^[5] . In these articles, a scoring model is proposed and applied it to evaluate current 13 DNS root servers how they handle large response.

Triggered by Geoff’s work [4] and [5], we formed the idea of scoring the Yeti DNS root server system using the same testing and metrics. The results are summarized in this document.

2. Repeat the Tests on the IANA DNS Root Servers

We first repeat the testing on the IANA DNS root servers. It is expected to confirm the test results, but may show some new findings from different vantage points and metrics. Using the same approach we use the dig command to send queries against each root servers with a long query name for a non-existent domain name like this to get a response of 1268 octets:

dig aaaaaaaa.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa +dnssec @198.41.0.4    +bufsize=4096

And what we see from each root server is shown in Table 1.

Table 1 – IANA Root Server Response Profile with large DNS responses

The structure of Table 1 is a slightly different from the table in APNIC’s blog article. There are additional columns to show the exact size of DNS reponse messages from each root server. We list them because we found them are different by 3 octets. A,E,H,J,K,L,M have responses of 1268 octets and others 1265 octets. After some digging, we found that different root servers behave differently due to the sequence of NSEC RRs in the response.

In our case the sequence “aaa.” as Next Domain Name in NSEC RR sometimes appears before it is used as a label, and then other “aaa” labels following can be compressed using DNS compression, and we see 1265 octets. Sometimes it does not, and we see 1268 octets.(Compression on Next Domain Name of NSEC RR is not allowed) There is no protocol requirement for ordering the RR for an optimal response. So we regard it as a trivial differences. But the compression issue on #23 in table 2 is nontrivial which will be introduced in next section.

There is another difference when we display the TCP MSS information in the table. In APNIC’s article it said that in IPv4 all the root servers offer a TCP MSS of 1460 octets and the value is 1440 octets in IPv6. It may be true if there is no middle-box or firewall which changes the TCP MSS intentionally. In our observation, the TCP SYN initiated by testing resolver carries the TCP MSS of 1460 octets in IPv4 and 1440 octets in IPv6, but the TCP SYN/ACK responded by root servers are all 1380 octets. As far as we know, TCP MSS of 1380 octets is a special value used by some firewalls on the path for security reasons (CISCO ASA firewall).

Let’s look at fragments!

As with APNIC’s finding, we found that F and M fragment IPv6. It is worthwhile to mention that in our test F only fragments UDP IPv6 packets and M only fragments TCP segment. F fragmenting only UDP can be explained that F’s DNS software sets IPV6_USE_MIN_MTU option to 1 and the TCP implementation respects the IPV6_USE_MIN_MTU, so does not send TCP segments larger than 1280 octets.

TCP MSS setting,in APNIC’s blog article it is suggested that TCP MSS should be set 1220 octets. It is observed that H already accepted Geoff’s suggestion as the time of writing. TCP MSS setting in root server is relevant because there are mainly two risks if TCP MSS is not set properly. One introduced in APNIC’s blog article is Path MTU (PMTU) Black Hole in which large TCP segment may be dropped in the middle but ICMP6 PTB message is lost or filtered. Another risk is not covered by that article but is also relevant. That is the behavior where the IP packets TCP sent for segments are fragmented by root servers if TCP does not respect the IPV6_USE_MIN_MTU socket option and IPV6_USE_MIN_MTU=1 in that server. Please see TCP and MTU in IPv6 ^[6] , presented by Akira Kato at the 2016 Yeti workshop.

M is fragmenting the TCP segment in this way! It is odd that large UDP responses of M are not fragmented, and unknown why M fragments only TCP segments.

As for the truncation behavior of root servers, our test shows the same pattern that in IPv4 B and G truncate the response, in IPv6 A and J join B and G to truncate the response. By definition truncation happens when DNS message length is greater than that permitted on the transmission channel. The truncation behavior of A, B, G and J looks odd because they truncate the response even when both query and response specifies a large EDNS0 buffer size. There must be a separate method to determine whether to truncate or not. Obviously for these root operators they prefer to truncate the response and fall back to TCP rather than fragmenting a larger response. This preference is actually stronger in IPv6 and IPv4, as A and J send truncated packets of 900 octets, possibly because the potential problem of fragments missing or filtering in IPv6 is worse than in IPv4 (as explained in detailed in APNIC blog article).

We observe that there is a Server Failure response (SERVFAIL) from E in IPv6 which happens only in China. We also observed C in IPv6 and M in IPv4 are unreachable as the time of testing. By trace-route, the problem is spotted by upstream network provides. It reminds us that DNS system - even the root server system - is vulnerable to network attacks or routing problems.

3. Tests on the Yeti DNS Root Servers

The next step is to dig the same query against Yeti DNS root servers. And what we see from each Yeti root server is shown in Table 2.

Table 2 – Yeti Root Server Response Profile to a large DNS response

We see no truncation for any Yeti root server. That’s means none of the Yeti servers have truncation logic apart from the server default truncation, based on the EDNS buffer sizes.

We notice that #2 and #14 accept Geoff’s suggestion to change TCP MSS to 1220 and reduce the risk for TCP segment fragmentation (although perhaps they were configured this way before this recommendation).

For #23 (running Microsoft DNS), the response is bigger than others because it does not apply name compression for the mname and rname fields in the SOA record - leading to an increase of 12 octets, and it also apply name compression on the root label itself, resulting in a bigger packet. Name compression is not a huge optimization but in certain cases octets saved by name compression could avoid truncation or fragmentation.

Note that currently Yeti implements MZSK ^[7] which produces large DNS responses due to multiple ZSKs. By querying for DNSKEY records with their DNSSEC signature, all Yeti servers response with a DNS message size up to 1689 octets and fragment the UDP response. When the +tcp option is added to dig - performing the DNS query via TCP - the result in the “Fragment” column is the same as that in Table 2 (#1, #3, #4, #7 fragment TCP segments). So in Yeti’s case there is a trade-off between whether to truncate the large responses or to fragment them. There is no way to avoid the cost brought by the large response (1500+ octets) with the existing DNS protocol and implementations. However, some proposals are made to address the problem by DNS message fragments ^[8] or always transmitting the large DNS response with connection-oriented protocols like TCP ^[9] or HTTP ^[10] .

4. Metrics for Scoring

Like Geoff, we can use a similar five star rating system for Yeti root operators. Of course, we won’t use the IPv4 ratings, since Yeti is IPv6-only. We can “borrow” 3 of Geoff’s stars:

• If the IPv6 UDP packet is sent without fragmentation for packets up to 1,500 octets in size, then let’s give the server a star.
• If the IPv6 UDP packet is sent without truncation for IPv6 packet sizes up to 1,500 octets, then let’s give the server a star.
• If the offered IPv6 TCP MSS value is no larger than 1,220 octets, then let’s give the server another star.

We can suggest two more stars:

• If the IPv6 TCP packets are sent without IP fragmentation, we will give the server a star.
• If the server compresses all labels in the packet, we give the server a star. (Not technically the “penalize Microsoft” star, but in practice it is.)

5. Scoring Yeti Root Servers

Using this system we rate the Yeti servers.

Table 3 – Starry View of Yeti Servers

If we can make setting the TCP MSS at 1220 a common practice then we should have a lot of “5 star” Yeti servers.

6. Conclusion

We found it interesting to replicate APNIC’s results, and were happy to be able to use similar ratings across the Yeti servers.

Regarding the DNS large response issue in general, operator can adopt Geoff’s suggestions on the IPV6_USE_MIN_MTU option, TCP MSS settings, and DNS message compression. However, good solutions are still unknown for issues like IPv6 DNS fragmentation in UDP for very large response. It is still unknown whether is it better to truncate or fragment the large response… more attention is needed in the network community because we are going to build the whole Internet on IPv6 infrastructure.

As always, please let us know what you think and what you’d like for us to look at in either the IANA or Yeti root servers.

7. Reference

[1] Yeti DNS Project, www.yeti-dns.org

[2] Geoff Huston, Evaluating IPv4 and IPv6 packet fragmentation, APNIC blog, January 2016.

[3] Geoff Huston, Fragmenting IPv6， APNIC blog, May 2016.

[4] Geoff Huston, Scoring the Root Server System,APNIC blog, November 2016.

[5] Geoff Huston, Scoring the DNS Root Server System, Pt 2 - A Sixth Star?, APNIC blog, December 2016.

[6] Akira Kato, TCP and MTU in IPv6 , Yeti DNS Workshop, November 2016

[7] Shane Kerr and Linjian Song, Multi-ZSK Experiment, Yeti DNS Project

[8] Mukund Sivaraman, Shane Kerr and Linjian Song, DNS message fragments, IETF Draft, July 20, 2015

[9] Linjian Song,Di Ma Using TCP by Default in Root Priming Exchange,IETF Draft, November 26, 2014

[10] Linjian Song, Paul Vixie, Shane Kerr and Runxia Wan DNS wire-format over HTTP, IETF draft,September 15, 2016

Workshop Agenda

1) Notes on software construction and reliability for privately signed root zones（Paul Vixie） slides

2) Invite talk: IDN introduction and current status（Marc Blanchet）slides

3) Yeti DNS Project status and development (Davey Song)slides

4) Yeti experiment and findings (Shane Kerr) slides

5) IPv6 issues in Yeti (Akira Kato) slides

6) Yeti tools: YmmV and PcapParser (Shane Kerr) slides-YmmV，slides-PcapParser

7) Invite talk: Entrada Introduction (Moritz Mueller) slides

8) Open discussion: Improving Root Name space(Paul Vixie) slides

Youtube links: https://youtu.be/SJG2dKm-aoA

Background

KSK rollover is one of Yeti experiments on Yeti DNS Root testbed. The purpose is to test the KSK rollover in large scale and to find any potential risks before the real KSK rollover to be performed in production network in future. The proposal of Yeti first KSK rollover experiment is documented in one GitHub page [1]. The first rollover happened at the time stamp 20160830073746 (Tue Aug 30 07:37:46 2016 UTC). In the same while the old KSK was revoked, and removed after 30 days.

DNSSEC Bug report

There is an issue spotted on one Yeti resolver (using BIND 9.10.4-p2) during Yeti KSK rollover.The resolver is configured with multiple views before the KSK rollover. DNSSEC failures are reported once we added new view for new users after rolling the key. (In Yeti KSK rollover plan, once new key become active the old key is revoked).

When we view secondfloor which is created before the KSK rollover.

  $echo -n secondfloor|sha256sum 
  b1629b86416e2208ce2492ea462475f77141a2c785e53d8cd64dbf9dabe9f01f - 
  $ cat b1629b86416e2208ce2492ea462475f77141a2c785e53d8cd64dbf9dabe9f01f.mkeys 
  $ORIGIN . 
  $TTL 0 ; 0 seconds 
  @ IN SOA . . ( 
  6490 ; serial 
  0 ; refresh (0 seconds) 
  0 ; retry (0 seconds) 
  0 ; expire (0 seconds) 
  0 ; minimum (0 seconds) 
  ) 
  KEYDATA 20160926180107 20150929074902 20160930050105 385 3 8 ( 
  AwEAAaP3gGQ4db0tAiDEky0dcUNGeI1aTDYP5NFxzhbd 
  pD60ZhKLVV4KyxPmoSNUpq5Fv5M0iBwK1Tyswsyq/9sM 
  SoZ8zx8aT3ho1YnPsSqQeJfjTT1WsX6YZ5Kw6B2QkjRN 
  a6OMGZ96Kn8AI/slqsw+z8hY49Sn3baeo9iJxHPzloNc 
  2dQkW4aLqzNEYxnuoJsthCfGrPSAXlUjY9m3YKIaEWR5 
  WFYQk770fT+gGWLk/54Vp0sG+Lw75JZnwhDhixPFaToT 
  DNqbHQmkEylq1XJLO15uZ/+RZNRfTXZKO4fVR0tMEbMA 
  ITqRmyP8xLXY4RXbS4J32gnenQbzABX8sQmwO7s= 
  ) ; revoked KSK; alg = RSASHA256; key id = 56082 
  ; next refresh: Mon, 26 Sep 2016 18:01:07 GMT 
  ; trusted since: Tue, 29 Sep 2015 07:49:02 GMT 
  ; removal pending: Fri, 30 Sep 2016 05:01:05 GMT 
  KEYDATA 20160926180107 20160809184103 19700101000000 257 3 8 ( 
  AwEAAbA0lBT1aDxwoNl7d/fXqFFBtL+VwBLqgOYHgAqr 
  nvhRvHs+GrTWZZ5gZu/0NeX4YGXmovT1nGpY/9oi30pD 
  vbzPluQXOKSVP/xr1KyLPp8pxiVqGe973F55fX4iQOUM 
  B2n2VXfIxSryTNYPz44Zltpa10WAVYzHpy3oxx0qZSeD 
  sdPHMNB7Ym0hBMY92cifWyQWifHbcgbFGf2mpwF00vAL 
  l92qhnvIORVZC/ihNNd7DvQtMLdUvSoQ0woC/EhqexXQ 
  v0bLlPkG55d37JoaVbWCEnWLZ+CT+Eei5U4VCqH+xCEv 
  OjT45ZQt0kfB3K4bwfh6D5EBleJ13z3pbUwBy0U= 
  ) ; KSK; alg = RSASHA256; key id = 19444 
  ; next refresh: Mon, 26 Sep 2016 18:01:07 GMT 
  ; trusted since: Tue, 09 Aug 2016 18:41:03 GMT 

When we view 6plat which is created after the KSK rollover.

  $ echo -n 6plat|sha256sum 
  1369c5fe9c97ce67fa0a4b3f25e6ceb86105045569eac55db54a6e85353d030b - 
  $ cat 1369c5fe9c97ce67fa0a4b3f25e6ceb86105045569eac55db54a6e85353d030b.mkeys 
  $ORIGIN . 
  $TTL 0 ; 0 seconds 
  @ IN SOA . . ( 
  4 ; serial 
  0 ; refresh (0 seconds) 
  0 ; retry (0 seconds) 
  0 ; expire (0 seconds) 
  0 ; minimum (0 seconds) 
  ) 
  KEYDATA 20160924170104 19700101000000 19700101000000 257 3 8 ( 
  AwEAAaP3gGQ4db0tAiDEky0dcUNGeI1aTDYP5NFxzhbd 
  pD60ZhKLVV4KyxPmoSNUpq5Fv5M0iBwK1Tyswsyq/9sM 
  SoZ8zx8aT3ho1YnPsSqQeJfjTT1WsX6YZ5Kw6B2QkjRN 
  a6OMGZ96Kn8AI/slqsw+z8hY49Sn3baeo9iJxHPzloNc 
  2dQkW4aLqzNEYxnuoJsthCfGrPSAXlUjY9m3YKIaEWR5 
  WFYQk770fT+gGWLk/54Vp0sG+Lw75JZnwhDhixPFaToT 
  DNqbHQmkEylq1XJLO15uZ/+RZNRfTXZKO4fVR0tMEbMA 
  ITqRmyP8xLXY4RXbS4J32gnenQbzABX8sQmwO7s= 
  ) ; KSK; alg = RSASHA256; key id = 55954 
  ; next refresh: Sat, 24 Sep 2016 17:01:04 GMT 
  ; no trust 

It is found that mkeys file of the new view (6plat) only contains the old key which is inherited from the managed-keys (tag 56082) in global setting (named.conf). But it is not inherited the new key (tag 19444 )from the managed keys database which is valid for KSK operation in RFC5011 context. That is why DNSSEC is failed in new view because old key was inactive and revoked.

However, by checking the manual of BIND9.10.4-P2, it is said that unlike trusted-keys, managed-keys may only be set at the top level of named.conf, not within a view. It gives an assumption that for each view, managed-key can not be set per view in BIND. But right after setting the managed-keys of 6plat, the DNSSEC validation works for this view.

Some thoughts

There are two preliminary suggestions for BIND users and developer.

1) Currently BIND multiple-view operation needs extra guidance for RFC5011. Right now ICANN decided to roll the KSK in 2017-10-112. The manage-keys should be set carefully during the KSK rollover for each view when the it is created.

2) It is recommended that in BIND implementation the new views should be operationally set to inherit the managed-keys from one managed keys database of a existing view. It is more natural way for Multiple-view model to adapted to RFC5011.

Reference

[1] Yeti KSK Rollover Plan, https://github.com/BII-Lab/Yeti-Project/blob/master/doc/Experiment-KROLL.md

[2] 2017 KSK Rollover Operational Implementation Plan, https://www.icann.org/en/system/files/files/ksk-rollover-operational-implementation-plan-22jul16-en.pdf

The Yeti DNS workshop is an annual meeting with reports and discussions of root DNS services, DNS tools based on experiences from the testbed and experiments in the Yeti DNS project. The first Yeti DNS workshop was held before Yokohama IETF meeting in 2015. The second Yeti workshop will be held before Seoul IETF meeting in this November. Now the venue and time is secured and we would like call for participation for people who are interested in Yeti and related topic. The detailed information of this workshop is as follows:

Date: Saturday, November 12, 2016

Time: 13:00-18:00 UTC+8 (04:00-09:00 UTC)

Location: Conrad Seoul, Studio 1-6F (IETF 97 venue)

Host: Beijing Internet Institute

Sponsor: Yeti Coordinators

Remote access https://www.gotomeet.me/yetidnsworkshop (need to Download a software for the first time)

Draft agenda

-Welcome address
-Notes on software construction and reliability for privately signed root zones（Paul Vixie）
-Invite talk: IDN introduction and current status（Marc Blanchet）
-Yeti DNS Project status and development (Davey Song)


Coffee break

-IPv6 issues in Yeti(Akira Kato)
-Yeti experiment, findings and software development (Shane Kerr)
-Invite talk: Entrada Introduction (Moritz Mueller)
-TBD: a Vendor's presentation on Yeti participation
-Open discussion

Yeti Root DNS testbed has been running for more than one years with joint effort of Yeti community. Now the system is composed of 25 NS servers for root zone and attracted IPv6 traffic from interested parties and individuals, like Universities and labs. Some alternative operation models have been experimented, like Multiple-DM, Multiple-ZKS, Big ZSK and KSK rollover, etc.

We also call for presentations from Yeti operators and researchers. Topics outside of the Yeti DNS project itself are also appropriate, if they are related to DNS in general.

Note: Please send a mail to Yeti coordinators if you are interested to attend the workshop. It will help better preparation of the workshop. Thanks!

The Floor plan for Yeti DNS Workshop

Introduction

The Yeti project would like DNS resolver operators to send us their query traffic. Because Yeti is an experimental network, DNS operators may not want to put their users on resolvers using Yeti. By using dnsdist, resolver operators can continue to use the normal IANA root servers and also send queries to the Yeti root servers.

To do this an administrator needs to set up a resolver that uses the Yeti servers which runs at the same time as the production IANA resolver. They will then add dnsdist as a DNS load-balancer in front of the resolver using IANA root servers and the resolver using Yeti root servers.

dnsdist is a highly DNS-, DoS- and abuse-aware load balancer. We can use the TeeAction to mirror DNS query to Yeti resolvers.

TeeAction This action sends off a copy of a UDP query to another server, and keeps statistics on the responses received.

dnsdist will only copy UDP query to other server. There is a figure depicting the dnstist mirrored traffic flow below:

|Client|<-->|DNSDIST|<--->Resolvers<--->Other authority servers
            |DNSDIST|<--->Yeti Resolver<--->Yeti root name server 

The dnsdist is installed betweeen client and a normal in-production RDNS server, so that the stub queries heard by that server will be mirrored toward a specified Yeti-capable RDNS server.People who is willing to use dnsdist to mirror traffic to yeti is expected to run another Yeti resolver. (The is also a list of registried Yeti RDNS in the Yeti webpage)

Note that there is no round-robin, load balancing, or RTT-based server selection for the dnsdist ‘tee’ action, and so, all mirrored queries will be sent to the Yeti Resolver server, whose IPv6 address must be hard wired in the dnsdist config.

Install

Please refer to dnsdist

Mirror dns query to Yeti resolvers

1) Yeti resolvers add ACL for dnsdist server.

2) dnsdist add rules:

addAction(AllRule(), TeeAction("240c:f:1:22::103")) -- default port 53
topRule()

Test

1) running in the foreground.

dnsdist -C /path/to/dnsdist.conf

2) daemon

dnsdist -C /path/to/dnsdist.conf --daemon
dnsdist -C /path/to/dnstist.conf --client # connect to control

Reference

1. dnsdist-github
2. dnsdist-README

It has been 6 months since our last virtual meeting, and more than 4 months since our Yeti DNS workshop. We thought it would be nice to have another virtual meeting to catch up on the status.

Date:  2016-03-24
Time:  07:00 UTC

As with the previous virtual meeting, this is mostly for Yeti participants or people who expect to participate. All interested parties are of course welcome.

We expect this to be around 1~2 hours long.

draft agenda & slides:

• Introduction of Status of Yeti system
• Report on Multiple ZSK (MZSK) experiment
• Next experiment: KSK rollover
• Next experiment: Root naming scheme
• open discussion

We setup a Jitsi link for our virtual meeting this time. You can check availability of this link any time before the meeting (chrome/firefox ).

https://meet.jit.si/GoodAnimalsObserveConstantly

Yeti monitoring with RIPE Atlas

1. Purpose of Yeti Monitorint Sytsem

Currently Yeti testbed lacks sufficient information to reflect the status of Y
-eti operation, in aspect of availability, consistency and performance. It
also lacks tools to measure the result of experiments of like introducing more
root server, KSK rollover, Multi-ZKSs etc. Thus, there is a need of monitoring
the functionality, availability and consistence of Yeti Distribution Master as
well as Yeti root server.

The basic idea is setting regualarly monitoring task in Atlas to query SOA
record of fifteen root servers through both UDP and TCP to check the consi
-stence of SOA record. Use Nagios plugin to periodically get the result of
Atlas monitoring task and parse it to trigger alert. Alert email is sent when
there is an exception.

2. check_yeti, the Nagios plugin

2.1 the design of plugin

chekc_yeti gets test results by Atlas API and analysis it.
then output status results and display it by Nagios Web interface.

get the result from Atlas, sample code:

stop_time=str(int(time.time()))
start_time=str(int(int(stop_time) - 3600))
base_url="https://atlas.ripe.net/api/v2/measurements/"
url=base_url + targetid + "/results" + "?start=" + start_time + "&" + \
                                       "stop=" + stop_time + "&format=json"
urllib.urlretrieve(url, outfile)

2.2 Checking Algorithm

• Atlas：
  a.set regularly monitoring task for 15 root server.
  b.each time use 10 probes.
• Nagios：
  a.get result of each root server in one hour
  b.analyse result
  c.return statuse code

  OK: over six probe returns OK result WARNING: OK number blows 4
  CRITICAL: None of proble returns OK
  UNKNOWN: no return data.

PS: nagios status code

STATE_OK=0  
STATE_WARNING=1  
STATE_CRITICAL=2  
STATE_UNKNOWN=3  

3. Deployments

3.1 check_yeti

Give the permission to execute it and put it into Nagios’s plugin directory.

3.2 hosts.cfg

Define monitoring Server

define host{  
    use                     linux-server-yeti-rootserver  
    host_name               bii.dns-lab.net  
    alias                   bii.dns-lab.net  
    address                 240c:f:1:22::6  
}    

Different servers should be defined seperately

3.3 commands.cfg

Define check_yeti checking commands

define command{
        command_name    check_yeti
        command_line    $USER1$/check_yeti $ARG1$
}

$USER1$ : nagios’s plugin directory $ARG1$： plugin input parameter, ID of monitoring task

3.4 service.cfg

Define check_yeti monitoring service

define service{  
   	     use                             generic-service          
   	     host_name                       bii.dns-lab.net  
   	     service_description             check_yeti  
   	     check_command                   check_yeti!1369633 
   	     }

generic-service: Define nagios monitoring templates such as checking interval(2 hours), alarming interval, alarming level
1369633 : ID of monitoring tast
Different servers should be defined seperately

3.5 contacts.cfg

Define alarming contactors

 define contact{
       contact_name      yeti          
       use               generic-contact               
       email             xxx@biigroup.cn
  }

contact_name : contactor name, will use directly in templeate
email: contactor, seperate by commas

3.6. Start nagios

servic nagios restart

4. Display Atlas monitoring status on website

1. seting dnsdomainmon tasks in Atlas, get zone ID
2. using Atlas API, refering https://atlas.ripe.net/dnsmon/
3. key parameter: zone: “3069263”
4. sample code

	 
	    <!DOCTYPE html>
	    <html>
	    <head>
	    <title>domainmon test</title>
	    </head>
	
	    <body>
	    <script type="text/javascript" src="https://www.ripe.net/++resource++ripe.plonetheme.javascripts/jquery/1.11.2.js"></script>
	
	    <script type="text/javascript" src="https://www.ripe.net/++resource++ripe.plonetheme.javascripts/bootstrap.min.js">
	    <script type="text/javascript" src="https://www.ripe.net/++resource++ripe.plonetheme.javascripts/template.js"></script>
	    <script type="text/javascript" src="https://www.ripe.net/++resource++ripe.plonetheme.javascripts/browser-update.js"></script>
	    <script type="text/javascript" src="https://www.ripe.net/modernizr.js"></script>
	
	    <script type="text/javascript" src="https://www-static.ripe.net/static/rnd-ui/atlas/static/ui/js/moment.min.js"></script>
	    <script type="text/javascript" src="https://www-static.ripe.net/static/rnd-ui/atlas/static/core/contrib/tablesorter/jquery.tablesorter.min.js"></script>
	    <script type="text/javascript" src="https://www-static.ripe.net/static/rnd-ui/atlas/static/core/js/jquery.form.min.js"></script>
	    <script type="text/javascript" src="/variables.js?v=Archangel"></script>
	
	    <script src="/easteregg.js?v=Archangel"></script>
	    <script>
	    DNSMON_PROBES_DATA_API_URL = 'https://atlas.ripe.net/dnsmon/api/probes';
	    DNSMON_SERVERS_DATA_API_URL = 'https://atlas.ripe.net/dnsmon/api/servers';
	    DNSMON_ATLAS_DATA_API_URL = 'https://atlas.ripe.net/dnsmon/api/atlas-data';
	    DNSMON_ATLAS_TRACEROUTE_API_URL = 'https://atlas.ripe.net/dnsmon/api/atlas-data';
	    DNSMON_ATLAS_NSID_API_URL = 'https://atlas.ripe.net/dnsmon/api/atlas-data';
	    </script>
	
	    <script type="text/javascript" src="https://atlas.ripe.net/dnsmon/dnsmon-widget-main.js" ></script>
	    <div id="domainmon"></div>
	
	    <script>
	            var dnsmon;
	            $(function() {
	                var hasUdp = true;
	                var hasTcp = true;
	                function onDraw(params) {
	                    var tab;
	                    if (params.isTcp) {
	                        tab = $(".protocol-tabs a[data-protocol=tcp");
	                    } else {
	                        tab = $(".protocol-tabs a[data-protocol=udp");
	                    }
	                    $(".protocol-tabs").show();
	                    tab.tab("show");
	                };
	
	                dnsmon = initDNSmon(
	                        '#domainmon',
	                        {
	                            dev: false,
	                            lang: "en",
	                            load: onDraw,
	                            change: onDraw
	                        }, {
	                            type: "zone-servers",
	                            zone: "3069263",
	                            isTcp: hasUdp ? false : true
	                    });
	            });
	    </script>
	    
	    </body>
	    </html>

This is a short summary for the people who are interested but not able to attend the workshop. The audio record on youtube

(1) The first speech was given by Paul Vixie(from 0:15:00 of the youtube record). The slides link

He is one of the founder of Yeti. He introduced his 10+ Years of Responsible Alternate Rootism, as a track of the time that lead to Yeti. He admitted he has been in the center of constrovercy of DNS and root system for years.

Paul Vixie’s presentation talked about the evolution of his involvement with the root system, starting from the concept& his experience of root system, anycasting, his idea of alternative rootism, ultimately leading up to the Yeti project. He insisted many times that to keep one namespace is not necessary to keep one set of root servers where the root zone is served. He proposed IANA to sign another zone parallelely to helped get IDN, DNSSEC and IPv6 out there faster. But it is viewed controversial and failed.

As note writer observed, it is true that due to DNSSEC encryption technology, the signed data can be authenticated by the data itself not necessary to the authority who publish it. So in Yeti project today, Paul and people with same interest turn to use its own key to sign the zone for a testbed instead asking IANA to sign the alternative zone.

(2)The second speaker is Keith Mitchell who is the president of DNS OARC. He was invited to speak about AS112 Operations. (from 00:48:00) The slides link

There was some discussion about researching using DNAME in the same way that AS112 uses DNAME, by adding a DNAME delegation representing a name from the special-use registry. While an interesting idea, there seemed to be consensus that this was slightly risky because it would actually be changing the contents of the root zone.

(3) Davey Song is one of the founders of the Yeti project, and he presented the current status of the Yeti DNS Project（from 01:09:00) The slides link

There is a informational draft song-yeti-testbed-experience to record the Yeti experience gained from the project which covers the most part of Davey’s presentation.

There was a lot of discussion about the DM model and the details of the process. This is documented and the source code published on the GitHub repository, but people still wanted clarification.

There was discussion about using Yeti in an IPv4 scenario.There was discussion about the duration of the project - 3 years.

The limitations of emergency fixes in the root zone was discussed. This is not much worse in the Yeti system than in the IANA root. If it becomes a problem, we can synchronize much more closely with the IANA root.

(4) Shane Kerr works for BII, one of the Yeti DM. He presented the current experiment situation in the Yeti project and activities in BII lab(from 02:32：00). The slides link

He first introduced in detail how Yeti DM model works and followed with a Multi-ZKS experiment for Yeti DM model. He then briefed us the KSK plan by ICANN and what is expected in Yeti testbed. There are also other related topics about hint management and DNS layer fragment.

One point of concern was with updating hints files, since that could allow an attacker (or operational mistake) to prevent a system from ever using the real root servers. DNSSEC validation makes this less scary. One suspicion was that broken resolvers were the problem, and that no amount of improving working resolvers would help much.

Regarding the DNS layer fragment, there is a joint document by BII and ISC. https://tools.ietf.org/html/draft-muks-dns-message-fragments-00

A proposal to investigate TCP-only root service was made, and a discussion around that.

(5) Stephane Boryzmeyer is a DNS expert, and has been actively involved with the Yeti project since its inception. He presented his observations from monitoring Yeti and introduction of various monitoring tool for Yeti (from 03:13:00). The slides link

Currently, Yeti testbed is operational, but lack of monitoring and fine-grained measurement for the dos quality of Yeti. Apparently, dig from time to time is not enough. From Stephane’s presentation he encouraged us try to use automatic tools for Yeti monitoring even thought it’s not for production usage.

He briefed us the state of art of DNS monitoring tools from his experience, varying from Icinga, Check_dig, Zonemaster, DNSmon, DNSviz and finally the RIPE Atlas. He suggested us using those tools for improving the system monitoring & measurement going forward.For example, providing a status page on the Yeti web page that could be used to update monitoring results.

(6) Dmitrii Kovalenko is a DNS operator at MSK-IX, and was an early Yeti participant. Recently MSK-IX set up a mirroring of resolver traffic, and Dmitrii was asked to present this(from 03：50). The slides link

There was discussion of possible additional work that might be interesting on the resolver query duplication.

Also noted that we should make it clearer how to submit a trouble ticket, as there were some issues during Dmitrii’s work that should have been tracked.

Other topics:

• There was a discussion of investigating ways to shutdown the project. Certainly we need to encourage resolver operators to join the mailing lists.

• We discussed future workshop plans. Currently there are none.

• We discussed the timeline of experiments. Currently there is none, but this will be proposed soon.

• Everyone should configure all of their resolvers to use Yeti! :)

Dear Colleagues,

In October 31st afternoon Yokohama, we held the first Yeti Workshop. There are 25+ participants (including online participants) there and we have a fruitful discussion in Yokohama. Firstly I would like to thank all the workshop participants and speakers who contribute this workshop. Thanks to Akira Kato who helped to find a nice meeting room for us. Special thanks to Paul Vixie, who arrived at Yokohama 2 days ahead to prepare yeti DM setup for the workshop.

Now all slides and record are posted into Yeti website. Please check the links below more information. I would like to quote a saying with consensus here during the workshop：”Yeti project is totoally driven by the community and the peopole”. Please send request, questions, and even push us! It’s a wonderfull experince and hope to meet more Yeti friends again.

Agenda(with slides):

• Welcome address (5 min)
• Keynote speech: From 2005 to 2015: 10+ Years of Responsible Alternate Rootism, Paul Vixie
• Invited talk: AS112 Operations Update , Keith Mitchell
• Yeti project Introduction and status update, Davey Song
• Feedback from Dublin ICANN meeting, Shane Kerr
• Coffee break (20 min)
• Research and Experiments in BII lab, Shane Kerr
• Monitoring and measuring Yeti , Stephane Bortzmeyer
• Yeti experiment in MSK-IX , Dmitry Kovalenko
• Open discussion (20 min)

Gotomeeting Audio record of this workshop

I also would like to share some pictures during the meeting.

Background

This document describes the process used to generate and distribute the Yeti root.

The Yeti DNS Project takes the IANA root zone, and performs minimal changes needed to serve the zone from the Yeti root servers instead of the IANA root servers.

In Yeti, this modified root zone is generated by the Yeti Distribution Masters (DM), which provide it to the Yeti root servers.

While in principle this could be done by a single DM, Yeti uses a set of three DM. These DM coordinate their work so that the resulting Yeti root zone is always consistent. (We need to avoid the case where not all DM share the same set of Yeti root servers, and so they produce different versions of the Yeti root zone with the same serial.)

Generation

The generation process is:

1. Download the latest IANA root zone
2. Make modifications to change from the IANA to Yeti root servers
3. Sign the new Yeti root zone
4. Publish the new Yeti root zone

IANA Root Zone

The root zone is currently downloaded using AXFR from F.ROOT-SERVERS.NET:

   $ dig -t axfr . @f.root-servers.net.

This appears to be the best way to get the updated version of the root zone as soon as possible.

TODO: The DM should check that the zone they are retrieving from IANA is properly signed, with something like ldns-verify-zone.

A new version of the Yeti zone is only generated when the IANA zone changes, which is detected by the change in the serial number value in the SOA record.

KSK secret

All DM share the same KSK secret material. This generated using the BIND 9 dnssec-keygen tool, and then sent via encrypted PGP to the other DM operators.

Modifications

The root zone is modified as follows:

• The SOA is updated:
  • The MNAME and RNAME are set to Yeti values
• The IANA DNSSEC information is removed:
  • The DNSKEY records
  • The RRSIG and NSEC records
• The IANA root server records are removed:
  • The NS records for [A-M].ROOT-SERVERS.NET
• The Yeti DNSSEC information is added:
  • The DNSKEY records
• The Yeti root server records are added:
  • The NS records
  • The AAAA glue records
• The Yeti root zone is signed

It might be worthwhile to use the serial value in the SOA field, however for now we duplicate the IANA serial value.

The list of Yeti name Servers is synchronized between the DM as described below.

Timing

Each Yeti DM checks to see if the IANA root zone has changed hourly, on the following schedule:

A new version of the Yeti root zone is generated if the IANA root zone has changed.

Synchronizing List of Yeti Name Servers

It is important that the root zone produced by the Yeti DM is always consistent. In order to do this, we use something like a 2-phase commit procedure.

A change to the list of Yeti name servers gets put into a PENDING directory on any one of the Yeti DM. This directory contains:

• the new list of Yeti name servers
• the time when the new list will be valid from
• a file for each Yeti DM which has confirmed the new list

Each DM will periodically check this PENDING directory. If the directory is present, then the DM will download the new information, add a file documenting that it has received it.

Sometime after the scheduled time arrives and before the next Yeti root zone is generated, each DM will check if the other DM have both received the new list of Yeti name servers. If they have, then the list of Yeti name servers will be replaced with the new one. If they have NOT, then an alarm is raised, and humans debug the failure.

In pseudocode, something like this:

sync_yeti_roots:
    loop forever:
        try rsync with PENDING directory with each other DM

        if PENDING list of roots != my list of roots:
            add a DM file for me in the PENDING directory

        if current time > PENDING scheduled time:
            if the DM file for each DM is present:
                copy PENDING list of roots to my list of roots
            else:
                PANIC (notify a human being)

        sleep a bit

We choose 48 hours as the current time to adopt a new list of Yeti name servers. This allows plenty of time for for DM administrators to fix issues.

Only a single PENDING change is possible at one time. This is an entire new list of Yeti root servers. Further changes must be held until the current set is applied.

Note that it might be possible to start using the new list of Yeti name servers as soon as all DM have received it. However for predictability and simplicity we will always use the scheduled time for now.

Problem Statement

Some problems and policy concerns over the DNS Root Server system stem from unfortunate centralization from the point of view of DNS content consumers. These include external dependencies and surveillance threats.

• External Dependency. Currently, there are 12 DNS Root Server operators for the 13 Root Server letters, with more than 400 instances deployed globally. Compared to the number of connected devices, AS networks, and recursive DNS servers, the number of root instances is far from sufficient. Connectivity loss between one autonomous network and the IANA root name servers usually results in loss of local service within the local network, even when internal connectivity is perfect. Also this kind of external dependency will introduce extra network traffic cost when BGP routing is inefficient.

• Surveillance risk. Even when one or more root name server anycast instances are deployed locally or in a nearby network, the queries sent to the root servers carry DNS lookup information which enables root operators or other parties to analyize the DNS query traffic. This is a kind of information leakage which is to some extent not acceptable to some policy makers.

There are some technical issues in the areas of IPv6 and DNSSEC, which were introduced to the DNS Root Server system after it was created, and also when renumbering DNS Root Servers.

• Currently DNS mostly relies on IPv4. Some DNS servers which support both A & AAAA (IPv4 & IPv6) records still do not respond to IPv6 queries. IPv6 introduces larger IP packet MTU (1280 bytes) and a different fragmentation model. It is not clear whether it can survive without IPv4 (in an IPv6-only enviroment), or what the impact of IPv6-only environment introduces to current DNS operations (especially in the DNS Root Server system).

• KSK rollover, as a procedure to update the public key in resolvers, has been a significant issue in DNSSEC. Currently, IANA rolls the ZSK every six weeks but the KSK has never been rolled as of writing. Thus, the way of rolling the KSK and the effect of rolling keys (including both ZSK and KSK) frequently are not yet fully examined. It is worthwhile to test KSK rollover using RFC5011 to synchronize the validators in a live DNS root system. In addition, currently for the ZSK 1024-bit RSA keys are used, and for the KSK 2048-bit RSA keys are used. The effect of using key with more bits has never tested. A longer key will enlarge DNS answer packets with DNSSEC, which is not desirable. It is valuable to observe the effect of changing key bit-lengths in a test environment. Different encryption algorithms, such as ECC, are another factor that would also affect packet size.

• Renumbering issue. Currently Internet users or enterprises may change their network providers. As a result their Internet numbers for particular servers or services, like IP address and AS numbers, may change accordingly. This is called renumbering networks and servers. It is likely that root operators may change their IP addresses for root servers as well. Since there is no dynamic update mechanism to inform resolvers and other internet infrastructure relying on root servic of such changes, the renumbering issue of root server is a fragile part of the whole system.

Based on the problem space there is a solution space which needs experiments to test and verify in the scope of the Yeti DNS project. These experiments will provide some information about the above issues.

• IPv6-Only Operation. We are try to run the Yeti testbed in pure IPv6 environment.

• Key/Algorithm rollover. We are going to design a plan on Yeti testbed and conduct some experiment with more frequent change of ZSK and KSK.

• DNS Root Server renumbering. We may come up with a mechnism which dynamically updates root server addresses to hint file; this is like another kind of rollover.

• More root servers. We are going to test more than 13 root name server in Yeti testbed and to see “how many is too many”.

• Multiple zone file editors. We will use IANA root zone as a source of zone info. Each of BII, TISF, and WIDE modifies the zone independantly at only its apex. Some mechinisms will be coined to prevent accidental mis-modificaiton of the DNS Root zone. In addition we may implement and test “shared zone control” ideas proposed in the ICANN ITI report from 2014. ICANN ITI report: https://www.icann.org/en/system/files/files/iti-report-15may14-en.pdf

• Multiple zone file signers. To discover the flexibility and resiliency limits of Internet root zone distribution designs, we can try multiple DMs with one KSK and one ZSK for all, and we can try multiple DMs with one KSK for all and one ZSK for each.

We are not

• We never and ever try to create and provide alternate name space. Yeti DNS project has complete fealty to IANA as the DNS name space manager from the beginning of its conception. Any necessary modifications of the current IANA zone (like the NS records for “.” ) will be dicussed publicly and given a clear reason.

• We are not going to develop or expriment with alternative governance models, regarding the concern arised in many occasions that a certain TLD (mostly ccTLD) will be removed intentionally as an additional option for punishment or sanction from USG to against its rivals. It maybe discussed or studied by different projects, but not Yeti. In Yeti we keep the same trust anchor (KSK) and the chain of trust to prevent on-path attacks and distribute root services based on the current model.

This is the Yeti DNS Project, an experimental testbed network of DNS root name servers.

This testbed network will be used to discover the limits of DNS root name service, including the following:

• Can root name service succeed if it is only connected via IPv6 (and never via IPv4)?
• Can we change the DNSSEC “ZSK” more frequently, perhaps every two weeks?
• Can we change the DNSSEC “KSK” more frequently, perhaps every six weeks?
• How many root name servers is enough? How many is too many?
• Can we add and delete root name server operators frequently, such as every month?
• Can the IANA name space be served by more than one set of root name servers?

Note that the Yeti DNS project has complete fealty to IANA as the DNS name space manager. All IANA top-level domain names will be precisely expressed in the Yeti DNS system, including all TLD data and meta-data. So, the Yeti DNS project is not an “alternative root” in any sense of that term. We hope to inform the IANA community by peer-reviewed science as to future possibilities to consider for the IANA root DNS system.

Resources

The latest zone file, public key, and description of the Yeti DNS project are placed in a repository in GitHub:

https://github.com/BII-Lab/Yeti-Project

There is a discussion mailing list for project participants, reachable here:

http://lists.yeti-dns.org/mailman/listinfo/discuss

Statistics

The DSC page of the Yeti root servers is available, where you can see the traffic of the testbed:

http://dsc.yeti-dns.org/dsc-grapher.pl?plot=bynode&server=Yeti-Project

Coordination and Participation

The Yeti DNS project was launched in March 2015 by representatives from WIDE, BII, and TISF, who now act as co-equal project coordinators. The role of the coordinators includes outreach to the community, cooperation with Yeti DNS authority name server operators (“providers”), and support for Yeti DNS recursive name server operators (“subscribers”). Initially, the coordinators will also operate the Yeti DNS distribution master servers.

The Yeti DNS project coordinators invite interested authority name server operators, recursive name server operators, to contact us by e-mail, either individually:

ljsong@biigroup.cn (Davey Song)
kato@wide.ad.jp (Akira Kato)
vixie@tisf.net (Paul Vixie)

…or as a team:

coordinators@lists.yeti-dns.org

About Us

BII Group — the parent company of BII (Beijing Internet Institute), a public interest company serving as BII’s Engineering Research Center.

WIDE — Widely Integrated Distributed Environment.

TISF — a collaborative engineering and security project by Paul Vixie.

Hello Yeti

“Hello “