Archives

IPv6 Working Group

26 October 2016

At 3 p.m.:





JEN LINKOVA: Good afternoon. Please take your seat. Welcome to IPv6 Working Group. Measurement Working Group is in another room, so if you, for some reason, are not interested in IPv6, you should be, so please stay.

So, we have actually quite busy agenda today, we have three presentations and we have some administrative things about co‑chair ‑‑ Working Group Chair selection. So, anyone has any comments on the agenda, someone really, really, really wants to present something? No. Great. Let me thank you, RIPE NCC staff scribers, people who help us to run this session today. So, before we start, I would like to ask you to rate the talks, because otherwise we have no idea what would you like to hear about. It's really, really helpful, thank you for everyone who has rated the talk on the previous sessions. We were looking at the results and exactly because of this, and not just because of this, we have one speaker who constantly getting the best rating for all talks, so I am glad to invite Geoff Huston to present something about IPv6 and DNS.

GEOFF HUSTON: Hi all, I am with APNIC. It's okay if you wanted doing to the measurement room because I will to a measurement talk and a v6 talk at the same time. So, this is actually about measuring IPv6, and the DNS. So I just want to remind you a little bit about how far we have come with IPv6 now, the technique that I use is actually thanks to Google, and I will reference, I will thank them, they have been very helpful. What do I is on line ad, it's a really innocuous ad, and in between the ad, sort of down there, hidden, is a script. Because every ad has scripts. They make things dance across your screen and that kind of stuff. My script is pretty simple: It says get these three URLs, just get them. V4 only, v6 only, dual stack. Now, there is a couple of things about those URLs that make this possible: The domain name is unique, and it includes a bit of you and me. So, caching won't work, DNS caching web caching. The DNS server, the only one on the planet is me. The web server, is me. So if your browser runs this script, I see you. I see your resolver and I see you. So, here is you for about 7 million samples a day and the beauty about Google, bless their little hearts is they give you a 7 million every day. Now, you might think Google do sort of average job with some things but remember this: 97% of their revenue, and they make a lot of money, hundreds of billions of dollars, comes from advertising. They do advertising really well. They find users where the rest of us just can't. What do we see there? Comcast has done amazing jobs in the US. Country in the world that has the most is still Belgium, there is Deutsche Telekom's work, work in Finland and Estonia, that is Greece, that is Canada, that is Peru, isn't it amazing Telefonica did some astonishing work in Peru and it is brilliant, at the retail level. They are still working hard to try and find that same leverage to get them going here in Spain. Portugal is a little further off. You can go and look at the numbers. We get a pretty good picture across the world of where v6 is.

If you put this all together you get around 7 to 8 %. I know Google say more but you have got to do some stats. There are an awful lot of people in China, and not all of them use Google to the same rate as the rest of us so you have got to balance things up and recognise that the numbers in China, while they might below, represent a huge amount of user population and that brings down the world averages, so when we say around 7%, that is what we mean. What it really says is, if I put up a web service v6 only around 7% will be able to get to me and the other 93% are stuffed. So that is kind of where we are with v6 at the browser level. But if you think about it and you guys do because that is why you are here, there is a lot more to the Internet than the web. There is a lot more to the Internet than YouTube or Netflix. So we are measuring web‑based metrics but how about other bits and pieces. Everything you do, almost everything, apart from DDoS attacks, everything else you do uses the DNS to start with. Because no matter what it is, no matter what you run, it normally starts with go to name. What does name mean? Ask the DNS because that is the thing that actually resolves the name. So there is a good question to ask: How well, are we doing with v6 in the DNS?

Now, there is all kinds of questions you could ask and I have seen folk answering all kinds of questions, taking the ALEXA top million, who has AAAA records, yeah fine. Who actually publish information about v6? Whose client resolvers can ask questions? And there is this whole issue of are we talking about v6 records across an IPv4 substrate? Or are we talking about using the v6 substrate for doing the DNS itself? And so because I do ads and because I can measure stuff, I was going to pick just one question here and really concentrate on that one question. So let's go down one level and I don't care about the ALEXA top one million, other folk might and they are welcome to it, wherever you are. I am not doing that. I am looking at the underlying infrastructure of the DNS. If you run a DNS resolver, can it ask questions in v6? And if you run an authoritative name server, will it respond? What does a query rate look like? So the question is, how much of that resolution infrastructure is actually capable of running v6? Now, the DNS is a car crash, a car crash on a freeway, it's really bad. To say it's mete stable non‑deterministic and chaotic is being pleasant about it. The DNS works only by accident and even then at random ways. The fact that it works at all is a miracle, and it's the only thing that makes me believe in magic because it's astonishing that you get an answer any time and even more that it's so fast. There is a huge amount going on. Normally, your ISP will give you not just one resolver to ask but two or if it's feeling enthusiastic, 3 or even 4, and your local resolver library asks one, asks two, asks three, so queries multiple and when you have got a forwarding recursive resolver, it might use resolver farms and load balances. There is an awful lot of stuff going on and one question can lead to an avalanche of questions that the authoritative name server. So actually, trying to work out what is going on is challenging. To my mind, the DNS looks a lot like this, because I am the authoritative server, I am sitting over here and looking back. What I have done, is I have made the user do something and what I observe is some results. So, this mess here is what we call the DNS. I have no idea what is in it, and neither do you. Queries have no TTL, they have no trace. I reckon there are forwarding loops. I would bet that there are 20‑year old DNS queries still sitting around on resolver loops forever, they never die, because they can't. There is also another thing you should remember, at least 40% of all the queries in the DNS are crap. They are nonsense and not real. Some of these is endlessly circulating things, query logs, mad resolvers, you know it. There is this ‑‑ that is mess, you can't describe it and I can't, but I see some resolvers, visible resolvers, so let's just look at that.

I am going to seed a known event and watch what happens. So, I talked about the ad platform and all that, you can read the slides. The next trick is how do I know a resolver is doing something? Now you might go that is a really easy question. This is the DNS. Nothing, nothing is obvious. All those cascading queries have no rational explanation. So, one thing I have noticed is glue, you all rely on it and use it a huge amount of the time. Here is a bit of glue for Bugatty in the root name system, not only does it say these names here are the authoritative servers for Bugatty, but for free because the DNS is so helpful, in the root zone of the DNS is this other bit of information which is not authoritative, but you use it. It's called glue, and it helpfully resolves those names immediately. Here is their addresses. So when you ask a root serve for I want I want Bugatty, the answer from the route server is, go ask these people, by the way use those addresses because they are probably the right ones. We use glue all the time, right? But it's optional. So, what if there is no glue? That is a good question. Because if there is no glue, you have got to stop, and go ah, this name server thing, what is that name? And resolve a second problem. And then you have got doing back to the first problem again once you have found the answer. Got you. Because what I can do, is isolate the resolver, because if I make that train of questions all pointing at me, I can see individual resolver behaviour, so what is going on is I start down one thing and get a glueless delegation, park that thought, go and do another thought. Here is the other thought, that is also me. It's resolving this other name. Once it's found the answer, it goes back to the original thought, right? Brilliant. What if I put v6 only right in the middle. You can only get that name if you have got v6 as transport. If not, out of luck. That's how I find individual resolvers. Because this is dual stack, so you are probably going to come at me in 4, who knows. That is dual stack, I am going to see if you complete it, but to resolve the name server, if you have got a v6 side in your DNS you are cool. If you haven't, never going to happen. So, what I see are those four queries. Dual stack query, dual stack query, query three, only works if you have got v6, no option. Query 4, dual stack. So, I have now got individual resolvers, except for Google. Because Google set up resolver farms and so what they have is a brain behind, a whole bunch of Anycast brains and the front end, a bunch of v4 and v6, a whole bunch of addresses that feed the one DNS process. So, it's not quite as simple as one to one addressing with server farms but I can still do a basic job of identifying who does what.

Ad campaign, thank you Google, between five and ten million ads a day. About 400 million results when I did this across July to August, is that enough? Probably. There is only one 3 billion‑odd users, 400 million must a decent cross‑section. There is a lot of resolvers out there, 345,000 resolvers I saw; other folk see as much as 4 million. Trailing edge, a long trailing edge of resolvers. These are the resolvers eyeballs use, eyeballs, because this is an ad that caused it so these are the resolvers that ask because of eyeballs. Of those, wow, 268,000 never gave me a v6 question for that particular query. So, these are the ones I like, 59,372 asked in v4 and asked again in v6. So I reckon, around 22% of resolvers are actually doing a decent job and are able to use v6 to make these DNS queries. 7% of users, 22% of infrastructure, well done. It's brilliant, users by 3 to 1. So folk who pay attention to the DNS appear to be doing a really, really good job and that is great. Right? Now, resolvers and users are not the same thing. So of those 22% of resolvers, how many people do they serve, how many eyeballs are actually accounted for here? Here is where I get to the fact that all resolvers are not the same. Some resolvers are much more equal than others. Here is the top, oh my eyes ‑‑ here is the largest 8,000, that's right, resolvers, by IP address. Google has many IP addresses. But this is the largest 8,000 IP addresses, 90% of all eyeballs. 90%. 15% of all users on the planet send their queries to Google's public DNS. Theirs does v6. So somewhere we can start thinking about this a little more. 194 million experiments asked the authoritative parent for the parent zone, 122 million eyeballs didn't make it, 35% of the eyeballs, 35% of the e‑ permits weren't appropriately weighted, so it's appropriate to users, use IPv6, so might be 22% of the resolvers, one‑third of the user population. Take that web nuts. This is v6 like crazy. You are beating DNSSEC, this is amazing.

So 35% of the users, what do dual stack resolvers do when given the choice, because the next question is, if it's dual stack, what do you prefer? The folk do happy eyeballs in the DNS, do they prefer v6? No. In a word. In measuring this, 25% of the experiments pass the queries through, all of those around 11% that were dual stack‑capable. It should have been around 17%. If you are really doing a random choice it would have been 17%, if you were biasing v6 it would have been up to about 20%, but it's only 11%. So I guess that there is still a certain amount of v4 bias inside the DNS.

So who are they? Of the v6‑capable resolver sets, 30% of those v6‑capable ‑‑ 36% of customers, 31.nine, sorry, of end users who do v6 in the DNS use Google to do it. Fine. 13% use AT&T, why? Because they use a resolver for all of their customers, they tend to have a lot of American customers, there are a lot of Americans out there, what a surprise, and so then resolver numbers are big. Comcast has a lot, around 40 or 35 million, so again they feature as well. Open DNS, much smaller than Google, around 3.4% of market share and you see others have done the work in the DNS but not necessarily with customers, look at Mexico, Japan, there is the first Chinese one, collar owe in Brazil, they have done v6 deployment. You can look at this, India, Bart art he will have done v6 and switched their resolver as well. Turn on v6 in your resolver, yeah? Be careful. Really careful. Fragmentation is a problem in v6, and we are incriesing use of DNSSEC big DNS responses cause UDP fragmentation in 6, this generates extension headers and what do we know about extension headers in the Internet in v6 Jen? They get dropped like crazy. If do you fragmentation in DNS responses in v6, you can expect drop rates of up to 30%. So you have got to be careful about how you do this. And let's also talk about why I did not see that in this experiment.

So, reliability. How much am I dropping in my experiment? I actually use three sizes of response to test it out. Under 515, right, nothing special; between 512 and 1280, which requires a little bit of DNS work but nothing much; and then the monster, below 1,500 but above 1,280. If there are problems we might find it here. But unlike many, I deliberately chose a local MTU of 1,500 for UDP. Deliberately moved it up. What happens when you move it up? Well, I see failure behaviours where I get some kinds of repetitions of queries, I get a certain amount of truncation. But basically the large packet is as reliable as the small. And the only reason why this is the case, the only reason why this works, is the high MTU setting. That local MTU of 1,500 objecting at the times makes a huge amount of difference. 1,280 is such a bullshit number. The network is perfectly capable of handling 1,500 objecting at the time MTUs, and thank God tunneling is disappearing. You can run 1,500 objecting at the times across most of the Internet, most of the time, as you see. Now, as far as I understand, 1,280 is 1,280 because it's 1,000 and 24 plus 256, according to Steve Deering. It's not a natural number, it's not a packaging number, it's not a number that makes any engineering sense, setting your MTU to 1,280 is good if you are running TCP, because you are trying to avoid path MTU black‑holing. It's shocking for UDP, do not do it. Understand? Do not do it. Because otherwise, that number there, that number the 1425 would not be 96%, below 70%. If you want the stuff to work, use a large MTU for UDP. What is the result? One‑third of the users will actually get to you if the only thing they can use for DNS is v6. Success. That is brilliant. But if you say okay, let's do it, just make sure that your local configuration is up to snuff. Keep your MTU large. Make sure if you do need an ICMP packet too big, that you get it. Clear out your ICMP filters. TCP, I'd still say if you can adjust your TCP down, keep that down. Because that makes the TCP part work. But beyond that, yeah, brilliant. One‑third of users can use v6 on the DNS. What a success story. Thank you.
(Applause)

JEN LINKOVA: Questions?

GEOFF HUSTON: It's really hard to set your UDP and TCP to different values and I thought someone was going to ask me about that, but if you think it's okay you know more than me.

SPEAKER: This is Anand Buddhdev from the RIPE NCC. Thank you for great presentation, some very interesting results here which I would like doing and investigate. I just have one comment about delegations and glueless delegations. It's more a clarification about your slide, really. In referrals glue is required if the name servers are in the same zone, so glue is not entirely optional. It is required in some cases, so in the case of Bugatty which is required, but for your experiment, it's not required.

GEOFF HUSTON: I was going doing back there and say that, yes, I had deliberately set it up so that the glue, the name of the name servers was off to one side, so I wasn't violating any standard in so doing it.

ANAND BUDDHDEV: It's fine, yes.

GEOFF HUSTON: There are a small number of resolvers that people use, not a lot, if they don't see the glue just go all fuzzy and stop. Guys, what code are you running, what century did it come from? Glue is not mandatory, it's an option. But it seems that everyone just expects it now. So yeah, thank you.

The other thing is a root name server. You would notice that that number at the bottom is very deliberately chosen because at one stage of rolling the root key which will happen about this time next year, we will be running that response size coming out of the root zone queries for DNSKEY. So as long as the root zone operators, including yourself, are taking very careful note of what I did to make it work really well, this should be pretty good.

ANAND BUDDHDEV: We shall take note.

GEOFF HUSTON: Thank you.

SPEAKER: From ICANN. Back in September 2004 ‑‑ wrote a document called RFC 3901 and ‑‑ v6 guidelines which is also BCP, and I was wondering is this document still current, I have a recommendation made where on, if you set recursive resolver don't set it v6 only. Is it something that should be revisited or ‑‑

GEOFF HUSTON: So I saw and read an RFC that I think you co‑authored, on the use of extension headers in v6 and store against the ALEXA top whatever, a very savage drop rate. I did that same experiment in this set‑up using the DNS, and saw about a 30% drop rate from my servers across to visible resolvers when I deliberately introduced v6 fragmentation. That is really bad. That is why I came up with this recommendation, okay if the response is big, you can't get around it. If it's more than 1,500 objecting at the times you are going to have to frag. If it's below that, if you set your local MTU low, you are inducing upon yourself massive damage, huge damage, 30% drop rates are horrible. So my advice is, if it doesn't talk about the difference between sort of data gram systems like UDP and TCP flow control, it should. You will then ask me, because you know this, what happens when I run quick, and I go, holy hell, I have no bloody idea, I think 1,500 should be okay for quick even though you need to run lower for TCP. But quick is kind of weird and maybe we need to do more v6 quick work to really answer that.

SPEAKER: Thank you.

GEOFF HUSTON: Which probably answered more questions than you asked.

JEN LINKOVA: More questions? No. Thank you very much.
(Applause)

So, let's continue talking about measurements, so people stay in this room, I like to see this room full. So the next talk is about the measuring the happy eyeballs.

VAIBHAV BAJPAI: So, Hi, I have only 15 minutes so this will be quick. Today I am going to talk about measuring the effects of happy eyeballs.

So this is a v6 Working Group so I am going to assume everybody knows happy eyeballs. And just in case, if this is not the case I have some background material tend of the slide, if you download the material you will have some background slides for the ‑‑ okay. So, the happen eye eyeballs of 300 milliseconds was chosen during a time 2011 and 2012 this was quite prevalent and this was largely attributed to failures called by Teredo, in fact people have shown like 3 and 4, that even in situations where these really work, they add considerable latency over the Internet. Thanks to you and v6 operations committee in general, we have seen that these transitioning mechanisms are start doing decline over the years and consequently, as a result, failure rates over v6 have dropped significantly, so Geoff has shown us that back in 2011 failure rates were around ‑‑ over v6 were around 40% which reduced to less than 4% last year, and even later v6 failure rates have gone counsel to around 2%. So what I am trying to get to is the v6 landscape is very different today from when the RFC was defined and you see this in the implementation of the standard of how this has become fragmented over the years. Google Chrome in 2011 introduced 300 milliseconds timer value which is being used today and on the other hand Firefox use zero ‑‑ so far use to use this history of previously latencies, but the recently changed the implementation last year by giving 25 milliseconds head start to v6. What I am trying to get to is, the standard implementations have become fragmented over the years and these timer values chosen by browser implementations are arbitrary. What is right timer value and we want to contribute something towards it. These are all the contributions that we provide so we actively measuring the ALEXA top websites and since 2013 and it has not ended and we noticed TCP connect times to dual stack websites have considerably improved over time. I will show you some of the results. As of May 201618% of these dual stack websites are faster over v6 with 91% of them being at most one millisecond slower so this is fairly good news for v6. We noticed 300 milliseconds timer will make nine nine% of all websites prefer v6 while making connection from the clients nine 8% of the time. However, even though there is strong preference to ‑‑ slower in nine 0 percent of the cases and so in this paper we recommend lower down the timer value for the changed landscape of the data to 150 and we show you margin benefit of 10% while maintaining the same preference what you see in 3.

How do we do this? We have an active metric which is used to measure TCP connect times and implementation of this which we call it happy. It is OpenSource simplified licence and available on website. This is how it looks like, if you run it on a system, and we use this test to measure the words ALEXA top 10K websites so these are some samples websites we look at and we cross compile the stats for OpenWrt platform, if you know that is a company based in London which is giving out these routers to people at home for doing broadband measurements. So we deploy this test on these probes and this test is running every hour since 2013 for the last three years. So we have these probes with the test and gave this probe to people, okay? And this is the trial so each point is a dual stack so these are probes hosted by people like you so thanks to you and to other people who are actually hosting this probe for me. Most of these probes you should notice they are sitting in residential settings.

So, let's look at some results. So, the first thing we look at is trends. So this is showing you the times series of it. CP connect times to some pop lab websites that I chose. Starting 2013 to 2016. And this is showing you difference in TCP connect times so v4 minus v6. So what it means is, if you see anything on the negative X Axis ‑‑ negative Y axis that is not good for v6 and what you see generally is back in 2013/2014 situation was not good so v6 was considerably slower to v4 and things have considerably improved over time so things look ‑‑ let me take an example of Facebook which is the green line on the top, so used to be around 100 milliseconds slower over v6 back in 2013. And now almost congruent over v4 and 6. The next question, this is how it used to look in history, how does it look today? So this is the distribution of all the ALEXA top 10K websites from May 2016, and so note in all the plots negative scale means that it does not ‑‑ not good for v6 so the positive scale which is on your right is are the websites which are faster over v6 so you notice that 18% of these websites are faster over v6 while 91% of the rest are at most one millisecond slower, notice the X scale is horribly zoomed in so only a millisecond difference, which is fairly good news. And 3% of the websites are at least 10 milliseconds slower and 10% 100 milliseconds slower so there is a trail where v6 is slower, but things are getting better.

So the next thing that we looked at was the entire contribution of all the samples that we collected during the last three years, around 189 thousand samples so look at the top lot, and this is showing you the distribution of it. CP connect times, the X axis is in log scale and what I am trying to get from this plot is look at the right side of 300 milliseconds, it is only 1% of the samples which cross 300 so only one% of these samples from the past 3 years that mattered for happy balls, this is only 1% of the cases where it came into the picture.

Effects of this, we actually see in the bottom plot where calculate the preference towards connecting all these ‑‑ all these websites over v6 and what we see is, a 300 milliseconds timer only 2% chance for v4 to successfully connect with happy eyeballs. This means 99% of ALEXA top 10K websites prefer making connection over v6 nine 8% of the time. Which is good. We are strongly preferring v6 thanks to happy eyeballs. But you also want to compare it to how it used ‑‑ how it will compare to v4 latency, right, so this plot is showing the cases where the clients are preferring v6 connection using happy eyeballs but how does it compare over v4. If you look at the bottom plot you will see that in 90% of the cases happy eyeballs is giving preference to v6 but it is actually slower than v4. But the news is not that bad because if you look at the top lot, the absolute difference between v4 and v6 is not that much. So in 30% of the cases v6 was at least one millisecond slower and in 7% it was at least 10 milliseconds slower. So the next question we asked was, why do we need happy eyeballs, maybe we are at the stage we can just completely disable it, right? This is what will happen if we do that. So, look at the red line in the plot and this is showing that you if you don't give v6 a fair chance to succeed today, 18% ‑‑ your v6 will drop from 99 to 18 percent and this is because of the case that I showed 18% of the websites are faster over v6 and this is what browsers like Firefox do, BCP connections, right? So the point I am trying to get to, we are not at the stage where we can completely disable happy eyeballs but we can reduce the value. So next we wanted to see perhaps we can actually reduce the value from 300 milliseconds to some other value which gives the same preference to v6 as is today but does not hamper v4 in situations where it is considerably faster.

So, in this experiment what we did was, we tried to control these two parameters, 99% of ALEXA top 10K should prefer v6 nine 8% of the time. Do not want to compromise on this. Tried to lower down the happy eyeball timer values. Look at the top lot, up until 138 milliseconds it holds, this condition, afterwards the preference starts to go down. So I gave it some buffer space and tried to suggest that perhaps reducing it to 150 milliseconds would be good timer value for today's 2016.

And if you look at the bottom plot, if you use 150 milliseconds and compare the absolute differences between v4 and 6 you will see get 10% advantage, 10% is huge because this is a three‑year long dataset, Kistelekiing of 18,000 connections. So this will give you 10% margin advantage while retaining the same preference over v6. And I just want to point out something: This was the previous plot so if you look at the time series, if we would have used 150 milliseconds from the very beginning it would have still worked so this goes to show that measurements should actually be used as a guidance for doing protocol in general.

There are some limitations. So the comparison reflects performance as seen over TCP on port 80 only. I don't know how it will look on any other. The measurements cover ALEXA top 10K website. And the results are biased by vantage points where my probes are sitting and largely centred in Europe, US and Japan. Most of the v6s only concentrated in these regions so there is a give and take here.

So, what did we see? Actually we saw that TCP connect times has considerably improved over time in the past three years, 18% of websites are faster over v6, while 91% are at most one millisecond slower. 300 milliseconds is very strong preference to v6, to 99% of websites while in 90% of these cases you are actually preferring slower connection over v6 and that is why we suggest that lowering the time doing 150 milliseconds which will give a margin benefit of 10% while retaining the same preference level which is ‑‑ that is about my end and happy to take your questions.
(Applause)

JEN LINKOVA: Questions? Comments?

GEOFF HUSTON: It's more of a comment than a question. How do you know as a network operator when to turn off v4 completely? It's a serious question. Because if you are running dual stack, then you have got some v4 traffic and v6 traffic, how do you know at what point the v4 traffic is redundant, not worth doing? The reason why I ask this, that was the aim of happy eyeballs. The whole idea was, that if I was dual stack and you were dual stack, and we could do both protocols, in the normal course of event we would choose 6. And so when everyone was doing that, there would be no more 4, and so we would know when to turn off 4 because there was no 4 left because happy eyeballs made it so. What it wasn't, was to pick the fastest. That was side effect. The whole idea was signalling to all of us that if you prefer 6 when you have a choice, you know when to turn off 4. If you don't prefer 6, you never know when to turn it off, because it will linger forever.

VAIBHAV BAJPAI: Right.

GEOFF HUSTON: So it's a side effect that you are picking faster and that might be a happy side effect but it's not the point.

VAIBHAV BAJPAI: So one thing that I am thinking about this, happy eyeballs is not giving the preference to v6, it's RFC 6724 so when you resolver the name, IP address will come first. It's happy eyeballs which is allowing to you do a quick fall back, right?

GEOFF HUSTON: No. The way happy eyeballs was meant to work, was you initiated two connections at once, and if you got v4 first, you allowed a wait period of up to 300 milliseconds something for the v6 connection to complete to the sin axe stage and chose that if it was still there. There are other ways of doing it. You delay the v4 connection by the happy eyeball interval and take whichever comes first, but the whole idea was to give v6 a bias of a certain amount of time so that, in the normal course of events, as long as v6 was working, you'd prefer it.

VAIBHAV BAJPAI: Right. Right. So there are two comments I would like to make. So the first, remember the first slide I showed you, but Geoff has shown us that failure rates have considerably gone down over v6 so yes, in that situation general happy eyeballs is not needed today so I see the point. So maybe. I don't know. This is what you have to deside.

JEN LINKOVA: Just a comment. The problem is when we are saying about failure rate is low, we are measuring average, right? Is there networks when it's 0 and new deployments when it actually might be quite high and if you don't have happy eyeballs there on your end devices ‑‑

VAIBHAV BAJPAI: Exactly yes.

JEN LINKOVA: I think we need them because people who are still rolling out, will need ‑‑ it does hide the problem yes. So I agree with Geoff, it's not about getting you to website or five milliseconds faster, it's about making sure you get there tend of the day without time out. 1
VAIBHAV BAJPAI: I agree with both of you and in the paper we don't recommend you turn off happy eyeballs. All three of us agree with each other. More questions? Thank you very much.
(Applause)

So we already see how v6 and DNS works together, how v4 and v6 works together. Now we are going to look at v6 from completely different perspective, and look how different countries are going to roll out v6 based on GDP. Alan.

ALAIN DURAND: Good afternoon. So that is study which I have made and Geoff Huston has provided a lot of input to this so I want to acknowledge his contribution to this. Questions that we studied with is, we hear a lot of success stories about IPv6, we also about some places where not so ROSIE and the question we were asking is, how is it really deployed across the board, is there a divide between rich countries and not so rich or is there a east west divide or north/south divide so wanted to look a bit further. So what can this tell us about IPv4? Is it time to retire IPv4 or not? And are we there yet, essentially?

So, our methodology is find some data and compare country. So when we compare countries, we ‑‑ economies usually use GDP per capital, which is gross domestic product per capital at that. That is classic measure. I start by doing only gross domestic product, which is output of a country, that gives a bias to very large countries or small, so created by using GDP per capital at that. I would like to look at out liars because we can learn a long. Data to see if it makes sense or not.

The first step was to find data. I found two sources, one is data from Geoff, from APNIC using Google ads distributed service; and this is a really good proxy for measuring penetration as in capability of using IPv6. The second set of data is from Akamai from the state of Internet study, and what they do is measure IPv6 hits on their proxy cache, so a hit on the cache means there is a file being transferred, means there is data flowing, so there is real traffic. So this is a proxy for measuring bandwidth, okay. I have both proxies I don't have direct measurement. Ideally, I would like to have a third measurement, which will be will IPv6 measurement volume measured at the service provider level or Internet Exchange points. So, signed a contract with another third party to go and get data from exchange point so next time I will do the study I hope to have this third layer of information to overlay. That is what we have as of today. So, where are we? Take all the countries, little hard to read on the graph but the slides are available so you can look at them, and I round them by GDP per capital at that, to the left I have richest countries and to the right are the poorest countries. So the richest country if you don't know about it is Qatar, about 130,000 something per year per person. Poorest country in that list, to the right, and side of graph here. So it's typical graph.

This data is coming from the world bank so I didn't make this up. On top of this the IPv6 measurement. So, in blue data coming from Akamai, proxy to bandwidth; in red, data coming from Geoff, proxy to penetration. So, the two are side by side so it's a little bit difficult to see the colours in here but we will develop more details later. The thing is, look at this graph and well, what do we see? Well, what we see is that there are visually two groups of country, a group on the left of of black line and the group on the right of a black line. The group on the left appear to be top 50, 50 more ‑‑ 50 richest countries per capital at that and if you are in the group on the left you seem to be having a good likelihood of having some kind of IPv6. If you are in the group on the right, well, not so good. So that seems to indicate a divide between rich countries and not so rich countries.

If you are curious about where the cut‑off; top ‑‑ both below you are not. There is a notable exception to this, it's not like if you are in a poor country you have no v6, that is not true. We have a number of countries like Romania, Brazil, Peru, Ecuador, Bosnia, Bolivia, Haiti, countries on the very far right are outliers, that is weird. And last time I made this presentation, somebody from Haiti was there and said that is not true, we don't have any v6 there. Developed a bit further with Geoff and the issue is geolocation, but for it's not good in v4 or v6, no surprise here. So it was two on the rigtht hand side, really outliers that should simply not be there. But the others like again Romania, Brazil, Ecuador, Bosnia, Brazil, those are real, are not expected, I was not expecting them at all. So, that is interesting.

Now, let's zoom in the top 50 countries, so the same graph simply just zoom in there. And the situation is not uniform either, it's not because you are in a rich country that you have IPv6, that is not true. You have six countries that are really leading the charge on IPv6. Those are Switzerland, the US, Germany, Belgium, Portugal and Greece. There are some countries that you will see on that list that you will expect to be leading the charge in IPv6 but have very, very little penetration.

If you look at the top 10 countries in the world, including the top ten countries deploying IPv6 in the world, regardless of being in the top 50 GDP or not, you will see you have two are not in the top, Peru and Ecuador. This is, again, very, very surprising. They are not in the top 50. So, let's look at the data now. If you look at those blue lines and red lines and blue is bandwidth as much as from Akamai or proxy from ‑‑ the red is penetration as measured by Geoff. First observation is that ‑‑ first two lines do not match exactly. Sometimes they are close and sometimes they are not. Sometimes they are very different. So, natural thing to do is to look at the ratio of the two. And in ideal case we will think penetration is the same as bandwidth. We will think that yeah most probably penetration is a little bit higher than bandwidth because some people may not use it. When people use it more and it is available it is kind of weird. But that is kind of what we are seeing. So this graph is ‑‑ I took all the countries that have at least 0.1% penetration of IPv6 and I just plot it in this, by this ratio. So looking at the ratio, again should expect to have ratio of 1, right? Now, to the left you have countries where IPv6 is available but essentially never used. The ratio is up to 900. This is just amazing. And you can say okay, small countries may be again some side effect of geolocation. But there are some larger countries here like right at the edge is Great Britain. The ratio is 7.5. It's available but only essentially one out of 12 people are really using it.

On the other hand of the graph, more outliers, countries have quite a significant usage of IPv6 although penetration is small which means that the few people who use it use it heavily. And that is also very surprising, Korea, on the right of this diagram is significant outlier and no explanation at this point as of why.

So in the middle of the graph to kind of, middle of the road here, you have countries that are ‑‑ usage is 2 to 8 times higher than what I was expecting, and then the countries that one two to two times, is kind of what I was expecting.

So, if I remove the outliers on the left and on the right, the average is 2.2, median is 1 .8 which means if you look at penetration number, if you have to ‑‑ if you want to have an idea of what is really the usage in terms of it's only half of it. So, the point I am trying make here if you simply look at penetration number it can be very misleading in terms of what is actually being used.

So, are we there yet? Is v6 a replacement for v4? The short answer is no, of course. There are positive signs. There are six countries that have both penetration and bandwidth number above 20%. I mean, five years ago, ten years ago, 20%, wow. That is fabulous. So that is great, I mean if this is happening in some places, but not everywhere. And v6 is going to bereft and be relevant for quite some time. This is not we have some success stories in places, repeated now everywhere. You have a large number of countries, especially the one with low GDP per capital at that where there is little or no IPv6 at all. And last word I wanted to make about was measurements; we see number of different measurements in different places. As I pointed out, measuring penetration is sometimes misleading. I feel it would be good, other people would do those measurements and try to figure out what should be the best practices how to combine those things so we can apples to apples and not apples to oranges. So there is a bunch of raw data you can go for it in your own spare time if you want to see where your country fits, there is a lot of ‑‑ and I think I am a little bit early, but that is the end of my presentation.
(Applause)

JEN LINKOVA: Thank you. Questions? Marco?

SPEAKER: Actually it's not a question, it's just trying to provide some input. I think in general countries with slower GDP, there are more incumbent operators, okay, so if just one operator for whatever reason, and I can talk, I am going to do it but I can explain if somebody want to after this session, it's a long history, what is happening in Peru and Ecuador and other countries, where I actually did some actions this those countries, including working with the government and that means the incumbent operators, that explains why, in those countries, there is a bigger deployment or earlier deployment which, as you said, it's not not the same as actual usage and so on but I think it's easy to correlate that situation.

ALAIN DURAND: This is an interesting point. It seems to me that, yes, governments have an impact on this. However, when I superimpose this study with another one where we look at the deployment of DNSSEC validation, we realise that the countries that are leading the charge are not the same. So, it would appear, and I am putting a condition on here, that those are technology bet that are made either by some country governments or by some service providers about where they should put their marbles.

MARCO HOGEWONING: RIPE NCC. Thank you for doing this again, I would say, because I have been looking into this a few years back and recently is it something comparable for a talk did I to the OECD. It's really interesting because I am always curious to say like yeah, would you expect richer countries to do better in IPv6 and as you mentioned there are certain outliers there. Now you mostly celebrate the winners, I am more of a negative disposition, always look like the ones that really fall out of that list. But again, I think this is comprehensive and worthwhile.

For future studies, what might be worth incorporating is to consider the full Internet penetration of the countries and mot the IPv6, but if there is more Internet in a country does this also sort of comparatively increase the chance that somebody has IPv6? You might even want to, and I am not sure, probably the ITU or ‑‑ try and locate the dataset to see if we can come up with a cost, if the Internet gets more expensive, how does that do with IPv6? I do like it, I don't think there is any correlation. There are more questions than answers usually following these kind of studies. But I do think it's worth, especially also for this group to focus on the outliers and both the winners and the losers, because it's then what is the success story behind the winners and why are some countries so falling behind? I think, based on your suggestion, based on what I have seen, I don't think GDP is a very big factor in whether do you IPv6 or not. I think open to the group figure out what is, but unfortunately I don't think it's the money, but that's ‑‑ it's clearly not the only factor.

MARCO HOGEWONING: Clearly not only the money but I am happy to take hints on what it is because I am trying to solve this puzzle for the last 15 years.

BENEDIKT STOCKEBRAND: Speaking for myself rather than as a Working Group Chair, I think it takes a couple of things to deploy IPv6. One is money, so there is a correlation there, obviously. But it also sometimes takes some provider whoever to run into problems like that story about Comcast not being able to manage their home boxes any more and what I find is really important, at least at smaller scale in enterprises, is, you need this one really, really annoying person just why don't we do IPv6 all the time? And you can have that at country scale sometimes as well. So, I guess yes, the GDP plays one ‑‑ you have to bring these factors together. And that would probably kind of explain the numbers you found, but of course then it's always easy to find explanations for the numbers that don't look like you might have expected.

ALAIN DURAND: Yes, that is true, and also following on the previous comment and we have your comment put together, it will be really interesting to do a second phase of that study and to go see what outliers, the countries that were not expected, why succeed and the countries we were expecting something we see little. It would be a fabulous project and if anybody is interested to work on this with me we would be happy to accept.

SPEAKER: Oliver from the RIPE NCC and I am reading out a question from the chat. Rob lister from LONAP says did you attempt any other metrics than GDP? The first one I tried was GDP, purely and the curve was essentially the same shape, there was a countries that were more to the left and more to the right. When I talked to some economist, they said when you use GDP, purely, the data is skewed so you should use GDP per capital at that. So that is a classic measure in economy.

SPEAKER: Thank you. He has a comment: Interestingly, newer and smaller countries which have new infrastructures are perhaps able to switch on IPv6 more easily, that is something to bear in mind. Thank you.

JEN LINKOVA: I think I have a comment. So I think obviously, money is not the only factor, right? But I think there is some correlation between probably how reach the countries and probably the model operators doing this like because I am pretty sure is big factor is operators using manage TCP or people just going to the nearest shop and buying some cheap box and keep that box forever because they don't want to spend another 20 dollars on buying another CP and that CP is very old software, I can remember in ‑‑ sometime ago I realised I was replacing CP 2010 died and it might take five, six years for it, and obviously you couldn't deploy v6 on that old stuff.

ALAIN DURAND: That is a good point. If you look at the countries on the right‑hand side of this diagram, in many of those places you have service providers that are trying to get to the cheapest thing possible and most of the time don't buy new equipment, they buy secondhand equipment that has been passed down from other service provider in other countries, so in rich country like the US or Spain or England and Germany, France, you may have some relatively short cycles for equipments that may ‑‑ IPv6 N those countries you might have to wait a couple of generations before the current equipment that does support IPv6 goes to them because they simply don't have the noun buy it.

JEN LINKOVA: Yes. On the last, I think it was Swiss, IPv6 conference when somebody made a comment please don't send equipment, you took off your network to some poor countries because the equipment most likely does not support IPv6, don't make their life worse.

MARCO HOGEWONING: I was told we have plenty of time. I have mentioned this before in terms of your comment about studying further, I mentioned this during the Plenary and I am going to mention it again. The current Internet governance best practice forum of IPv6 is actually trying to look into those outliers and trying to come up with certain reasoning why certain countries out perform others while would you expect otherwise so I would welcome you to contribute some of your observations and thoughts into that process. We are currently reviewing our first draft but happy to take more input so please be welcome. Regarding Jen's comment and that is probably impossible because I don't think we would ever even with the total group, break it together. There is probably a likely correlation between ownership of the CP, is this provider supplied /ORPBD and control, T R 69 in France, are you able to reflesh a large quantities of CPs to enable IPv6 support. I am afraid we will never find a consistent data source that tells you who is doing what but it would be nice to sort of at least, as somebody already remarked, this is often the countries that we see perform really good is indeed one strong incumbents switching it on and in that sense it probably is worth looking manually to see what the market situation is in that particular country. So I would indeed recommend looking into it. It might not be a nice graph but it would provide valuable input.

ALAIN DURAND: That is I think the only way to do this and go talk to people in those countries. In one of the countries, on the left‑hand side that has some nice spike, I talked to some of the folks there, and how did you get that fabulous number? And the answer was, well, we have three service provider in the country and we got together drinking beer and decided to do it. That was what it took in that place. I am sure that is same thing cannot be replicated everywhere, but it's an interesting thing.

MARCO HOGEWONING: I think indeed in the particular country I think you know what I am talking about, is indeed the fact that over a beer I can decide to deploy IPv6 to a million customers because I have the ability to push that button and I know that other providers simply don't have that, they can provision IPv6 but sort of waiting for the clients' CP to come back for a prefix and that may as well take forever.

ALAIN DURAND: Well, we have about a billion CP on the planet, give or take. And they cost anywhere between 40 dollar and 100 dollar, if you don't factor in the cost of a ‑‑ so, billion times four, 40 is 40 billion, a very, very expensive proposition, so every time CPs can be flashed remotely, that is very good. However, this is in categories of consumer electronic devices that usually are never flashed again, and remotely it's even rarer. So maybe something to take away for this IOT type of things when we have seen last week type of damage that can be done, thinking about having software that can be, or firmware that can be remotely upgraded is probably the key to deployment of the next technology, not just IPv6.

MARCO HOGEWONING: Either that or lower predictions because 10 there are a boxes are remarkably sturdy in blowing up.

SPEAKER: Dot RS domain from Serbia and going to just note something from the ball cans region. So GDP is one of the ‑‑ based statistics in the world parameters but considering this it's really not enough. You have to have politician reasons in account because, for example, Bosnia, it's a great, great example. Although, what you call poor countries, that is like prejudice a little bit, they don't have money and they don't have knowledge of how to implement new infrastructure. So first thing that I think you are missing here is a variable, is it migration of all the infrastructure or is it completely new infrastructure? IPv6 in new infrastructure, piece of cake. And if you read, I don't know, the papers would you do it like that. If you are a poor country you need credit, you go call EU. EU says okay, here is a credit but people from EU are going to make this implementation in country, then smart people from EU experience and say okay, only IPv6. Actually that is what happened in Bosnia. Bosnia as you know was war torn country, still deeply divided, in very small country you have three small countries, under‑developed on some sort, and only thing why you have this big penetration of IPv6 there is because you have, okay, stop fighting, you need to invest to have high quality, brought somebody who knew how to to do it, and on IPv6, similar stuff happened in Romania, Romania did not have big IP seen scene developed and it merged with the time with IPv6. In Serbia if you have this problem, we have infrastructure, we know from IPv4 for ages and we had to survive all the wars and stuff and now why we stick with IPv4, because it is very expensive to migrate. IPv6 as a new thing, it's less expensive than to migrate to IPv6. So, this is a good talk and a good paper, but really, if you want to get more in details and find some of the solutions for this phenomenonal question, when do we shut off power and switch to 6, you need to put more and more details and it has a hell of a work. Thanks.

ALAIN DURAND: It's a very good comment. Thank you very much.

Blake Willis: Speaking for myself. Could it be that this data would also correlate with how much IPv4 is left in these given countries and the ability and/or willingness of those operators to buy more versus deploying IPv6? Be hard to ‑‑ harder data to collect, I mean could you at least get like IPv4 research ‑‑ resource transfers with RIPE or something like that but ‑‑

ALAIN DURAND: That could be interesting. I would, however, observe that of the ‑‑ all the transfers are global, they are global within region and even inter region so from all the brokers I have talked to, they are telling me it's not that they are taking an address from country X and only selling it to country X, go from place X to place Y. So, the availability of addresses for the market is essentially global now. Now, the question to be, what is a predominance of legacy addresses that is in the particular country? That could be something that we could start to look at.

MARCO HOGEWONING: Commenting on Blake. I don't have hard data, I don't have like fancy presentations, I did look into it for parts of the RIPE service region. I did not find any signals that would warrant further studying of what you mention. Helpfully, our colleague, Geoff, does maintain a table which basically counts the number of IP addresses per capital at that comparing that to the same statistics he provides on IPv6 penetration. I couldn't find anything that would warrant an extensive study to try and ‑‑ maybe I overlooked something.

SPEAKER: I have one more comment from Rob on the chat. He says: Many countries in ‑‑ many countries in Africa are jumping straight to mobile. This may have some relations IPv6 deployment when these mobile operations switch on because they never have wide fixed line/CE deployments. Thank you.

ALAIN DURAND: I would simply make an observation that I have not seen those countries making a big spike into this graph. Maybe next year.

JEN LINKOVA: Thank you very much for such interesting talk.

ALAIN DURAND: Thank you.
(Applause)

JEN LINKOVA: Now, I have news. One bad news and one good news and I will start with the bad one. To keep you in the room. So, our co‑chair is going to step down and is not willing to be re‑elected unfortunately, apparently we have done something, didn't behave very well. So, please join me with thanking Anna for all her hard work. Don't hide, come here please. You are still Chair as of 10 minutes. So thank you very much. It's a shame you don't want to stay but ‑‑ and I think we ‑‑ wave small present I think for you so you will remember us for the rest of your life.

BENEDIKT STOCKEBRAND: Sit back, have a glass of wine and enjoy the show. Thank you very much.
(Applause)

For those of you who don't remember how we got into this situation, the three of us, like what was it two‑and‑a‑half years ago, because Anna had way more experience with Working Group chairing which really got Jen and myself up to speed getting all this sorted out, so there was really, really helpful. Thank you again.

ANNA WILSON: Thank you both, you are far too kind, it's not because I don't like you or the group, it is a wonderful experience to chair a Working Group. I recommend it to anyone in this room and it's bben a pleasure particularly to chair with you both because do you all the hard work and you don't need me, but thank you so much, it's a wonderful opportunity and I look forward to seeing you at future RIPE meetings and commenting from down there. Thank you.
(Applause)

JEN LINKOVA: I promised you good news. The good news is we have a willing Chair, form, only one so we don't need the problem of how to select one. It's very easy, so please welcome Raymond, who was so brave and kind to help us with chairing this Working Group.

Raymond: Good afternoon colleagues and friends in the room here and everyone else listening somewhere. I would like to thank, first of all, for all the support I have got by the list and by, in the meeting here and everywhere else. A little bit about myself, I work for an ISP in Finland, we are doing our best trying to be something with v6 but we are not yet ranked very high. This is not my first RIPE meeting, I think it's almost 20th I am in so I have been around for a little while and I know most of you in this room, I have at least seen you or talked to you. The reason why I actually want to do this thing is, like, actually this morning in Address Policy was already mentioned that there are some views of some people that v4 is actually a thing that is dying out. We have known this for 20 years. We fully denied it and we still continue fighting about the last small pieces of crumble of v6 that is still there. I understand why, it's not easy to change your habits and I think that I have some plans on maybe doing something on this Working Group to change the attitude on this. I don't know how, I don't know when and I don't know what, yet, but trust me, I will find something.

So, that is basically the whole idea. And this is actually the introduction I wanted to give to you. Thank you.

JEN LINKOVA: Don't go away, I need to give you something, we need to change your badge, yellow colour. Welcome to the club. And this, I think, concludes the session, unless someone has anything else to say? Okay. Thank you, please rate the talks, please come back tomorrow at 5 p.m. for more interesting talks. See you tomorrow, guys.
(Applause)

LIVE CAPTIONING BY AOIFE DOWNES RPR

DOYLE COURT REPORTERS LTD, DUBLIN IRELAND.

WWW.DCR.IE