IPv6 Working Group
27 October 2016
5 p.m.
JEN LINKOVA: Good afternoon. Please take your seat, fasten seat belts, turn off electronic devices might also be a good idea.
So, welcome to IPv6 Working Group, second slot. We have quite a busy agenda. So let's try to be on time. By the way, before we continue, I'm sorry, I know everyone has heard it many times this week, but, when presentation is going on, microphones are not active. After the presentation, we'll ask for questions. If you have a question, please raise your hand. Don't press any buttons, just raise your hand. When the chairs is asking you to speak, then you press a button and ask your questions. Please do not press a button on the microphone when someone else is speaking because you will cut the person off and it's very impolite as we know. So if you accidentally do, just apologise.
And I would also suggest that if you think you are going to ask questions, could you please sit a little bit closer to us, so we can see you when you raise your hand, because I think it might be a bit harder to see you guys on the back.
And please rate talks, so we could know what you like and what you don't like and we will build agenda for the next meeting accordingly. So, the first presentation of the today's session is Marco from Microsoft will tell us about IPv6 only depoloyment in Microsoft.
.
MR. KEANE: Good afternoon eye name is Marcus Keane. I work in the IT group in Microsoft and we look after the infrastructure and network for the corporate network.
I'm based in London but originally from Dublin. So, thank you for having me here.
So, this is what we're going to talk about today, we're going to have a brief overview of the network, our IPv6 dual stack status, talk about our move to IPv6 only and the reasons for that. We'll also discuss some of the problems we haven encountered, in protocol problems and design problems. And finally, we'll conclude with some of the future directions, where we think we want to push or deployments and where we think things might go.
So, in terms of the network, it's fairly large. It's kind of divided up into two regions, mainly, so the Redmond campus in the US, and the regions. So the regions, network America, EMEA and Asia sort of all a single BGP BIS. We have our on on premises data centre, and most of our tail site connectivity is through carrier MPLS. We sort of talk about that in a little bit later on because it has ramifications for some of our design issues.
So, Microsoft has been doing IPv6 in some capacity really since 1993. Really I guess it was since 2006 that we have sort of done it in earnest, and I guess that was coincided with the deployment of Windows Vista when that came out and ran in beta. We did it with a combination of ICE owe tab and dual stack native, to some selection locations and the intention was to facilitate sort of testing for the developers and so forth at the same time, we enabled peering, Internet peering in all three regions and obviously in Redmond. Most of our peering is with AS 8075. AS 8075 is the AS supporting the online services in Microsoft, so, Microsoft .com, Bing etc. In terms of what is actually deployed, all new networks today, as are deployed, they are deployed with IPv6 and that's reasonably straightforward over the years a lot of networks were deployed through expansions and so forth and they didn't get IPv6 deployed, so there was a recent effort to retro fit those networks with IPv6 dual stack. But there is still a residual amount of networks in terms of labs and co‑managed networks. So, that will take, I think quite a little bit longer, a bit longer than some of the corporate networks that we covered recently purely because it's very difficult to voice to IPv6 on labs and co‑managed networks because you are not quite sure what you are going to break.
So that brings us really to IPv6 only and why. I mean, the IPv4 deletion story is well known so I'm not going to rehash it except to say that we are subject to the same limits and we have had quite a bit of public v4 space, but we're out of that. So initially, over the last few years we even had some public space on the corporate network interimey but behind the NATs, the firewalls etc., but we are surrendered that for online services. So the corporate network now is completely addressed out of 1918 space.
So the big problem we have is that we're pretty much out of 1918 space, which ‑‑ so we can see that sort of of coming over the horizon fairly soon and it's really just down to expansions and I guess, a related problem is overlapping 1918 space, so, we do have overlaps with Azure, and as you are aware, there have been some acquisitions in the last couple of years and they have had their own installed base of 1918 space, as we merged the two networks together you have to talk and you have overlapping 1918 space. That's become strategicey to say the least. Operationally, it's very complex. Certainly there has a potential of being fragile.
So, I guess related to that is just generally dealing with the complexity of dual stack. In terms of you know help desk calls, network support, dealing with two stacks is expensive and difficult, so, we would like to sort of reduce that and get rid of that issue whenever we can. So because of that, I think we have a strategic goal to get as much as of the network on to IPv6 only as possible. And our management is write up to the top it's very supportive in that goal, so that's pretty good for us as engineers.
IPv6 only, we started a couple of years ago, there was a testbed in an IT building in Seattle and really, it was sort of an opt‑in network, we had a lock at the wireless guest network and on the Corp network wired and wireless, and really it was just sort of testing, or enabling a couple of ports, so for people like engineers and even IG M could experience IPv6 only and really kind of see what worked and what didn't work. So that was you know reasonably successful.
We used a non redundant NAT 64 and DNS 64, and you know had sort of equipment in the lab to sort of support that. Because we really we were just experimenting we decided we were going to play around with different address acquisition schemes, and we sort of had a look at the SLAAC and DHCP stateless. For the guest network we were obliged to to DHCP stateless and RDNSS and we'll have a look at that later on because that led to some design issues for us.
So, the next stage, once we had sort of bedded down the approved the concept ID building in Redmond, we thought we need expand this and get more exposure to other customers for start and other networks. So we decided we needed to target some production networks. So, the initial focus was on wireless guest. We sided okay we wanted to target some production networks, the obvious question is which networks are you going to target and where? So the wireless guest one was an obvious choice for us because purely the application profiles are a lot simpler, it's simply a case of you know, if you are going to do e‑mail you are going to do web and so forth. A secondary reason and probably no less important is the fact that for the wireless guest network, the expected level of SLAs is less than it would be forth corporate network. So if we are to experiment and we knew this was, this would be potentially bumpy, there'll be a less of an issue if there would be for the safety for the corporate network. And obviously with the corporate network you have got quite a different array of different applications that you needed to support. So, probably I think, and the reason ‑‑ for that reason we decided the wireless guest was good because we could move faster. We wanted to get that exposure quickly.
I mentioned that the expected level of SLA or service level was lower, but the wireless guest network today for v4 is fully redundant, so that means ‑‑ you know, redundant default gateways, redundant obviously routers, firewalls, etc. So, we made a conscious decision that whatever we did with the v6 only wireless guest we had to replicate that redundancy. So, to that end, we ensured that the design had continue with VRP and we had redundant DNS 64 and NAT 64.
That takes care of what and how, and the next question was where? We decided we would move, expand within the IT buildings where we were and actually move to sort of some production networks and then secondly, which was more problematic, was moving to other buildings in Redmond campus.
And I'll go into the reasons why that was slightly more problematic in a second.
So, this is our problem. This is our main problem that has driven a lot of the design decisions. As you are aware, there are a couple of mechanisms for acquiring DNS resolver information in IPv6. You know, you can either do RAs with RDNSS or you can do DHCPv6. As you are probably also aware there is a buy forcation in support for both of these schemes with Windows only support DHCPv6 and android only supporting or DNS. Clearly as we targeted the guest network, we had to support all disparate types of platforms and Windows
TORE ANDERSON: An dried were one of those, so we had to basically support rDNS and DHCPv6. So that end we deployed a parallel DHCPv6 infrastructure, when I say parallel, parallel to the corporate one, and then we also needed to support rDNS S on the routers. Within the IT buildings in Redmond, that was easy enough. Because the routers that we had there, we knew supported rDNS S already because the existing testing had shown that. The problem that we had encountered was a lot of newer routers that we had that we had in Redmond campus as building routers, they were very, very big route reflector boxes but they didn't support RDNSS, so we engaged the vendor and the vendor ‑‑ they have running code that supports RDNSS. The problem with that is it wasn't going to be available to us in the time frame that we wanted. So we decided we needed to come up with something else, otherwise just wait.
So, solution was that if we figured ‑‑ if the local default gateways do not support RDNSS, well why not just move the default gateway closer towards where the peering is and where the egress is. So, to that end, we built a layer 2‑VPN overlay using EVPN and effectively moved the default gateway northbound to support this. And then on the centralised pair of routers we support RDNSS and JCB 6 and VRDP. So it's much easier to look at the ‑‑ you can see this, but the green things at the bottom are the wireless controllers and they trunk into the redundant switches, who in turn trunk up to the are you doesn't building routers, this is down the bottom here. Now, we're imposing a Q and Q tag here, and the reason for that is that typically ‑‑ well actually, in all the buildings, the standard for wireless guest access is using a standard VLAN. So as we were creating this overlay and sort of merging it up here, we didn't want to create this huge big layer 2 domain. So for that end basically we thought we'd have a unique tag imposed per building here and then we just run it up here and these routers up here are doing Q in Q on the sub interfaces. Obviously they are doing RDNSS and DHCPv6 relay and VRP. The obvious question S this is a little unconvention for sure but we figured for ‑‑ I mean forth actual application it wasn't bad, because the traffic profile we expected was going to be pretty much north /south. If there's any peer‑to‑peer traffic, it would be contained within the building, or within these controllers, we weren't expecting any, very much peer‑to‑peer traffic between buildings. So we weren't terribly worried about tromboning. Obviously for the corporate network that's going to be slightly different.
I mean, it works pretty well. We're, we have got a couple of issues to look at in terms of convergence. But the next question is why did he with choose EVPN? Because it's nice. But I mean really, in honestly we can do single active here and we don't need to play any tricks with span entry to get the blocking that we needed, so really we can use EVPN using the same ES I here, it's doing blocking on both sides, so it's nice single active and we get the full redundancy we wanted.
So, there are some issues still. So, as I mentioned, most of the WAN connectivity is carry MPLS. What we have is we have an our doing e.g. P with the carrier, we are MPLS network and it traffic gets dumped out. The problem is obviously that we don't have any MPLS in the middle, our or own MPLS so we can't do MPLS VPN. So we have had a look at it and see what can we do? If the hardware support it there may be a possibility of doing something with VXLAN. The question remains is what do you do in terms of control plane? We can't do EBGP EVPN with the carries, we have to do some BGP multi‑hop. These are the kind of things we're going to have to look at. The other thing is probably more important we have quite a large installed basis of older land routers that will never support RDNSS, the vendors told us that they will not get RDNSS, so we need some other solution for that.
Now I mean the question is okay, does this model lend itself to the sent rised default gateway model? And the answer is no not really, because typically, these, you might have a building attached to the MPLS backbone, and then there is nothing really to go upstream. So really, the only possibility is really going to be something with suit owe wires because they don't support VPLS or EVPN, we're stuck there.
The only problem is, and you know, I'm looking around at how we might do this, is, with some pseudo‑wire redundancy, that's tricky particularly with some of the.platforms that we're looking at. That's a work in progress, but we need to do something because we have got this quite big install base.
So, the existing status. The equipment we have in Redmond currently is basically lab equipment. So it's not full production, but we have purchased production equipment which is much beefier and it's in the process of being installed now. So, that will take care of Redmond. And then for outside of Redmond we want to start replicating this, and again the decision we decided we were going to do it in Europe first for no other reason that basically I am based in Europe so if there is any issues I can work on them in the same time zone.
So, we will be deploying sort of of a NAT 64, DNS 64 in Europe in Dublin and in London.
So, in terms of centralised default gateway. We will do probably at the moment we're thinking maybe one per region, but that works well for Europe because the geography is nice and exact. It might become more compact in North America and even more so in Asia, with the stretch of geography it a little bit more challenging. Particularly you have got regions like sort of China and India that need to be locally survivable. Probably you need to have multiple installations of this.
So the future plans:
As I mentioned, we are going to deploy the redundant NAT 64, the DNS 64 to other regions. Part of the things we want to look at is get a feeling for how much the traffic is going to be. Because really we are basically making, you know, estimates based on the level of traffic we know going through the firewalls and we say okay. If we can take a certain percentage of that, we'll get a better idea once we start moving more networks to v6 only as to exactly how much capacity we will need on the NAT 64s, that will feed as input into the decision‑making process then for the other regions, North America and Asia.
Again, for the same reason, the centralised default gateway solution is probably, you are going to have a lot more instances that we might have in Europe and our experience in Europe will help us with that decision.
I think probably once we start get going and we distort of make it into production offering, I think we'll probably be able to move quickly. So the next obvious thing is, well, what about other networks? And the corporate network is one that we'd like to target. So, again, we sort of you run through the same sort of decision process and say well, what kind of networks are you going to target on the corporate network? Where? And how? So, I think for some networks it's reasonably straightforward so if you have HR, finance or whatever, all in the same segment you can say great I can target that segment and make it v6 only because the application requirements in principle should be reasonably straightforward. The problem starts when you get other types of customers such as product support, or developers, that they may not like you removing v4 and they may have good valid business reasons to have v4 on that segment, so you can't really unilaterally remove v4.
The other thing is, particularly if you look at say the building we're in in London, there is quite a mixture of different users all in the same sessionment so you might have someone from finance in front of you and a developer behind you. It's very difficult to say you know what's going on in that segment in terms of I can definitely go v6 only or not. So, it's more problematic than if you did do it there is a possibility that you are going to break someone and you are going to get some complaints. I think we are going to have to deal with that.
The final thing I think is the ‑‑ the corporate networks. Again I say Redmond and Europe will be again the first targets I think for the same reasons I mentioned before.
Regarding our Internet first strategy. This is kind of, and I'm sure you're aware, that a lot of companies now are looking to sort of off load for small tail sites off load Internet traffic straight away rather than back hauling it to a central location. Economically that makes a lot of sense in a lot of locations, particularly in some parts of the world where connectivity is quite expensive.
Needless to say that obviously that brings with it other challenges, and I think one of those ‑‑ obviously if you are going to have local peering with the provider in some capacity, you are also going to have your firewall, but obviously we're going to have NAT 64, we are wise there's no point, we can't be back hauling it back to a central location. That kind of sort of I guess lends itself to a service training, sort of N F v scenario. We are kind of looking at that figuring out we can do that then for small tail sites.
I think probably it should be doable.
I think I finally just sort of of conclude that, I think we're really kind of feeling as we go, because this is all very new for us. But I think it's definitely worthwhile from a strategic point of view, it's difficult to say you know because there's no immediately obvious return on the Capex and opex, but I think it's definitely in terms of experience gained and the fact is that when we do run out of space, which we will, then we will be well situated to deal with that and I don't think, we won't be panicking as would be the case if we hadn't engaged in this. So over you'll I think it's been a very worthwhile project to engage in. I'd certainly encourage everyone else to look into it.
That's really all I had for it you. If you have any questions, I'm happy to take them.
JEN LINKOVA: Thank you Marcus. Questions?
AUDIENCE SPEAKER: Natalie Treneman, RIPE NCC. I have a question. You briefly mentioned Capex opex, I ask this question always to everybody to did IPv6 deployment. What was the part of the IPv6 deployment where you had to spend money on?
MR. KEANE: DNS 4 and NAT 64.
AUDIENCE SPEAKER: On equipment?
MR. KEANE: Yeah. So, obviously the opex part is going to be a training element to it and support and so forth. But I think in the grand scheme of thing it's probably not an awful lot. It's expensive don't get me wrong, but it's worth it.
AUDIENCE SPEAKER: We have been quite successful on mobile phones with turning IPv4 through IPv6646 X slot or whatever it's called. Does Microsoft have plan to implement the software for the office PCs where we can then implement IPv6 only in the network and tunnel IPv4 through it? This would ease our burden in deploying IPv6 in office LANs where we have this flat of applications a few hundred in larger enterprises.
.
MR. KEANE: The question is are we ‑‑ would we enable a C lat is that what you are saying which would do translation or is it actually tunnelling?
AUDIENCE SPEAKER: I would prefer tunnelling, because if you translate from IPv4 to IPv6 you always lose some information in the header. But it doesn't really matter, it should be something, a simulation of IPv6, IPv4, on the PC and IPv6 natively on the LAN but IPv6 only, so we don't have dual stack environment in our LAN segments because we want to do new network design there, we want to do micro segmentation in the LAN, we have enough addresses today we have big LANs and it is quite difficult to my grade in a dual stack environment and we have many many old applications that will never ever get IPv6. So, having some tunnelling mechanism implemented by Microsoft. Tested by Microsoft and then ready for production would be really nice and you could be the first testbed, couldn't you?
.
MR. KEANE: I'm not aware of any plans to build a C lat in the OS, but I see what you're saying, I guess, but there is the argument, I mean I guess it's sort of analogous to to the happy eyeballs, are you suppressing information? You are hiding the failure effectively. I mean... a good question, a good question.
AUDIENCE SPEAKER: It's not a questions, it's actually contributing to the answer. Jordi pallet. Actually the C lat client, it's not a tunnelling, it's a translation, and the right place to deploy it will be the wire LAN environment like residential for offices, the CPE and in fact in the RFCs describe it that way. The the a different situation but it's not meant in principle to be deployed in every PC.
MR. KEANE: I guess you can emulate it you know if you ‑‑ I guess in principle if you put the same 1918 space and then ‑‑ again, you'd have to put your 46 if you think about it in a campus, you'd have to put it in a hub location. I think it would be hard toing manage. That's just me.
AUDIENCE SPEAKER: Just a question about how you interact with your own development teams, so you have built a huge testbed for IPv6 in enterprises and you use your own software there. So, do you work well with them to fix issues you might find or did you find any? And just, are there any plans to use your experience in this deployment to generate educational material for your customers?
.
MR. KEANE: Good question. I'll take the first question first. Yeah, we do, we have an active conversation with the product groups. Because I think when all different types of product groups, I think because we're finding stuff out and we need clarification and stuff and we feed problems back, some bugs that we found. And I think generally I think they enjoy and they learn from us and we can get a better understanding of what the product is doing from them. I think it's been pretty product I have.
Regarding the second question, we hadn't any plans. I think we're happy to talk about it as, in sort of environments like this. But, that's not a bad idea. I think probably people could benefit from some experience.
JEN LINKOVA: Any more questions? No. Thank you very much Marcus.
(Applause)
So, yesterday we were discussing ‑‑ yesterday we had a great presentation from Geoff Huston about DNS on IPv6, and actually I was thinking yesterday we have another very nice protocol widely use and we apparently do not know, at least I don't know, what the situation with IPv6 and IPv6 only... so, fortunately Andre kindly offered a talk on exactly this topic. To please.
ONDREJ CALETKA: I am from CESNET, it has been founded 20 years ago, so that's why there is a 20 on every slide. And today, I would like to tell you something about one of my personal contributions back to the world IPv6 launch day which happened on 6th June 2012. It was actually ‑‑ well, it started with a domain name called ‑‑ it means doesn't work. So, I had this domain name of doesn't work, so I thought okay it would be cool actually to have a website that would not work for many users because it would be operated IPv6 only. So I made this IPv6 only website just by publishing only AAAA records for this domain name. It was one of approximately dot CZ domains publishing such records. It's still really a lightweight static website with some CS S tricks to show you whether you are capable of IPv4, whether you are browser prefers IPv4 or picks IPv6 and so on. And it actually was quite successful. So after the launch, I analysed this web page at the IPv6 day conference in Prague, and there was a contact e‑mail address on the website which was using the same domain name which meant that the domain name was IPv6 only, and most people trying to contact us as authors had actually got their mail bounced and they have to use some other means of contacting us.
So, I said okay, it's worth making an automated test machine, so I deployed something that I call‑out reply mail checker test at does not work EU. It does not mean that I have something against the EU, remember it was 2012, so it was not meant like that, it was just like funny, an unregistered domain, I actually quite like the EU for the record.
And I also add no, ma'am reverse record checks and as I said the English version is available on this website.
So, the thing S how to do an e‑mail checker. There are actually many abilities that you should check, you should test and those are quite unrelated, so the first one is whether to deliver ‑‑ whether somebody can deliver mail to v6 only destination, an IPv6 only domain name. This is probably the simplest way to test. You just create a mailbox where you just publish AAAA records and the network will do everything for you. The thing is, there is a difference whether it's IPv6 only destination like the domain name or whether it's actually IPv6 protocol is used because this is somehow automatic, not completely, something a little bit difference. But the difference is if you can send mail, whether you can also receive IPv6 only envelope address and IPv6 only end body from address, you will see it's actually not that simple.
So, what my solution to this currently deployed and feel free to try it by sending a blank mail to test at does not work dot EU. Is that if you succeed with the sending those e‑mails to this IPv6 only destination, you will get actually two responses. One respond will get you through dual stack transport from an IPv4 only domain. So, because this is what I thought it's like most ‑‑ the probability that you will get this mail is quite high. And the other response you will receive if you are lucky, is a response from IPv6 only domain, the domain doesn't work dot EU, with used only IPv6 transport. So, most ‑‑ or some you will not be that lucky. In this case, there is another robot that actually passes delivery status notifications from the mail server that it's IPv6 only. In that case it got some undifferable notice that this notification. This notification is actually forwarded back to you using IPv4. So you get the reason why we were not able to send you the response.
So as I said feel free to try it and while you are trying it, I will tell you what's behind the scenes. It's quite simple.
The core is Postfix mail server, which has this ability to have multiple personalities, so here I use two personalities, one it dual stacked and used for regular mail transferring of everything. And the other one is actually IPv6 only. And those personalities are selected by different envelope address of originating mail. Then there is brook mail, which you call some external script depending on incoming mail and this brook mail call some custom script I wrote years ago and this is all this logic of auto replying over both protocols. So it's quite simple.
And this is, if you are not that lucky, so this is how the response, how the automatic response looks like. You will see, you get a generate a text saying sorry, although you were able to send a members of the jury to IPv6 only address ‑‑ because I said, you can only trigger these responding in the machine by actually delivering mail to IPv6 only destination. So if you are not able to deliver to IPv6 destination only because you will not get anything because I cannot tell that you were trying something.
So, if you were successful in sending members of the jury to IPv6 only address but there is a problem delivering reply from that address. And it also attaches the exact descriptions from the delivery status notification so just here the most typical example is that host or domain name not found. Name or service error for some MX record name because there is no AAAA record for such MX server, even though you can send e‑mail to IPv6 only destination cannot receive e‑mail from IPv6 only destination because your DNS likes AAAA records for MX.
During the time I was ‑‑ this project is going, I was debugging some weird issue with some random e‑mail providers and one thing I was told, it's maybe could be problem is actually difference between two standardising SMTP. The first one is proposed standard from 2001, RFC 2821 and it's defines everything about S over IPv6, like over IPv6 used in SMTP communications. But for the case you are delivering to the server ‑‑ to a domain name or destination name that has no MX records, there is just explicit statement that only A RR should be treated as if it was as I say an implicit M RR. So there is nothing about AAAA records, so this is probably something left over from the previous specification that was IPv4 only. This has been fixed in the newer standard RFC 5321 which is draft standard from 2008, and this one is saying it more generalically that if the list of MXes is returned the address is treated as if it was associated with an implicit MX RR with a preference 0 pointing to that host. Why I'm talking about this is that the first time there was no MX record, the web server that serves the web page also served as the mail server. So, I thought it's not necessary, and what was quite funny was how g‑mail reacted to this. Because when there was no MX, the g‑mail bounced immediately every attempt to send their mail with saying that there is no MX record. So, what I did was that I added a self referring MX record, which is referring to itself with priority 0, which should be equivalent, according to the newest RFC. But this that case Google started to hold the message in queue for three days and every 24 hours it bounce with failure that DNS server returned answer with no data. It was like temporary error to is tried Tor three days and after three days it gave up and bounced the mail. Fortunately this has been fixed around August 2013, so since then, g‑mail is completely IPv6 only capable.
On the other hand, by the time g‑mail started receiving e‑mails over IPv6 but they will not receive it without reverse DNS record.
That means that you can have a problem if you have for instance broadband connection where you want to deploy SMTP server and this is how I find out there was as much thing as personalities, because in the link, your block was, or mailing list there is a solution how to deliver to g‑mail over only IPv4 to work around this issue.
I have some stats for the operation of this mail responded err. So first stats I made about four months after launching this service. Unfortunately, unlike Geoff Huston for instance, I don't have any information about delivery attempts, because they are only in DNS and I don't do, don't own all DNS servers hosting the zone. By the way DNS is dual stacked so I don't check the DNS IPv6 only ability. But, nevertheless, I received about 100 messages from different domains. By the time there was automatic check of returned path, but I did a manual check to find out the results are actually quite bad, because I actually found 71 different IPv6 addresses of mixed servers and from the 71 only 50 actually accepted TCP connection to port 25.
So, the problem here was that most Linux distribution shipped mail transfer agents to IPv4 only so people, even those who deployed IPv6 to their servers, they actually missed out that the MTA is listening only on IPv4 circuit. Fortunately this has been fixed and by the time also I haven't found any free mail service that would be able to use v6 only e‑mail, even though some of them was already IPv6 ready.
I did a new statistic right now, there is a little gap because I changed the system how it worked so I don't have exact data in between 2012 and 2013. So, in this year end something, I still don't have any information about delivery attempts but I got 303 test messages, and I must say that it got much better, because only 39 bounces, so, which means that somebody who was successful in delivering to IPv6, was not able to receive the message over IPv6. 34 of them was this what I already showed you, no AAAA record for MX. The five others was somehow different errors that could be or could not be related to IPv6. The problem with service not listening on IPv6 is obviously gone because new line distribution is all right set up the MTAs correctly.
So this is ‑‑ now I have an example of really strange server behaviour. This server is IPv6 enabled, so, I have this example of Telianet session that I can connect via IPv6, it doesn't show but the connection is over IPv6. The server is responding, I am saying hello. It greets me with my IPv6 address, you see I'm ‑‑ this is from IPv6. But then I try to send mail from IPv6 only domain name, it will tell me that the sender's pest MX has no IP, please contact the network administrator for assistance. You will see even though the server is capable of receiving over IPv6, it doesn't handle properly a domain name that is IPv6 only.
So, this is probably everything about the mail service and I have two more slides about other aspects of operating IPv6 only websites. The first thing was, yes, we all like DNS and we like to encrypt everything. The problem is where to get certificate for IPv6 only website. By the time in 2012 the only free option for certificate was start SSL or € sign, both of them are IPv6 only, both of them are dead because since last week the certificates wouldn't be trusted in Firefox next year. What I did was I temporarily deployed an IPv4 MX to get a validation e‑mail so I could get a certificate, and I repeated this every year.
Then there was Let's Encrypt this new project, it launched late 2015, even though it was in 2015, it was IPv4 only. So it was quite problematic. Almost impossible, but it actually deployed DNS based validation in 2016, in January. So, by the time I was able to use DNS and the full support is there already since July.
So for this websites like it doesn't work, I deliberately have no redirection to HTPS. Because you sometimes you get different results. This provide side effects. So even though your mobile data connection is IPv4 only you can actually reach IPv6 enabled websites by enabling this accelerator because Google proxy will do the IPv6 gateway for you. But it only possible, for http not for http S. The other thing is quite helpful to have IPv6 only website is that you can check your websites. This site is telling you if the site supply or down. It says the site is down. On the other hand, the test of Dane works perfectly with IPv6 only since the first time I tried. SSL labs famous web also has added support. What is actually quite not that funny is that there is S T S pre‑load web page for loading Chrome browser, it's operated on the Cloud. This is prefix but the Google included doesn't support IPv6. Really!
So, my other domain which is this fancy one, is actually applied manually. The same problem with browser extensions called http S everywhere, which has Travis CI which doesn't support IPv6 only so the requests fails in this continuous integration due to this problem. The same brad new browser which was launched two months ago is IPv4 only and only partially fixed to support IPv6. Again, due to some limitation in Amazon services or I don't know where.
That's basically everything from me. And if you have got any questions, I hope I will be able to answer them.
AUDIENCE SPEAKER: Jan Zorz: I think this is brilliant. Thank you for doing this service, and while you were speaking, I tested it, and I was watching at the log files on my mail server and I saw that there is some complex stuff going behind because you're testing on both protocols. And the most amazing thing is that not even the grey listing did confuse your system. So, it reported correctly back. Well done. Thank you.
ONDREJ CALETKA: The greyest thing was the case why use the real dot MTA for delivering it because you have it already. If I programme my own MTA I would have to deal with the various things and stuff if you use Postfix you just have it already done by somebody else.
AUDIENCE SPEAKER: Hello. Do you gather data that would allow to make the future RIPE meeting a presentation about how different anti‑spam system behave if you can identify them or what do anti‑spam system, how do they react when sending IPv6 only. You had the first ‑‑ it's not actually anti‑spam, it's just e‑mail validation, you had the first step done. Do you have data that would allow you to go even further?
ONDREJ CALETKA: Well, I'm not sure whether I can somehow automatically make this data, because what I showed you like here was, really, I discovered this manually by looking into logs, so it was quite problematic to do it like properly without spending lots of time or looking at every single e‑mail. But I can try to do something about it.
JEN LINKOVA: More questions? No. Thank you very much.
(Applause)
So, our next presentation is from Carlos who is going to tell us about IPv6 at RCTS.
CARLOS FRIACAS: Hi everyone. It's really true that cannot see anyone from here.
So, my name is Carlos, I work for the Portuguese Enron. Lately I have moved into security, but I still have a great interest in IPv6 deployment.
So, this is not a technical presentation. This is just a small status presentation about how are network and the v6 part of it.
So this is a small description. We also provide advanced connectivity services for our members. Fibre, and VPNs which we brand as RCTS plus.
This is currently our map. So, covering all the country also with inter‑connections also to the Spanish research and education network, the European education network and also the local internet exchange.
So about IPv6, we have been doing it for white a while. We did start even in the last century, Millennium, whatever, so we are also serving not only universities, but also schools, and just since kindergarten and so on. We think we have a very stable IPv6 connectivity. And an important detail about our setup is that we are hosting CGN cache nodes. So this is where we have identified that we see an important part of our IPv6 traffic coming from.
Some numbers. We currently have 77 members. Mostly are universities. Unfortunately, not all universities are and not all members have IPv6 connectivity enabled. They could have done so already, but I guess, they still have some other priorities.
I also went to the trouble of looking at traffic data. And some of the members already have traffic peaks above 100 megs, and there is only one which has a sustained traffic of above 100 megs of IPv6 during a full day.
So, this is also some details about our addressing plan, because we were one of the first networks to get addressing, we started off with a /35, then we moved to a /32 and we happily evolved into a /29 like most of the other people right now. So we do assign /48s. We still ‑‑ well, besides, we are academic, we also have this merger ‑‑ we also go through this, we will see inside our membership, there is also some merger phenomenon, so some universities, I think mostly due to budget restriction have been merging. So, when that happens, we don't take the /48 back from any of those organisations merging, we just let them keep all of the v6 space.
So, this is the topology two slides are examples of our traffic in an overall fashion. So, with a peak of ‑‑ well that is from early June before the vacation, so, during the vacation, typically on an Enron, the traffic goes down a bit because people go away, so we have v6 above 1 gig, which is they think fairly okay. 1 gig versus a bit more than 10 gig for v4 traffic. So, this next slide just shows that in mid‑October, the traffic levels were pretty much the same, just small changes but nothing that we can say some university has just started doing v6. So... and there was practically no change.
My last slide is about services. So, this is ultimately why we are doing v6, is that we want to provide services and v6 in fact impacts almost all of our services, but I would just like to speak about four of them.
On the bottom right corner you see speed metre. So this is a tool that we have available for public to perform measurements through a browser. This has worked since its first version on v4 and v6. We have also EduRoam, which is which is the way people roam between organisations inside our network but also inside our countries academic networks, so this basically depends on having v6 on this service basically depends on universities deploying v6. We also have one of our big projects in terms of budget is the online knowledge library, which is called B on, and it's funny, because sometimes we know that people are using v6 because they report problems about some v6 prefixes that are not pre‑known by this service database, so there is still ‑‑ I would say an archaic way of accessing this service and it's really based on source IP and not only source IPv4, but also by source IPv6 sometimes.
Last but not least, the security certs RCTS is our security team which for my surprise is not still with, around 10% v6 traffic, doesn't see too much issues, too much incidents, too much events related to v6. But I seriously expect that will change in the near future.
So that concludes my presentation. I'm happy to take questions.
JEN LINKOVA: Questions? No. Okay. Thank you Carlos.
(Applause)
Now, I think one of the main problems with deploying IPv6 is people sometimes think that just to do whatever they have been doing with v4 and just configure longer addresses, but it's a bad idea in general but sometimes they need to configure something and they will configure it in v4 because it didn't exist in v4 and they don't know how to do that, and now Jan is going to tell us about new initiative to help people to do things properly.
JAN ZORZ: Thank you. I forgot how bright the light is here. I just have five minutes apparently, so we talked about this document at best current operational Task Force on Monday, and this was the page that was shown on Monday. Now we got three new quarters. So, Gert Döring and these men joined this document. What is this document about? I travel a lot and talk to operators around the world. And these people ask me two very common questions these days: They say, when we enable IPv6 for our end users, how big should be the prefix delegation, the prefix that we give to users and how should we allocate it to them, delegate it statistically so every time they get the same prefix or we can just do the pool on the BR S or something like this and then they will get the prefixes different each time?
So, these are the two most common questions these days and I got a bit fed up with answering it every time all over, over, again with the same answer. So, I said look why not write a document about this and see how much people agree with what is being said.
So, I basically put this together in two days. I just did the brain dump from what I usually respond to a document, and it began to be six pages document now.
The aim of this document is not to be longer than eight pages so we will have to work wonders.
What is the table of content? It's three parts. First one is size of the end user prefix delegation is /48 /56 or something else. Then we go into the numbering the connection link between our network and end user CPE. These are sort of like different options that we have and why one is better than another.
Then we have prefix delegation optings. These are basically prefix that is used behind the CPE.
And this is like a few options and why one is better than another.
And why this one is not good.
Then we have end user prefix assignment static or dynamic. Why dynamic is easier than static and why stick prefix assignments are recommended.
Then the third point is this is more or less he is a co‑chair of home networking group at the IETF. Here we will explain what is envisioned future of home networking on IPv6. So, why we should choose wisely today and deploy the right solution in the right way, so, because of of this stuff, so, we will not have to go back in a few years time and redo the whole thing, because this costs money.
I have been very, very quick. So, the process, the document is on the Google documents platform because I wrote all my ‑‑ all started Google docs because it's very nice for cooperation.
So, we need, operators with practical experience to quarter or opt‑in as an approve reading at the end and we will put your name and your company into sort of like these operators agrees that this is a good advice and valid advice. And help from IPv6 Working Group, that is you and I'm here now to check the technical validity of the document before ‑‑ the aim of this is that we can publish it as a pick‑up or BCP RIPE document at the end.
Whoever wants to read it, type in this, or press the button or download my presentation and press here. You will end the "read only" mode because it's shared, everybody can read it but cannot edit it. And my question is, do people think this could be a useful document and whoever wants to contribute or say something...
Do we have time for questions?
JEN LINKOVA: We have plenty of time for questions. Natalie?
AUDIENCE SPEAKER: Hi. First of all, thank you so much for picking this up. We get this question a lot in the training courses, it's actually one of the most asked questions in the training courses, and also from Jordi's survey, we see that people sometimes get the weirdest prefix sizes and distributions. So, I think this really important that we get this pick up document out and the sooner the better I think, because people are doing IPv6 deployments and now start making really weird decisions. So thanks for that. And I think we would be happy to help with, when it's finished, sort of, to give it a RIPE number.
JEN LINKOVA: More questions? Was it a question? Jan, I have a question. How would you like to proceed? Would you like to people who would like consult tact you and then when the document is ready for review you send it to v6 Working Group mailing list to get comments?
JAN ZORZ: Yes, that would work. Whoever wants to, you probably have my e‑mail address is there. Send me and e‑mail and we will work it out. I think we need to ‑‑ because people is now deciding on IPv6 and we need to give a good advice so they make a good decision.
JEN LINKOVA: Thank you.
(Applause)
So, it looks like we are doing a great job in deploying IPv6, right, and we have a lot of presentations and by the way, we almost filled the first slot for the next meeting with agenda, so if you would like to present on the next meeting, I would recommend you let us chairs know in advance.
So, the question is: What's the end of the game plan? What do we consider a success? And I think Anna has something to say on this topic and then we will get an open mic discussion.
ANNA WILSON: Thank you Jan, yes, I am back. I have a problem people. Something has been niggling at me for a while now and I didn't know what it was, and I have cracked it. I don't know what I want. And I don't think you know either. Because, I have checked. I have asked. Over the past few days, people who have ‑‑ I have been chatting to some people and every now and again I'll bother someone by going, this v6 thing, what does success actually look like? And the only thing I know is that everyone has a different answer.
Someone said to me, well, when the Alexa 100 is dual stacked, maybe we're done, maybe that's how we can be confident that v6 has taken off and it doesn't need to be helped any more by, you know, by Working Groups and conferences and things like that. Maybe that's when we know that v4 is officially dead.
And maybe that's true or maybe it will take the Alexa 1,000 or ten thousand or million. But that's really about the web isn't it. Is dual‑stacking the web enough? It might be. But I don't know exactly what the next step is after that. Maybe it's enough on its own and maybe everyone else, maybe all the Internet people will go oh my God if Google and Facebook and everything are doing v6, maybe we should too. Maybe not, begin that that bits already happened and they haven't.
I have seen this. Once half of my network traffic is v6, maybe we're done. Maybe half? Maybe it should be all? Maybe you really have to persist until every single possible device on the Internet is running IPv6 dual stacked or no? Do we really have to go that long? Is that ever going to happen? And if it's never going to happen maybe there's something in between that's good enough? What that is. Maybe we should combine these two? Google tells us how much of their at least websites visits are v6? Maybe we need to reach a critical point or turning point there, is 50% the right one? But again that's a very web‑ish. Maybe we should go deeper? Maybe we should look at you know home networks? Mobile phone at least in the US and certain other countries are doing very well. So, you know, maybe we need to get home networks dual stacked and therefore the reason I ask these things in this way is because every single one of these would require very different strategy on how we approach it.
If we're talking about hassling router vendors, that's a very different thing to hassling large website providers which is a very, very different thing to try and work out what we need to put in place for smaller developers and smaller website owners and the infrastructure that they use. Those are three different things that would be require to have three different approaches. Perhaps we need to do them all. But I don't know yet if we do or not.
And I wonder how much of this is because we kind of work off what we know. We have our network at home and we have very happy that we have v6 working on it and so should everyone else. Or do we really need to be looking at some of the other things that are more difficult and harder to think about and will probably take a lot longer to pull off and I don't really know how to do some of these things, particularly enterprise networks? And by the way I tell people these days, university networks are basically enterprise networks, because they use a lot of same hardware and they have a lot of same reliability requirements. There is a very close comparison there.
Or maybe we're going about this the wrong way. We work oft things we can measure and if there's one thing the people in this room can measure, it's traffic. But maybe traffic on the Internet as it is isn't the right answer. One of the of the most interesting things I have noticed over the past while is, I mean since about 2010 people have been talking about these things and how we are in the post‑PC era and stuff like that. And it's one of those practically buzz word terms that no one really notes what it means. I don't want to solve that one here at the same time as this one.
But, if we are heading into a world where there are a 6 digit number of things that we would call PCs which includes all the Macs and Apple I see glowing in front of me right now. But three or four or five or six or seven billion phones, and we get all the phones, is that good enough? Does that actually take the pressure off? And again, I'm very wary of things like focusing on what we know or what we measure as opposed to what might be most effective.
And it seems to me one of the most effective ways to go about solving that problem is if you can convince the major phone vendors, I should say the major operating system, phone operating system vendors to either ensure that all their apps will work in a v6 only network by a technical means or by app store review requirements. And that's already done. Because as I understand it, the latest version of Android have 4 of 4 XLat built in, that would mean if you want to deploy v6 only network and all your clients are Android, you're in business and the app store now has an explicit requirement that it must also work in a NAT 64 situation. That seems like that end of the deal is done. Now, to see the actual traffic change, we have to either wait or find a way to stimulate all the app developers to the back end support that. But is that actually a goal? That something that needs to happen Tor is it enough that the scarcity in mobile networks is probably going away if everyone buys one of these two operating systems and is dealing with apps from those two stores and has that technology in place, is it actually done?
I'm not sure. So I want to ask you guys, we can bring the lights up I'd be very grateful, because I want to stop talking now. I want to ask you, what's your picture here? We're not going to get consensus today but that's okay I'm not trying to write a policy today but the way I think perhaps the best way to think about this and break it down to see what we might do in the future is to think about what is the actual goal we need to achieve here? Is it about web traffic? Is it about IP traffic? Or is it something more business‑ish? Is there any kind of a deadline and when we should try to achieve that? Perhaps there isn't or perhaps there are things that we know if we don't get it done by this date ascertain thing will be possible? And what will be the effect of that? And the reason I ask that is if we got the Alexa top 100 or 1,000 dual stacked, would that persuade anyone else to do the things they need to do? Or would it make it easier for other people to do the things they need to do in the same way as we might think about app store requirements and something else? So I'm going to go down there because I'd like to hear from you and grab a microphone and see what you think. What's your answers to these questions?
AUDIENCE SPEAKER: Whilem. I don't think this is a sports game. Success is not in our case whether we win 3 nil and this is not a religion, getting new supporters, I think success of IPv6 is if we get it into the networks quite smoothly without breaking them and getting the world economy around this cliff so that we have a smooth transition. This is not something about 50% or 60% of traffic, the world economy has to move and we should think in business terms and not in technical terms.
ANNA WILSON: So, do I understand correctly if you are saying let's not worry about when, let's not worry about v4 scarcity at this point because it's a done deal but the important thing is to make the tradition as smooth as possible is that right?
AUDIENCE SPEAKER: I think it is important to have a smooth transition, because otherwise companies companies break and drops are lost, things like this. This is important for people not running 50% or 60% of traffic or Alexa 100 sites. This might be a signal to a few persons or companies to move, but our real goal is to get this new technology in the networks and get the old technology out and not to celebrate on half‑way when some of the companies have introduced it into their networks. I think this is not our success and it's not a sports game.
ANNA WILSON: Thank you.
AUDIENCE SPEAKER: Oscar: I think we are ‑‑ I wouldn't clarify domain when I don't have to care about the IPv4 any more. When support can use come transition technology, they can use the work‑around to which Internet in a way. So I mean, that would be a definite to go, done for me at least.
ANNA WILSON: It's an amazing goal but my question would be, what are the steps to get there?
AUDIENCE SPEAKER: No good answer.
ANNA WILSON: That's a tough one.
AUDIENCE SPEAKER: Hi. Carlos. My simple answer would be the goal is all you listed and way, way more, but the session ends at 6:30, so cannot do this. By when do we need to achieve it? We just need to keep doing it and what is the effect? There is a huge, huge list.
ANNA WILSON: I'm going to take the privilege of having a microphone in front of me right now. I'm not sure I agree with you Carlos on that one because I think focus is very important. I think if we try to do everything, then nothing gets done quickly. And I really have the feeling that ‑‑ I mean my own personal opinion is that the lex a websites are probably not such a big deal, compared to certain things which will make a difference for the Internet how it looks like in 2021 or 2022. I'd actually like to understand what that looks like and if it turns out it's going to be mostly these things, I'd take a very different approach than I would if it's mostly laptops sitting in front of us.
AUDIENCE SPEAKER: Benedikt Stockebrand. I don't think there is any individual kind of goal. I mean, if I put the Working Group Chair hat on, the obvious goal to achieve is that we shut down the IPv6 Working Group and instead start vintage IPv4 burial sort of Working Group maybe, or even shut that down. If I think about software development, when is the time that I don't have to worry about all the trouble that NAT causes to things like zip or whatever. That's quite an achievement. But because I had a hit of head start I had a bit more time to think about this because I knew about this a bit earlier. I think v6 is successful by the time that pretty much everybody who understands anything about anything related to networking simply assumes that when we talk IP, it's IPv6. That's probably the one thing. And that's down to the last end user who has never heard of what an IP address is or whatever but simply it assumes things to work and assuming so, actually assumes implicitly that it's working on IPv6. All these numbers, measurements, that are helpful, they are nice indicators for us but it's really not that much. I mean talking about when is IPv4 dead? U DD P? They are still in use somewhere. So just watching those doesn't really make all that much sense.
SANDER STEFFANN: I think Benedikt said basically what I had in mind as well. What I'm feeling is, what I feel like success is the moment when somebody can no longer viably ignore IPv6. I still see new products coming up that are just IPv4 only, and they sell pretty well and this is still a viable business. The moment when it's no longer viable to have an IPv4 only product is the moment when I think we are so far on the way to success that it no longer matters.
ANNA WILSON: Thanks Sander. I know there is I want to observe, I have now heard two contradictory things because one person is saying the most important thing is the transition is smooth and I would interpret what you are saying Sander as v4 needs to become uneconomic. Infinite time they are probably contradictory things.
AUDIENCE SPEAKER: A website of a bank that I was involved in the project, they actually deployed v6 because the CGNs were giving them too much trouble. So, while it would be cool if we have a smooth transition, the reality is that a lot of organisations will only start doing v6 once v4 starts hurting so it won't be completely smooth.
ANNA WILSON: That's interesting, thank you.
AUDIENCE SPEAKER: Gert Döring. I'm quite happy that Sander didn't steal my point, because I know he is thinking along similar lines. I think the goal is to make IPv4 addresses have no value again. The fact that integers cost money these days is a mistake. By when do we need to achieve it. Geoff Huston tells me we should be there in four and a half years so the big crash is coming, and if cannot make IPv4 a legacy technology by the moment we run totally out, it will not be smooth. So, what's the effect on IPv4 addresses having no longer any value? Basically, you can afford to run a network v6 only. You don't need v4 addresses any more. That would be an effect that I would like to see, that people just build v6 only things, because they know it's going to work.
AUDIENCE SPEAKER: Jan Zorz again. So this IPv6 transition project, if you want to call it, project that is going on for so many years now, is basically a project that consists from three parts. First part is awareness where we talk about how cool IPv6 is and that we should deploy it and things like this. I think some countries, or operators, or content providers came from, came out from awareness phase into a deployment phase and learning phase that is the next part of the project. The last part of the project is business as usual, right. When we know how to run it, we know how to operate it and everything works. I think that the goal is to try to bring the rest of the world, those countries and those operators and those people who are still in awareness phase into a deployment phase. I think awareness phase has been long enough at least. And I think we need to get people to start to deploy as they are, document the best practices, document how to do it properly, and just get the rest out of the awareness phase where I believe it is quite nice being in awareness phase because there you just talk about it, you are paying a lip service. In deployment phase you have to actually deploy it. You know, that sometimes hurts, but let's set this as a goal, to get people out of the awareness phase. And, for example, for operator, the goal should be and success that their users even don't notice when they get the IPv6 address, it should be completely transparent for them, right. Otherwise they may get in trouble. Thank you.
ANNA WILSON: That's actually pretty concrete thank you, because there's enough deployment out there that we should be able to get a good picture of what was it that moved you from awareness to deployment or what's stopping you. When you ask people what's stopping them I think often you don't get the right answer because we don't know necessarily what's actually stopping us yet, but for people who moved there must be a reason that they did. The thing I'd like to explore there really is who are the most effective people to try to move, which parts of the industry do you need to get moving? Are the big websites important, are the small websites important, the AP developers, important, you know, something else.
AUDIENCE SPEAKER: Les, I'd like to go back and ask, is this the right question? What is the goal of the DNS?
ANNA WILSON: I like the sound of this.
AUDIENCE SPEAKER: What is the goal of the DNS Working Group? Wouldn't you say that's practically deployed? So I think there is no single goal. I think everything that everyone has been saying here is a part of the overall question that we need to accomplish, so, one goal will be short‑term and one goal will be long‑term, but it's a step in the right direction, and this Working Group I think is about taking steps in the right direction, and some day we will be in a much better place than we are today, but have we achieved the goal? Probably not. There will still be work to be done.
ANNA WILSON: Life's complicated. I like your thinking.
AUDIENCE SPEAKER: Hello. I don't really agree with Jan ‑‑
AUDIENCE SPEAKER: Okay. Just to take a dimetric for a success. I completely agree with Gert on the objective and what they said. I think the only metric that we should follow it the size of the IPv4 routing table the day it starts to go down is the day when we have success for IPv6.
ANNA WILSON: Getting people to start advertising, that's a big deal.
JEN LINKOVA: Okay. Benedikt and I'm cutting the line because we have two minutes left and I don't want to stay between you and the next BoF.
BENEDIKT STOCKEBRAND: About Wilhelm's idea about doing a painful smooth transition, I think that's pretty much hopeless at this point. That's a very human thing, because when do most people go to the dentist when they can't stand the pain any more. We have got the same situation with the IPv6 these days and I think success, and as well ‑‑ and the third question, what is the effect is, success is when the pain is gone!
JEN LINKOVA: Thank you. One last one.
AUDIENCE SPEAKER: Hello. Ruediger. I don't completely agree with Jan that we should go out of the awareness phase. The awareness phase has to last up to the point where IP means IPv6. From right now most people when you say IP, they understand IPv4. The day when you say IP, it will be IPv6 and everything is by default on IPv6 probably maybe eventually supporting IPv4, maybe? At that point we can get out of the awareness phase and we know that everybody knows that we are done with the transition. Thanks.
ANNA WILSON: Okay. Thank you. We should wrap‑up. That's helped my niggle. Thank you very much. It's clear obviously we all have different goals. And we need to find a way I think to converge those in some product I have way, I don't know yet what that is. I think what I'm going to do is I'm going to take what we have heard in the last 20 minutes, which I think is some really useful stuff. See if there is something we can do with that and I think you are absolutely right, one goal is never going to be enough for a group like this. But let's go from there. Thank you very much.
(Applause)
JEN LINKOVA: Thank you, Anna, for starting this very interesting discussion. So I hope everyone goes a kind of action item to go home and make us closer to the end of the game. And please before you leave, please rate the talks, please, we do need to know what to bring for the next session.
Thank you very much. See you in six months.
(Applause)
LIVE CAPTIONING BY
MARY McKEON, RMR, CRR, CBC
DUBLIN, IRELAND.