Hacking Web Servers to Make Them More Secure and Faster Using Open Standards
6:55PM Jul 28, 2020
How can you be out there promoting open standards like TLS and IPv6, if your own websites don't support these standards? Should there be a step-by-step, recipes, or default configs out there that make this easier? Our next talk is a dive into how the Internet Society tackles these very challenges. Please welcome Dan York with Hacking Web Servers to Make Them More Secure and Faster, Using Open Standards.
Hi, Welcome. Greetings. Thank you for watching today and choosing to come in and talk about this and to be here and be part of this conference. So, I want to begin with a little bit of a story about why I'm here, what we're talking about, and the mess that I found myself in, and the situation that was there. So, I work at the Internet Society; we're a global nonprofit working to bring about a secure and trusted internet for everyone. And I manage a team that manages a whole network of websites, all sorts of sites that perform all sorts of different functions and purposes. Some you may have heard of, some you may have not, but they're around there. We also believe that, you know, open standards are our fundamental building block to have an open Internet, we've got to be there, that's part of what's there. And, you know, this is what's made the internet, whether it's the fundamentals of TCP IP, or whether it's, you know, DNS, WebRTC, SIP, any of the things that are out there, that helped, that have brought us to where we are. There are certainly proprietary protocols, there's lots of other things out there, but these are the protocols that have brought about the internet that we use. So, certainly, we as an organization, you know, we want internet standards to be everywhere. And, including our own sites and services. So, a while ago, a couple years ago actually, I learned about a site called internet.nl and I would quite honestly encourage anyone watching right now to go there. It's just literally internet.nl and if you go there, you can test your site. internet.nl is an initiative, it's supported by the Dutch Internet Standards Platform. We at the Internet Society are also part of that but it's a range of organizations that are there to develop a test platform for basically three things right now: one is, how well does your connection work; a second is, how does your website work; and third is, how does your email system work. And they test for the things you see here, IPv6, DNSSEC, HTTPS, various different options and pieces like that. And I would encourage you again, I just opened up another window do check it out, and you may be surprised by what you see. As a side note, I would mention to you that if you're interested, this project is an open source project, it's on GitHub. The folks there are very interested in getting pull requests or feedback on issues, things like that. And so, I would encourage you to check that out. But coming back to my story. So, about a year and a bit ago, I was running a test across that whole matrix of websites you saw to see how well were these sites really supporting the open standards, you know, if we are an organization that is encouraging the usage of open standard throughout the internet, we ought to be supporting those. Well, when we looked at it, we found that in many cases we were, but in some, we were not. And some of it was known, some of it was a surprise. But all of that was there and so, I had talked to the people who actually operate the servers, a colleague of mine, namely Gregg and our IT team and I, said, hey, can we get these fixed? And he said, sure, I can go off, we can get those fixed up. No problem. Dan, we'll have it, we'll have it for you in a week or so, or something. So, so things went on
And a little bit more. And I was starting to wonder. And Gregg came back and said well, so, about those web servers.
The challenge that Gregg found is something we've seen in the larger space, is that many people, you know, want to support the latest standards, they want to do something in some way. But they don't necessarily know how, nor do they know why. And the challenge that Gregg certainly found was that there's so many sources of information, but some of it is out of date, some of it relates to older things. Some of it is, you know, and there's different ideas around you know what's good. What does a good site look like, how does it work and Gregg also was like, you know, I can't find any reference servers, I just want to do a, you know, I want to do a dig or curl against the server, and be able to see what does it look like, what does good look like, so he could understand what did he have to do to go and make that work. So, as part of this, you know, his basic message back to me and to others within our organization was, you know, we need to make this easier for people running websites. If we want to have open standards out there in a wider way, we need to have it accessible, easy to work with. So, this year we as an organization set up a project that we call the Open Standards Everywhere project, and Gregg and I are both involved in this project to try to move it forward, and to work with it. We have four components to it, really. We've already built and deployed a number of reference servers so people can see. Website operators like Gregg can be able to go and find something that is good and can be able to then go and test it and see what it looks like. We also are then going to be documenting our work, and this is where I'd love some help from the HOPE community, and I'll talk a bit about that. We want to promote that documentation so that others can understand what they can do. And then we also want to lead by example and make sure that our sites are, as supportive of open standards as possible. We are using two test frameworks, actually, one is internet.nl and then another is a site called HTTP2.Pro, which supports the HTTP2 protocol. And it's really, I think, important to look at. These are the protocols we're looking at, you know, IPv6 and DNSSEC, HTTPs or TLS, as we call it, Transport Layer Security, especially TLS 1.3, HSTS, some others like that. If you've gone to that test site at internet.nl and done the test, you'll find that you may not have done well, hopefully you did, but if you didn't, you may see that you're, I would just tell you you're not alone. We did a test earlier of the Alexa top 25 sites, coming off of that and saw a wide range, and very low scores on many of the sites that are out there, and that's true across others. There are sites that score 100% and you can find them on there. Now in this project that we're doing, we had to scope it, we scoped it carefully that we're focused on how do we provide recipes for system administrators, like Gregg, that are focused on the connections to the web server. We were focused on TLS, on HTTP2, on DNSSEC, on IPv6, these things that are working, basically all the way up to, when the data starts getting exchanged between the server and the web browser, everything below that, if you will, if you think about it in a typical network chart, is where we're targeted at. So, we're not looking at things like website, accessibility, or usability performance, although I'll come back to that. There's a very interesting piece around that. We also scoped it to look at kind of three types of servers. We started out with a focus around self-hosted servers, you know, if you're running Apache or NGINX on your own server, whether it's on an actual server, or whether it's a virtual machine, or something like that. We started writing out instructions and recipes for how to do exactly that. And that's, if you go to the GitHub repository, I'll mention in a few minutes, you'll see that we have instructions there for how you can configure this on Apache and NGINX. We also were working on instructions for content delivery networks and CDNs. Now, what's interesting is, as we started to roll this out to our chapters, the Internet Society has 130 Chapters and Special Interest Groups around the world, as we started putting this information out and seeing how our Chapters and Special Interest Groups were, you know, would receive this, what they could do with it. We found that the common comment coming back saying, you know, that's great, but we host our website with, you know, pick your hosting provider, whether it's Wix, or GoDaddy, or WordPress.com, or Squarespace or, you know, pick your provider. And so, they said, you know, it's great that you have these instructions, but ultimately all I can do is maybe configure some settings, or send an email to the support team, or something like that. And so, we said, well you know, we really need to come up with another line of documentation for people at hosted environments. And I should be clear, with the documentation we're not trying to recreate, you know, there's a lot of incredibly great information out there. We're trying to make short, concise recipes that then point to other documentation for people to learn more. I would mention, if you're not working in the world of content delivery networks, it's, I do want to just take a word about that, because we found that a number of people weren't fully familiar with what a content delivery network can do. If you think of a traditional website, all the connections from web browsers go directly to a web server. So, if you make the changes at your web server the changes happen there. But if you use a content delivery network, or CDN,
Then, what happens is, your website, your web server, connects to the CDN, which then loads that content out into all of their edge servers, all around the world. The web browsers actually go and hit the edge servers instead of hitting your original web server. And this is typically done for performance reasons and gets you to a lower latency and all of that, but it also has some interesting aspects where the CDN can actually be providing IPv6, or HTTP2, TLS, DNSSEC etc. Even if your original server does not. I've done this, I have a couple of websites hosted with a hosting provider who has basically told me they will never, they're just, they have no interest in moving to IPv4. They haven't quite said never, but it's low low low on their list. The cost for me to move a couple of these legacy sites to another hosting provider, because it's a completely different system, is a lot, and I haven't felt justified in doing that yet, so, I'm using a Content Delivery Network in front of it, and it's doing all of those changes there. As I mentioned on the slide, some CDNs are free, others have a cost, but they're one way that you can go and help work with us. So, talking back about what the project is doing and where we're looking for help and pieces. We started out, building for reference servers, just in VMs at a hosting provider that we could be able to show what does good look like. All of these are scoring 100% on the internet.nl test, they're Apache and NGINX, with or without a CDN, and working with that. They're publicly available. You can get to them, you can see them out there, and you can connect and see what a good website looks like. There's nothing on them, they're not running, I mean, they're just a, it's just a standard web page that says a little bit about the project, but they're there so that you could use Curl or Dig or something, and be able to see exactly what is, looking at that. Then, our goal, and where we're at in the project right now, is documenting how we set up those servers and pulling together the list of the best documentation in the ways that we can find things. So, right now we're focused on some initial web pages with tutorials, the quick recipes I've talked about. We're also in the process of recording screencasts so we can make those available for people, we're linking to other tools, other resources that are out there. And also providing some materials about why this is important. All of this documentation is available on GitHub, just in a repository there OSE-documentation. Once we get the documentation to a point where it's more complete, we'll move copies of it over onto the main Internet Society website. We'll keep it on GitHub so we can continue to evolve it, but we'll also translate it into French and Spanish. We'll get those pieces out there so people can be able to work with. And this is really where we would welcome help, is in reviewing this documentation, helping us, if you go to GitHub, to the repo, you'll see that we have documents there, you'll see we have a number of issues that are there, many of those are around the hosted documentation, but we're looking to have, you know, to build this out over the next few weeks ahead is what we're at right now, to get this documentation out there. So, we'd love your help, love pull requests, love text. Some of the hosted documentation we're trying to identify which hosting providers provide IPv6, DNSSEC, TLS 1.3, pieces like that, and so we'd actually really appreciate feedback from you all who are here and others that you may know, around, what are hosting providers that provide for instance IPv6, as one type of thing it's there. I would also mention too, that as we work with this, we are going to be translating into French and Spanish but certainly because it's out there on GitHub in certain ways, once we're at a good spot, where we feel it's fairly final, we'd love to have translation help into other languages, ways that it could be available, community translations, that could be available in other places as well.
obviously, then we want to help promote this information, help share this information wider, so that more people can be able to go and make their websites more secure, and actually faster, using some of these new protocols. We're looking to do presentations, like I'm doing here, you know, and being able to interact with people, love to interact with folks, and write articles, participate in, we've participated in several podcasts, we're looking to participate in more. If you know of events in the security community, in the open source community, where this kind of information would be useful, we'd be glad to do it. I can do a presentation such as this, where I'm keeping it fairly high-level, although when we get to the Q&A, I'm glad to dive into the details of any particular protocol, if you'd like to. We've also done tutorial sessions, where we've gone in and walked people through, precisely what do they need to do within an Apache or NGINX server, those kinds of things. So, we'd love to interact with anybody from the community, if you've got ideas, or places where it would be good to have a presentation such as this, or a more detailed dive into some of this, you can just contact me at York@isoc.org and I'd be glad to speak with you more. On the promotion side, what's interesting is, for the first time, in a long time, there's actually some interesting business cases that people can make for moving to some of these new protocols. Apple in their WWDC, the Worldwide Developer Conference, earlier this year, gave some fascinating numbers out of the work that they've done finding that in the connections and the work that they were doing, they were seeing that IPv6 was 1.4 times faster in the connection setup, than it was over IPv4. Similarly, when using HTTP2, they were finding that it was 1.8 times faster. And honestly, sometimes to enable HTTP2 is just a simple couple of changes in the configuration file and you're done. But it can provide almost, you know, almost a 2X speed in the requests that are being handled by a web server in some kind of way. They even found that TLS 1.3, you know, part of the goal of TLS 1.3, and if you're not, I would hope that people in the HOPE community are familiar with TLS, but it's what we used to call SSL or, you think of as HTTPS but Transport Layer Security TLS 1.3, one of the goals of TLS 1.3 was to have a faster connection setup. And that was proven in Apple's stats to say that they're 1.3 times faster in what was going on here. So, for the, you know, this is good information that can help all of us who are advocating for these, to be able to make the business case that we need to make to others within the organization about why they should spend time, energy, money, whatever is needed to go and make these kind of changes. And what's also happening in here is, a couple of things, that are, the folks over at Google can help with, one is Google has recently come out with a,
well, Deloitte came out with a business case, or a white paper, that was commissioned by Google, a study, that they went and surveyed about 37 different high traffic retail and travel sites, and luxury sites, they surveyed them over a four week period, and found out that just by decreasing your load time by .1 seconds, you would wind up, they found that conversions grew by 8 or 10% for some of the sites. The key point was, there was a business driver, and as you look at the study in the reports, it talks about how this led to an increase in income, in revenue, that was being based off of this kind of information. The other thing to notice is, what's down at the bottom of the slide, is three metrics, which you may or may not be familiar with, but if you, unless you do a lot of work in website performance, but these are what Google is calling it's common web vitals. And if you go to web.dev/vitals, you'll find out more information, but perhaps most significantly, Google has indicated that these three factors, user experience factors, will all become a Google search ranking factor sometime in 2021. They haven't indicated when, and they've also indicated that they will give six months' notice when they are going to make this change. But the key point is that these will become ranking factors. Now, the minute you say these things can't become ranking factors, suddenly people responsible for websites, and potentially for the budget behind websites, suddenly start to notice, you know, TLS took a long time to get to the place that it's at right now where, you know, in many cases we have upwards of 80-90% of all sites are being [...] over HTTPS, over TLS, they're encrypted connections, and a huge factor in that has been the evolution over the last five years of the Let's Encrypt certificate authority which has made TLS certificates available for free. You know, for, and if you're not familiar with Let's Encrypt go there, letsencrypt.org, it's the free certificate authority, helps you go and do this. Let's Encrypt played a huge role in this and another huge role was the fact that Google said that HTTPS, that having your site available over HTTPS, would be a ranking factor in search engine ranking. And with Google's predominant role in search engine ranking that caused a lot of people, and especially a lot of people from the marketing and business side, to say, oh, hmm, maybe I should pay attention to all of those security guys who were talking, and gals, who were talking about, you know, this TLS thing. What is this? How do I make it work? Well, the same thing is going to be happening around performance because sometime in 2021, these user experience factors will be more of a ranking factor, according to Google, in their search engine results and that may cause, you know, people coming to you saying, well, how can we make our sites faster at which point we can start to say, well, one way will be, let's look at how we go and support some of these newer open standards that do provide these kind of performance improvements, that folks like Apple have been able to measure. So, that's an interesting new factor around this. Another part of the project obviously is what we are going to do in our own self, how can we lead by example? And that's where I began this, right? In the beginning, all we want to do is make sure we were practicing what we were promoting, and how we could work with that. So, we did audit our sites, we're continuing to audit them, we've now been able to get our sites up to 89% on the internet.nl testing. Interestingly, a significant factor for us is that our content delivery network that we use, our CDN, needs a number of changes to it, in order for us to be able to go and fully get to that 100% that we're trying to get to. But we're working on it. That's our goal. Let me just end with a couple of notes and then I'd love to throw it out to questions. So, if you've got some questions please feel free to get ready to ask them in the chat.
The project is obviously going to continue, we're not just sitting here making documents and that's it. We have a couple of different goals. One is that we'd like to go beyond just providing documentation, to potentially providing some Docker containers that people could use if they just wanted to go and set up a fully standards conformant kind of website that would work very quickly and easily. This is kind of one of our stretch goals we'd like to see as the project moves forward. We also will continue to watch internet standards because, of course, the web standards don't stay still. They continue to evolve, and they continue to go in different ways. Some of you may or may not be aware of the QUIC protocol, I saw at least another talk here that was talking about it. And the QUIC protocol will bring again some security and performance increases and it will be used, it's the basis of what's being, what's called HTTP3, HTTP over the QUIC protocol. And as that starts to, as it reaches maturity, both in terms of standardization and getting out there into web servers and web browsers, we're going to look at how do we bring that into this project and how do we help create some documentation to go and do this, and how do we make our own web servers speak HTTP3 as well. Another area that's happening, is there's some work within the Internet Engineering Task Force, the IETF, around website packaging. How can you make it so that you can be able to package content and make that available to people, more rapidly and more easily, especially if you think of people browsing in mobile environments, or people with lower bandwidth, or who are looking to work offline, or different kinds of places. How can you package up a website in a way that can make it so that it can be, you know, easily used and consumed in that kind of fashion. We're also looking at being involved with the hackathon such as the ones that the Internet Engineering Task Force is involved with, if people have other developer conferences or hackathon areas, we'd love to talk to you about that. How can we help with continuing to refine these documents, with helping make sure there's running code to go and work with this. This is, we started in 2020, talking about web servers, but our goal is to really help push open standards far beyond that over the course of the next five years or so.
After we do web servers this year, and develop these quick recipes, we will continue to support web servers, over the next years ahead, as those standards continue to evolve. But we're then looking at what comes next. There's mail servers, there's a number of very solid, you know, standards out there, DMARC, DKIM, SPF, things like that, that are out there and being used today. How can we help people who are continuing to operate their own mail servers with making it that much easier? That's a target we're looking at probably for 2021 in part because the people at internet.nl, the team there, have a testing engine and quite honestly, they've already developed a significant amount of documentation around this, so we may partner with them to go and help promote that work in different ways, it's one of the areas we're looking at. We also, you know, there's a lot of effort going on with DNS servers these days, DNS over HTTP DoH, DNS over TLS, there's even people looking at DNS over QUIC, you know, how can we help get those standards, more widely deployed. That's our goal, is what can we do to help get those open standards to be out there everywhere they are. Time servers are also something, I don't know how many people are aware that there's a new standard out for Network Time Security. The Time Protocol, NTP, the Network Time Protocol, that we use as the basis for so much of what we do and it's critical to our security infrastructure. It's absolutely critical for things like TLS certificates. If you don't have accurate and reliable time, you can wind up having attacks that can go against, you know, the time and get into different issues with certificates and such. There's a project that we have at the Internet Society around Network Time Security helping foster some of the implementation and deployment of that. There are a number of people working in that community. And so it may be something that we will look at, we're also thinking about, you know, is there a role that we can play in helping, you know, increase the deployment of some of the Web RTC standards, others, quite honestly, I'd love feedback from you all here, and from the wider community around where, you think we should spend our attention next, where could we help the most people, you know, get to faster, more secure servers in general, and the pieces that are there. Let me just finish by saying there are ways that we'd love to have people get involved. One is just, test your sites, I think, helping people be aware that there are sites out there like internet.nl that allow you to see what you can do, how you can improve your site. They provide, on their site, information about the cyber security guidelines coming from a couple of different places that help explain why it's important to, from a security point of view, to use these new standards. Again, as I mentioned, I would really appreciate help with our documentation. I'd love to see people come to our GitHub repositories, star our repos, watch our repos, you know, provide feedback on some of the issues, you know, text if you want to, I'm glad to accept pull requests and bring information in and see where this goes. So, you know, if you can share this information, share that this project is out there, that we're trying to do this kind of work, that we're looking to try to help website operators get that documentation so they're not looking, trying to find it all over the place. We would love your help in sharing that. If you'd like to dive more, you can join the Internet Society, membership is free. And we do have some additional forums and lists where we engage with people who want to dive a little bit deeper into the project. We do believe that an open Internet is based on open standards. TCP IP is why we're all communicating, why we're in this call, why we're having this conference online, and why all of us have been able to go and work remotely, or work from home, or to all this stuff in this crazy world of pandemic. We need to continue to have it, based on that, we need to help get these open standards out there, and I would ask for you to help us, join us in that. So, with that, I will turn it over to questions and I'll leave my contact information up here. I'm just email@example.com; I'm on Twitter is @danyork, I'm on Mastodon, I'm on GitHub and I'm on Twitch as well, so, number of different places. So, thank you for listening and do we have any questions coming in?
Yes, indeed, you're watching Hacking Web Servers to Make Them More Secure and Faster, Using Open Standards, with Dan York, of the Internet Society, and we do invite those of you attending HOPE 2020 to ask Dan York questions now. Please post those questions to the livestream Q&A channel on our matrix server, and they'll be relayed here. We've already got so much, really, good conversation, and the questions happening in the channel so, I'll just dive right into them here. A member of the audience asks, apart from depleting the IPv4 address space, is there any real advantage to having IPv6 enabled? I find that when I have it enabled, I have all sorts of routing problems and disabling it fixes it.
Well, I, so, I would say, there are. Let's talk, send me a message, I can point you to some people who can provide some resources to help with that routing issue that's there. It sounds like you've got some challenges that are perhaps fixed on there. One thing that's interesting on the IPv6 side, we're seeing a number of different reports around the performance increases, the speed increases, the parts are there. You also, we also have seen that increasingly in mobile environments, and this is where the performance also comes in, in many of the mobile environments that are being deployed right now, they had no choice but to deploy on IPv6, they simply don't have enough IPv4 addresses and so they're deploying as IPv6 only mobile networks and this is happening in Asia, it's happening in United States, and happening in other places. And so, they actually have to go through IPv6 to IPv4 gateways at the edge of the network in order to get to IPv4 resources. So, as again, when you go back to that performance, you know, and especially as people look at how you shave any kind of thing off of there. If you can provide native IPv6 connectivity to your v6, to your web servers, etc., you can shave that tiny amount, again, it could be microseconds, whatever, that's going through those gateways to mobile networks, so to me, that's a big argument. I've seen other folks who are interacting with people in some other parts of the world where they actually need it in order to interact with partners that are over there.
Another member of the audience asks: How can we be using the latest TLS version, like TLS 1.3, when CDN providers like CloudFront didn't even support it until a few months ago? There seems to be a pretty big delay there.
Exactly. And that's actually one of our challenges. When I showed my little chart, that showed that we're not at, we're only in 97% for some of our websites, it's because some of the CDN providers are not providing TLS 1.3. My big answer to that one is, we got to keep asking them, because, you know, as we went and asked some of our providers, you know, they claimed oh, you're the first person who asked us, to which we say, yeah, right, but we need to keep asking, keep raising those support tickets, keep doing that, because that will cause, if the CDN providers, you know, if we can take away that excuse of, well, nobody's ever asked me, and I think these days we can also do that on social media and those kind of things as well, and say hey, when are you going to be supporting this?
And in relation to that, could you speak to the issue of interoperability with older web clients in relation to TLS 1.3?
we hear, we had this issue, we had one of our new websites we put up, and we put it up, and one of the folks who was involved that was putting up said, hey Dan, I got that new site to work at 100%. And, and we said oh great, and we looked at it, we discovered that the way that we'd been able to do that was that it supported TLS 1.3 only, and I said, well, wait a minute, you know, TLS 1.3 is brand new, in web terms, we can't just say we're going to jump to that. We do need to put in there that we need to get back to TLS 1.2 as well. There's no reason to support 1.0 and 1.1, those are dead and gone and should be buried. So, we need to be 1.2 on and, you know, the reality is, we're going to have to support 1.2 for some period of time, you know, as we make the transition, there's just no escaping that.
We're asked, how reliable and trustworthy are security benchmark systems like the Qualys SSL Labs test?
Oh, those are great too. And, those are another, actually, for some other work we did, we actually evaluated using the Qualys test and some others. And then ultimately, we decided just to use internet.nl as our test framework because it kind of covered the range of stuff we had. But, I also, I like a lot of those other tests that you have, Qualys is one, there's several others that are out there because they give you an even deeper, a deeper dive into some of the various different TLS options and the pieces that are there. So, I think they're a great additional test to use.
You spoke a bit of Google and their metrics, whereas, how is Google coming up with these metrics, is there any public input to this?
Well, if you go and look at their web.dev/vitals, they go into some extremely long length around how, what these are, and what the pieces are. I don't actually know the answer, to say how much public input they had into these or those kinds of pieces, but they were very clear on a post earlier. So, they've been using a couple of these metrics in different ways over the past several years, but now they've been very clear that these three metrics are ones they're going to start using as part of this overall page ranking going on, and sometime in 2021. So, but, you know, as far as what public input, I can't tell you on that one.
Do you know of any free software tools that can run self-hosted, or locally, to test metrics like this?
I don't off-hand, although I will say that, again, the internet.nl tool is open source, they do have, you can run it in a Docker container to run it on your own system, and especially, I think, you need to be on a Windows box, it doesn't work quite right on a Mac box, there's something that you need to do in there, or Linux box you could do it either, I think, on Linux or Windows but not Mac, because I was trying to make this work, I wanted to run it locally and do all my own testing right there. But, if you look at their GitHub repo, they do have a, you can download the version and work with it there.
Regarding IPv6 performance, we are asked: Could it be that IPv6 only appears faster because the providers that implement it happen to be the better and faster ones, or do those measurements control for that already?
It's a very good question. I don't know the answer. It very well could be that some of those are the bigger providers or the folks who are doing more of that. I don't know, but it would cause, it would be a good question ask.
I also see. Oh, go ahead.
Please, you're controlling the questions, I'm sorry, I now have, I'm in the chat so, now I can see these things.
No worries at all. We did have a comment, I thought was interesting, which is, that training is needed. This person writes my main email address is 47 characters, one would expect banks, as an example, to be able to handle my email address. Obviously, some of their website programmers have little clue about the standards; unknown standards are of little use.
Yeah, yeah, and that's for the 47-character email address. Yeah, there's a whole area, if you look up universal acceptance, there's a whole area that's already being pushed by ICANN, the Internet Corporation for Assigned Names and Numbers. If you look at what they're, there's a whole initiative there around universal acceptance and encouraging people and developers to be sensitive to these kinds of things and to be, and to realize that, for instance, you can have domain names in non-Latin scripts, and you can have other different kinds of things that, you know, or super long TLDs, right? We're no longer in the era of three letter TLDs, we're, you know, dot whatever, super long names. So, that's a whole space there, but yes, it's a challenge, and training of developers and helping developers understand why they need to be concerned about this, is a real hard, it's a hard road, it's a hard road to go down.
We've got lots of good questions coming in, please keep those questions coming in on the live stream Q&A channel. Our next question is: Do you have standards initiatives? Do you have complimentary training initiatives?
Well, so, actually, it's a good, interesting question. So, we are, we, The Internet Society, are looking at developing some courses that would, or at least a course, that would go along with this particular bit of work, that would help provide an online training course, that would let people go through and work through this process with, to help work on the server so, we're developing that. There's many other people out there as well who have courses that are around there. So, I'd be curious to talk a little bit more, if you have ideas drop me an email or contact me on Twitter or Mastodon or something, I'd be glad to talk with you more.
And just for the sake of putting it out there, could you give your contact info, your web info, social media?
Sure! It's firstname.lastname@example.org. I'm also on Twitter as danyork, I'm on Mastodon as danyork@Mastodon.Social, and I'm on GitHub as Dan York, and Twitch as DanYork324, and I think that's probably good enough.
Excellent. Our next question is: Pushing the use of CDNs makes website traffic faster but it also creates a single point of failure, as we saw even a few days ago, when CloudFlare had an outage, is there any way to get around this?
Yeah, use a couple of CDNs.
I mean, I joke, but actually there are people doing that, for precisely this reason, which is super challenging in and of itself because then you have to deal with DNS and all sorts of different challenges of how you make that happen in reality. But it is a fundamental change that's happened in the architecture of the internet and it's fascinating. Jari Arkko and some other folks from the Internet Architecture Board, wrote a whole draft around consolidation and sort of the architectural impact that we're having on the internet right now. And it's a real challenge, because if you want performance these days, you have to think about, if you really want to go down the performance road, you have to pretty much think about a CDN in some way, because our movement to HTTPS, our movement to encrypt everything and TLS, which I fully want, I mean, I want that. The challenges, it removed the entire caching infrastructure of the globally distributed internet. Because it used to be, right? We all would run, you know, squid proxy caches or something on our local systems and we just cached all the HTTP, all the web pages, when we provide them back, and odds are that our ISP was also providing caching servers at some level and probably somebody else in between so, we had caching everywhere throughout the globally distributed web and internet and stuff. Well, a side effect of going to TLS, HTTPS, is that all of those things are gone. You can't cache it because the only connection is from your browser to some TLS endpoint. And so, nothing in between has any visibility to that. So, it's a glorious win from a security perspective, and from a privacy perspective, but it has a performance impact because the caching no longer works. So, in order to do that, and not drown your own web server, you basically have to scale your HTTPS/TLS infrastructure, and the way to do that is to use some CDN. Now, there's free ones, there's open ones, there's lots of different places, but it does create single points of failures, you know, it creates issues that are out there. Now, you can say, you know, again, you've got any casted servers out there, you've got lots of different places so, the CDNs themselves are, in theory, extremely robust, extremely all that. But again, we saw it, we've seen it with the recent outages, it's a challenge that we all have to go and work out.
There's some commentary going on in the chat about the fact that, like CloudFlare for example, at least with some of their plans, makes you host your DNS with them, which interferes with the idea of redundant multi CDs.
Exactly. Exactly. Yeah, you want they're free version, you got to host their DNS, you want to pay more, you can control your DNS, and you can, and there are people, you know, doing the multi DNS, multi CDN thing, right, and figuring out how to go and do that and that gets, you know, all sorts of complexity. It's fascinating from a network perspective but, you know, who knew we'd be all dealing with multiple secure overlay networks.
So, what do you see is like the biggest barrier, the biggest hurdle, to more adoption of these open standards?
I think the biggest adoption is understanding the standards, you know, understanding that they're out there, you know, it's funny, we're talking to people about HTTP2, and just saying, you know, look, it has a number of features, that it does, it does multiplexing, it does, you know, it can push content down to the server like, you know, if you know that you need the HOPE logo when you visit the HOPE page, then whether, rather than waiting for your website, your web browser, to request the logo, the server can just push it and say, oh hey, you want the page, you're going to get a logo too. There's some things that can do there but people just didn't even know that it was a thing. And they didn't know that they could go to their Apache or NGINX configuration file and basically change one line and restart the server, and they'd now be operating at a much faster way to go and do this. So, some of it is just, it's education and helping people understand that. And back to, you know, to my colleague Gregg, it's helping people find simple, easy recipes to go and do this. So, you know, I would put out a general plea: this is what we need, you know, we need simple, easy documentation and help people understand that, whether you're, whether you help us out on our project, whether you do your own project, whether you just write it for, you know, distributions, or for open source software, that's a key thing, is we need people to understand how easy it can be to go and implement this stuff.
And someone asks, have you seen a good multi CDN reference architecture?
I have not, but if you drop me a note, I know some people who probably can, probably can help me with that though.
I also saw somebody mentioned a note in there about, you know, do you go for the latest Qualys test, or do you piss off your customers and, you know. That is one of the challenges, right, with any of this sometimes is, you want to go and make something more secure and sometimes the tradeoff is that maybe you can't. So, somebody mentioned I saw on there, that some older clients still only support TLS 1.0 and 1.1, like older versions of Android. Well, you know what, guess what, I'm sorry, they're not going to be able to get to my websites, because we've dropped support for 1.0 and 1.1 and we've also mandated that you must connect over HTTPS. So, effectively I am making a choice that I want that, over that population of people still using those very old clients. I'm okay with that. Some businesses may not be. I would really actually want to, I would want to know what percentage of people that is, because that actually kind of does concern me a little bit.
It is of course not a new story in infosec to have the biggest hurdle be, like, I need the users to upgrade something, Right? Yes, exactly. We've got a few minutes left. Someone asks: How do you gauge the effort for the mail server part, given the state we're in today and the quality of the tooling or support of modern protocols?
Yeah, so, we're just starting to look at, so, we're starting to look right now at what would be involved with that and what kind of work we could do, how we could help people with getting more usage of DKIM and DMARC, SPF, those kind of things. And part of that is, part of that work to gauge it, is understanding what the adoption is right now, you know, across the industry and I'm, in my queue I'd love to find some research that's really good about how much, you know, what's the deployment level out there. You know, how can we, you know, is there, what's the space in there, what can we work with that. We know there's some tools like what the internet.nl folks have for testing of those standards. I know they have some documentation, other people have written that, but that's kind of the experience I'm looking at over the next few months, is to try to figure out is this the right place for us to start to go? And if you have mail server related questions, I'd be curious to know what those questions are if you want to drop me a message. I don't know that I can answer them today, but I would be curious to know what they are, because it may help us as we start to look at, you know, do we want to take on mail servers in 2021?
Your questions would be helpful to me.
Yeah, we do have someone in the chat, who says I have like 20 mail server related questions, but I don't want to open that Pandora's box.
We have like one minute left so, no.
As we wrap things up, I think we've got time for one more question and that is: Why if all the software's and service mail providers want us to embrace DMARC don't they offer native tools for that?
Well, that is one of the reasons why I think we need to look at that for the next 2021.
You know, I don't have an answer for you today, but I think it's a very valid question and, why is there not support, if they're going to do this. The other question there, why do companies ignore that? You know, yes, those are all questions that we need to help push those kinds of standards forward if we want to do it. Especially, the mail part, to me, is also an avenue that if we want to, if we want to allow an open, decentralized, distributed internet, where people are running their own mail servers and pieces like that, we need to make it easier. I'll admit, I don't run my own mail server anymore, because I just couldn't keep up with doing it as a one-person thing, but I know many friends and others who do. And we need to make it easier for that, and we need to continue to make it so that open standards are there so, we don't wind up with just a small pool of centralized mail environments, or web environments, or anything like that so, it is why we need to have open standards everywhere.
I'll just wrap it up with one more comment from the chat which is, someone says: Dan, your last comment reminds me when, for example, some banks demand that the end users use Internet Explorer.
Yeah, well it's, you know, and although we're seeing that now today too, right, we're seeing some sites will, you know, they work best with browser X, and pick your browser in there but that's a challenge, the larger consolidation issue.
Well, thank you, thank you for having me.
Thank you. Thank you very much for joining us and enjoy the rest of HOPE.
Thank you all.