Keynote: Linus Torvalds in Conversation with Dirk Hohndel

Keynote: Linus Torvalds in Conversation with Dirk Hohndel - https://www.youtube.com/watch?v=OM_8UOPFpqE

Guten Morgen. Simlich Voldoch Trotsein aber Leider Goddess. So let's do this in a language that more people understand. My name is Dirk Horndel. I'm the head of the open source program office at Verizon. And you are? And I'm Linus. And we've been doing this pony show with Dirk now for what, 20 years? No, no, no, really. It's only been since 2012. And we do it because I am not a great speaker and I get very nervous and I hate doing it. So the way to get me in front of an audience is to have somebody else make up the discussion and we'll just do a q and a kind of thing. And the key part is that he doesn't know what I'm going to ask him so he doesn't have to prepare, which makes it so much more fun for one of us. So lovely weather you guys have here. It was a wee bit windy yesterday as I was walking across the door now the Danube and things were going horizontal. Nice. It's definitely. You created some excitement for us. Thank you. This is your first time in Vienna? Yes. No, I was actually really looking forward to it because my daughter was here in April and saw the Easter decorations and sent pictures of beautiful downtown Vienna. And then we flew in on Saturday and decided that I'm not going to do any sightseeing this week, I think. So the whole plan was to have Sunday to take in the sides and have some good food and. Yeah, none of that happened anyway. 33 years of Linux. It's pretty cool. How exciting. You released 611 yesterday. Yep. What's interesting, what should people know about 611? I don't think we, I mean for the last almost, well, not 20 years, but 15 years we've had a very good regular cadence of releases. And it has resulted in the fact that we're doing, I'm doing the releases every roughly nine weeks. And that means that releases are not exciting and they're not supposed to be. They are timely and they are hopefully very reliable. But exciting is not, I think, the goal. So for me the excitement of doing a release is that now the day after we're doing all the new code for the next release. And that's the fun part, although not so fun necessarily when you're traveling. Yeah. So you decided to do the merge window on the road while traveling. And for those who are checking online, while we were waiting back there, he was doing merges on his laptop. So it is just in time delivery of the Linux kernel. But overall, if we look at trends, for many years we've had mostly drivers, a lot of file systems, some hardware support. So where is most of the code coming from these days? Well, it's always been mostly drivers. I think more than half of the kernel is driver support, literally the point of a kernel. Most people should not care if, unless you are a kernel developer, because what the kernel does is abstract out all the hardware details. And obviously some of the hardware details are the actual cpu architectures and things like that. But in the end we only have what, twelve architectures? I forget we have thousands of drivers. So every single release the bulk of the changes are just new hardware support for drivers and fixes for old hardware support for drivers. And that kind of goes without even being hugely mentioned. I do tend to, when I make releases, I do the statistics of which driver areas are the major ones. But honestly the excitement is, and it's, at least for me, it's the very core kernel. And I have been doing this for literally a third of a century now, and I'm still surprised by the fact that we're doing very core development. Like half the stuff I've merged today has been fairly low level virtual file system code, and a lot of the discussions have been about still memory management and things that have been around forever. But the continued expansion of the hardware base, but also the continued expansion of the user base means that we're still doing very core development in, in all these areas that you would expect to be ready after a third of a century. So are we getting the new scheduler this time? Yes, so I have that in my poll queue. I'm probably not going to get to it today, and tomorrow is the kernel maintainer summit, so I may not get to it tomorrow either, but it's in the queue. A lot of the core developers are actually here. Well, some of the kernel core developers are here. So I got a lot of the pull requests early last week and a number of the developers said hey, I'm sending this early because next week I'm going to be traveling and I don't want to do development. So I'm stuck with that part. So one of the exciting things that also is happening this year is the celebration of the real time Linux project finally being completely in the kernel of a very brief development cycle of about 20 years. Yeah, I don't actually, I don't think I have the pull request for the final bits in my mailbox yet, but I'm expecting that to happen this week. So I mean, some features and I keep saying this, that people think that kernel development is very quick because every release schedule, release, and we do that every three months or so we have between ten and 15,000 commits going into the kernel. So there is a lot of development going on. But a lot of it has been is stuff that is now ready and it may have been developed over months or years or in some cases decades. So while kernel development is very active, it doesn't necessarily mean that you can get a new feature file system or anything into the kernel very quickly. There's often a long development period that people seldom even see because it happens not exactly in private. It is being discussed on the mailing list and things like that. But you don't kind of see the background work that goes into a lot of these things that then suddenly appear in the next kernel. 20 years is still over there. 20 years is way. Yeah, that's very unusual. We definitely have had. I mean the issue with the real time patches is that they've been touching every single area in the kernel and that tends to be when you have issues and where it takes longer to integrate because you have to convince all the people involved and you have to get all the drivers updated and you have to just do a lot more work if you're not able to compartmentalize the code into one specific area. But there isn't really anything else that compares to that. There isn't another 20 year project that's happening. No, I think RT Linux has been, I mean it's famously been in the works for a very long time. Another topic on the kernel and that is creating a lot of conversation, let's say, is obviously rust. And one of the maintainers stepped down last week saying, and I quote non technical nonsense as the reason for stepping down. And interestingly, there was quite a bit of argument, let's say between the folks from Asahi working on the Apple silicon GFX drivers and the DRM scat people. So why is this so hard? I think I actually enjoy it. I enjoy arguments. I think one of the nice parts about Rust has been how it's livened up. Some of the discussions and I mean some of the arguments get nasty and people do actually, yes, decide this is not worth my time. But at the same time it's kind of interesting and I think it shows how much people care. At the same time, I'm not quite sure why Rust has been such a contentious area. It reminds me of when I was young and people were arguing about Vi versus emacs. They still are maybe they still are. And I've just. Should we have a poll here in the room? Yeah. No, no. But for some reason, the whole rust versus c discussion has taken almost religious overtones in certain areas. Well, so a big part of this is that, from a design philosophy perspective, if you want to actually write memory safe rust code, there are certain architectural requirements. And it seems that a lot of the older, and I don't mean age, I mean, just having been around for a longer time, maintainers are fairly resistant to accepting the fact that if you have a rust subsystem, it requires you to have certain failure handling which they don't want to add because you say, oh, that's a bug. That should never happen. I mean, we have been doing this for over three decades and seas, in the end, a very simple language. It's one of the reasons I enjoy C and why a lot of C programmers enjoy C, even if the other side of that picture is obviously that, because it's simple. It's also very easy to make mistakes, and Rust is not. Rust is a very different thing, and there's a lot of people who are used to the C model, and they don't necessarily like the differences. And that's okay. In the kernel itself, almost nobody, I mean, I won't even say almost absolutely nobody, understands everything. In the kernel, I don't. I rely heavily on maintainers of various subsystems, and there are only a few areas where I get, personally, very much involved. And I think the same is true of rust and sea, is that you should expect that not everybody will understand rust. And on the other hand, not all the rust people necessarily understand sea. And that's okay. I think it's actually one of the strengths we have in the kernel that we can specialize. And some people care about drivers and very specific drivers, and that at that, some people care about specific architectures, and some people like file systems. And that's how it should be, and that's how I see Rust. But clearly, there are people who just don't like the notion of rust and having rust encroach on their area. And I think it's been interesting, and I'm not afraid. I mean, people have even talked about the rust integration being a failure. And I'm actually. I don't think a. We've been doing this for a couple of years now, so it's way too early to even say that. But I also think even if it were to become a failure, and I don't think it will, that's how you learn so I see the whole rust thing as positive, even if the arguments are not necessarily always. Yeah, and I mean, the challenge really is the idea that a memory safe architecture makes certain assumptions about the infrastructure. And it's the infrastructure people like DRM Sket as the example where it has become very public. It's the infrastructure people who seem to resist some of the changes. To be fair. I mean, we've done a lot of changes to the C infrastructure too. The kind of code, what the kernel does is not normalcy. We have. It's not just that we've write things in a certain way. It's also that we have a lot of tools on the C side that enforce infrastructure rules that are not part of the language. And part of this is our locking safety. We do have actually a lot of memory safety infrastructure on the C side that is not technically part of C itself. It's part of our infrastructure for the kernel, where we have debug builds that make the kernel go much slower, but it will test a lot more of the memory safety stuff. So on the seaside, though, we have been able to kind of incrementally add it. And I mean, we have a lot of that kind of support, but we've had a long time to add it and that kind of has allowed people to do it without the outcry. And the rust change is obviously a much bigger and much more in your face thing. And so you said that it's too early to say it's a failure, and I agree with that. But one of the things that I've seen lately is a certain amount of interest in bottoms up, grown up from the start, rust kernels as an alternative to Linux. So is that a potential outcome if we continue to see struggle to get Rust into Linux? That there will be something like redux or Maestro or one of the other ones that is popping up, that we will have an alternative universe built around the different kernel. I think that's a, regardless of the Linux use of Rust, I think if you're doing operating systems, you really don't have very many choices when it comes to languages, because you require a language that can deal with system issues. And that automatically means that your language choices are very limited. Unless you want to write an assembler, you're going to use C or one of the c like languages, or you're likely going to use Rust. And I don't think, I mean, Linux is not everywhere. We are very widely spread, but after three decades, Linux is also very big. And we've always seen these embedded systems in particular who just want something smaller and simpler and not quite as fully fledged. And that's when you would use rust or C or anything else. And that will never go away, I think. I mean, obviously we see things like Zephyr and other embedded, deeply embedded kernels, but as a general purpose os, I think we can say Linux is absolutely everywhere. It's today there is a complete Linux distribution running on most 5g modern chips. So in your iPhone is a chip that runs as its firmware Linux. So it is everywhere. I mean, yes, we used to have this joke. I mean, many many many years ago we used to talk about world domination and that joke became reality and isn't funny anymore. And I'm okay with that. But at the same time, I mean, nothing lasts forever. And I'm sure some clueless young person will decide how hard can it be and start his own operating system in rust or in something else. And if he keeps at it for many, many decades, or she and many many decades, hey, I'm looking forward to seeing that. To save you from bad quotes. When you say some clueless person, that is a reflection on your own past. Oh, absolutely, yeah. You have to, you have to be all kinds of stupid to say, I can do this, right? Because it turns out, yes, I'm still doing it 33 years later. And a, I couldn't have done it without all the literally tens of thousands of other people anyway. And the only reason I ever started was that I didn't know how hard it would be. But that's what makes it fun. So that is actually the perfect bridge to the next set of questions that I have for you. So you said the maintainer summit is tomorrow and an evergreen topic that I think comes up at every maintainer summit and certainly that I bring up in our conversations every year, every other year is this bigger question of burnout. So the maintainers are an aging group. Strangely, some of us have, you know, not quite as much or the right color of hair anymore. Hey, gray is the right color. Gray is a great color, yes. But the Linux ecosystem, the kernel development community, there is absolutely no sign of slowing down. It's accelerating in many ways, rust being one of them. And so the question that I always ask myself is time is kind of cruel stop for us. And as the maintainers get older, as burnout becomes a bigger problem, is it about time to talk about mini Linus? About the successor, the next generation? We have talked about that forever. And people are probably, some people are probably still disappointed that I'm still here. No, I mean it is absolutely true that kernel maintainers are aging. And the positive spin of that is how many projects do any of you know about where you have maintainers, not just me, who have literally been around for over three decades. It is very unusual. So then saying, hey, people burn out and go away. Yes, that's true, but that's kind of normal. What is not normal is that people actually stay around for decades. That's the unusual thing. And I think that's to some degree a good sign. But obviously it also does mean that as a new developer, as a young person it can seem kind of hard to become part of this when you see all these people who have been around for decades. At the same time, I think in open source the Linux kernel is actually very unusual in just how many developers we have. So it is very true that we have maintainers who have been around for a long time and who are getting gray. But at the same time if you actually look at the statistics of the developers we have very few if any projects have 1000 developers being involved in every single release and the releases being every three months or so. So I think we do have a fairly healthy developer subsystem. The whole monkey dance about developers, developers, developers, we've got them. The fact that we also have these old graying people around, I don't see that as a huge problem. I'm not trying to say it's a problem. So what I'm trying to say is you've been doing this for 33 years. I don't want to be morbid, but I think in 33 years you may no longer be doing this. Possibly. And even though I would love to still do the conference circuit in 33 years. But we come in on our walkers, it's going to be awesome. What I'm saying is in order to be the king of Linux, the main maintainer, you have to have a lot of experience. And the backup right now is Greg Kh who has even less hair than the two of us and is about the same age as we are. Greg, are you in the room? I apologize. You can punch me later. But what I'm saying is how do we get the next generation to gain the experience so that in 1015 2030 years your role can be handed off to somebody else? It's that sustainability that I'm asking about actually. So I think part of the issue with us having a lot of developers is that we've always had a lot of people who are very competent and could step up. I mean you mentioned Greg. But the thing is Greg hasn't always been Greg. Before Greg there's been Andrews and Alans and after Greg there will be Shannons and Steves. Who knows? There are people around who have been around for decades. And the real issue is you have to have a person or a group of people that people in the development community can trust. And part of trust is fundamentally about having been around for long enough that people know how you work. But long enough does not need to be 30 years. We have very core developers that are like top level maintainers for major subsystems that have come up in just a few years. It's not instant, nobody's going to claim that. But there are new people who come in and three years later they are a main developer. It is not impossible at all. Being a maintainer is not as glorious as it sometimes can seem, right? And there's a lot of people who would be more than happy to get new people in and come and help. And sometimes people are more afraid of kernel development than they really need to be. I am sure a lot of people are currently googling Shannon and Steve as first names of kernel developers to know who you're talking about. They were random names I made up. Just out of interest, how many people here are actually kernel developers have sent in patches. I mean kernel developers are odd, we know that, but I mean we're not so unusual as to be unheard of. And it is the most interesting thing you can do, I'm convinced. Still. Yes, we all know kernel development is the peak interest of things a person can do. So why are people laughing at me? I should not have repeated that sentence. What is interesting to me is to now extrapolate from this and look at the larger open source community. Because one of the things that we've seen just in the last three, four years is that a lot of the large companies who have very, very aggressively funded open source development for so many years seem to have stepped back a little bit. Lots of layoffs, lots of companies noticeably reducing their involvement. And since many of them are sponsors of this event, I will not name them, but it's easy enough to figure out. And so at the same time we see more and more attacks on the fabric of how open source works on this trust model you just talked about by threat actors, be it criminals, be it nation states, whatever. So to me this overall question of how can we improve the health of the idea of being a maintainer and open source is something that I'm very interested in. And one of the things that Linux has done so amazingly well is to have had an incredibly deep bench of a lot of people involved and a lot of people who aren't afraid to speak up and say hey, this is wrong, or I don't like this, or this is a mistake. And is there a way to take what we have learned in Linux and apply this to a broader set of projects? I actually think that question is very pessimistic in itself because I remember when I was young, long, long ago, and open source as a name was only very new. And I think the open source communities today are much better off than we used to be. And yes, there are technology layoffs and they seem to be very cyclical. And it's not necessarily always a nice industry to be in. But at the same time I feel like open source is doing very well. People take open source for granted now in a way that they absolutely did not just two decades ago. And maybe that then has resulted in some more negative commentary and more attacks. But I would say that one of the strengths of open source is that it is now so ubiquitous that as a new programmer in particular you can use open source as a way to enter this not so nice industry always and make connections without necessarily having gone to the right schools or having the personal connections that you often need in many other industries. So I think open source is doing really well. I agree, open source is doing well. It's much more this question of how do we create more sustainability. But one thing that you just said, really I find interesting new people coming in. So when we all started a few decades ago, it was a much smaller, much simpler world. Today everything is hybridization versus reality. And before anybody thinks, I'm not just talking about AI, we have so many projects that seem to be all about the quick buck, about the quick exit versus things that are making a difference. If you were starting today, do you think it would be easy to find interesting, rewarding, long term useful projects? I don't think that has ever been easy. I mean what has happened is that there's a much bigger mass that can do so and all the infrastructure is there. So it's so easy to basically do a few web clicks and you have a GitHub repository and you can start doing open source. And you don't have to explain why you're doing open source because people take that for granted now. So there's a lot more of these small projects that you would never have seen 30 years ago because it's so much easier. But it was never easy to find something meaningful that you could really spend decades doing. And I don't think that's easy today either. You just have to come up with something that you're interested in, but at the same time, you're not the only one interested in it. That's. I think one of the problems is that people say, hey, do what you love. But if what you love is something that nobody else cares about, you're not going to create the next big, successful, open source project. So you have to love something that is meaningful and you have to find it. And that's sometimes hard to do, especially in the tech industry, where so much of it is about the hype and everybody is following everybody else, like lemmings off a cliff, trying to chase the next big thing. And I don't think that's a successful strategy. I think you need to find something that isn't what everybody else does and excel at that and be the first to do something slightly different. You called me early. A negative. I was hoping we would end this conversation on this wonderful, inspiring highlight, Linus, telling the community where to go and make a difference. Instead, we're ending on lemmings falling off a cliff. I have zero minutes left, so unfortunately, I can't rescue this with an incredibly clever question. But I think we'll just leave it here and tell all of you you are not the lemmings he's talking about. And enjoy the week in Vienna. The weather is supposed to be much better. And thank you everyone, for making it in with all the flooded roads and tunnels and whatnot, and have a great show. Thank you.