Guten Morgen. Simlich Voldoch Trotsein
aber Leider Goddess.
So let's do this in a language that more people
understand. My name is Dirk Horndel. I'm the head of the open source
program office at Verizon. And you are? And I'm Linus.
And we've been doing this pony show with Dirk
now for what, 20 years? No, no, no,
really. It's only been since 2012.
And we do it because I am not a great speaker
and I get very nervous and I hate doing it. So the way to
get me in front of an audience is to have somebody
else make up the discussion and we'll just do a q and a
kind of thing. And the key
part is that he doesn't know what I'm going to ask him so
he doesn't have to prepare, which makes it so much more fun for one of
us. So lovely
weather you guys have here. It was a
wee bit windy yesterday as I was
walking across the door now the Danube and things
were going horizontal. Nice. It's definitely.
You created some excitement for us. Thank you. This is your first
time in Vienna? Yes. No, I was actually really looking forward to it because
my daughter was here in April and saw
the Easter decorations and sent
pictures of beautiful downtown Vienna. And then we
flew in on Saturday and decided that I'm not going
to do any sightseeing this week, I think.
So the whole plan was to have Sunday to take in the sides and
have some good food and. Yeah, none of that happened anyway.
33 years of Linux. It's pretty cool.
How exciting. You released 611
yesterday. Yep. What's interesting, what should people know
about 611? I don't think we, I mean for the last almost,
well, not 20 years, but 15 years we've had a very
good regular cadence of releases.
And it has resulted in the fact that we're doing, I'm doing
the releases every roughly nine weeks.
And that means that releases are
not exciting and they're not supposed to be. They are
timely and they are hopefully very reliable.
But exciting is not, I think, the goal.
So for me the excitement of doing a release is that
now the day after we're doing all the new code
for the next release. And that's the fun part,
although not so fun necessarily when you're traveling. Yeah. So you
decided to do the merge window on the road while traveling.
And for those who are checking online, while we were waiting back there,
he was doing merges on his laptop. So it is just
in time delivery of the Linux kernel. But overall,
if we look at trends, for many years we've had mostly
drivers, a lot of file systems,
some hardware support. So where is most of
the code coming from these days? Well,
it's always been mostly drivers. I think more than half of the
kernel is driver support, literally the
point of a kernel. Most people should not care if,
unless you are a kernel developer, because what the kernel does
is abstract out all the hardware details.
And obviously some of the hardware details are the actual cpu
architectures and things like that. But in the end
we only have what, twelve architectures?
I forget we have thousands
of drivers. So every single release
the bulk of the changes are just
new hardware support for drivers and fixes for old
hardware support for drivers. And that kind of
goes without even being hugely mentioned.
I do tend to, when I make releases,
I do the statistics of which driver areas are the
major ones. But honestly the excitement is,
and it's, at least for me,
it's the very core kernel.
And I have been doing this for literally a third of a century
now, and I'm still surprised by the fact
that we're doing very core development.
Like half the stuff I've merged
today has been fairly low level
virtual file system code,
and a lot of the discussions have been about still
memory management and things that have been around forever.
But the continued expansion
of the hardware base, but also the continued expansion of the
user base means that we're still doing very core development
in, in all these areas that you would expect to be ready
after a third of a century. So are we getting the new scheduler this
time? Yes, so I have that in my poll
queue. I'm probably not
going to get to it today, and tomorrow is the kernel maintainer summit,
so I may not get to it tomorrow either, but it's in
the queue. A lot of the
core developers are actually here. Well, some of the kernel core
developers are here. So I got a lot of the pull requests early
last week and a number of the developers said hey,
I'm sending this early because next week I'm going to be traveling and I don't
want to do development. So I'm stuck with that
part. So one of the exciting things that
also is happening this year is the celebration of the real
time Linux project finally being completely
in the kernel of a very brief development cycle of
about 20 years. Yeah, I don't actually, I don't
think I have the pull request for the final bits in my mailbox
yet, but I'm expecting that to happen this week.
So I mean, some features and I keep saying
this, that people think that kernel development is very quick because
every release schedule, release, and we
do that every three months or so we have between
ten and 15,000 commits going into the kernel.
So there is a lot of development going on. But a lot of it has
been is stuff that is now ready and it
may have been developed over months or
years or in some cases decades.
So while kernel development is very active,
it doesn't necessarily mean that you can get a new feature
file system or anything into the kernel very
quickly. There's often a long development
period that people seldom even see because it happens not
exactly in private. It is being discussed on the mailing list and things
like that. But you don't kind of see the background
work that goes into a lot of these things that then
suddenly appear in the next kernel. 20 years is still
over there. 20 years is way. Yeah, that's very unusual.
We definitely have had. I mean the issue
with the real time patches is that they've been
touching every single area in the kernel and that tends
to be when you have issues and where it takes longer to
integrate because you have to convince all the people involved and
you have to get all the drivers updated and you have
to just do a lot more work if you're
not able to compartmentalize the code into
one specific area. But there isn't really anything
else that compares to that. There isn't another 20 year project that's
happening. No, I think RT Linux has been, I mean it's
famously been in the works for a very
long time. Another topic on the
kernel and that is creating a lot of conversation,
let's say, is obviously rust. And one
of the maintainers stepped down last week saying,
and I quote non technical nonsense as
the reason for stepping down. And interestingly, there was quite
a bit of argument, let's say
between the folks from Asahi working on the
Apple silicon GFX drivers and the DRM scat people.
So why is this so hard?
I think I actually enjoy it. I enjoy arguments.
I think one of the nice parts about Rust has been how it's
livened up. Some of the discussions and
I mean some of the arguments get nasty and people do actually,
yes, decide this is not worth my time.
But at the same time it's kind of interesting and I think it shows how
much people care. At the same time, I'm not quite
sure why Rust has been such a contentious
area. It reminds me of when I was
young and people were arguing about Vi versus
emacs. They still are maybe they still are.
And I've just. Should we have a poll here in the room? Yeah. No,
no. But for some reason, the whole rust versus c discussion
has taken almost religious overtones in
certain areas. Well, so a big part of this is
that, from a design philosophy perspective,
if you want to actually write memory safe rust code,
there are certain architectural requirements. And it seems
that a lot of the older, and I don't mean age, I mean,
just having been around for a longer time,
maintainers are fairly resistant to accepting
the fact that if you have a rust subsystem,
it requires you to have certain failure handling
which they don't want to add because you say, oh, that's a bug. That should
never happen. I mean, we have been doing this for over three decades
and seas, in the end, a very simple language.
It's one of the reasons I enjoy C and why a lot of C programmers
enjoy C, even if the other side of that picture is
obviously that, because it's simple. It's also very easy to
make mistakes, and Rust
is not. Rust is a very different thing,
and there's a lot of people who are used to
the C model, and they don't necessarily like the differences.
And that's okay.
In the kernel itself, almost nobody,
I mean, I won't even say almost absolutely nobody,
understands everything. In the kernel, I don't.
I rely heavily on maintainers of
various subsystems, and there are only a few areas
where I get, personally, very much involved.
And I think the same is true of rust
and sea, is that you should expect that not everybody will understand
rust. And on the other hand, not all the rust people necessarily
understand sea. And that's okay. I think it's actually
one of the strengths we have in the kernel that we can specialize.
And some people care about drivers and very specific drivers, and that
at that, some people care about specific architectures,
and some people like file systems. And that's how it should
be, and that's how I see
Rust. But clearly, there are people who just don't like
the notion of rust and having rust encroach on their area.
And I think it's been interesting,
and I'm not afraid. I mean, people have even
talked about the rust integration being a failure.
And I'm actually.
I don't think a. We've been doing this for a couple of years now,
so it's way too early to even say that. But I also think
even if it were to become a failure, and I don't think it will,
that's how you learn so I see the whole rust thing
as positive, even if the arguments are not necessarily always.
Yeah, and I mean, the challenge really is the idea
that a memory safe architecture makes certain assumptions about
the infrastructure. And it's the infrastructure people
like DRM Sket as the example where it has become very
public. It's the infrastructure people who seem to resist
some of the changes. To be fair. I mean, we've done a
lot of changes to the C infrastructure too. The kind of code,
what the kernel does is not
normalcy. We have. It's not just that we've
write things in a certain way. It's also that we have a
lot of tools on the C side that enforce infrastructure
rules that are not part of the language. And part of this is
our locking safety. We do have actually a lot of memory
safety infrastructure on the C side that is not
technically part of C itself. It's part of our infrastructure
for the kernel, where we have debug
builds that make the kernel go much slower, but it
will test a lot more of the memory safety stuff.
So on the seaside, though, we have
been able to kind of incrementally add it. And I mean,
we have a lot of that kind of support, but we've had
a long time to add it and that kind of
has allowed people to do it without
the outcry. And the rust change
is obviously a much bigger and much more in your face thing.
And so you said that it's too early to say it's a failure,
and I agree with that. But one of the things that I've seen
lately is a certain amount of interest in bottoms
up, grown up from the start, rust kernels
as an alternative to Linux. So is
that a potential outcome if we continue to
see struggle to get Rust into Linux? That there will be something like
redux or Maestro or one of the other ones
that is popping up, that we will have an alternative universe built
around the different kernel. I think that's a, regardless of
the Linux use of Rust, I think if
you're doing operating systems, you really don't have very many choices when
it comes to languages, because you require a language
that can deal with system issues.
And that automatically means that your language
choices are very limited. Unless you want to write
an assembler, you're going to use C or one
of the c like languages, or you're likely going to
use Rust. And I don't think,
I mean, Linux is not everywhere.
We are very widely spread, but after three
decades, Linux is also very big. And we've always
seen these embedded systems in particular
who just want something smaller and simpler and
not quite as fully fledged.
And that's when you would use rust or C or anything
else. And that will never go away, I think. I mean, obviously we see
things like Zephyr and other embedded, deeply embedded
kernels, but as a general purpose os,
I think we can say Linux is absolutely everywhere.
It's today there is a complete Linux distribution
running on most 5g modern chips.
So in your iPhone is a chip that runs as its
firmware Linux. So it is everywhere.
I mean, yes, we used to have this joke.
I mean, many many many years ago we used to talk about world domination
and that joke became reality and isn't
funny anymore. And I'm okay with that. But at
the same time, I mean, nothing lasts forever. And I'm
sure some clueless young person will
decide how hard can it be and start his own operating
system in rust or in something else.
And if he keeps at it for many,
many decades, or she and many many decades,
hey, I'm looking forward to seeing that. To save you
from bad quotes. When you say some clueless person, that is a
reflection on your own past. Oh, absolutely, yeah. You have to, you have to be
all kinds of stupid to say,
I can do this, right? Because it turns out,
yes, I'm still doing it 33 years later.
And a, I couldn't have done it without all the literally tens
of thousands of other people anyway.
And the only reason I ever started was that I didn't know
how hard it would be. But that's what makes it fun.
So that is actually the perfect bridge to the next set
of questions that I have for you. So you said the maintainer summit is
tomorrow and an evergreen topic that I think comes up
at every maintainer summit and certainly that I bring up in
our conversations every year, every other year is this bigger question
of burnout. So the maintainers are an
aging group. Strangely, some of us have, you know, not quite
as much or the right color of hair anymore. Hey, gray is the
right color. Gray is a great color, yes.
But the Linux ecosystem,
the kernel development community, there is absolutely no
sign of slowing down. It's accelerating in many ways, rust being
one of them. And so the question that I always ask
myself is time is kind of cruel stop
for us. And as the maintainers get older,
as burnout becomes a bigger problem,
is it about time to talk about mini
Linus? About the successor,
the next generation? We have talked about that forever.
And people are probably, some people are probably still disappointed that
I'm still here. No,
I mean it is absolutely true that kernel maintainers are aging.
And the positive spin of
that is how many projects do any of you
know about where you have maintainers, not just
me, who have literally been around for over three decades.
It is very unusual. So then saying,
hey, people burn out and go away.
Yes, that's true, but that's kind of normal.
What is not normal is that people actually stay around for decades.
That's the unusual thing. And I think that's to some degree
a good sign. But obviously it also does mean that
as a new developer, as a young person it
can seem kind of hard to become
part of this when you see all these people who have been around for decades.
At the same time, I think in
open source the Linux kernel is actually very
unusual in just how many developers we have.
So it is very true that we have maintainers
who have been around for a long time and who are getting gray.
But at the same time if you actually look at the statistics of
the developers we have very few
if any projects have 1000
developers being involved in every single release and the
releases being every three months or so. So I think we do
have a fairly healthy developer subsystem.
The whole monkey dance about developers, developers,
developers, we've got them. The fact that we also have
these old graying people around,
I don't see that as a huge problem. I'm not trying to say
it's a problem. So what I'm trying to say is you've been doing this for
33 years. I don't want to be morbid, but I think in 33
years you may no longer be doing this.
Possibly. And even though I would love to still
do the conference circuit in 33 years. But we
come in on our walkers, it's going to be awesome.
What I'm saying is in order to be the king
of Linux, the main maintainer, you have to
have a lot of experience. And the backup right
now is Greg Kh who has even less hair than the two of
us and is about the same age as we
are. Greg, are you in the room? I apologize. You can punch me later.
But what I'm saying is how do we get the next
generation to gain the experience so
that in 1015 2030 years your
role can be handed off to somebody else? It's that sustainability
that I'm asking about actually. So I think part of the
issue with us having a lot of developers is that we've always had
a lot of people who are very competent
and could step up. I mean you mentioned Greg. But the thing is Greg
hasn't always been Greg. Before Greg there's been Andrews
and Alans and after Greg there will be
Shannons and Steves. Who knows?
There are people around who have been around for decades.
And the real issue is you have to have a
person or a group of people that people in
the development community can trust. And part
of trust is fundamentally about having been around
for long enough that people know how you work. But long enough
does not need to be 30 years. We have very
core developers that are like top level maintainers
for major subsystems that have come
up in just a few years.
It's not instant, nobody's going to claim that.
But there are new people who come
in and three years later they are a main developer.
It is not impossible at all. Being a maintainer
is not as glorious as it sometimes
can seem, right? And there's a lot of people
who would be more than happy to get
new people in and come and help. And sometimes
people are more afraid of kernel development than they really need to be.
I am sure a lot of people are currently googling Shannon
and Steve as first names of kernel developers to know who you're talking about.
They were random names I made up. Just out of interest, how many people here
are actually kernel developers have sent in patches.
I mean kernel developers are odd, we know that,
but I mean we're not so unusual as to be unheard
of. And it is the most interesting thing you can do,
I'm convinced. Still. Yes, we all know kernel development is
the peak interest of things a person can do.
So why are people
laughing at me? I should not have repeated that sentence.
What is interesting to me is to now extrapolate from this
and look at the larger open source community. Because one of the
things that we've seen just in the last three, four years is
that a lot of the large companies who have
very, very aggressively funded open source development for
so many years seem to have stepped back a little
bit. Lots of layoffs, lots of companies noticeably
reducing their involvement. And since many of them
are sponsors of this event, I will not name them,
but it's easy enough to figure out.
And so at the same time we see more and
more attacks on the fabric of how
open source works on this trust model you just talked about
by threat actors, be it criminals,
be it nation states, whatever. So to
me this overall question of how can we improve the
health of the idea of being a maintainer and open source is
something that I'm very interested in. And one of the
things that Linux has done so amazingly well is to
have had an incredibly deep bench of a lot of people
involved and a lot of people who aren't afraid to
speak up and say hey, this is wrong, or I don't like this,
or this is a mistake. And is
there a way to take what we have learned in Linux and apply this to
a broader set of projects? I actually think
that question is very pessimistic in itself because I remember
when I was young,
long, long ago, and open source
as a name was only very new.
And I think the open source communities today are
much better off than we used to be.
And yes, there are technology layoffs
and they seem to be very cyclical. And it's
not necessarily always a nice industry to be in.
But at the same time I feel like open source
is doing very well. People take
open source for granted now in a way that they
absolutely did not just two decades ago.
And maybe that then has resulted in some more negative
commentary and more attacks. But I
would say that one of the strengths of open source is
that it is now so ubiquitous that
as a new programmer in particular you can
use open source as a way to enter this
not so nice industry always and
make connections without
necessarily having gone to the right schools or
having the personal connections that you often need in many
other industries. So I think open source is doing really well.
I agree, open source is doing well. It's much more this question
of how do we create more sustainability.
But one thing that you just said, really I find interesting new
people coming in. So when we all started
a few decades ago, it was a much smaller, much simpler
world. Today everything is hybridization versus reality.
And before anybody thinks, I'm not just talking about AI,
we have so many projects that seem to be all about
the quick buck, about the quick exit versus things
that are making a difference. If you were
starting today, do you think it would be easy to find
interesting, rewarding, long term useful projects?
I don't think that has ever been easy. I mean
what has happened is that there's a much bigger mass that can do so
and all the infrastructure is there. So it's so easy to
basically do a few web clicks and you have a GitHub repository
and you can start doing open source. And you don't have to explain
why you're doing open source because people take that for granted now.
So there's a lot more of these small projects
that you would never have seen 30 years ago because it's
so much easier. But it
was never easy to find something meaningful that
you could really spend decades doing. And I
don't think that's easy today either. You just have
to come up with something that you're
interested in, but at the same time, you're not the only one interested
in it. That's. I think one of the problems is that
people say, hey, do what you love. But if
what you love is something that nobody else cares about, you're not going to create
the next big, successful, open source project.
So you have to love something that is meaningful and
you have to find it. And that's sometimes hard to do,
especially in the tech industry, where so much of it is
about the hype and everybody is following
everybody else, like lemmings off a cliff, trying to
chase the next big thing. And I
don't think that's a successful strategy. I think you need to
find something that isn't what everybody else does and
excel at that and be the first to do something slightly
different. You called me early. A negative. I was hoping we
would end this conversation on this wonderful, inspiring highlight,
Linus, telling the community where to go and make a difference.
Instead, we're ending on lemmings falling off a cliff.
I have zero minutes left, so unfortunately, I can't rescue
this with an incredibly clever question.
But I think we'll just leave it here and tell
all of you you are not the lemmings he's talking about. And enjoy
the week in Vienna. The weather is supposed to be much better. And thank
you everyone, for making it in with all the flooded roads
and tunnels and whatnot, and have a great show. Thank you.