You are many things and I can't explain it properly, so you should introduce yourself.
But yeah, your microphone's right here. Yep. Press hard
for the next slide backwards and a pointer if you need it. Great, thanks.
Thank you so much. Hi, everyone, I'm Peter.
I'm going to kind of take things in a bit more of a philosophical kind
of direction. You know, I work for a research lab these days,
so that's kind of my tendency.
The lab is called Ink and Switch. We are interested
in carrying on kind of the work of Engelbart and Lick
Leiter and Alan Kay and these people. We're interested in how
computers can augment human intelligence.
And that's very highfalutin. It's a lot of fun.
We get to dig into all kinds of weird corners of software and
experimental interfaces and so on. But before that, I have written a
lot of production software. So I'm not just speaking to you from kind of an
ivory tower perspective. I was one of the early employees
at a platform as a service company called Heroku.
I've run a lot of postgres databases.
I've worked in game development as well. I've built titles for DS and gba.
I worked on this one, Teenage Invasion of the Alien Brain
thingies for a Victoria, B.C. game company. And I've
worked on desktop software like Songbird, which was a
cross platform media player that lost fairly decisively.
And besides that, I've also worked in research,
spending some time on, for example, this ship here, the Sir
Wilfrid Laurier. It's a Canadian Arctic research icebreaker.
So I spent some time in the Arctic and that actually helps inform my
perspective on things too. But we'll come back to that. So I want to start
by kind of saying, hold on a sec. Before we get
into talking about why we can't make simple software,
we should probably define our terms a little bit and talk about what
complexity actually is. So the first thing I want to clarify
is that complexity is not difficulty.
Just because something is hard doesn't mean it's complicated. The ideas of calculus
might be very difficult to internalize, but they're actually very simple
and elegant. And I also want to draw a distinction between
something being complex or just being kind of
big. A box of Lego is not necessarily
a complicated thing, though you can make a lot of things with it.
Complexity occurs when your systems interact with each
other and when it takes a lot to
get things done because of that. More specifically, I think
the problem with complexity is that your systems can become
unreasonable. When I Say unreasonable. I don't mean in a
colloquial sense. What I mean is that you literally cannot
reason about them. You don't have the ability to predict
what's going to happen. A lot of the time that's a problem
because you got work to do and the thing is crashing and you don't know
why. But a complex system can also be generative and
surprising if you harness that complexity in positive
ways. Last, before we get into it, I want to make
just a very small detail point for the more
pedantic in the audience. Some people draw a very
specific difference between complex systems from sort
of chaos math and like emergent complexity. We'll talk
a little bit about that. As distinguished from complicated systems
where you have a lot of just kind of mess.
I'm kind of going to float back and forth across both
of these without a lot of distinction because I think in this context either
one can cause similar effects.
Okay, so we're going to begin
with a non comprehensive look at
complexity. More specifically, we're going to cherry
pick some examples from fake industrial scenarios where
you can begin with something simple and end up with something complicated.
So I'm going to start with the most basic of all
reasons why your software gets complicated. And that's the first
version of your code. You just write the happy path, everything's glorious. And.
And software can be really simple when it lives in an idealized world and everything
is fine. And so, you know, here's like a really simple imaginary
pseudocode web thing. Great, we got a get request to our
application. We're going to go fetch something from the database and say, hello there.
This is a hello World app of web apps. Server apps.
Great. And it works fine if user 12 is me and user 10
is my friend Keiko and user Fu is. Wait, what?
Right, we got to validate the input.
Okay, so we'll validate the request now.
Okay, so we're validating the request and that's good. And geez,
I'm not really handling it if we don't validate the request.
And wait, do I have to do this on every request? I guess I
have to do this on every request. And am I doing it the same way
on every request?
In fact, this exact pattern of validating
user input and checking your arguments can lead to something that Bradis
and Patterson call shotgun parsing. Shotgun parsing is kind of the
idea that you're parsing your input, but it's like someone's taken a shotgun
to your code base because it's just blown through the whole
code base. It can lead to very serious security vulnerabilities because
you might carefully check an input in one place but
not in another. And there's a Great talk from Brewcon 2012 about
this. So of course now we're
going to be a little more careful. So we're going to also, if the
database is down, we don't want to send a 500. We're going to fail elegantly
as you go. The more of these cases that you have to handle
and the better the quality of your code is, the more
conservative, defensive, thorough that you are at
this point. Now the actual logic of this
method is beginning to disappear into all the edge case handling.
We're well off the happy path now.
You can clean this up a little bit. Imagine here in this case that
we have a slightly better set of libraries,
more thoroughly written set of libraries. We've moved the complexity down out
of this method and we're saying, well, the database won't throw an exception
because exceptions are ugly and subtle to handle well.
It'll return null if it can't do the job for whatever reason. And we're going
to say, well, there's a good type system
that's handling parsing the input args before they get to us, so we
don't have to worry about that anymore. And the value of well written libraries
and languages, type systems and so on is that they can really
simplify your code, make it less complex by preventing you from
have to carry around all, all of this mental state
everywhere that you go.
My observation of this world is that vigilance is not a strategy.
We talked earlier about how if a system fails,
it's not oh blameless postmortemist. It's not that the developer
failed, it's that the system failed. If you have security vulnerabilities
in your code because you didn't handle input, the answer isn't to berate
yourself for not handling the input. It's to step back, to level up and
say, well, hold on, why do I have to do this manually every time?
Strategically you want to minimize the scope of failures and
design them out of your system. One way to do that is to have a
type system so you don't forget to check your nulls. There are other
approaches. I don't think this is universally true, but it's one way.
Let's talk about scale. We all know as an
array gets bigger, you can have an O of n log n kind of sort
algorithm. Great. Well, scale's easy. Let's Just imagine an admin panel
for a web application. You list your users. Looks kind of like this. Go to
your database, select all the users, and then in your front end, just show them
all. Perfect. Easy. It's fine. You've only got 10 users. Who cares?
Okay, well, now we're at 10 to the 4. Okay, you've got a thousand users
or so. Okay, well, okay, we can't just render all
that on screen. Okay, well, we're going to use one of those offset SQL
things, right? We know about this. And then we're going to need pagination,
I guess. And you got to click through. Okay, yeah, sure. We can still do
this. This is fine. You don't want to request all that. It's a pain to
scroll through. Okay, well, now we're 10 to the six users. We got a million.
Actually, that offset thing, that's a problem if you have a lot of
users, because in order to generate page 850,
it actually selects all of those rows and scrolls through them and just
takes all of those results and just chucks them on the floor until it gets
to the ones that you asked about. And actually now, even though nothing has
changed about our problem statement, like,
not only are the performance characteristics of the system starting to
change and the user interface is starting to change, but the
way that we interact with the problem is changing too. Because if you've got
a million users, the way to find a user is not to just like,
keep clicking next until you find them one page after page
350. I'm on the J's. Okay, click forward to change the
query. ARG to force. We've all done these kinds of things when the tools didn't
keep up with our admin system systems.
But the point I'm trying to make here is that once
you start to reach 100 million users, it's not
even the same kind of problem anymore. The complexity
that's snuck in here is now that we have different responsibilities.
There are ethical, there are legal, there are policy responsibilities. You probably
have a team of people at your organization who are responsible for dealing with
abuse. Some of these people are terrible, and you want them off your platform and
you can't find them. Right? Like, it's a different thing.
And it's complicated because the environment has changed as much
as anything else. So everything changes
when your scale changes. It's not just like an algorithmic thing,
it's also the kinds of things that you're doing change.
And, like, in general, my advice to someone who's built a lot of high scale
production systems through many orders of magnitude is that
it's as harmful to build for the system of the future as
it is to build an insufficient system for the
present. If you build the tools for 100 million users, when you
have 100 users, they're going to be completely unusable too. And you're going to waste
so much time trying to solve problems you don't have yet that you're going to
miss out on all the benefits of being able to just look at all your
users and see who they are and talk to them. So you need to be
scale appropriate and kind of be looking ahead.
Another way software can get complicated is with leaky abstractions.
Here we have a platform that's a little bit skewed and so complexity
can kind of bubble up from under the waterline through these imperfect
abstractions. This is how you copy a string. It's a nice
handmade low level. This is an excerpt from Kernighan & Ritchie 1988 2nd
edition C programming Language. This book was such a revelation for
me how it showed me the way kernel functions
were really implemented. No, I'm just kidding. They're not really implemented
like this. They say here that this is how a C programmer
would prefer to write string copy. The C
programmer in question has never forgotten to null terminate a string. I see.
But this is how string copy is actually implemented as
of 2022 for Alpha in
Linux. And actually this isn't how string copy is implemented. This is like one
page of assembler. There's like 300 lines of
assembly. And really importantly and intriguingly,
although the interface remains the same,
modern CPU architectures have so many more constraints
around memory, alignment and everything else that although the interface
is preserved, you can pass any pointers in and get values back. The kind
of performance characteristics of doing an unaligned versus
an aligned string copy will be dramatically
different. And so complexity
here I think we can kind of call this a win.
We've got this super powered computer under the hood now, but it's still kind
of feels like that old jalopy they were running in 1988.
And that's cool that it works. But in
a sense this API that's so simple is
actually undermining the value of the system. And there's a lot of magic
happening to kind of hide this. And so I feel very mixed
about this because on the one hand it's sort of like SQL databases where
you can add an index and change nothing else and things get faster.
But there's like A really important subtle
lie that's being told here. But maybe this is good and maybe
it's bad. This is like a value judgment kind of question.
And whether you understand this is happening could be completely irrelevant, or it
could be the difference between the success and failure of a project.
Okay, so these are kind of like technical complexities
that come in. I want to talk about other sources
of complexity, different kinds of complexity, and one really
big source of complexity in my projects. Maybe you're smarter than me, and that's
cool, but it's when you have a gap between the problem you think you
have or the problem you used to have and the problem you have right now.
And again, we're going to use a toy example to illustrate this. So you've got
a users table, just same app as before.
And you know your users. There's a bunch of fields. We got a first name
and a last name. And how do you get the first name and the last
name? It's easy, guys. You just split
on a space. As a Van
Hardenberg,
you can see how this might get you in trouble. And in fact, on my
California driver's license, it said that my middle name was Van for
many years. And the problem here is that
the model that you have for the problem doesn't
actually map the problem domain successfully.
So what do you do?
And there are lots of things you can do. You can rewrite from scratch,
you can patch around it. And actually, just as an aside for
everybody in the room, the W3C has a wonderful essay
about personal names around the world. And just if you
take one little thing from this talk, don't use first name
and last name. Don't use family name. People's family names aren't accurate. Some people
have them and others don't. What you do is you have one field that is
what is the full name? And then the other one is, if you need it,
what should we call you? It's like a much better model.
Also, Unicode,
I know that's its own set of complexity, but.
And so there's only so many things you can do
when you have this problem. And I had a really specific, concrete example
of this in a distributed system recently where, like, I had a mental
model of how the system behaved, and my tests demonstrated that
it behaved that way. And then in my production environment, it behaved completely
differently because of behaviors that were not
modeled by my understanding of the problem. And this is part of why
distributed systems are so prone to this is because there's so
many new free variables. And like Articulation
points where things can be fast or slow or where they can fail
or arrive out of order and all these kinds of things. And so when
you have these kinds of model reality gaps, you have to bridge them somehow.
And so what can you do? Well, you can fix the problem, you can improve
your understanding and rewrite everything. And when you can
do that, it's probably best.
You can't really always do that, though. And more to the point,
you may still not actually, you could see the problem, but you may still not
actually understand the solution. And so what can you. Well,
you can hack around it. I've put Van Hardenberg into a
lot of text boxes without a space in it where the
first letter is capitalized and the H is lowercase, and that's not how I spell
my name, but it is how a lot of systems spell my name. Or you
can ignore the problem, maybe it's fine, and genuinely, it could be
fine for your use case. You're just like, well, I guess that Peter
guy can suck it. He'll deal with having his name spelled wrong. And I do.
These are options that you have. And there's like a great meme
series out there on the Internet lies programmers believe about pretty much
anything. Like all time zones are one hour apart,
half an hour later in Newfoundland. And so
I think this is a very fundamental source
of complexity. But where things get really bad is
when your problems start to multiply off each other.
And this of course, is where you have kind of like
compound interest on complexity. So what happens
if your problems dimensionalize against each other?
So again, imagine this web application. You've got a bunch of different browsers to
support. You've got a bunch of different runtime environments to support,
different screen sizes, network speeds, different OS or browser
versions. And so if you want to actually understand what's
happening, all of these things multiply against
each other. So you don't just have one runtime environment,
you might have one code base, but you have lots of different contexts.
So how can you know that you actually have correct functioning code?
Because the old joke about Docker like, well, it worked on my computer.
Well, I guess we'll ship your computer. Well, even that doesn't really work because once
you get out into the real world, there's different memory context,
so you're up against different loads or whatever else, right? You don't actually configure control
the whole environment. And so this is what really starts
to kill you with complexity. And it's where all of those smaller
inconsistencies play off each other to create an unknowably complicated
environment. And this is why you see so much
popularity and amberjan were kind of hinting
at this, like, you don't use the native APIs, because God help you if
you're trying to figure out how to map all of these totally
different environments down to one consistent thing. And one
solution here is just to only have one thing as best you can and to
minimize the difference between those environments. And that's why you see sort of like ostensibly
lazy electron apps from big companies. Because even worse than
having to support all these different things is having to coordinate all the different people
and teams to try and build features, right?
And we've all, many of us have been there, right? Like, if you have an
iOS and an Android team, like, how do you get them to ship the same
feature in the same quarter? Right? That's tough. It's tough.
Okay, so I hope I've
convinced you now that complexity
is like a complex problem itself and
that it manifests in a lot of different ways. So we're going to change gears
a little bit and talk about seat belts and
more specifically, why seat belts don't save lives.
I have an asterisk here because this is a little bit of a controversial research
topic. But the upshot of this
is that in at least some studies, they found that after seatbelts
became mandatory in cars,
people still died on the roads at similar rates.
That was really surprising because seatbelts
are a good idea. They keep you safer in the car. Somebody proposed
this idea called risk homeostasis.
The idea behind risk homeostasis is that actually
we die at a rate on the road related to how much risk we're willing
to take. And so if you now have a seatbelt and you're driving a
car that's safer, you're going to go a little faster and you're
going to take the corners a little tighter because you feel better.
The issue with making people safer was that
they then took on more risky behavior, conserving their total
risk tolerance. And so I think we see a similar thing
in software, right? I call this complexity homeostasis,
which is to say that if you have kind of a system that's evolving
over time, you know, everything's fine, we're going along,
life is good, and then we add some more things and,
you know, we're still happy. And then, oh, yeah, now it's.
I don't like it anymore. It's not good. It feels bad.
Now it's time for that rewrite that we were hearing about. Right. It's time to
bring things back. Ah, okay. Now we can
go back to making things complicated again.
And so this is kind of like a set point. So homeostasis is the process
where our. Or it's any process where you sort of maintain a set point.
And it's commonly used to talk also about how our bodies regulate temperature,
but I think our organizations regulate complexity.
And so all of us have different intuitions. And when people talk about wanting things
to be simple, there's like an aesthetic preference here. And different people
perceive that and pursue that differently.
And like Duveen's quest for just the
right number of opcodes that we heard about yesterday is a great example where,
like, in a certain sense, the correct number of opcodes is one,
and you could do that if you really wanted to, but then you've got kind
of like the CISC op code model where actually the correct number is hundreds because
you're maintaining backwards compatibility. And anything that can squeeze you either
better numbers out of a benchmark or more CPU sales
is acceptable. How you perceive what constitutes
complexity or how much complexity you're willing to tolerate is
an individual or an organizational decision.
Some things can actually move that up.
Like, I have abandoned projects because they got complicated and annoying
to work on. But, like, some people might just tolerate
that and not perceive it. I've worked with some brilliant people who write the
most complicated, insane, convoluted code from my perspective.
But when I sort of go to them and I'm like, this is so complicated
and weird. Why did you do it this way? They don't see it because they're
just much smarter than me. They're able to hold that complexity in their head
comfortably. And so for them, it doesn't feel that way.
Another way, a system can become more complicated if it's worth a lot
of money, right? Because, like, you can just hire another poor
schmo to sling Java into the code base. I mean,
has anybody here worked on an EA Sports title?
Yeah, I'm so sorry.
I have heard some things, man. And it ships every year
on deadline, without fail, right? Like, that's not an environment
that leads to healthy refactoring or reduction of complexity.
But you know what? It makes a boatload of money.
And so they'll hire people to work on it because they can afford to.
And you'll work on it because they'll pay you enough, because they know.
They know, right? And so part of this
as well is that also if you just have more people working on things you
can tolerate more complexity. I can hold part of it in my head and you
can hold part of it in your head. And I kind of want to distinguish
like there's sort of the breadth of complexity and if you have
a well factored system, you can decompose the complexity either into
layers or modules. Right. Like each individual LEGO
brick is simple, but you can build very complicated things from it.
Right. And so that's sort of the system complexity versus the component
complexity. CSS is incredibly
complicated. And by that what I mean it's complex. You can't tell from looking at
one rule what it will manifest as in a final document.
Okay, well we can just solve this problem with better tools, right? Well, no,
because as we've talked about, we have this homeostasis
point that we fall towards over time. We can choose where to set
it, but we do move towards it. The research
on this actually goes all the way back to the 1860s. This is
Jeffons paradox, which was like, why did coal consumption
not decrease when engines became more efficient?
The answer is people did more work.
I feel very much that this is the
inevitable conclusion we have to draw from looking at the problem, which is that the
degree of complexity of a system is tied to who we are and what we're
doing over time.
When we buy back some complexity by using better tools or
by picking a simpler environment, we're going to spend
that out again eventually.
Okay, so let's talk a little bit about some theories of complexity,
because I'm from a research lab and I like to read
papers. So this is not a new
problem. Have people heard of cyclometric complexity before?
Yeah, we're not going to do this. You're welcome.
But I do think that although you, you know, sort of the naive pursuit
of like software metrics as a way of measuring what
we're doing is not wise. I think that the conceptual
understanding that like when you couple things together that that can
have a cost and be problematic is worth thinking about. In fact, people have
been looking at this and thinking about it since the 1980s. This is
Mayor Lehman in 1980's IEEE Volume
68. So like, this is not a new domain. And so even back
then this edition was from 1980. The first paper published was
in 1974 on these laws of software engineering. We're not going to read
these super close. It's more that I want you to realize
that this kind of recognition that
systems grow to meet growing needs if they're successful
is not a new idea. It's a common problem. And whether you're working
in building tools or languages, you might start with an
elegant and simple thing. But as a system has
more demands on it, it responds by adding,
that's a very common, and it's a reasonable consequence. And if you don't do those
things, the thing will often starve and die.
One great paper, if you haven't read it yet on this topic,
is out of the Tar Pit by Moseley. This is from 06.
This paper differentiates between what is
accidentally complex versus essentially complex.
Essential complexity is the irreducible,
non eliminable part of your system.
If you are trying to model certain physical processes,
like smoke or fire, like the physics part,
where you're actually doing the work, you can't get rid of that. You don't want
to get rid of that. That's what you're here for. But the accidental complexity
is all that other stuff, like, oh God, we got to compile this so
it works Like, Windows 11 has changed the ABI
for this DLL. Like all that stuff where,
you know, we heard about Davine hoisting the phone
up the mast to download 11 gigs
of something,
right? That's mostly accidental complexity for our purposes.
And so it helps as you're working and thinking about complexity to.
To think about where you're spending that budget and whether you're being deliberate
in terms of how you're adopting complexity.
The other thing I want to refer to is this
great essay from the Berkeley DB team in the Architecture of Open
Source. And honestly, I highly recommend reading this if
you just like software engineering, the architecture of Open Source software
is a cool series. Basically what they do
is interview open source communities about their
software and have them write essays about what they've done, and then they publish and
share those. But I think this BerkeleyDB chapter in particular
is exceptional, and I think about this all the
time, which is that software architecture degrades
with changes made to software. So you might have the most
elegant, brilliant, carefully planned system in the world,
but it does not exist at a single point in time in a vacuum.
And as new demands come along, this architecture will decay,
and so it requires a constant shoring up. And when you have big interfaces
that you've invested heavily in and they're straining, it can be extremely
expensive to change them. I also just love
this because it's a little more handmade specific kind of vibe.
The Excel team motto, at least according to Joel Spolsky,
is find your dependencies and eliminate them.
I have been told on reasonably good authority that Excel
actually is built with its own C compiler, that they didn't even want to
rely on other teams within Microsoft and to the degree where they've built
their own compilers. I'm not
saying you necessarily should do that. I'm not saying you necessarily
shouldn't. You got to think about how you're spending your budget.
The last point I want to make on this theme is again,
that complexity isn't necessarily bad. Complexity can
lead to all kinds of wonderful emergent properties.
The Legend of Zelda,
Breath of the Wild. It has had such a wonderful
community grow up around it precisely because they managed to tame complexity
with their chemistry engine. So there's a lot of emergent gameplay properties
and experiences that come out of interactions,
complex interactions between systems. But the systems are factored in a way that
enables and empowers us. And of course,
if you're a roguelike fan, people have been doing this forever
there. Okay, so now
what?
We have now looked at complexity. I've told you,
you can't get rid of it. So we're just going to
have to live with it. And better tools won't save us.
So how are we going to spend this complexity budget that
we have? I mean, one approach is
you just put your head down and you work away and pretend it's
not a problem. And that's a pretty common solution to all of
our problems in life. So we could do that. But I'm going to maybe try
and propose a few ways we can cut the Gordian knot here. And so the
story of the Gordian knot was that there was this sort of this
ox cart with a really complicated knot on the handle.
And people were like, oh, whoever unties this
knot will rule all of Asia. And many people had come.
And then Alexander the Great came along and depending on the version of the story
you hear, he just like, okay, cool. And he cut it with a knife,
right? And so instead of solving the problem by like
trying to be really smart or work really hard, like can we just cheat and
change the rules? I think that's a better approach.
So one approach is to do what folks here like to do,
just start over. You can't actually get rid of this problem, but you can reset
the clock on it. Just build a new one.
Start from the beginning. Everything's easy in the beginning. It's only when you have
users and features that you have complexity.
So make a new programming language,
start a new vm. It's good. You can't really change this long run
pressure. But like Genuinely, you can reset the clock.
And I think that's part of why things like Excel are popular
and successful is they don't have package managers. Right? Like every time you make a
new Excel document, it's a brand new universe, you know,
with none of the misery or mistakes that you made before.
It's like the forgiveness of the blank page, you know, that's.
Then there's something to that. I went to a talk by John Romero
at Strangeloop not that long ago, and he talked about how ID
Software had been making these shovelware games
as contractors, or that the ID software team, him and Romero
and Carmack, had been making these sort of shovelware games.
And their attitude was like, well, you should just always start from scratch.
You get really good at it, you can be really fast, and each time
you do a little bit better than the one before. And we do, I think,
do better. I'm not. I love learning new languages
and building new ecosystems. And I think a big part of the reason why they're
successful is because when you have a new language, you do have the
opportunity to clear away the
standard library and also just the ecosystem of ideas and people and
sort of start from a fresh starting point. Now, this won't cure complexity
in the long run, but it can get better,
at least in the midterm and we can get further. So I think, you know,
that's pretty cool and it's also pretty fun. So, like, I'm all for that.
Please, please do more of that. And to
some extent, it's sort of like, you know, it feels like
we live in these unbreakable regimes where,
like, Unity, you know, rules or where, like, the big companies,
Google and Microsoft and everybody rules. But that's the kind of thing
that's true until it isn't. And there's this lovely quote from Ursula Le Guin about
capitalism, which is sort of the generalization of these problems.
Capitalism was itself an invention,
and I'll just read it. We live in capitalism.
Its power seems inescapable. So did the divine
right of kings. And any human power can be resisted and changed
by human beings.
We should not doubt that
the environment can be changed. The environment was
created, it will be changed. We don't know how or
when, but it is inevitable. Okay,
so you can start over, revolution, burn it down,
or, you know, just do less. Do less
with less. Do I have my play date? Do people have
these? Does anybody got a playdate? Yeah, this is great. It's black
and white. There's only one platform. It's small.
It comes in one color. It only has one color, black or white,
depending on how you think about it. Doesn't have a lot of buttons,
doesn't have much ram, doesn't really have any. There's a little
bit of networking and there's only one hardware platform, so you don't have to worry
about compatibility problems.
Ish. There is the emulator, which behaves
differently, and due to sourcing problems,
they had to get a different microcontroller, but it transpiles
natively. They're doing a really good job. Anyway, it's a heck of a lot simpler
than building for PS5, you know, Xbox,
PC, Switch, 3Ds,
all on one code base. Like, it is genuinely awesome. And because it's
so small, it's like, you know, kind of appealing. And when
you have less scope, you can choose to spend that energy, that complexity
budget. You can put it into polish, right? And like on a small
platform, a small idea can shine.
So, okay, well, we can't solve complexity,
but we can make things worse.
This is actually from Augmenting Human Intellect by Engelbart
in 1962. And it's demonstrating that you
can in fact change people's experience with design
interventions. I said change, not improve.
So, yeah, you can. We've made things pretty bad in terms of software development.
This is my favorite slide always to show. This is the
cloud, native cloud, Cloud foundation,
Cloud, native landscape. I always get this wrong. This is
ostensibly, I think, $20 trillion worth of companies.
Anyway, please memorize all of this. There'll be a quiz at the end if you
want to make a Web application in 2022.
I had to zoom out the browser just to fit it all on screen for
this one. So I'm going to talk a little bit now about our
research, which kind of ties back into this because I think it might be interesting.
Our approach is less the kind of like reboot model
and more of the maybe do a little less with less model.
And one of our research interests
kind of builds on these sort of tools for thought.
A big part of what we're
interested in as a research group is not simplicity versus complexity or decentralization
or anything like that. But we do find that we have to work
in that space because what we want are tools that we
have agency over, that we have ownership of, that are ours and
can't be taken. And so we call our research there
local first software. And the idea is basically
that instead of having to go learn that whole chart of technologies
that you can't actually do any of that stuff
if you just run it on your computer, because that stuff's all in the cloud.
So if you want to build software that works on your computer,
not only do you not get to use all that
stuff, you don't have to use all that stuff.
And so this, you know, it's sort of like simplification,
like via amputation. So we just cut
off most of the cloud and then we build things locally.
And so we've dabbled in a bunch of different platforms over the years.
Right now we're building things in the browser, but storing everything in
IndexEDDB. But one of our core beliefs is that
collaboration is such an important part of thinking
and making and doing that it should be like a really fundamental
part of our platforms. And so we've been exploring technology
that allows you to build software that runs on your computer,
but then collaborates with other people, even allowing online
offline kind of cuts. And the
model underpinning that is a data structure called automerge.
It's sort of like a portable versioned JSON, like data
structure. You could think of it as like, get for your data.
I'm happy to talk at length about this stuff,
but I don't really want to harangue you too much on
our particular kind of development interests. But it's
really incredible just how it feels to build software
that's fast and simple and runs on your computer.
If you, like me, have spent the last chunk of
your career building cloud services and running
things in the sky. I had this really memorable outage
where Amazon turned off US east one dirty
because the generators didn't come on in a hurricane. And we were
sort of like three days into some godforsaken
system rebuild from backups of everybody's databases.
And my coworker turns to me and he goes,
I'm fixing computers that don't exist in
a data center I've never been to for people
I've never met.
And he just had the like thousand yard stare in his eyes and I was
like, yeah, do you need some more coffee? Yeah.
And so, you know, like, in a sense working on this stuff is almost like
penance for me because I created all these single points of failure.
Anyway, so let's recap a little bit of all
of this. How do we live our lives in this complicated world? Right?
So complexity occurs when
our systems have internal interactions,
right? Complexity doesn't mean, oh, there's a lot of stuff.
It's when all that stuff starts to bump against each other and cause
unpredictable Outcomes. And complexity
is also a natural consequence of system incentives.
If you have a lot of people using a thing and you're listening to what
they need and you're evolving as you
learn more, you're going to end up with something complex.
Better tools won't change this.
The complexity is a consequence of who we are and the choices we make.
Sure, you can. You can burn through your budget faster. But ultimately,
where your project ends up on the complexity scale is more about how much time
and how many ideas are invested into it than anything else.
So we can't beat complexity, but we can get beaten by it.
And so what are our coping strategies? Well, we can start over.
We can eradicate dependencies. We can cut scope,
do less. Right. We can simplify our architecture. And a
really big one is being conscious of and learning to kind of identify when
you're getting into these multiplicatory environments,
right? Like once you start porting things to multiple platforms and having
to build those, like, per platform abstractions, how do you manage that complexity back
down? How do you isolate complexity? And a big part of
success is isolating complexity. But I
also want to, you know, give a shout out to gazing into the abyss.
And, you know, you can, you can go, go for it, right? Like,
embrace complexity, harness it. Yeah. It's a deal with. With an elder God.
And you may accomplish great and terrible things, but at a great and terrible price.
You know, that's fine, but when you do this, you got to be
real careful, right? And you want to
be really deliberate about how and when you
take on that complexity. So I guess in closing,
we can't solve complexity, but we can build better
software. Or to put it in the words of cyclist Greg LeMond,
it never gets easier. You just go faster
there. And that's all.