WEBVTT 00:00.560 --> 00:04.400 I've spent my career in commercial software development. 00:06.480 --> 00:11.360 I've had occasional encounters with the military 00:11.860 --> 00:15.480 or government world. Remember working on a project at Lockheed way 00:15.980 --> 00:17.600 back in the day? Does Lockheed still exist? 00:19.360 --> 00:20.880 Yes. Okay, good. 00:22.640 --> 00:24.000 Glad I didn't sink them. 00:26.330 --> 00:30.170 So some of the things that I was hearing 00:30.670 --> 00:34.170 this morning were outside of my usual sphere 00:34.670 --> 00:40.450 of concerns. But much of what I heard was spot 00:40.950 --> 00:44.330 on and the constraints are very similar. 00:45.930 --> 00:49.690 And I also solved a long term puzzle. 00:50.010 --> 00:53.330 The way my brain works is I'll take a problem and I'll stick 00:53.830 --> 00:57.620 it in the back and I'll churn on that. And sometimes 00:58.120 --> 01:01.020 the answer comes within minutes, sometimes it's months, 01:01.740 --> 01:05.460 sometimes it's years or even decades. And so 01:05.960 --> 01:11.100 at the end I'm going to reveal a refinement 01:11.600 --> 01:14.940 of an idea that I've been working on for 10 or 01:15.440 --> 01:19.060 15 years that I think I cracked this morning 01:19.560 --> 01:21.500 as a result of listening to General Whiting. 01:24.060 --> 01:26.980 But I'm just going to leave that out there to make sure that you pay 01:27.480 --> 01:31.180 attention to the whole thing. Is that okay? By the way, 01:32.140 --> 01:35.380 if anybody has questions as we go 01:35.880 --> 01:39.580 along, I'd rather have you raise your hand and ask right away 01:40.860 --> 01:43.340 than sit there and wonder about it. 01:44.140 --> 01:46.540 Asking questions is a form of leadership. 01:47.660 --> 01:51.520 I remember being a freshman in college and 01:51.680 --> 01:56.000 we had a, we had an experimental discrete 01:56.500 --> 01:59.600 math class and Professor Ken Ross, 01:59.840 --> 02:04.240 I still remember him at the whiteboard was 02:04.740 --> 02:07.760 not being terribly clear. And one day he was a little late coming in, 02:08.260 --> 02:11.960 this is probably three weeks in and the whole class was there and 02:12.460 --> 02:15.360 we're sitting there and just kind of the glazed eyes, 02:15.600 --> 02:18.240 everybody looking forward, trying not to make contact, 02:18.920 --> 02:22.600 eye contact with anybody else. And finally somebody 02:23.100 --> 02:26.680 said, does anybody know what's going 02:27.180 --> 02:30.520 on? And everybody said, no, no, we're completely, 02:31.160 --> 02:34.360 completely close about what's going on. And it was such a 02:34.860 --> 02:35.160 moment of. 02:37.880 --> 02:41.360 And then Ken Ross came in and we 02:41.860 --> 02:44.920 just blasted him just like nobody knows what's going on. 02:45.420 --> 02:49.410 Stop whatever you were going to lecture about, please don't explain what 02:49.910 --> 02:53.210 was going. And it was great for everybody. And he ended up writing a very 02:53.710 --> 02:57.570 popular discrete math textbook. And I would like to say that our class really 02:58.070 --> 03:00.650 had a lot of responsibility for that happening. 03:01.930 --> 03:04.410 So if you have a question, 03:05.450 --> 03:08.890 comment, please raise your hand. We have 03:09.370 --> 03:12.090 mic runners ready to go. Go around. 03:13.210 --> 03:15.210 So my name is Kent Beck. 03:17.770 --> 03:21.530 I've been coached by Gene Kim, who many 03:22.030 --> 03:25.930 of you will know that I ought to introduce myself 03:26.650 --> 03:29.930 in the following kind of way. I don't like it, 03:30.090 --> 03:34.050 but Gene says so. So I 03:34.550 --> 03:38.130 was instrumental in bringing patterns and that Pattern style of thinking to 03:38.630 --> 03:41.960 software development. Programmer testing. 03:43.320 --> 03:47.880 Programmer testing frameworks like Junit was my 03:48.380 --> 03:51.720 original invention. I refined it with Eric Gamma and it was then copied 03:52.220 --> 03:55.000 a million different ways. Test driven development, 03:55.640 --> 03:58.440 which is an age old idea. 04:00.040 --> 04:04.280 I recently found a book from 1957 04:04.360 --> 04:10.370 called Something like digital programming where 04:11.170 --> 04:14.890 the distinction between binary computers and decimal 04:15.390 --> 04:20.050 computers had not been worked out yet. Which I thought was like, 04:20.550 --> 04:23.330 oh, I guess that must have been a question at some point. 04:23.730 --> 04:27.450 But late in that book it talks about how do you write programs? 04:27.950 --> 04:31.410 And it talks about you go to the users and you ask 04:31.910 --> 04:34.950 them for some check items, 04:35.590 --> 04:38.950 that is give me some input output pairs 04:39.030 --> 04:42.710 from your domain and we'll make sure that the computer 04:42.870 --> 04:47.950 program matches those input output items. And I thought that's 04:48.450 --> 04:51.590 test driven development right there. So I can't claim 04:52.090 --> 04:56.630 to have invented tdd, but I 04:57.130 --> 05:01.630 maybe rediscovered it or something like that and 05:02.130 --> 05:05.430 then refined it. Extreme programming is 05:05.930 --> 05:09.110 my baby. I was 05:09.610 --> 05:13.430 one of the signatories of the Agile Manifesto. I wasn't just 05:13.930 --> 05:17.430 one of. I was the first signatory of the Agile Manifesto 05:17.990 --> 05:18.870 alphabetically. 05:22.950 --> 05:26.110 And then since then I've spent a lot of time 05:26.610 --> 05:28.790 coaching high potential engineers. 05:30.390 --> 05:34.310 I'm deep in the augmented 05:34.810 --> 05:39.470 development world. I'm working with one 05:39.970 --> 05:43.470 of the frontier labs, which is its own set 05:43.970 --> 05:48.350 of interesting constraints. I mean it's 05:48.850 --> 05:49.270 one of the first. 05:52.390 --> 05:56.470 I work at the intersection of technology and sociology 05:57.350 --> 06:02.110 and this is one of the first times when it's 06:02.610 --> 06:07.230 not a press play kind of problem on how a 06:07.730 --> 06:11.270 team that's building a frontier model or building a sequence of frontier models, 06:11.770 --> 06:15.390 how they should actually work. I'm having to really think, I'm having to learn a 06:15.890 --> 06:17.910 lot, which I appreciate. 06:20.550 --> 06:24.070 So that's my background. I'm coming here to 06:24.570 --> 06:28.630 talk to you about craft and the role of craft in an augmented 06:29.130 --> 06:34.030 development world. I always say augmented development because 06:34.510 --> 06:36.590 to me this is still a human process, 06:39.310 --> 06:42.830 at least so far. I still have a lot 06:43.330 --> 06:46.430 to add to the decisions that 06:46.930 --> 06:50.360 the GENIE is making. I call it the GENIE because it grants 06:50.860 --> 06:53.640 your wishes, but it ends up not being what you really wanted. 06:54.920 --> 06:58.760 Just as a reminder that that's the essential relationship 06:59.260 --> 07:02.280 here, I'm going to make some requests, I'm going to get something back, 07:03.080 --> 07:06.920 it's going to seem really cool and then at the end of it 07:07.420 --> 07:11.080 I'm going to be a little bit disappointed. So that's 07:11.580 --> 07:11.960 how we work together. 07:14.950 --> 07:18.510 I wanted to say I'm not going to bury the lead. 07:19.010 --> 07:22.870 The one thing I would like you to take away from is 07:23.370 --> 07:27.430 there's a tendency now when people talk 07:27.930 --> 07:31.270 about augmented development, where they make a statement 07:31.350 --> 07:34.790 like, now that the genie, they don't 07:35.290 --> 07:39.270 usually call it the genie because they're not going to be that cynical now 07:39.770 --> 07:42.310 that the genie is better at coding than humans. 07:45.570 --> 07:49.330 And I'm calling balderdash on 07:49.830 --> 07:53.730 that statement, that attitude that just because 07:56.290 --> 07:59.970 GENIE can create syntactically correct programs 08:00.530 --> 08:04.770 doesn't mean that they're better than a programmer 08:04.930 --> 08:07.090 or even that they actually work. 08:08.940 --> 08:12.940 And one of the things I always loved as an autistic person, 08:13.020 --> 08:16.060 that I loved about programming is this 08:16.300 --> 08:19.260 binary red, green. Oh, 08:19.760 --> 08:23.580 that's so relaxing to be able to say, this program 08:24.080 --> 08:26.620 doesn't work. I fix a bug and now it's fixed. 08:29.180 --> 08:32.900 And that's the programs that we get out 08:33.400 --> 08:37.430 of augmented development 08:38.310 --> 08:41.350 oftentimes just don't work. 08:42.630 --> 08:46.390 They're plausible. Jeanne's really good at plausible. 08:47.270 --> 08:49.510 But actually working. No. 08:51.750 --> 08:55.910 And we've all read these breathless 08:57.670 --> 09:01.520 accounts of what? Well, we spent $10,000 09:02.020 --> 09:05.280 in tokens and we got a working C compiler out of it. 09:06.000 --> 09:10.560 How amazing is that? And then you start digging and you realize that 09:11.060 --> 09:15.040 working C compiler, hello world, doesn't actually 09:15.199 --> 09:19.360 work. Or there were 09:19.860 --> 09:23.840 a few remaining bugs, and every time one of those bugs got fixed, 09:23.920 --> 09:25.760 three more bugs were introduced. 09:28.730 --> 09:30.490 If you dig a little bit deeper, 09:31.850 --> 09:35.370 it's close to a C compiler. It's C compiler 09:35.870 --> 09:39.770 ish, but it doesn't actually efficiently 09:40.270 --> 09:43.130 compile C programs that then run, 09:43.210 --> 09:47.410 which seems to my old fashioned way of thinking to 09:47.910 --> 09:51.650 be the criteria we ought to be using. So that 09:52.150 --> 09:56.230 idea that somehow it we've just replaced programmers 09:56.730 --> 10:00.030 because computers can do what programmers used to do is 10:00.590 --> 10:04.030 simply wrong. Now maybe someday it 10:04.530 --> 10:08.750 will be less wrong, but today it's 10:09.250 --> 10:13.310 simply not true. And that means your relationship as a 10:13.550 --> 10:17.030 programmer or as a manager of programmers or as 10:17.530 --> 10:20.800 a purchaser of software, if there's 10:21.300 --> 10:24.880 a heavy presence of the 10:25.380 --> 10:29.520 GENIE in the production of some software, you have to take a cynical 10:30.020 --> 10:33.200 view, you have to take an adversarial view and say, 10:33.700 --> 10:37.520 okay, you say this works, you made 10:38.020 --> 10:41.720 yourself happy. But did you make me 10:42.220 --> 10:46.190 happy? How real is that? How hard do 10:46.690 --> 10:50.270 I have to push? So the kinds 10:50.770 --> 10:54.590 of things that you do as a programmer to make sure that a program 10:54.750 --> 10:57.830 works by construction or that 10:58.330 --> 11:02.150 a certain class of bugs is simply not possible. Genie doesn't have 11:02.650 --> 11:06.030 those tricks. So things that you used to be able 11:06.530 --> 11:10.110 to infer, trustworthiness that you used to be able to infer, 11:10.590 --> 11:14.110 you simply just can't assume that that's true anymore. 11:15.470 --> 11:19.230 Which now I've been through 11:19.470 --> 11:22.990 a number of Cycles. One of the advantages of losing 11:23.490 --> 11:27.070 all this hair was I've been through enough cycles where people said, 11:27.390 --> 11:30.830 oh goody, we don't need programmers anymore. 11:31.870 --> 11:35.350 And as programmers, it behooves us to think about why they 11:35.850 --> 11:39.560 keep trying to get rid of us. Kind of at a strategic level. 11:41.880 --> 11:45.080 But also it's never been true, you know, because. 11:49.240 --> 11:53.079 Computer programmers are highly technical artifacts 11:53.579 --> 11:57.640 and we'd love for, say, business people to just be able to speak 11:58.140 --> 12:01.880 in a plain language to the computer so 12:02.380 --> 12:06.120 that we don't need programmers anymore. The business people 12:06.620 --> 12:10.800 can just state their requirements in plain language and the computer will just compute it. 12:11.520 --> 12:15.520 And that, of course, is the high level description of cobol. 12:16.720 --> 12:20.000 But that cycle has been going over and over again. 12:20.560 --> 12:23.600 And here we are at the next round of that cycle, 12:24.880 --> 12:28.960 which is why I'm not worried about having a job. That and because 12:29.460 --> 12:34.140 much of my job is coaching. It turns out that young engineers make 12:34.220 --> 12:38.180 exactly the same young engineer mistakes that young engineers 12:38.680 --> 12:42.460 have been making since the time of the Romans. And they can 12:42.960 --> 12:46.060 use someone who comes along and says, yeah, 12:46.620 --> 12:50.220 yes, you're kind of weird, but you're definitely not alone. 12:50.380 --> 12:52.540 And you know what? I've seen worse. 12:54.300 --> 12:57.740 And that's a very soothing thing for a young, 12:58.700 --> 13:01.250 nearly overwhelmed programmer to hear. 13:06.210 --> 13:10.130 So that word craft that's in the title of my talk is 13:10.630 --> 13:14.850 an interesting word. And I have a mixed relationship 13:14.930 --> 13:16.370 with that word craft. 13:17.650 --> 13:22.930 I've written at least three books which 13:24.370 --> 13:28.370 can all be described as craft oriented. 13:28.750 --> 13:30.110 Okay, you're a programmer. 13:32.750 --> 13:36.270 What is it that, like, how can you put your whole self 13:36.590 --> 13:38.270 into the act of programming? 13:40.590 --> 13:44.150 And all three of those books are completely obsolete and should be thrown 13:44.650 --> 13:48.510 away at this point, which is an odd thing to do. Feel it towards 13:49.010 --> 13:52.110 the end of a career. But there you have it. 13:53.790 --> 13:57.550 The kinds of activities that we used to use 13:58.050 --> 14:00.870 to express craft. Careful naming, 14:01.990 --> 14:05.790 careful decomposition of logic into little 14:06.290 --> 14:09.670 pieces that all compose together to create some larger effect. 14:12.470 --> 14:15.670 Indentation, use of white space, 14:18.390 --> 14:21.830 optimizing program structure for human 14:22.330 --> 14:25.760 understanding. Those aren't 14:26.000 --> 14:29.160 decisions. Those used to be decisions that had a 14:29.660 --> 14:32.880 lot of leverage. Now they tend to have a lot of leverage over time. 14:34.160 --> 14:37.440 So five years later, someone would come across some code and say, 14:37.840 --> 14:41.240 oh, thank goodness I can read and understand this, and they would 14:41.740 --> 14:42.240 feel good about it. 14:45.520 --> 14:49.600 Those decisions don't have the same leverage 14:50.100 --> 14:53.610 that they used to have. And that's just true. It doesn't mean 14:54.110 --> 14:57.930 that humans don't need to understand code. 14:58.410 --> 15:01.770 I think there's still a lot of leverage if I have to choose 15:02.410 --> 15:05.770 a system where the humans can read it and a system where the humans 15:06.270 --> 15:09.810 can't. I'd much rather have a system where the humans can read 15:10.310 --> 15:13.610 and understand about it. But I also have to recognize that that 15:14.010 --> 15:17.290 reading and understanding is going to happen in the context 15:17.790 --> 15:21.370 of I've got this genie, it can explain stuff to me 15:22.490 --> 15:25.690 if it's not lying to me, which is always an option. 15:26.170 --> 15:29.370 But if it's not lying to me, I'm going to be able to understand this 15:29.870 --> 15:34.089 more quickly. But that, 15:34.570 --> 15:37.530 that detail oriented which feels so good. 15:38.330 --> 15:41.570 Oh my good, you know, I'm like, how does this go together? 15:42.070 --> 15:45.620 And then I realize, oh, I inline these things and I extract those 15:46.120 --> 15:48.820 things and I rename these. Ah. Then it all becomes clear. 15:49.220 --> 15:53.340 That moment felt really good to me and 15:53.840 --> 15:56.660 there's just not payoff for that particular moment. 15:58.420 --> 16:01.940 Now, CRAFT was, let me say. 16:06.260 --> 16:09.540 From my perspective, that word craft was hijacked 16:10.640 --> 16:14.960 by a group of people who wanted, 16:16.320 --> 16:20.280 again this from my perspective, wanted to go in their little cave and 16:20.780 --> 16:24.680 they wanted to be programmers and it was just them and the computer and 16:25.180 --> 16:28.800 they would craft themselves until they 16:29.300 --> 16:30.400 were ready to be finished. 16:33.760 --> 16:36.790 This is the flip side of my relationship with that word craft. 16:37.740 --> 16:41.100 I really hated that. Oh no, it's not done yet 16:41.900 --> 16:44.620 because I haven't finished with my craft. 16:45.740 --> 16:49.500 Well, sometimes if the cost of delay is 16:50.000 --> 16:53.740 high, an ugly program that gives you feedback is 16:54.240 --> 16:55.500 exactly the right thing to do. 16:57.900 --> 17:01.660 Other times if the cost of delay 17:02.160 --> 17:05.670 is low, this software is going to live for the next 20 years, 17:06.630 --> 17:10.470 then absolutely. That Swiss watchmaker get 17:10.970 --> 17:13.670 everything exactly right is going to pay off over time. 17:14.550 --> 17:17.910 But the idea that as a programmer 17:19.110 --> 17:23.350 time doesn't matter compared to your experience 17:23.590 --> 17:26.910 of programming, I just disagree 17:27.410 --> 17:32.330 with that strongly. And I didn't like what that 17:32.890 --> 17:36.730 side of programming did with that word craft. 17:37.770 --> 17:41.210 But we have to recognize now 17:41.290 --> 17:45.010 that CRAFT is going to mean 17:45.510 --> 17:49.690 something different. It's still 17:51.370 --> 17:55.370 a valid concern, it's still a valid discipline, 17:56.680 --> 18:00.080 but it's going to show up very differently. And I'll talk as I 18:00.580 --> 18:03.240 go along more about how I see that change happening. 18:13.000 --> 18:14.920 So let me draw a little bit for you all. 18:18.040 --> 18:22.250 I've noticed that if 18:22.750 --> 18:26.610 you look at my GitHub, so I've been programming augmented 18:27.110 --> 18:30.490 development like back into programming hardcore every 18:30.990 --> 18:32.770 day for maybe the last 18 months. 18:34.370 --> 18:37.890 I saw Gene Kim and Steve Yeage give a 18:38.390 --> 18:41.850 demo and they were so excited. I thought I have to get back 18:42.350 --> 18:46.290 into this. And I gotta say, I absolutely love 18:46.450 --> 18:49.900 programming with the genie. I've always had 18:50.540 --> 18:53.740 bigger ambitions than my technical skill 18:54.300 --> 18:58.220 could achieve and now I can try ridiculously 18:58.720 --> 19:01.420 ambitious projects and make a lot of progress. 19:03.260 --> 19:07.260 I didn't say accomplish them because 19:07.740 --> 19:11.860 on my GitHub you'll see that there's a name of a project and 19:12.360 --> 19:15.590 then Project two, Project three, 19:16.470 --> 19:20.310 Project four. Because I kept hitting the wall, I kept 19:20.810 --> 19:24.430 getting to that place where I couldn't fix one bug without breaking 19:24.930 --> 19:28.630 something else. And then new project, new project 19:28.790 --> 19:33.230 two, new project three. Now it somehow never 19:33.730 --> 19:36.790 occurs to me to put a one by that 19:37.290 --> 19:40.950 new project. I just assume this time it'll be different. 19:41.630 --> 19:45.150 This time I'll be able to actually get this thing completed. 19:45.390 --> 19:48.710 But no, I have to release over and over and 19:49.210 --> 19:52.670 over again. So here's 19:53.170 --> 19:56.270 how I've come to think about this. 19:59.470 --> 20:02.430 If we look at progress. 20:03.950 --> 20:07.150 It took me a while to get to this visualization, 20:07.470 --> 20:09.880 by the way. This is. This is one of those ideas that went in the 20:10.380 --> 20:12.480 back of my head and I had to work on for a while. If we 20:12.980 --> 20:14.920 look at progress on features, 20:17.000 --> 20:20.720 what we see is it goes fast at first and then it goes slower 20:21.220 --> 20:25.640 and slower and slower and slower. And that's the like 20:26.040 --> 20:29.800 what goes into that dynamic, 20:30.440 --> 20:32.280 what goes on the other axis? 20:35.410 --> 20:39.650 And sometimes I call it optionality, sometimes I call it futures. 20:39.810 --> 20:43.730 So it's easy to see futures versus Features. 20:44.290 --> 20:47.690 And then I knocked out one of my front teeth and then I had 20:48.190 --> 20:51.690 to call it Futures versus Features and it was a really terrible choice. 20:52.190 --> 20:54.130 But then I got the tooth replaced, so now I'm good. 20:55.570 --> 20:59.330 The vicissitudes of age so here's what 20:59.830 --> 21:03.140 happens. We always start with a wide variety 21:03.640 --> 21:07.540 of things that we could do but we have no features. 21:08.040 --> 21:11.740 And then we implement something and we burn some of our futures 21:12.240 --> 21:15.420 in order to realize that feature. At the very 21:15.920 --> 21:19.220 least we're going to have to maintain backwards compatibility if 21:19.720 --> 21:22.980 we add the next feature, which will constrain what we can do a little bit. 21:23.860 --> 21:26.980 Or we'll decide we don't like that first feature and we'll 21:27.480 --> 21:30.640 have to take it out and then again constrains a little bit what we do 21:31.140 --> 21:35.200 next. But what happens next is 21:35.700 --> 21:39.440 we burn some more futures in 21:39.940 --> 21:44.879 order to get our next feature, and then we burn some more and 21:45.379 --> 21:48.800 then eventually we get down to the point where 21:49.300 --> 21:53.120 we have no options for change at all because everything is such 21:53.620 --> 21:57.590 a big mess. And then that's when I start Project two or 21:58.090 --> 22:02.910 Project three or Project four. So now 22:03.410 --> 22:07.350 we've made a lot of progress in software development. It used to take us 100 22:07.430 --> 22:11.350 people 10 years to get to this absolute 22:11.510 --> 22:15.150 zero state where we can't make any changes without breaking 22:15.650 --> 22:19.150 something else. And now with AI, I can do this 22:19.650 --> 22:22.310 in the privacy of my own office. One person, 22:23.750 --> 22:26.870 one week, and I can make a complete mess. 22:28.470 --> 22:32.430 So that's a big productivity enhancer to 22:32.930 --> 22:36.350 get to a project that can't be changed with far 22:36.850 --> 22:40.270 less in the way of investment. But what's the 22:40.770 --> 22:44.510 alternative? What else can we do? So that's what 22:45.010 --> 22:49.600 I want to create that first feature. 22:50.960 --> 22:54.360 It's always going to burn some futures. Nothing you can do about 22:54.860 --> 22:56.800 that. But now we have a moment. 22:57.760 --> 23:01.160 When I was in music school, there was a story 23:01.660 --> 23:05.400 about Pablo Casals, who was one of the greatest cellists 23:05.900 --> 23:09.080 of the 20th century. And the cello is a very physical 23:09.580 --> 23:12.430 instrument. It's big, it moves a lot of air, takes a lot of feel, 23:12.580 --> 23:16.340 physical strength to play the cello. And Casals 23:16.420 --> 23:19.620 had a piece that was a long run of 23:20.120 --> 23:20.500 16th notes, 23:24.100 --> 23:26.820 on and on and on and on. 23:27.780 --> 23:32.180 And someone asked Casals, don't you get tired playing 23:32.820 --> 23:36.340 all those 16th notes? And he said, no, 23:36.980 --> 23:38.980 I rest between the notes. 23:41.490 --> 23:45.490 And for me, as a young classical guitar major trying 23:45.570 --> 23:48.810 desperately to play fast enough, the idea that 23:49.310 --> 23:53.410 there was a between the notes was just mind boggling. 23:54.050 --> 23:57.770 Now it turns out Casal's never said this. The story's made up, 23:58.270 --> 24:00.370 but it's such a good story that I have to tell it. 24:02.050 --> 24:05.170 But here we have a moment. We've completed a feature. 24:05.930 --> 24:07.930 The genie said, okay, boss, 24:09.530 --> 24:12.370 you know, the finger guns, okay, 24:12.870 --> 24:15.610 boss, I finished that feature. Do you want me to implement the next thing? 24:16.110 --> 24:21.330 And we all have the opportunity to take 24:21.830 --> 24:25.130 a breath and say, huh, maybe I'm not ready. 24:26.010 --> 24:30.050 And that gives us a chance to 24:30.550 --> 24:35.730 add back some of the futures and maybe even a little bit more before 24:35.890 --> 24:37.650 we implement the next feature. 24:39.490 --> 24:42.530 And then again with the finger guns, and then again, 24:42.610 --> 24:44.770 we can say, no, 24:46.610 --> 24:49.570 I'm going to refactor what's there. 24:50.130 --> 24:53.610 I'm going to eliminate some duplication, I'm going 24:54.110 --> 24:58.930 to optimize some of this for 24:59.430 --> 25:03.000 readability. I'm going to throw away what 25:03.500 --> 25:07.160 I just did and implement it again in a different way, see if that turns 25:07.660 --> 25:11.600 out differently. There's a lot of things that you can do in 25:12.100 --> 25:15.640 that moment between features that are going to add value to 25:16.140 --> 25:19.440 the project. Because the economic 25:19.760 --> 25:23.840 value of a project is the sum of the 25:24.340 --> 25:27.400 features that it currently has and the futures, 25:27.900 --> 25:30.230 all the options that for what it could do next, 25:31.590 --> 25:35.550 I can add value by adding to the futures. And so 25:36.050 --> 25:38.550 this is the trajectory that I want to create, 25:41.110 --> 25:44.789 to go back and forth between adding features and 25:45.289 --> 25:48.790 improving the futures of the project. Now, the challenge 25:49.290 --> 25:53.670 with this Is adding features is visible, 25:54.150 --> 25:57.680 legible in that seeing like a state sense. 25:59.280 --> 26:02.840 Everybody can see the progress. They want the features, the users want 26:03.340 --> 26:05.760 the features. Now they have it. Woo. Thank you very much. 26:06.960 --> 26:09.040 Futures is harder to account for. 26:10.240 --> 26:14.120 You may have added to the theoretical value of the software, 26:14.620 --> 26:19.320 but the fact that you could make a change that 26:19.820 --> 26:22.860 you may or may not want to make seems 26:23.360 --> 26:27.340 like gilding the lily. Why didn't you just 26:27.840 --> 26:31.300 implement the next feature? Well, because if I just implement the next feature, 26:31.860 --> 26:34.980 I'm going to go down to zero and everybody hates that. 26:36.580 --> 26:39.700 So this is how I think about the challenge of augmented, 26:39.860 --> 26:44.260 not just augmented development. This is the challenge of software development is 26:44.660 --> 26:48.100 balancing the investment between the next set of features 26:48.870 --> 26:52.310 and the futures which determine what it is 26:52.810 --> 26:56.150 that you can implement next. With the GENIE 26:56.650 --> 26:59.830 involved, the whole thing ramps up. 27:00.470 --> 27:04.390 The timelines are shorter, the opportunity 27:04.469 --> 27:08.350 cost, you know, if you just say, yeah, implement the next feature, 27:08.850 --> 27:11.910 you're going to get it really fast. Something like it, 27:12.630 --> 27:16.480 maybe that works mostly. 27:17.440 --> 27:21.200 And if instead you go and future 27:21.280 --> 27:24.320 proof your software, 27:24.560 --> 27:28.000 you remove the sources of error, 27:31.360 --> 27:35.080 it's harder to account for that. It's not written in 27:35.580 --> 27:39.200 any spec that says, and this will be easier to change in the future. 27:40.650 --> 27:44.650 Nobody seems to ask for that. They're all looking for that project 27:44.730 --> 27:48.130 that gets finished. Whereas for 27:48.630 --> 27:52.090 me, finished, like I don't want to change, this software is a 27:52.590 --> 27:56.010 failure. I want to write software that 27:56.890 --> 28:00.570 inspires any number of new ideas 28:00.810 --> 28:04.210 to me, that's more valuable, but it's harder to 28:04.710 --> 28:06.680 write a spec for that. And I'll talk, 28:07.560 --> 28:10.960 I'll get cynical about spec driven development before 28:11.460 --> 28:16.280 this is over, I promise. So this 28:16.780 --> 28:20.200 is the trajectory I'm trying to create. If you 28:20.700 --> 28:25.000 step back from this as a trajectory far 28:25.500 --> 28:29.400 enough, you don't notice the back and forth. It just looks 28:29.900 --> 28:33.400 like the software is getting more features and 28:33.640 --> 28:35.960 more options at the same time. 28:37.560 --> 28:41.800 If you try to, as an implementer, try and actually implement more features 28:42.300 --> 28:46.120 and more options at the same time, it turns out that's a bigger problem 28:46.620 --> 28:49.760 than fits even into the combined brain of 28:50.260 --> 28:53.880 the GENIE and the human. The more effective strategy is to go back and 28:54.380 --> 28:58.160 forth between the two. We're going to make some progress, 28:58.660 --> 29:01.360 then we're going to consolidate, we're going to make some progress and we're going to 29:01.860 --> 29:05.060 consolidate. You can't wipe the knife while you're cutting. 29:06.020 --> 29:09.300 You have to cut and wipe the knife and cut and wipe the knife, 29:09.780 --> 29:12.900 but you have to wipe the knife. You know, 29:13.400 --> 29:17.260 if you went into a professional kitchen and you said, you know, you're wiping 29:17.760 --> 29:21.500 the knife and it's pretty much clean already. Why don't 29:22.000 --> 29:25.580 you just stop wiping the knife? Then the chef is going to offer to use 29:26.080 --> 29:28.980 the knife on you and then wipe it. So, 29:30.600 --> 29:34.600 you know, the commercial kitchen has developed these strategies 29:35.100 --> 29:40.200 that work out well. And in 29:40.700 --> 29:44.360 software development, I mean, one of the things about extreme programming that I thought 29:44.860 --> 29:48.920 was the biggest compliment I could have gotten was this is what 29:49.320 --> 29:53.000 really good teams look like in the wild. This isn't 29:53.080 --> 29:56.500 theoretical. This is an observation of 29:57.000 --> 30:00.660 what teams really look like. And that was my goal 30:01.160 --> 30:04.580 in extreme programming. If I said anything creative or 30:05.080 --> 30:09.020 I said anything innovative, it was definitely a mistake. And I wanted to take it 30:09.520 --> 30:12.700 out. I wanted proven techniques. 30:14.700 --> 30:18.620 But now we have this much more powerful tool that 30:20.470 --> 30:24.630 can go much faster, that can create 30:25.110 --> 30:28.550 opportunities for feedback faster than we're 30:29.050 --> 30:32.550 prepared to gather that feedback or analyze that feedback. 30:34.070 --> 30:37.910 And the temptation is there to just 30:38.410 --> 30:39.430 let it go full speed ahead. 30:42.870 --> 30:46.750 And for me, this is dangerous anytime 30:47.250 --> 30:49.830 I've tried fully agentic. 30:50.710 --> 30:54.550 The dark factory. The dark software factory. And the 30:55.050 --> 30:58.230 dark doesn't. It's not meant like dark, 30:58.730 --> 31:01.989 like evil. I was kind of hopeful. The dark 31:02.489 --> 31:05.630 software factory, like, oh, I get to finally write evil software. 31:06.130 --> 31:09.550 And no, it turns out that's not what they mean at all. It's just 31:10.050 --> 31:13.670 there's no human involved, there's no feedback. And that 31:14.070 --> 31:16.790 always goes off the rails. Always. 31:18.150 --> 31:22.470 And here's the problem. It's amazing. While it's going off the rails, 31:23.590 --> 31:27.350 the crash is so entertaining, right? You're like, 31:27.750 --> 31:30.230 wow, how did you get this much? 31:31.670 --> 31:35.150 How did you do this much? And it's still complete 31:35.650 --> 31:39.040 crap, but there it is. 31:40.000 --> 31:43.920 So the challenge for me and my personal practice 31:44.420 --> 31:47.440 for the people that I coach, for the teams that I coach, is how do 31:47.940 --> 31:51.600 you create this space between features to go back 31:52.100 --> 31:56.880 and enhance the futures of the work that you're doing and 31:57.120 --> 32:00.680 to do that in this environment where it's so 32:01.180 --> 32:05.040 tempting to just implement the next feature. The next feature. The next feature feature. 32:11.840 --> 32:15.920 So the comment is by pacing yourself, you're giving yourself the 32:16.420 --> 32:20.240 chance to form understanding. And that's exactly right. 32:21.200 --> 32:24.520 The best software that I've ever, that I've worked 32:25.020 --> 32:29.200 on is software where I tried 32:29.700 --> 32:34.000 to maximize learning, which threw 32:34.500 --> 32:38.080 off software is a side effect as opposed to I'm trying 32:38.580 --> 32:41.840 to produce software and sometimes accidentally 32:42.340 --> 32:46.400 I learn something because I messed up your 32:46.900 --> 32:50.600 habits. If you're trying to maximize learning, your habits look completely different, 32:51.100 --> 32:54.680 the pace looks completely different. You're not afraid to throw stuff away 32:55.160 --> 32:58.680 when it gets kind of messy. This is One of my early experiences 33:00.560 --> 33:04.120 working with Ward Cunningham, who was my mentor when 33:04.620 --> 33:08.280 I came straight out of school, the biggest piece of luck I ever 33:08.780 --> 33:15.680 had in my life. Ward invented the wiki and 33:16.400 --> 33:18.160 we had developed some code. 33:20.240 --> 33:23.880 It did what we wanted to, but it was late in 33:24.380 --> 33:27.480 the day, we felt a little bit like, I don't know. 33:27.980 --> 33:31.710 And he just reached over and turned off the computer and I about jumped out 33:32.210 --> 33:35.790 of my skin. Like we implemented this stuff and it was kind 33:36.290 --> 33:39.030 of working and we only felt a little bit bad about it. 33:40.310 --> 33:44.110 How could you destroy that? So I kind 33:44.610 --> 33:47.870 of huffed my way home and came in 33:48.370 --> 33:52.070 the next morning and we re implemented everything that we had done the 33:52.570 --> 33:55.990 entire previous afternoon in about 15 minutes. 33:56.490 --> 34:00.470 And it was absolutely clear and we understood it. And that 34:00.970 --> 34:04.990 was the light bulb moment for me. This is a learning process that 34:05.490 --> 34:06.950 throws off software as a side effect. 34:08.630 --> 34:12.630 Whereas the temptation in the Dark factory, you don't learn anything 34:13.030 --> 34:16.870 except at the end you learn how unreliable 34:17.370 --> 34:20.390 the genie really is. But we already knew that. 34:21.030 --> 34:23.990 So why is it that I need to keep learning that I don't know? 34:26.960 --> 34:31.680 So that's what I'm trying to create. This trajectory 34:31.840 --> 34:35.480 where futures and features go 34:35.980 --> 34:39.040 together in a little bit back and forth. 34:41.120 --> 34:45.120 Something I've heard echoes of earlier in the day is 34:45.620 --> 34:48.720 a distinction that I want to make that. 34:53.609 --> 34:57.289 It's coming back. This is again one of these advantages, 34:57.769 --> 35:01.089 disadvantages of experiences. You see stuff and you go 35:01.589 --> 35:05.049 like this again. Okay? And this is 35:06.488 --> 35:10.729 the distinction between one shot development and iterative. 35:10.889 --> 35:11.449 Demel. 35:13.689 --> 35:15.849 Sorry about that. Iterative development. 35:18.430 --> 35:21.590 Sometimes my mouth gets a little ahead of my brain, or maybe it's the other 35:22.090 --> 35:25.590 way around, but here we are. There is 35:26.090 --> 35:31.510 such a temptation in spec driven development to 35:32.010 --> 35:35.550 go down the path of I'm going to write a spec 35:35.710 --> 35:39.150 and the genie is going to produce the software and 35:39.470 --> 35:42.710 if the spec is good enough, the software is going to be good 35:43.210 --> 35:48.580 enough. And there's a word for that that's 35:49.080 --> 35:52.300 exactly waterfall development. Where it didn't work, 35:52.940 --> 35:56.420 it never worked. If you go back to the original 35:56.920 --> 36:00.660 Winston Royce paper about the waterfall, there's a 36:01.160 --> 36:04.780 diagram with the waterfall and it 36:05.280 --> 36:08.540 says right there on that page, of course this 36:09.040 --> 36:13.470 doesn't work because decisions you make later are going to inform decisions 36:13.970 --> 36:19.830 that you made earlier. So the next page has waterfall 36:20.330 --> 36:23.190 going down and feedback coming up. 36:23.990 --> 36:27.589 But it's the power of an image is 36:27.590 --> 36:31.110 everybody looked at that waterfall and thought, wouldn't that be awesome? 36:31.510 --> 36:34.990 If I could just write a spec that was good enough and 36:35.490 --> 36:38.960 then the software would be finished and that's what stuck? 36:39.520 --> 36:42.480 That's how careful you have to be when you're explaining things. 36:46.000 --> 36:49.280 What about formal methods? Great question. 36:50.320 --> 36:54.000 I have enjoyed. So when I was in graduate 36:54.500 --> 36:57.800 school, I was in music school and computer science school 36:58.300 --> 37:02.240 every other year and I just ended on the wrong year. 37:02.480 --> 37:06.440 So I'm on stage here during the day instead of in the evening. 37:08.520 --> 37:10.520 Feels good to be in a music venue though. 37:13.320 --> 37:16.440 I was a TA for the proof of 37:16.940 --> 37:20.680 program Correctness class. So I have those 37:20.760 --> 37:25.080 patterns baked into my head and I use them informally 37:25.400 --> 37:29.640 all the time to do some data flow analysis or control flow analysis 37:30.140 --> 37:33.810 and define some invariance and so on. 37:33.890 --> 37:37.410 But I hadn't done formal proofs until 37:38.130 --> 37:41.970 maybe a year ago and a friend introduced me to Lean 37:42.370 --> 37:46.610 and Lean with the genie and I 37:47.110 --> 37:50.570 tried it out. The challenge 37:51.070 --> 37:53.570 I have with formal methods is twofold. 37:54.530 --> 37:57.890 One, it still feels like a one 37:58.390 --> 38:02.500 shot process. I have some formal 38:03.000 --> 38:06.180 specification and I want to prove some properties of it. 38:07.860 --> 38:11.820 And I do a whole bunch of work and I prove the properties. What happens 38:12.320 --> 38:15.860 if I change one element? Oh, I got to 38:16.360 --> 38:20.100 wind back. And even if it's faster because I have the GENIE helping, 38:21.060 --> 38:26.150 it's still a drag on 38:26.650 --> 38:30.510 my ability to change. And I want to encourage, I want to build 38:30.590 --> 38:33.950 systems that encourage change instead of discourage change. 38:34.670 --> 38:38.670 So that's my first one. And the second one 38:39.170 --> 38:42.590 is even with formal methods, there's still a gap 38:43.090 --> 38:46.430 between this mathematical model and implementation. 38:47.150 --> 38:50.390 Okay, I can have some formal specification, I can 38:50.890 --> 38:54.880 derive some properties, I can prove those properties. Properties hold for this formal specification. 38:55.360 --> 38:59.200 And now I want to turn that into a running 38:59.700 --> 39:03.680 program. There's still a gap in that derivation 39:03.840 --> 39:07.760 of the program from the formal 39:08.260 --> 39:11.560 specification. And I don't know how to 39:12.060 --> 39:15.360 bridge that. Okay, I want to turn that into C code, I want to turn 39:15.860 --> 39:18.480 that into C or ARM 64. 39:20.170 --> 39:22.890 You can program an assembly language now. 39:24.410 --> 39:27.650 It's so cool. It's so. I mean, that's where 39:28.150 --> 39:31.450 I started, my dad and I soldering together a 6800 39:31.950 --> 39:35.290 machine. You can do that now. You can just say, well, 39:35.530 --> 39:39.330 okay, implement this data structure and do it in x86 39:39.830 --> 39:43.450 assembly. Chung chunga chung ding. I will come some. 39:44.420 --> 39:45.140 It's amazing. 39:49.140 --> 39:52.900 Figuring out all the different ways I can torture the genie is 39:53.400 --> 39:57.260 just so entertaining. Okay, let me make this even harder and more 40:00.900 --> 40:04.460 So the second part of the to 40:04.960 --> 40:09.020 finish that thought about formal methods, the second part is that bridge that 40:09.520 --> 40:13.460 gap between here's a specification and here's an implementation that matches 40:13.960 --> 40:17.380 that specification. Specification, as far as I can tell, is something 40:17.880 --> 40:21.260 that hasn't been fully bridged. 40:21.740 --> 40:25.820 Now the good news is you can get to 40:26.320 --> 40:30.780 low defect densities with a combination of 40:32.060 --> 40:35.980 examples, automated tests, carefully chosen and 40:36.940 --> 40:39.580 foolproofing of the design. 40:42.130 --> 40:45.450 If you work at those two things pretty hard, 40:45.950 --> 40:47.970 you can get software that is trustworthy. 40:48.770 --> 40:52.050 Unfortunately, the GENIE wants to make shortcuts, 40:52.770 --> 40:56.610 it doesn't want to foolproof the design and it 40:57.250 --> 41:01.010 will do things that subvert the intentions of test 41:01.510 --> 41:05.530 cases. You have to watch for returning 41:06.030 --> 41:09.450 constants, you have to watch for just all 41:09.950 --> 41:13.450 of the nasty things that you can imagine that a 14 year 41:13.950 --> 41:17.330 old programmer would do just to say, yeah, it works, 41:17.830 --> 41:21.570 boss. So it is possible 41:21.650 --> 41:24.930 to write though trustworthy program 41:25.410 --> 41:29.050 with the genie. Ah, so I got into that 41:29.550 --> 41:32.450 one shot versus iterative. 41:32.930 --> 41:36.740 So are you making some software that 41:37.240 --> 41:41.620 has value deployed in the world that you then change and 41:42.120 --> 41:45.700 now it has more value in the world that you then change and then how 41:46.200 --> 41:50.020 now it has more value in the world? That's one paradigm versus the 41:50.420 --> 41:53.380 I'm going to write a spec and the spec's going to be so good that 41:53.880 --> 41:56.500 the software is going to be good. Oh, it's not. 41:57.300 --> 42:02.570 I'm going to write more of a spec and a bigger spec and a until 42:03.210 --> 42:07.250 the spec is big enough and complicated enough that 42:07.750 --> 42:12.490 the software is correct. And there's 42:12.810 --> 42:16.330 times to do both. If I'm writing a little app, 42:16.650 --> 42:20.250 one shot is fine. If I'm writing 42:21.050 --> 42:24.570 as I am a virtual machine for a graph database 42:25.070 --> 42:28.140 with an embedded programming language, that's not good enough. 42:28.540 --> 42:32.780 It's going to have to be something. Where I 42:33.280 --> 42:36.780 have some software, it has some value, I see a way to improve it, 42:37.180 --> 42:41.660 and so on. The people who are getting 42:41.820 --> 42:45.740 really excited about spec driven development just reminds 42:46.240 --> 42:51.180 me, it doesn't sound like they've ever had a billion users because 42:51.420 --> 42:55.100 the kinds of things that you have to do, if you have a really large 42:55.600 --> 42:59.620 installed base and a lot of traffic to change software 42:59.780 --> 43:02.660 like one shot and then, oh no, never mind, 43:03.940 --> 43:07.700 let me one shot it again. That's just never going to work 43:07.860 --> 43:11.620 because you've got. You have continuity problems, 43:12.580 --> 43:15.140 you have data migration problems. 43:16.500 --> 43:21.260 You're better off assuming that you're going to be iterative all 43:21.760 --> 43:25.510 along the way. Now I said that 43:26.010 --> 43:29.990 I was going to show you an update of a 43:30.490 --> 43:34.470 diagram and I'm 43:34.970 --> 43:37.470 excited because anytime I get one of these, 43:41.630 --> 43:45.190 I have a puzzle and I can finally solve it. Then I'm 43:45.690 --> 43:46.910 a very happy person. So. 43:51.640 --> 43:55.280 Us programmers, I'm a 43:55.780 --> 43:58.680 programmer mostly and so that's the way that I speak. 44:00.920 --> 44:02.840 If we want to look at what we do. 44:05.800 --> 44:09.680 There's this sequence where as a programmer, you put 44:10.180 --> 44:14.930 in a certain amount of effort and 44:15.430 --> 44:18.610 then you have, say, a new feature 44:19.110 --> 44:22.850 that you can deliver to your customers, and that's output. 44:24.130 --> 44:28.690 And then the customers use that feature and 44:29.190 --> 44:32.770 their behavior changes. After all, if we produce 44:33.270 --> 44:35.970 software and nobody's behavior changes in any kind of way, 44:37.730 --> 44:39.650 we didn't need to produce the software. 44:41.660 --> 44:47.260 And that's outcome. And one of the big changes 44:47.980 --> 44:51.540 in my life was moving to the 44:52.040 --> 44:55.180 web and being able to observe what customers actually 44:55.680 --> 44:59.180 do with the software. Before then, you had software 44:59.680 --> 45:03.220 on a cd, you'd send it out. Are people using 45:03.720 --> 45:06.960 my new features? I don't know. There was no way of knowing. 45:07.040 --> 45:11.040 So just being able to observe outcomes was a big step 45:11.540 --> 45:15.040 forward. How much of what we did wasted effort 45:16.160 --> 45:18.560 because nobody used the feature at all? 45:20.000 --> 45:23.760 But here's what I learned today is 45:24.560 --> 45:28.000 outcome for the customers. 45:28.080 --> 45:31.200 The people using the software leads to, 45:33.000 --> 45:36.520 and I used to use the word impact, but I love this word mission 45:39.320 --> 45:45.560 leads to changes in our accomplishment 45:45.880 --> 45:49.720 of the mission. And the mission is shared between the 45:50.220 --> 45:52.920 people making the program and the people using the program. 45:55.720 --> 45:59.210 And here's where we get to the tyranny of metrics. 45:59.710 --> 46:02.970 People will be amazed. The genie produced 200,000 46:03.050 --> 46:06.170 lines of code. That's a measure of 46:06.670 --> 46:08.970 effort. It has nothing to do with the mission. 46:10.410 --> 46:13.850 And whether the mission is making enough 46:14.350 --> 46:17.770 money for my next yacht or I don't have a yacht. 46:17.930 --> 46:21.770 Just using that as an illustrative example or 46:22.170 --> 46:25.650 making the country safer. The fact that 46:26.150 --> 46:29.050 we've generated 200,000 lines of code is irrelevant. 46:29.210 --> 46:32.810 And 200,000 lines is not 10 times as much 46:33.310 --> 46:36.650 as 20,000 lines from the mission perspective. 46:37.210 --> 46:41.090 It's just irrelevant. And yet it's 46:41.590 --> 46:45.370 the easiest thing to measure. And so we look for our keys 46:45.870 --> 46:49.770 under the lamppost, and the GENIE 46:50.270 --> 46:54.730 is able to produce so much output with 46:55.230 --> 46:59.850 so little effort, it's easy to see that as productivity, 47:00.010 --> 47:04.050 because after all, productivity, the very definition 47:04.550 --> 47:07.770 of productivity, is the ratio of output to input. 47:10.730 --> 47:14.170 And whether it's lines of code, which we've 47:14.670 --> 47:17.650 known was balderdash for a very, 47:18.150 --> 47:22.160 very long time, and somehow it's back PR 47:22.480 --> 47:22.880 count. 47:26.240 --> 47:30.040 Anything which is a measure of effort or a measure 47:30.540 --> 47:34.520 of output is, per definition, not a 47:35.020 --> 47:38.840 measure of mission. The problem with measuring mission is it's 47:39.340 --> 47:43.680 hard to do. Oftentimes value from 47:44.180 --> 47:46.480 the mission is bursty. 47:47.690 --> 47:51.410 Like you don't see it until a crisis happens, and then you realize that you're 47:51.910 --> 47:55.490 ready for the crisis. So it's 47:55.990 --> 48:00.010 hard to measure from the level of the mission, 48:01.690 --> 48:04.890 but that's what actually matters. So the 48:05.390 --> 48:08.810 fact that the genie can give you a lot of effort, 48:09.050 --> 48:12.450 and now I can just spin up 10 genies, and now I have 10 48:12.950 --> 48:16.750 times as much effort. Yeah, but it doesn't 48:17.250 --> 48:20.550 matter. Doesn't mean that we're making progress 48:21.350 --> 48:24.630 on what really matters to people. And the earlier 48:24.710 --> 48:28.310 in this cycle you measure, the more 48:28.810 --> 48:34.510 likely you are to see Goodhart's 48:35.010 --> 48:38.350 Law. You know, when a measure becomes 48:38.850 --> 48:42.540 a goal, it ceases to be a measure. I think Goodhart 48:43.040 --> 48:46.780 was an optimist. It's not just that the measure 48:47.280 --> 48:50.620 ceases to be a measure, it's that people will twist the 48:51.120 --> 48:53.660 entire system out of shape. 48:54.540 --> 48:57.700 They'll make it worse in order to get better 48:58.200 --> 49:01.140 measures. So it's not that you just, 49:01.640 --> 49:04.620 you lose visibility by turning measures into goals. 49:04.940 --> 49:08.780 It's, you know, you lose progress on the 49:09.280 --> 49:12.780 mission by doing that. The later in this cycle you go, 49:14.300 --> 49:16.140 the harder it is to attribute, 49:17.420 --> 49:20.940 hey, the mission is now 7% better, 49:21.440 --> 49:24.620 and I own 0.075% of that. 49:26.140 --> 49:29.660 You know, it's hard to say what you did versus what 49:30.160 --> 49:34.390 he did versus what she did, like, which of us deserves 49:34.890 --> 49:39.990 the credit. And this is the challenge we 49:40.490 --> 49:42.470 had from earlier today. 49:43.910 --> 49:47.350 How do you incentivize progress on the mission? 49:47.829 --> 49:51.630 And I'll close with just a few words about this. I've had the great 49:52.130 --> 49:56.470 good fortune of having a few of my ideas spread 49:56.970 --> 49:59.380 out in the world of software development. 50:00.500 --> 50:04.340 And what I know about that process is 50:07.140 --> 50:11.620 you have to find an expression 50:11.700 --> 50:14.820 of what you're trying to accomplish of the mission. 50:15.700 --> 50:19.540 You have to find an expression of the mission that speaks to people's heads 50:20.040 --> 50:25.610 and their hearts. And then you have to repeat it and 50:26.110 --> 50:29.410 repeat it and repeat it. And that 50:29.910 --> 50:33.370 repetition that. Not getting bored with telling the story of how 50:33.870 --> 50:37.530 software development can be over and over again 50:37.770 --> 50:40.890 is part of the skill 50:41.390 --> 50:44.610 of leadership. To tell that same story for 50:45.110 --> 50:49.050 the thousandth time as if it's brand new. And then 50:49.550 --> 50:52.310 someday somebody will come to you and say to you, 50:53.190 --> 50:56.750 I have this great idea. And they'll tell you your 50:57.250 --> 51:00.870 idea without realizing it was 51:01.370 --> 51:05.030 you, without giving you any credit whatsoever. And that's 51:05.530 --> 51:10.390 the moment that you've succeeded as a leader. And I think we have these tools 51:10.890 --> 51:14.310 that have the capability of transforming the value we create in the world. 51:15.190 --> 51:19.120 And that's the challenges for finding those 51:19.620 --> 51:23.560 missions into which the tools fit for 51:24.060 --> 51:27.600 human purposes. And I'm looking forward 51:28.100 --> 51:31.440 to finding out what comes out next, to doing my little 51:31.940 --> 51:35.040 bit to shape what that is and to seeing what it is that you do 51:35.280 --> 51:38.800 that also shapes the future. Thank you so much 51:39.300 --> 51:40.000 for your time and attention,