Can we test it? Yes, was can! - Mitchell Hashimoto - https://www.youtube.com/watch?v=MqC3tudPH6w

Hello, everyone. Thanks for coming in. I'll try to make it positive, I guess. I think it's optimistic, but let's just dive right into it. So this is something we've all heard before. You know, a PR is opened. Maybe it's trivial, maybe it's not trivial. Who knows? Maybe the author proactively said this, maybe they didn't. Maybe you ask why there aren't tests, but usually you. You'll see it. This word, this can't be tested, often followed by the phrase, but it's OK because I just verified it manually. I've said this before. You've said this before. Someone in this room has probably said this in the past 24 to 48 hours. It's fine. And the truth, however, is usually this can't be tested yet or. Or replace yet with easily. And easily is okay, because sometimes you can make the argument that something is could be tested, but the work involved in testing it is not worth it. Tons of stuff like that exists. Maybe if there's a catastrophic bug, it's not an issue. Maybe you're just running this in, you know, reproducible environments. It's not. Not a big deal. That's all fine. There's loads of those situations. This talk is not at all about when to test. I'll let you make those value decisions yourself. What I want to talk about is how things can be tested. So my goal with this talk is to really give you a tour of concepts and strategies that can be applied to a variety of different situations. Try to give you some, like, pattern matching capability and how you might be able to apply those. And I think it's very similar to. I think all of us in here would agree that there's. There's value in learning many different programming languages, even if you work every day with only a single one. Learning multiple different languages tends to make you a better programmer. Similarly, I think that just being exposed to different testing challenges, even if you may not directly hit the examples I show you, I think it'll make everyone a better tester. And so that's really my goal today. Okay, so you understand the goals of talk. Why should you sort of listen to me? What's my experience? I'll go through this relatively quickly. My background is I started a company called Hashicorp. We made a bunch of tools. I was part of all those initial engineering teams. So it's either I'm sorry or you're welcome. I don't know if you love them, feel free to tell me if you don't Love them then I didn't do it, it was somebody else. But yeah, I did that for about 12 years. And importantly, I think I was part of a lot of the initial testing laying out a lot of the initial testing strategies for this software and sort of some of the properties there, they tended to be network systems. Mostly there was a large scale in terms of usually nodes or that kind of scale, that dimension, those axes. And there was security sensitivity since one of the things we made was vault. More recently, I left Hashicorp at the end of 2023, if someone didn't know that. But more recently, for the past few years, I've been working on a project called Go See, which is a terminal emulator. It's not a company, it's just a passion project, it's just for fun. But it has dramatically different properties. This is desktop software. It's cross platform across Mac OS and Linux. It has GPU rendering in it. And I think this is interesting because each of these experiences has given me these dramatically different testing environments. And I'm someone who loves testing. So I wanted to figure this out and I want to share sort of how what I, what I've figured out over the past 15 years. All right, so going back in 2017, I gave a talk at Go for Con titled Advanced Testing with go. And as the title and venue may suggest, this is fairly GO specific or this was heavily. A lot of examples were GO here. But over the years a lot of people have come up. This is sort of one of the most watched talks I've ever given. A lot of people have come to me and said, even if I don't write go, I found value in watching this talk. And it's been almost 10 years and I felt that having a sequel to this talk would be a good idea because I've learned a lot more, I've experienced more complex situations and I was sort of cut short in this and I wanted to talk about more. But an important part of that is I'm trying not to overlap at all with this talk. So there's one concept that overlaps with this only because we're going to use it in the future. Things I talk about. But if some of the things I talk about maybe seem like I'm jumping into something advanced, this has a lot more. Even though this has advanced in the title, this has a lot more elementary concepts. And even if you don't write GO like that, if you like this talk, I think you would like watching that one as well. Okay, so let's start Getting into it, here's what we're taught, right? Like in school or in testing chapters of programming books or something like that. We're sort of taught that there's some operation, there's some output, you run it and you expect or assert that output. And some stuff is like this. And if the stuff, if what you're doing is like this, then great, it's easy, we all have a great time and generally we'll write tests. But the issue is that let me make sure it gets right. The reality is that in a lot of real world cases you might run into something where add is actually sub processing to a shell, running BC for some reason and you wanted it to work on Windows and now it doesn't work on Windows. This is silly. This is not reality. The only ecosystem that would possibly do this is probably JavaScript. But the whole point is that reality is actually messy, right? Like most tests really don't come down to that simple run a thing, get an output, you're happy situation. At least in my experience. A lot of the things that I actually care to work don't fall into that category. They fall into these categories where they have side effects or there's a complex state of the world that needs to be created in order to even run it in the first place. Networking, other devices like GPUs, concurrency and, and so on and so forth, the list goes on. And I'm not going to go into detail about this. I just hope that people agree and have noticed this in their own code. And a lot of these are what give rise to you work in one of these categories and say, hey, this couldn't be tested, but it works maybe. Ok, so the last thing I want to talk about, one more setup slide before we sort of get into it, is the two sides to testing. So there's two parts to testing. They're both equally important. This is also gonna be the slide format. So if the slide header starts with a test tube, we're gonna be talking about testing strategy. If the slide header starts with soap and bubbles, we're gonna be talking about testability. So testing strategy doesn't need an explanation. That's how to test something. Without this test don't exist. Everyone gets this one, it's easy. The second one is just as important. But the one that I find, regardless of engineering experience gets ignored more often is testability. If software isn't made testable, there actually are cases where you can say this can't be tested and you would be telling the truth. So you need to pair testability with actual testing strategy in order to be able to test everything. And the way you achieve testability generally are things like proper code structure, making the right APIs available, making something friendly for automation, things like that. So I'm gonna be talking about both of these intermixed in this again with the slide design guiding it in order to get us to be able to test more things. All right, so let's get started with our first testing strategy. Oh, another important point. The slides are roughly in order of simplest to more complicated or more advanced. So if you feel that some of this stuff is like, oh, this is obvious and this whole talk's gonna be obvious. I suspect it won't. I hope it won't. But based on earlier talk, this is a pretty advanced group, so maybe so, but let's get started with the first one. Snapshot testing. Sometimes this is also called golden files testing, ground truth testing, other things like that. Snapshot testing is. I used to call it some of those phrases, but I feel like snapshot testing is more of the norm nowadays culturally. So that's what I'm gonna call it here. The scenario where you want to do snapshot testing generally is when you have some sort of complex output format where programmatically coding the comparisons is difficult. That's, I think if you are familiar with snapshot testing, the one that you'd be familiar with a lesser discussed scenario. But I think possibly more powerful for snapshot testing is that it often gives you better diffs to understand why something failed. And I'm going to show you an example of this. So here's. Oh, this came out way smaller than I expected. Here's an example directly from Ghosty, my terminal emulator. When I previewed this locally, that image was really small and the code is really big. So I don't know. But the code, you don't need to read it. The code is zig. I'm going to use zig examples in this. This talk is totally language agnostic. It doesn't matter, so ignore that. But basically what we're doing here is go see has an embedded sprite font within the terminal emulator. There's a bunch of glyphs that terminals render that depend on the grid size to be perfect. The ones you're probably most Familiar with are PowerShell type glyphs, like the arrows. If you end up using a font. If you see this right now in your terminal, you use a font and the arrows don't quite perfectly match up with your grid size, then the terminal's not doing this. What Ghosty does is those glyph code points we programmatically rasterize, given the current grid dimensions, so they're always pixel perfect. This is not uncommon. A handful of terminals do this, but the issue is that how do you test this? How do we make sure it works? Specifically, one of the first things we found was that we had an off by one error when the grid size was odd in either dimension and we were one pixel off. And so one of the ways. Oh, and also new glyphs are introduced all the time. We've changed out the library of how we rasterize and things like that. How do we make sure we don't regress this stuff? This is a perfect case for snapshot testing. What we do is we embed a. We render a bitmap of every glyph that we programmatically rasterize, we commit that directly to the repository and we compare it pretty straightforward. That's the snapshot and that's how we compare. What the code was trying to show you there is that there's a basic step at the bottom where we read this ground truth, we run the actual rasterization and we just compare it. There's nothing fancy there. But one of the things I talked about is that one of the helpful properties of ground truth testing, snapshot testing is better diffs. And in this case, since our comparison is an image, we could apply standard image diffing techniques to this. And we do do this also in the repo. So when the test fails, we also generate the diff and dump it to the file system as part of the test run. And again, I thought this would be a little bit more readable. But what I did was I artificially modified our code to draw vertical bars, one pixel off to the right. It produced a failure. And the diffs on the right, it's a bit hard to see, but there are some green lines in there. That's the diff that gets generated. And the thing that I want to point out here is that this is a more helpful diff because prior to this we did have some tests and it would just basically say it didn't match or this pixel in this place was wrong. One thing that snapshot testing gives you is snapshots usually have more context than a single assertion. Like, yes, this one pixel is wrong, but here's all the pixels so you could see. And usually standard assertions in like a unit test type environment don't also include the greater context. And snapshots tend to so in this case, I could see very clearly that vertical bars are an issue. I could see it's across multiple glyphs. And me, with my experience with the code base, when I see something like this, immediately think some common function that draws vertical bars is wrong. And that's what I ended up, you know, breaking here. This doesn't just apply to images. Images make for an easy visual example. But for example, for Terraform, about a decade ago, one thing that we ran into was that the first step that Terraform does is builds a resource graph that gets executed. And we had a bunch of tests around building that graph and we would get failures saying this expected node, this expected edge doesn't exist and it would be very difficult. We were spending a lot of time as engineers figuring out what the graph changed. So one of the things I changed was even though it wasn't hard to program those assertions, the diffing was very hard for us. So I ended up starting to dump the expected graph and the graph we got and then doing generating a dot format that actually colored the edges that were missing red, the bonus edges, green vertices, so on. Then you could load the whole graph and debugging these things became much, much easier. That was actually a text diffing format. You could render that into an image, but that was just text stiffing. But again, since we're missing one vertice, but we could see all the vertices, the greater context around it, debugging became easier. So I think this is actually the bigger benefit of snapshot testing, but something to keep in mind. Okay, this next section I didn't mean to keep in here, so I'm actually gonna skip it. That one overlaps as well with the GopherConTalk, so you could look into that one. It was also kind of like an opinionated one that people tend to get upset about. So it's fine. We should skip it anyway. Okay, let's talk about a testability subject. So soap and bubbles isolating side effects. I actually think if, if there's no other part of this talk that you pay attention to, this is the number one force multiplying strategy that sort of exists to make code testable. This comes up over and over in different contexts. And the scenario is this you have some sort of behavior you want to test, but it's reliant on some sort of external IO or otherwise complex system that's sort of filled with side effect type behavior. And this is a really common case of this can't be tested. And what you're looking for, the Goal with this is within this I O complexity state soup, you're trying to find the purely functional behavior stuck in there, extract it out, reorder things in order to get something that you could mostly test. In this case, you could usually get most of the complexity tested as a result of this. What you get is something that is obviously testable. I'm going to show you an example of this. You get something that's obviously testable, but it becomes a garbage in, garbage out type of test. And what I mean by that is you can't in this case simulate the external side effecty things. So you're gonna artificially provide those inputs. And so if you provide garbage and you test garbage, you're gonna get garbage out. You need to make sure that the inputs that you're gonna be providing to these sorts of tests are actually realistic in real world. And again, that's conceptual. Let's see it in practice. Okay, so here is something. We're gonna start visually and then see code, but I actually doubt we're gonna see any code given the slide sizes. But let's try to understand this visually. Here's a simplified but real example from Go see Again, one of the things a terminal emulator has to do is probably the main thing that it has to do is when you press something on the keyboard, it has to encode that into some format which is then sent to your shell or whatever running program. And then it does whatever basic things, like if you press Control R at your shell prompt, you tend to expect a reverse search to show up, so on and so forth. Early versions of Ghosty had a keyboard input handler that looked like the above image. Basically, what would happen here is a user would press a key, we would read mouse state, which seems odd, but depending on mouse state, it actually affects whether a key should be encoded at all. Like if you're actually actively highlighting something and you press certain keys, you might want to move the highlight, shift the highlight, things like that. So we would first grab mousetate, check, respond in some way. Then we would check keyboard state, because we need to know what keys are pressed, what modifiers are pressed. Is it a repeat, is it a first press? You know, things like that, do stuff around that. Then we would read terminal settings. There's a variety of settings that affect how it's encoded. Are we doing legacy encoding, kitty encoding within each of those? Are we encoding alternate Unicode code points? Are we encoding control as differently? There's a bunch of stuff. And then finally we encode the key and then write it out to the actual PTY that we have. And this was untested because setting up mouse keyboard state, stuff like that is non trivial. And I originally approached this as oh, this isn't a testable thing. And I punted it to a full end to end test. I figured one day I would probably spin up a VM or something, synthesize inputs and assert something. I just punted it away. We'll figure it out later. But then I sort of got punched in the face enough with this code. Constant regressions happening here. A lot of complexity that I realized I had to do something, even if that something was that VM based test right now. And I actually sat down and focused on I need to make this testable because this can't go on. And what I realized is what these colors are showing. If you could see the colors, the yellow is the stuff that's dependent on external logic and the green is the stuff that, that doesn't need any, is sort of pure. It just has some inputs and gives you a set of outputs and doesn't touch any external systems. And it's really easy to see here because it's colorized. And I simplified it into distinct categories of function calls and also made it alternating. But hopefully people could understand that the reality of this function at the time was that all of this was intermixed and we would grab state when we needed it and run conditionals on it. And it wasn't, at least to me. It took me a few hours of really staring at this code to see the shape of suddenly something emerge. And what I saw emerge was this at the bottom. If I actually took all of this stuff that grabbed external state, moved it to the top, turn into a read process, write order, then I could isolate that, provide artificial inputs there, test this green thing that is pure, just has inputs and gives you an output and then it becomes that testing 101 expect add one plus two equals three sort of environment. And in this case most of the complexity, most of the bugs, most of the issues was in this green thing. So we were able to really dramatically eliminate a bunch of issues. And also that made it much easier to fuzz and things like that. And I'm not going to talk about fuzzing in this. There's enough of that here. Yep, that's what I expected. This is the actual code from the key encoder. I wish you could see the bottom. The bottom is more important, but the top just know that each line, I think you could all see lines, just not the text that's in the line. Each line is a piece of state, whether it's structure, boolean, an integer character. It's a piece of state that's needed to do the key encoding. And what I'm just trying to visualize here is how much state is actually required for a terminal to produce a valid key encoding. There's something like. In total, there's something like 15 different fields here. Some of it is produced by the operating system, some of it's produced by the terminal, its internal settings. But the bottom is an actual test I copied out directly just verbatim zero edits to show the types of regressions that we can now test against and what the bottom test is doing. Just believe me, and you could look at the slides later. It's testing how we encode a certain input from a Russian keyboard layout with kitty keyboard settings encoded to also add alternate Unicode characters and predicts expects we get the right thing. This is the reason I had to test this because every time I would fix a feature or fix a bug or implement a feature, I would regress some to me very foreign layout that I of course was not just running adult speaker type Russian. And also getting this specific kitty layout is very difficult. So you know, this is very common. There's Russian, Japanese, Chinese, Hungarian, like all sorts of very language specific test cases in there to make sure we constantly do the right thing. And I think because of this, Go See has one of the most complete and stable sort of encoding key encoding features out there. And so this helped a lot. And this is sort of the key point of isolating the side effects. And basically every section from here on out is going to continue to show examples of this. We're going to take snapshot testing, we're going to take isolating side effects and we're going to bring them together to do something more and more complicated. All right. GPUs. What a crazy time for GPUs. So thankfully I don't work at OpenAI, so I'm not going to talk about AI at all in this talk. I'm just going to talk about GPU programming in general. It could apply to AI, but in my case it applies to rendering and some compute. But I want to talk about GPU programming and GPU testing actually. So background Ghosty is a GPU rendered terminal emulator. And what that means is when you run the terminal, that main thing you see with your cursor blinking, that whole thing is just an image that I'm rendering via the gpu. We also do A little bit of compute on the gpu. It was my personally, it was my first foray into writing any kind of GPU code, starting a few years ago. Prior to that, I lived purely on the cpu. And given that, when I started coming into it, one of the first things I asked was, all right, how do I test this? My initial feeling and response was the classic can't be tested. This seemed like a very obvious thing that I should spin up a VM and take screenshots of. I was rendering after all. So that seemed to be the right solution. But for years our renders didn't have unit tests. And for years, similar to key encoding, we would just whack a mole regressions constantly in the renderer, in the GPU code. And so finally I sat down and said, I need to figure this out. Resources on how to test GPU logic is surprisingly scant. Like if you do a web search of how to test GPUs, it's one of those rare things where Google the first page is just completely garbage. There's one response that has an idea that's kind of interesting, but it was just an idea that no one implemented. So it kind of leads you in a direction that you then have to figure out on your own. I went from that to going to the Kronos Group, Apple and Microsoft and downloading the reference material for DirectX, direct 3D, metal and OpenGL, Vulkan and all their docs, anything they could provide me in big text format. And I did command F searching, LLVM assistance searching on that. Everything I looked for test verify, snapshot, bug stability. I looked for all these terms and the amazing thing is the total reference material across those three vendors for their language specification, driver specifications and so on is something like 4,000 pages of PDF. And I didn't get a single hit on any of those terms. Like the word didn't pop up one time. So as far as I could find, there are zero official resources on how to do testing with GPUs, and very, very few people have even cared to ask the question, at least publicly, to where it was indexed. So that's where I was left. I took that mostly as a challenge to see if. Well, I feel pretty good about testing. I like testing. Maybe I could figure this out. And I think I did at least something that works for me pretty well. So this is where I'm just going to add a quick disclaimer that you could probably tell based on my experience here, that I'm not an expert on GPUs. I never worked on complicated 3D games. My terminal emulator is one of the only things beyond like toy advent of code examples that ever used a gpu. I'm not quite sure that my techniques here actually generalize that well. In my defense, Ghostly does have about 15,000 lines of renderer code split across metal and OpenGL. We have separate renderers for both systems, and that includes both the CPU code and the shaders for the GPU. The shaders are about 2,000 lines. So 13,000 on the CPU, 2,000 on the GPU. That's what I was working with. That's what I was trying to test. The core realization I came up with here is that GPU programming requires two sides. I'm going to go into that background. There's the CPU side to prepare the data and then process the results usually. And the GPU part that actually runs the shaders. And I felt that we can test each of those in isolation and specifically really clearly a GPU is just a pure function evaluator, which makes testing really easy, but setting up the workloads really hard, which is kind of a funny thing to run into. But let's go ahead and look at this more visually and start with the GPU side, the CPU side. We could actually read this basic thing. So as a point of background, for those less familiar with GPU programming, the CPU does have to do some work to prepare the gpu. The CPU has to put together the right data, the right steps, basically these little job descriptions that it then eventually offloads and submits the GPU and says, here's a bunch of crap, go do it and I'll come back, or you tell me when you're done, I'll read this later. That's the really general way to think about a gpu. The work the CPU does in preparation for the GPU can be roughly thought of as this top function. It's a function that takes in some sort of state of the world. If you're doing rendering, this might be called the scene state. It's what monsters are on the level. Where are they? Where's the camera? Where's the player? What planet am I on? That's the scene state that exists. It brings in the scene state, and as a result it produces three sets of values. It produces a graph, so you get vertices and edges, and it produces data attachments for that graph, which nodes need which access to what data. And that's a gpu. The graph is just a graph. It has vertices, edges, and the vertices are Operations, right? It's stuff like vertex shading, fragment shading, compute shading, things like that. Edges are the data dependencies between the steps. The vertex shader is going to produce some sort of output that the fragment shader needs to bring in, or the vertex shader needs, the fragment shader needs access to a certain texture, things like that. There's these edges that exist in the graph and the data attachments are literally byte buffers. Historically you'd probably call these textures. More modern graphics APIs out there, Vulcan, Metal and later direct 3D, they all tend to really just call these buffers now because it's really just a set of bytes. Sometimes the bytes have structure to them. They're RGBA with dimensions and stride and you have a texture. But sometimes they're just bytes because you're just computing stuff. So data is just bytes and therefore you could do whatever you want with it. So to test the CPU side, we have to apply that technique of isolating the side effects. In this case, the side effects are all the API calls to the GPU itself in order to submit, prepare and submit this workload. And so what I ended up doing was creating an intermediary where we bring in the scene state, we produce the graph and the data, and, and then I assert that the graph has the right shape, which I have a bunch of experience with the terraform and the data. I do snapshot testing, bringing that back because it's just bytes. And so we're able to do some structured sort of snapshot verification on that. And then after that there's a small amount of simple untested code which just translates the graph and the data attachments into GPU API calls. That stuff never really changes. I'm happy to keep it untested until there's end to end tests. And in this way we're able to test that given a certain scene state, we're producing the right workloads for the gpu. But we're not sure the GPU is going to do the right thing with that. But this is still one big part of the equation. I mean, this is 13 out of the 15,000 lines of code that go see does in order to render a scene. So this is a big one. Then on the GPU side, it's visually much simpler. GPUs have no access to disk, they have no access to networking, they, they have no access to any other peripherals, they only have access to their own memory. And so a GPU by definition is pretty much just a pure function evaluator it has some sort of computation, it has data and it outputs data. And that's something that's really juicy to test, really easy to test. But the funny thing about it is, like I said, the hard part is actually submitting the workload. So the general idea with the GPU side is I want to artificially construct some set of input buffers that my pipeline's going to expect. I'm going to ensure the output buffers are CPU readable. So instead of writing to a frame buffer that might never come back to the cpu, it's going to render this screen. Don't render the screen, render to this other CPU readable memory I have. Then I submit actual GPU work. We run unit tests on the cpu. On the cpu, we're going to run unit tests for the GPU on the gpu, submit the actual GPU work and then compare the output buffers. Just hand waving, you know, snapshot testing, actually parsing the data, whatever you want to do, but compare the output in some way. That's the general idea. It's hard, right? So in practice, what I found and what I've really only gotten to work well enough that I've shipped it, is full render passes with snapshot testing. What I do is I artificially create the scene state, usually a very small terminal, like a two by two terminal. I send it to the actual gpu, I get an image out and then I compare images and I expect pixel byte, equivalent images. It sounds kind of like end to end testing. It is in a certain way, I would say it's not quite a unit test, it's more of an integration test, but it's very, I think it's still a very robust, powerful test for two reasons. One, it's much faster. So getting a window made, submitting GPU work, grabbing a screenshot and, and comparing it is instant on any modern computer. It's really, really fast and it is very robust. In this case, we're really tightly controlling the input scene state. We're running one render pass, we're grabbing exactly one frame of results out of the other side and comparing against it. We don't have to worry about standard end to end tests with timing and synthesized inputs and, you know, window positions and chroming and like all this other stuff around the edges, like we get exactly the image perfectly cropped for what we're trying to compare and it's a single frame. So it's very robust. It works. For the future, I do want to test shaders in more isolation and I've already done a bunch of this. I'm going to kind of go through a few of the things I've done as proof of concepts and they work, but they all have these trade offs I'm not quite happy with. So I haven't actually shipped this in any way. So that's my disclaimer for this, that the proof of concepts all do work though. So the full render pass obviously tests a full input to image, but individual shaders themselves have quite a lot of complexity, or can have quite a lot of complexity. There's conditionals, there's loops, there's obvious edge cases that I see that I want to test in some way. And it's sometimes hard to elicit those edge cases through an initial scene state, or at least to visualize them in a resulting output state. So I want to get closer to a unit test with shaders. And so to do that I've been trying a variety of techniques, trying to figure out the best way forward. Again, these are all things that I haven't found a lot of people trying that much. I'm not going to talk about each in detail because I'm still learning quite a bit about it, but I'm going to just cover the high level ones. People know more details about this, I'd love to hear about it. But for example, sort of just a couple concepts. OpenGL. I know a lot of people are hyped about Vulkan and things like that, but OpenGL still works. OpenGL has this feature called transform feedback and what it basically allows you to do is capture the output of some shaders, some types of shaders, not all of them into a CPU readable buffer. It's actually a really nice API because you literally just add strings, say the variables that you want, and it just grabs them and throws them in order into an output buffer. And that's perfect. I don't know what this was made for, I don't know, I've never seen it. I did a source graph search and things like that. I don't see it used as very few API calls. It doesn't really exist in Vulkan, so clearly there wasn't enough value to move it forward. But it is kind of perfect for testing some kind of shaders. And that's an issue. It only applies to some on the metal side. Metal doesn't have transform shaders, neither does direct 3D. So what I found I've had to do there is extract shared logic into compute shaders. So non rendering, just compute shaders, make Each side just call this shared library and then run it through a compute shader and kind of build my own transferred feedback mechanism. That requires a lot more code. That requires code restructuring of GPU code, which standard for testing, but isn't great. And it works. The nice thing about that is that works for every kind of shader, no matter what. But you have to be able to extract it into a compute shader, which I've never found you can't do. So there's a lot of promise here. I unfortunately just didn't get to a production state of the future side. So I don't know, hopefully one day I could blog or talk about it and have it all figured out. But I think the result of this is I feel confident that we're now able to test our renderers. The full render pass thing I have is very robust and fast. And you don't actually need a GPU like hardware to do it because you could run it against software drivers. We could assume the drivers work, that we're not trying to verify drivers here, so just run it against software drivers. And again, we're just running one frame. So it's super fast. And yeah, it's interesting and I think it highlights snapshot and isolating side effects really well. Okay, the last sort of topic I want to talk about is VM testing. There are some things that do end up requiring this specifically to test those yellow boxes that I had earlier. The only way to really do it is to really make it happen. And the only way to simulate things like keyboard and mouse and other types of events is through things like VMs, not just keyboard or mouse. Right. This is also network failure stuff that antithesis is really good at. Disk failures, things like that. It's sort of best done through a hypervisor layer. I'm gonna apologize here because we're gonna mention nix. And I know a lot of people feel that Nick's enthusiasm is pretty exhausting and don't want to hear about it. So I am sorry. In my defense, I feel pretty confident that I have a good grasp about dev test environments, virtualization, containerization. Like it was my whole career for like 15 years. And I don't know any other technology that could achieve what I'm about to show you very well. So I'm going to use nix and I'm going to. And I'm sorry, but not sorry at the same time. Okay, so let's first talk about VM testing without the nix part. Okay, so if you're like having an emotional reaction, we'll start here. Okay. There are some things, like I said, that just require an end to end test. And VMs are the best for that. You can maybe get away with containers, but just use them interchangeably if you want. But in this case, I'm just keep using the word vm. VM testing lets you model really complex, pretty much arbitrarily complex states of the world. Specific kernel software versions, specifically the interplay between those. For me, on the desktop side, it lets me simulate different locales, different keyboard layouts, excuse me, things like that. And you sometimes just need them. So the idea is that you spin up a full vm, you actually run software, synthesize events, and then somehow assert that what you wanted to happen happened. That's usually through screenshots or SSH commands. You know, SSH commands would be like, this process is running, this file exists, this file has this contents, whatever. But those are usually the two mechanisms you do it. Okay, now we bring in the nix. Why do we have to bring in the nix? I'm glad we could read this actually. So I'm bringing in Nix because Nix provides a full first party testing framework that has access to Nix. And these three properties are really important because first party, Nix actually uses this to test Nix itself. And so it's not going away. It's running every day. It's running right now. There's thousands of jobs queued up right now by the next project in order to run these types of tests. Second, it's an actual test framework. It doesn't just define how to spin up a vm. It has a full API for writing tests and asserting they pass. And that's important because it's not just like run a Docker image or something like Docker provides the runtime for a container, but it's not gonna give you any of the tools to actually test in that case. This is giving you both sides of the equation. And then third, oops, sorry, go back one. Since it is powered by Nix, this is a benefit because you get full access. Well, the language is probably a detriment, I'm gonna be honest. But the access to Nix packages is a benefit because you get access to basically every version of every piece of semi popular software that has existed for the past decades. And this is really important because you could pin specific versions of everything down to kernel libc, everything. So this is the only way I've found, when the most annoying desktop users of all time, Debian users, bring an ancient version of long term support software and say, this thing doesn't work. This is the only way I've been able to actually verify it works. So let's take a look at what this looks like. The first step for any VM test is actually define the machine. I put optional pluralization because Nix lets you define multiple machines and do networking between them. We're not going to talk about that. That's like the whole talk. It's probably like a whole degree for machine configuration. It's just. And these air quotes are doing a lot of work. It is just the Nix OS configuration. You could put anything in there that you would configure a full NIX installation with. So that means, like I said, that means anything. Kernels, drivers, users, packages, everything. Step two is actually defining the tests. The tests are written in Python, they're not written in nix. So you actually put a string or embed a file with your tests here and Nix gives you this full Python API that gives you some nice high level stuff like waiting for systemd units. Since Nix OS uses systemd. Actually you could even do ocr. You could wait for certain text to appear on the screen and it just handles that for you. And it's just Python. And one of the things you get out of this is you actually can access a repl, so you could have the VM and just use the REPL to be playing around with your tests. So that's good. In this case we're defining a test that the ALICE user. I forgot to mention this, a previous one, we installed Firefox for Alice. The ALICE user has Firefox and Root does not have Firefox, obviously a toy example, but you could do anything here. And then step three is you have to run them again, hand waving going on here, as you have to do whenever you talk about Nix. But basically the important thing is you have a mechanism built in to the framework runtime to run the tests, to get a repl, to debug the tests, to develop the tests, to run a single test. Everything is sort of like there for you within a single command of some sort. And so this is just a full end to end thread thing in order to handle VM tests. And like I said, I just haven't found any other thing that's focused on providing this level of flexibility with a focus on testing. Right. There's tons of frameworks, I built some of them to spin up machines, spin up VMs, but not to just complete the end to end part of it in practice. What am I actually using Nixos VMs for? And again, stuff you really can't test without a full sandbox environment. The thing that really triggered me, the thing that really made me do this was input methods. I'm a fairly calm person, but input methods on Linux made me pound the table a few times. For those that don't know, an input method is basically any sort of how you input sort of certain Asian languages, emoji keyboards are an input method. It's the ability to input any character that's not represented on your physical keyboard is an input method. Handwriting as well as an input method. On Linux, the input method, the input method framework, the windowing system slash compositor, they're all developed by different people. And this is where I think the struggles of Linux really shine. Because of this very specific versions of different things just behave wildly differently. And it drove me crazy. So I needed to use VM testing to test input methods. I love Linux, I use NEXT os, but that's in contrast to something like Apple where you could really clearly tell there's some vertical mandate that all these things must work together in lockstep. So much easier as an app developer, but, you know, it's what I have to work with. And then there's other stuff, sort of desktop integrations, making sure that Open and Ghostly appears on right click. How can you possibly test that without actually like taking a screenshot and right clicking and taking a screenshot? These are things that also just break constantly in Linux with various version upgrades. So perfect for VMs. There's a lot of complaining about Linux up here, but it's just like this is the work you have to do to make, in my opinion, a stable desktop app experience for desktop Linux. I'm just going to put this here for later. Take a picture or just download the slides later. Here are resources where you could actually learn a lot more about VM testing. It's extremely powerful, but like everything in Nyx, the learning curve is like a sheer vertical cliff. So, you know, if you want to traverse the wall from Game of Thrones, then this is the resources that you're going to need to use. I think the benefit is the payoff from doing this. For the right type of testing that you need, there's nothing that compares. And so that's it. Thank you. I know we only covered five topics here, but I think it was a lot. Most importantly, again, if we got nothing else out of it, isolating side effects is a super powerful technique and I wanted to show multiple examples of that. And that's what I tried to do. And if you want more, see the go for contact from 2017, so thank you.