Andrew Ng on Coding Agents and Deciding What to Build

Andrew Ng on Coding Agents and Deciding What to Build - https://www.youtube.com/watch?v=qeMMnOiF51A

You know, I'm actually very excited about the future of databases. I know you're going to ask me another question. Let's share one thing. I feel like right now the world is going through a massive data center build out with a lot of AI training and inference data centers being built. I suspect something that a lot of the world does not yet appreciate is that in the future a lot of core infrastructure, be it computers, storage, storage will be used not just by humans, but by agents. And that agentic usage of all this infrastructure will drive massive demand. And so in addition to the AI data center buildup for training and inference, I think a lot of databases have been built for human usage today and we can only look at or query so much data. But with agentic usage, I think the demand for core infrastructure for computer will, will continue to skyrocket. So actually very bullish on this category as a whole. Yeah, it's a, it's an amazing space. I mean we feel very fortunate to be in the thick of it and actually we, we had a partner day yesterday where we invited a lot of people who work with builders or developers and basically everyone's saying, yeah, we're, we're running out of compute. We need like the amount of builders pouring into the world is just going crazy right now. Yeah, yeah. So actually I'm going to start you're. Because you have one of the most impressive backgrounds that, and I couldn't memorize it all, so I'm just going to. I know you're very humble. I'm going to read out for anyone who doesn't know, you're the founder of Deep Learning AI. You're a general partner of an AI fund, managing partner of AI Aspire, Chairman and co founder of Coursera professor at Stanford University. Previously you were the founding lead of Google Brain, VP and chief scientist at Baidu, Director, formerly director of the Stanford AI Lab, and you are part of the TIME 100 AI list for the most influential AI persons in the world. So first of all, that's crazy. What a life actually. How do you have the energy to kind of keep reinventing yourself and keep doing or do all these things? I feel like the answer I have is the obvious one, which is I've been really so blessed to work with so many great people consistently across so many organizations. I feel like, but for the people I get to work with, I would hardly manage to accomplish anything myself. So I've just been really blessed. That's cool. That's cool. Okay, so I was going to talk about skills Map. I was going to try explain it, but you are first and foremost a teacher, so maybe you can tell us a little bit about skills, map what you're doing at the moment and then we can dive deep. Yeah, I think the most exciting profession to be that's very rapidly at this moment in time is AI engineering. And by, by that I don't mean people that take on an AI engineering job title, but rather people in all sorts of job roles, including software engineering, front end, back end, DevOps mobile that will benefit from having AI engineering skills. So my team at D1AI did a lot of work over several months to try to map out what are the skills that are needed to do AI engineering well. And so we wound up categorizing, we said we looked at over 10,000 job descriptions, job postings, did dozens of structured interviews, had ran a bunch of our own surveys and synthesized views, talking to partners as well. And we wound up figuring out that there are four key skills needed to do AI engineering well. One is the ability to build and deploy AI applications and the difference between AI and non AI applications is they're uncertain. And so how do you manage that? The second is software engineering fundamentals. It turns out that even with AI writing a lot of code for us, people that understand software engineering fundamentals, understand what's the read write ratio, what's a transaction, you know, people can steer coding agents much better that way. Third is the ability to use coding agents. And then lastly is the ability to is a skill of shaping the bill because developers are no longer playing a role where someone tells them what to do and they just implement it, but instead are participating in deciding what to build and also influencing the process by which something gets built. So what I'm seeing in the job market today is developers, either people with an AI engineering title or people with other title that's gained some AI engineering skills are in great demand and we just can't find enough people with these skills. So deeplendai is excited to work upon this to help more people become skilled at AI engineering. Cool. And the skills map itself actually first of all, starting with skills, why are you focused on skills in particular? I think you have a view on skills that they're one of the more important parts. Do you think that they're going to be critical as we go along or can the agents just become so good that we don't need to focus so much on providing our own custom skills? How do you see this part evolving? Yeah, so you know, I think about there's some Things we do today that I hope an agent will take over for me. For example, you may remember three years ago, there's a lot of hype about the prom engineering job and how much money someone could learn make by being a prom engineer. And I think at that time it was already clear to actually give Chris Manning credit for saying this very clearly even back then, which was this idea that we need humans have arcane incantations to prompt an ELM to do something that's going to go away because an ELM was going to learn how to make those same arcane incantations. And so even today I see some things that, like how do you manage the sword form of agents? Some things humans will be needed to do. Some things is more of a one size fits all that. I think AI agents will manage other AI agents better and better in the coming years. And one of the key things that distinguishes whether AI will take this over or not is whether or not humans have a fundamental context advantage. It turns out when all of us are building applications, we know a ton of stuff that AI agents do not know. You may remember a conversation with a customer where you said something and there was a funny expression on their face, or you may remember something that you learned five years ago reading some market report saying people want this and not that. I find that humans have a bewildering vast range of contexts and remarkably good at pulling out this context and make key decisions. And so for that set of decisions where humans have a fundamental context advantage of, until someday where I don't know, like slap a camera on us and monitor the facial expressions of everyone we talk to and have, that's a very difficult path. So I think there's a lot of the task with humans with a fundamental context advantage that we want humans to do for a long time and then giving people the skills to do that valuable work to complement the wonderful work the coding agent is doing, that seems to be the path that we'll be on from a long time to come. And this is why with, I think you've probably seen with coding agents being the wonderful things they are, the number of job postings, the software engineering is up because as software engineers and AI engineers become more productive, we want a lot more of them. And businesses, I know, even my teams, we just can't seem to find enough skilled AI engineers. And I think this will continue. Yeah. So let's dig into that. Let's imagine two developers who are, who are too strong developers, but one gets better results than the other. What would the stronger Developer be doing that the weak one isn't doing today. You know, this is a great question. When we're mapping out the AI engineering skills map, that was one of the questions we repeatedly asked ourselves, which is we actually interviewed a lot of people and one of the standard questions you asked them was think of a great developer, think of a less good developer. What is it that a really great developer does compared to the less good one? And a lot of the answers were what we wound up distilling into the AI engineering skills map. And it's actually more than one thing, it's actually a lot of very different things. But just to give another example, the first of four key skills we highlighted was the ability to build and deploy AI applications. And it turns out that the key differentiator for AI applications compared to traditional software applications is they are uncertain. Right. Whether you prompt an LLM or train a supervised learning algorithm, you don't know in advance what outcome you get. And so skill AI developers have come up with or are familiar with a set of techniques to monitor the performance of these systems, often in a statistical way, and to drive a disk error analysis and evals loop to make progress in a systematic way. And so it was by asking questions like that and reflecting on our own experiences that we then tried to systematically map out the, not one or two, but frankly the very large number of gaps between skill developers and novice level developers. And we tried to distill that into what we call the AI engineering schools map to then try to work systematic to help people go from your less skilled expert level at all of these really valuable skills. Is the skills map itself for anyone to use, mostly for software engineers, people with background, how should people think about the skills map itself? Yeah, so we published Deep Learned AI. We published a weekly newsletter called the Batch. So for five weeks in a row I wrote about nothing but the AI aging skills map and actually know other companies that are now looking to structure curricula around REI English Skills, which is actually pretty cool. But I feel like for individuals there's a way to understand what being excellent, which many of you already are, but where you're already excellent in other places where you want to become more excellent. And then one thing deep learning AI is working hard on is to then create those paths for people to become skilled at AI engineering broadly. But I think the educational space, both deep learned AI and other companies still have work to do to help people develop clearer paths to become skilled broadly at all of these AI engineering schools. I'm curious, what are you Seeing in terms of people who are pouring in, is it experienced developers? Basically there probably is some sort of curve where there's these experienced developers who don't want to let go. I mean, myself, I loved the getting in the zone of coding and it became a long time. It took me a long time before I was like, okay, now I just let the agent do it. Are you seeing this shift slowly happen where people are like, okay, I'm giving up and I'm shifting all of it to skills first. Yeah, it's interesting actually. You know, um, I, I don't know, it's funny, I actually don't miss coding by hand. Yeah, yeah, yes, sir. But actually, do you, do you miss the. I don't know, do you get like in the zone? You know, when you just kind of lock in and it's problem. You had to go to stack overflow and copy and paste 50,000 things. Maybe you didn't. I did. And then, you know, you spend three hours on a problem and finally it clicks. I don't kind of get that vibe, I guess anymore. But yeah, I also. Yeah, yeah, I bet you, I bet lovie have this problem too. Remember before AI, we used to have a list of projects we wanted to work on but couldn't finish. Now my list, I'm going to guess yours too has gotten way longer now that AI has supposedly made us more productive. So I find that for myself, there's so many projects doesn't work on and don't have time to work on. And even as our capabilities grew up, certainly my personal ambitions grew maybe even faster than our capabilities. And so, yeah, it's a different type of vibe. I think starting a coding agent before you go to bed to see what it comes up with the next morning is a different way of getting into the zone, I guess, than debugging some weird JavaScript syntax. But I have to admit, I seem to like the new vibe more than he owes one. Yeah, yeah, we're definitely getting more done, that's for sure. Actually, like, I'm curious how you design a curriculum in such a rapidly changing market. Do you foresee that you're going to have to change this curriculum every single year, every month? How are you iterating on it? You know, it's interesting. I feel like the hype is that AI advances so fast, everything changes every three months. Um, of the AI engineering skills map, the one piece that does change the fastest is coding agents. And the way we code now is very different than three months ago, very different than six months ago. So the use of coding agents is changing very rapidly. But when I look at the other parts of the skills that I find really valuable, the way we build and deploy AI applications, there's a lot of fundamental knowledge about how to steer these systems, how to deploy them, how to monitor them, how to update them, how to that is actually changing much slower than the PR hype says. One of the funny things, at least before Astra, I think Astra may have changed, I'm not sure. It turns out most AI systems, they're not very good data scientists and not very good AI engineers. You kind of ask it, what should I do in a project? And for the most part, frankly, data science, right. I now don't do my data science work by hand. No more coding up pandas, data frames, whatever I'm still using on the backend, but I now use AI agents to do a lot of my data science work for me. And then frankly it just makes really weird decisions, comes to really weird conclusions. I hope that frontier models will get better, but somehow there's a lot of understanding the fundamentals to make trade offs. And I see that a lot of engineering is making trade offs, right? Cost, latency and then speed, reliability. And the challenge is until you understand the fundamentals, you often don't even know what trade offs exist to be made. Which is why when someone vibe codes without understanding, very often the coding agent will make a bunch of trade offs that you didn't even know it made a trade off and just bites you later. So I find that people that are good engineers understand the fundamentals in many, many parts of the skills needed just make much better decisions. But I'm also encouraged at what's now possible to do. You know, when you write code without understanding. But I think really good engineers just do such vastly better work than, you know, inexperienced engineers. Yeah, yeah. And actually on that one, like as part of the skills map you talk about shaping the build. I found this kind of an interesting concept. Maybe you can explain what you mean by shaping the build. Yeah, so as building has become easier, the bottleneck is less building, but deciding what to build. And so previously there were used to be a lot of, let's say front end engineers, but now front end engineers with AI assistance can also do a lot more backend than was possible. I'm proceed much stronger on backend and front end, but now frankly I'm maybe like a bad front end developer with AI systems. But so a lot of developers have gone from a more narrow role to being a full stack developer. But then even beyond that, it turns out that if a developer builds something and needs to wait for a product manager to get customer feedback and steer what's built next, that's much slower than the developer themselves has a little bit of product instinct, a little bit of product gut. Okay, talk to a couple customers or users and get feedback and keep driving the process forward. So in development, what I'm seeing is a lot of people are expanding their roles from just implementation to deciding what to build. And it does not mean engineers should become product managers, but it does mean that engineers, they can make some product decisions or more architecture decisions are much more productive than engineers that played the more narrow side of role from a few years ago. And so we call this shaping the build where engineers that can influence product decisions have some product guts also help to think through the build process. Like is this a good time to build an mvp? Or is it a good time to invest in building really robust architecture and thinking about scale? Or is this a good time to go out to customers to get feedback? Or this is a good time to do some research to better understand what backend technology stack you use. So developers that can steer the building process end up contributing much more to the overall comfy or project. How much then should they spend time kind of designing the system? One of my favorite skills, I don't know if you know, it's this like grill with docs creates these like architectural decisions and it spends an hour grilling you exactly what you want to build. Like how much time do you think people should be focused on upfront, what they want to build, how they want to build it versus like just go build your MVP and then fix it later on. Iterating? Yeah, I think, yeah, it's a great question. The challenge answering that question is that the answer is stage dependent. And it depends a lot on what Jeff Bezos calls one way door versus two way doors. But directionally, in the early stages of a project, you know, frankly, whatever idea you have right, type it into coding agent, build it, see what you get. Soles can do it in a safe way and don't leak sensitive data or just don't solas their protection for bad outcomes. But I find that a lot of projects, one of the challenges about AI engineering is the right decision is very stage dependent. I find that for early stage projects, solids, you can do it in a responsible, safe, safe way. Build a quick and dirty prototype, see what happens, throw it away. Building a quick and nervous product, see what happens and drive rapid learnings and then there's also a time and place for thinking carefully about architecture, figuring out what's your rewrite ratio, what are the customer requirements, what is sensitive data, thinking through your security infrastructure and then doing that. So one thing I've seen is I've known a lot of engineers that are really good at the zero to one startup phase. I've also known a lot of good engineers good at the mature systematic phase. But sometimes, depending on the stage of project you're on, the skill and the speed and how to think about velocity versus protecting from downside, risk shifts and even the user coding agents you see, a lot of people talk about how, oh, you should always plan it out, write a long spec and there is a place for that. And then sometimes there's a place if you know you're going to go to bed in 20 minutes, just bang out some stuff, have a coding agent work for a few hours and see what you get the next morning. And I think that stage dependent answer has been very tricky for people to learn. Okay, I need to switch gears and I need to ask you a question because I was chatting to Swix yesterday from Legendspace. I don't know if you know him, but he's here covering the sweep and he said something spicy, I might misquote him, but he said something like, oh, schools are dead. Or the way that we're changing, the way that we have to learn has to completely change. I'm sure you get this question a lot. I'm curious on your take on how will learning change put us forward 10 years? How are we learning? Surely does it look the same as what we're doing now in schools? Completely different. So it will change. So there are two observations. So first I'm going to say something obvious but maybe sound controversial, but chatbots are terrible for learning, right? I love OMS, use them a lot. Kudi Pro AI at the same time, on university campuses and in K12, we know that students that use LLMs do better on homeworks but worse than the final exam. And educators have known this intuitively. Students have known this intuitively, but the rigorous studies are now increasingly coming out that show that cognitive offloading to an LLM, meaning if a student lets an LLM think for you, you get more work done, but you learn less. I think even though the rigorous studies on this are coming out in K12 and in higher education university in formal educational context, I think people still underestimate the degree to which this is happening among working adults. I find that I love my coding agents, helps me do a ton of work. But the reduced thinking I have to do, I think there are certain things that. And then to be clear, there are ways to use arms that do improve learning. Just like I think there are ways to use TikTok that's great for learning. But the vast majority of ways that the vast majority of people use social media does not result in learning. And the vast majority of ways the vast majority of people use OMS does not result in learning. Now the flip side of this though is I think we can do better. So the teams I work with, including Coursera, Udemy, Deep Learned AI, Learn Vector, are working hard to create new learning experiences that are not focused on getting stuff done, but rather on developing, rather on focusing on accelerating human development. Here in Silicon Valley, we're very excited about our front end AI labs and accelerating AI development and that's great. Very much in favor of that. In Learn Vector, which was a recent announced company funded by, with Coursera's initial investor. I'm actually really excited about not just accelerating AI development, but accelerating human development. And the way that we're operating is building Learn Vector into a frontier AI, into frontier human development lab. And as one example, I feel fortunate that Coursera participated in changing education 15 years or so ago. But today, looking back on that experience, which still works, it feels very much one to many where we create one course and everyone kind of takes the same or very similar course and it works. Nothing's wrong with that. But the technology now exists for us to create much more one to one experiences and it will be much better. So Learn Vector hasn't yet announced exactly what products we have, but I'm actually very optimistic from the internal prototypes we have and we will have more to share in months. That's super exciting. Okay, I want to leave the crowd with something tangible. So let's say you could design a way that everyone here could spend the next four weeks upskilling to become the best AI developers. How should they spend their time for the next four weeks? Where should they go? What should they learn? What tools should they adopt? Anything that comes to mind. I know. So you know, in the case of AI engineering, deeplanddey's focus is to help developers become skilled at AI engineering, or if you're already skilled, to become even more skilled, working hard on, even improving what Deep Learned AI offers. But I actually find myself taking a lot of Deep learned AI courses myself to learn what are the frontier tools? Kind of what's the latest in the voice stack or in Generative UI or You know, building local memory for agents. So I find that maybe as an insider, I see a lot of ways I wish deep learning were even better, but I think that my own learning, actually, a lot of it has been. Has been with deep, learned AI. So I hope you all will find that hopeful cool and keep building. I think. Yeah. I think. I think all of you are builders, Right? I find myself. I still find myself waking up in the middle of night too often to check what my coding agents up to. Do you fire off on your phone as you wake up in the middle of the night? Yeah, I don't know. I think coding on the phone has been so wonderful, but also maybe. I don't know. Yeah. Do you text or do you use voice to chat to your coding agents? I use text a lot more than voice to coding with the usual Mac Mini. Okay. Yeah. Okay. That's cool. There's a bit of inside baseball. I'm very curious about your setup, actually. Okay. First of all, I know that you had a very strict schedule, and so we're incredibly grateful that we could fit you in and you could fit us into your timing. So thank you so much for coming. Coming to select. It's really a pleasure and an honor to have you. Thank you so much. And just say, you know, of all the credentials you rattle off with me just now, the thing that's going through my mind, my one credential you didn't mention was I'm also proud father of two children, so. And so the reason I need to dash is because I actually need to go pick them up from school. That's super important. Okay, we'll let you go there. Thanks so much, Javi. Thank you. Into noise hardware and a regularity.