I Spent $10K+ to Test Every AI Coding Agent (Augment, Claude, Cursor, Devin, OpenAI)

You're probably on one of two paths. The first one is that you're very skeptical about the AI tooling because you've tried a bunch of different ones. You've lit a bunch of different tokens on fire, but at the same time, maybe your employer or a demo online has you thinking differently. You know, you should be doing it. Or the second camp is that you already are using AI coding tools every single day. Maybe you have most of your use cases covered and that's great, but you think there's probably room for improvement. In this video, the intent is to get everyone on that first path onto the second one and everyone on the second one to learn a couple things that I've come across. A little bit of context about me. My name is Parker X. I've been working in technology building startups for over a decade. Had one that had a successful exit where I led the product, the design, and a lot of the technology. After we were acquired, I consulted with the CEO to help them with their rollup strategy, which involved hundreds of engineers, dozens and dozens of products, and yeah, there was a little bit of AI. So, I've done a lot of stuff. It wasn't just at that company. After that company exited, I rolled in some cash to try my own hand at different startups, learned a lot about market dynamics, what not to do, and ultimately decided I wanted to be an engineer. So, after three years of using AI pretty much every single day to help me learn, I've gotten pretty good at it. And I make YouTube videos about it, which means I have this added benefit where people give me their tools and I get to test them for free. Some people pay me to make videos about them and I disclose that, but I'm allowed to say whatever I want because I want to be honest with you guys. Let's jump into talking about all the different tools that I've used. And I'm going to cover the six different use cases that are very common when you're building products. And yes, I took some notes because I want this to be a helpful video that has a little bit more shelf life than most of the videos on YouTube from my incumbents. So, we're going to cover bug fixes. How do you deal with those? What model should you use if the model changes? What type of model should you use? We're going to cover how to deal with stale branches and merging PRs because that's a new problem that we have is if you're really leveraging these agents, you can wake up with 7 to 10 PRs and they're really good. We're going to cover planning and how to think about it. What a PRD looks like, the different types of PRDs. My background was for six almost seven years as a product manager, so I know a few things about PRDS or product requirement documents. We're going to cover the code execution and what it actually takes to get the AIS to work, what not to do, what to do, how to save time and keep all those patterns that you love to use every day in line. We're going to talk about design and what it takes to go from zero to one on a net new design while drawing inspiration from other brands and other products that you already love. We'll also talk about design fixes. When you come back and you see and you're like, "Wow, this is not that good." What do you do then? And all of this will be through the lens of two different modes of using AI coding agents. The first one is what I call active AI coding. And that means you're at the desk, you are working one, maybe two, maybe three at a time with different threads and actively watching what it's doing. And you'll notice that the skill there is actually code review in real time. And then we'll talk about the second mode, which is passive, meaning you are having a background agent, a remote agent is what some companies call them, but you're doing a lot of that planning up front because you're not going to be there to be able to watch it. So you want to put as much work in up front as possible. And like I said, because I make these YouTube videos, I've gotten access to every tool that they have under the sun. I've gotten tools access to the ones that aren't even out yet. So, I'm authority on the subject. I also operate a community quick plug. It is called VI or Vibe with AI. And we have over a 100 engineers who just eat and breathe and love talking about AI. So, it's not just me. This is also engineers from Microsoft, engineers from Google, engineers from high growth startups that are all coming together to try to come up with what's with what's best. So I'll start with a quick deck that I put together for our community last Friday. Should be about 5 minutes, but it should provide more context before we jump into code. So the first thing to understand and it was kind of a revelation when I thought about it is that we as the people pushing AI every day are actively defining it. The concrete is not set and we need to come up with primitives, patterns, frameworks of thinking through how to use these tools because anytime a new model comes out have to think about what won't change, right? So that's why I brought it back to these six areas. And so this quick walk through should definitely save you a lot of time and hopefully a lot of money. So I spent six weeks. I said I'm going to build this school competitor. And what it looks like now is you have your posts. You have your different categories of post. I don't know if it's running. So I should probably probably check that. Let's get it running. And I said I'm going to do this in two weeks and I'm going to do it only with agents. and it did not work. But over the last week of doing what I'm about to show you in this video, I was able to get to here. So, it's pretty featurerich with what you'd expect. We have the different categories and they're all separated and they're quick. And then if I click into a post, it's going to load up the post detail and it's going to be a little bit slower cuz we're in a next dev environment. We're going to see an error here. Looks like some thing around the dependent link, but we can fix that. But you have the different links that are highlighted. You can see different comments in here. This is before I had some of the styles in here. So, you can see that they're a little bit off, but you can go ahead, you can share, you can save this. So, that would go to your bookmarks. You can report a different post, which would go to me, the administrator. I can see the history of the post if there is one. Obviously, you can edit that. There are different types of posts. So, text, image, video, link, edit the category, all that sort of stuff. Uh, you can delete the post as you'd expect. You can reply within these with either a an attachment or an emoji or a gif with gy. You can also upvote things. So that uses optimistic updates. So there's all this stuff around the posts. It also has a calendar that's tied into a group calendar for weekly calls that we have. You can see that that's pretty nice. And this would publish. You can create the new events around those. You can say if they're physical or if they're virtual, what type of event is it? All these sorts of things. Then you see the members. So, we have a member directory in here. You can click in to a user. So, I'll type in Parker. And you can see all my posts, my comments, and my votes. You can link out to all the different things. I can edit my profile. All of it's in here, right? You can see other people's bookmarks, their recent activity. So, a bunch of stuff in here. You can also notice at the bottom, you can see when the next call is, you can see when the last push of meaningful code was, how many people are online. There's some modules around learning and whatnot. basic notification system has who's online right now, autoplaying for videos, has the road map of what we're going to build next. It has recent news that's pulled in from we're going to use this as a use case in the video to make better because Devon, well, it did do the job technically. This is too bland, too boring. The design looks bad, but it does work. We have who's online now. We have the road map of things that we're going to build next. So, this is just pulling from GitHub. You can report a bug. It will automatically grab the environment information. That's pretty helpful. You can upload screenshots, recordings, it brings them straight into a nice little ticket for me. And then you can also request a feature. So that's nice. If you wanted to look at the full road map, then it would be in here. That's what it was that we built. And how did we get there? What did we learn? Right? So the first thing that I learned is that you want to pick simple tech. When you're using agents, they've been trained a certain way on a ton of information, right? And it's basically they threw it into a blender, all this information into a blender, and they're not great at learning new things. There's a really good talk by swool John Carmarmac where he speaks about how they're really bad at learning and he compares our state-of-the-art models with a six-year-old learning how to play Doom. And it just makes AI models look pretty foolish because it takes billions of dollars and a lot of time playing the game to be as good as a six-year-old can be in just a matter of minutes. That said, pick the things that they know really well. You can rule around this, but why would you want to when you don't have to? you'll save time doing the stuff that they know and shipping more code without having to fight it. So simple things if you're in the JavaScript ecosystem, grab npm. I know you don't love that, but grab that, grab eslint, use something that you know deeply. That was my big mistake was I first went and I built this and I painted the canvas myself. I figured out how to use tan stack start just okay enough to get it done. I had never used Astro before. I had all these things. I said this is going to be great. But they I didn't know anything about it. So I was just fighting it the whole time. So it defeated the point of why I was doing this. And I think in a second I'm going to talk about how you should start at the end when you're building something new. Also when you're building even a feature what is it that I want? What's this result that I want out of this? And that would have helped me save a lot of time because the result that I want from this is an easy to use codebase that agents can drive. So this is the opposite of that. This is a two week thing probably longer. The second thing is I would have started with a feature tree. So, if you've ever been acquired before, if you've ever done a reverse engineer on somebody else's product, then you've probably built one of these. In my case, I just make spreadsheets and I just list out what are all the different features of what this product has that I want to have in my product. And so, literally everything. So, in my case, I just said, "Oh, I'm going to make Reddit plus school plus some chat GPT features with train GPTs on certain topics that people want to learn." But I didn't dig into what each one of those would require. So version history on posts, edit history, the ability to report stuff, all these things that I do want so that I'm at least, you know, I have the table stake features. I I can be at the table. I didn't list those out. So I underestimated the amount of stuff needed because I was like, "Oh, AI." And this is an example of a feature tree. I just use spreadsheets. The next thing is you want to paint the canvas. So when you go to start a project, you'll probably find that bootstrapping it from zero when you have a blank canvas with AI is not the move. It can do it, but it's not going to be able to follow patterns that you already know. And the guys from the startup did a really good talk on this. I'm not going to play it, but you should go listen. Dax talks about this where he's like, "When I get to something that I want an AI to do, it's usually a split between there's a dumb thing and there's a hard thing, and I give it the dumb thing." And so when you have enough meat on the bone, so to speak, with what you're building, then it has patterns that it can chew on and replicate. The one thing I did select because I wanted to simplify the stack was I picked up Convex. And they're not sponsoring this video. I wish they would because I love their product, but they allowed me to remove a lot of things. So that was the one big difference. And then I ended up with this which is not the same type of graph as the one that you saw before because there are a bunch of nodes but this shows all the things in it. So very simple for me to work on. I should have started at the end and I think you should do this too when you're working with agents so you can understand why is it that I'm doing this? What is it that I want? So I wanted an easy to understand codebase that I'd only have to grab dependencies when I had to. I wanted it to be fun to work on. That's a big one. so that the agents can drive 90% of the feature development and then I can focus on things that it can't do which is let's actually innovate here. Can we make it so that when you land on the website you can just talk to the form and it fills it out. I saw a great demo of that in one of our show and tell meetings and vi to do it and I want it to be incredibly stable with rigid testing and linting patterns so that everyone that wants to contribute to this project can. And then why am I doing it? Well, it's so that I can share my learnings. That's a big part of it. I like to educate people. I started a YouTube channel when I was 14 called Parker the Pyro and I taught people how to do chemistry. So, I have a long track record of liking doing this. I'm 32 now. I wanted to prove to myself that I could do it in less than two weeks. And so, we're on the 11th day of this build and I didn't work on it for probably three in there. So, a little over a week of actual dev work. And then I want to drive my understanding of how these agentic workflows or even agent workflows can and cannot work. And then I list out the actionables of it. So that's what I would have done. I would have started from the end a little bit better. And now we'll go through some of the highle stuff which is when you're doing remote and background agent work. I got access to Devon. They're not sponsoring this video, but they gave me access to the tool. Shout out to them. And I was really impressed. If I'm going to bed and I have things lined up that are already issues that are already ready to go, it's a great way to get work done because their way of setting up their environment is very thorough. Some people look at it and they're annoyed. It's like, oh, it takes me a half an hour to set this up. It's a really worthwhile half hour because it can just spin up separate VMs of wherever the snapshot was at for your last master or main codebase. and it's really good at that. I'd say out of all the PRs, I think I've shipped maybe 30 of them through Devon as a starting point. One of them messed up, but that's on me. I tried to push it a little too hard. And then I'm reaching for Augment for merging stuff. I'm kind of getting into the next part of this video, but Augment, they do sponsor the channel. Full disclosure, I don't use it for everything. I treat it as the merge master because it's still it's slower but it's like the tail of the tortoise and the hair where cloud code will fly through stuff and it won't work. I know if I go to augment for merging stuff and having like basically two big code bases or decent sized code bases are very small but if I need a merge I know I can rely on augment. If I need a bug that 03 can't go and get I know I can rely on augment. So I just I have it in the back pocket. I'm using it every day. not for everything. Then if it's something that's smaller, I'm going to reach for cursor and 03 because that's inclusive in their pricing. It's a little bit better. You get more leniency on the pricing there. And then I'll use Opus 4 for active feature work with cloud code. So let's jump out of this and talk about each use case. So debugging, we have different levels to it, right? So you might find that if it's something simple, you can just pick whatever tool. But it starts with having good observability. You can reach for Pino for logging. You can have observability through Versel or any of your providers. Logs always work. Traces work as well. But you need the evidence of what's going on. And so I found that for small bugs, I can just quickly maybe open up one of a new cursor instance and I'll have 03 selected. And you can see here I had some of these tests that were messing up. So I wanted it to just go and do that as well as the linting. So I'll go in and I'll just say fix our tests because this is what cursor does with their bugbot product. They just have a a bot that sweeps in behind a PR. So in the case with a small bug, I would just drop and you know fix this bug with 03 because of the way that Bugbots's been doing it. And Bugbot comes with Cursor Pro. You simply point it at a repository and then when you have PR comments come in, it's automatically going to fire. If you don't have it set to automatic, which is on by default, then you can mention Cursorbot in the comment and it will fire. It's free until I think July 27th while it's in beta. Highly recommend it. Cursor doesn't sponsor me. It'll drop a quick I emoji and then depending on how many lines of code it's reviewing, it'll answer within two to 15 minutes with a comment. So you'll see here there are a couple bugs in here. Coming back from Devon, I click on fix and cursor. When this opens up, it will just say fix this bug dot dot dot space and then all this context. So the combination of a small bug fix with just fix this bug. It works pretty well with 03. Now if I needed to uplevel that, that's when I'll grab augment. And what I can do is I'll open augment up. You can see I used this for merge recently. I'll open it and I'll actually grab this prompt. Now, you see that it's in this Claude commands. I don't use Claude for debugging, but it's where I host a couple of these prompts. So, what I can do is I'll put this in here, and it says, "Reflect on the five to seven different possible sources of the problem. Distill those down to the one to two most likely, then add logs to validate your assumptions before we move on to implementing the actual code fix." So, then I will put in whatever relevant logs are in there. I'll tag all the different files that could pertain to that. And I will not click enhance prompt. I won't because I already have it the way that I want to. And if you want to, you can click it just to see what happens. It won't count against any of your credits, just so you guys know. The next one, which I alluded to earlier real quick, is just for merge. So, you can see most of these in here over the last 7 days are for updating branches and dealing with merge conflicts. That's my go-to with augment. It always does it right. It saves me so much time and so much effort with merge conflicts. And this matters when you have a bunch of different PRs. You wake up in the morning and you have seven to 10 of these things. You want it done right. I've never have it run into an issue. I have had it run into issues with everything else. I don't know why, but yeah. And they do, full disclosure, sponsor this channel, but they're goated at this task. The next one we're going to talk about is planning. So there's a lot of ways to do this. I've tried all of them as an ex-product manager for a long time. I spent most of my days just basically researching what's the best way to plan out this next set of work. You know, cuz as a product manager, your job is to sit between three business functions. And why am I telling you this? Well, it's important because you might not know how to write a PRD. You might not have the background of why they exist. But you sit between the business needs, the design team, and the developer team. And so you have to be thinking about all of those. And the product requirement document can look different at every company. But at the end of the day, it's a distillation of what problems are we going to solve for the business, ideally lading up to a goal. And so that way when you go to do the work, you are the one as the product manager defining why are we building this? And that's why I mentioned that earlier where you start at the result. It's really nailing down why is it that we're doing this? It gives purpose behind it. Ideally, it's tied to a big customer problem or a problem internally with your team. But starting with why and then figuring out what you're going to build, right? That's when the solution comes in. You can do up to three of them. I know superbase when they do a request for comment and you go and you submit something you have to write three of them. I don't do that but it's a nice practice. And so then the engineers they figure out how. So I'm going to show two different ways that I like to do planning for new work. The first one because I know everyone doesn't have access to all these tools. You can do this in any tool but I'm going to show you two different flows that I lean on. So the first one is using cloud code and it's using this issues prompt. If you're not familiar, there was a really good YouTube video by the people at the company called Every. And it had a great title of something like how two engineers ship enough stuff or whatever. I don't know. But just every is the name of the company. Type in every cloud code. It's a popular video. But I took their prompt from this video and I kind of edited it. But what it allows me to do is I can just freehand some feature advice or feature require. uh I can freehand some feature that I want. So I'll type in issues and then I just start to type out what is it that I actually want because an issue can be an enhancement, it can be a bug, it can be all of these things, but I'll use this for smaller features where I don't think it's going to take a lot of work. I think that this will do a good enough job and so I'll just let it go and do its thing. It's going to create a nice issue. It's going to scan the repository as you can see in here. It's going to go scan it. and it's going to look for the issue template. It's going to look for similar files. It's going to do all this stuff because most likely you've done work like this in the past that will be similar. So, it's providing the context of upcoming work. It can try to kill or keep or combine an existing issue and that's really nice. So, that's the way that I do this is I'd plan it doing this and then I'd actually make sure it looks good. Right? So, this is active development. Now, the other one that I use is just straight up cursor. So, I have this file in my cursor rules, and it's called planning. This came from a developer who spent $6,000 a month on cursor max, which is crazy. And he's actually legit. He's not just a marketing guy that probably hasn't built anything because he worked at Stripe. He's got a track record. Now, you can get this by going in the description below, but what it does is it provides a format that you must follow where it breaks the work into phases. So, I can show you here on the lefth hand side. I have a a bunch of different folders that show you the things that I want to build, the things that I've already built. But I'll basically make a new plan. I'll just do plan. RST if you're not familiar. I wasn't. RST is a different flavor of Markdown. You can use markdown. This is apparently more expressive. I just use it because he used it. The benefit of it is when I hit commandB, it doesn't affect me from or it doesn't prevent me from opening and closing the sidebar because it's not trying to bold things. So, that's why I like it. But, it breaks things into phases and I found that I'm spending the majority of my time actually just doing the planning because if you get this right and I'm sending this off to a remote agent, I'm going to do this one. If I'm doing the issues one, that's for active development. on trying to get through and go and go and go. For something that I'm going to give to Devon, this is so important that you're taking the time and you're making sure all these phases look right. There will be mistakes. I will say that 100%. So, competency is at a premium. It has always been at a premium, but it's just so important to remember, you're not going to be the Michael Jordan of AI coding if you don't know how to code. Sorry. So, make sure that you're watching this as you're doing the planning. I'm going to be using 03 in here. You can see over here I have 03 opened and it gave me the first version. Now I'd go back and I'd check for mistakes and I'd see what's going on. You can see in this case I said come up with a plan outlined here. So I had already written some stuff up top and then I say use the rule and update it. And here's some images for inspiration. So it's going to use vision. It's going to see oh this wants to look a little bit more like that. I can pass in different commands for downloading an icon library for instance because a lot of the icons on our website have these little animations. So there is a library for that and I just go and I grab this command. So it looks something like this. I can pop that into the plan because it has full access to the terminal which is great. So this is where the sauce is guys. If there's one area of planning you don't need anything else. You just need to build your road map by if it's small stuff, use the issues template. If it's bigger stuff, you want to use this planning prompt to then really unlock the way that you're thinking about this. It's not all about the prompt. It's about you and how you use it, right? That's just that's the truth. So, the next one is execution. Let's get into execution because there's going to be two modalities here, right? We have the active and we have the passive. So when I'm doing execution, I'm going to offer different tools because not everybody has these, but I'll open up cloud code if I'm actively going and I can have it run a plan. So I'll say I'll tag the plan and I say let's get started. Before I do any of this, I always set it to opus. So if you have opus, you should be using opus. Absolutely. That's one hard part about augment. I'll be running augment in tandem, but it's going to be using forsunit. So, I'll have both of these going. I personally like the diffs that I see out of cursor a lot better than cloud code. So, I'm questioning if I'll even keep cloud code. Might try out cursor max next month. But, it's actively watching it. You're not kicking it off and walking away. You're not scrolling through Twitter because you're going to find areas where maybe it did veer away a little bit from what your original plan was. Again, the better the plan, the better the output. Now, if I was going to go and spin something up with Devon, then I will want it to actually create an issue for me because it has a linear integration and a GitHub integration. So, the outputs of these plans will then need to go into GitHub if you need those remote background agents going. And the way that you decide what should go to a background agent and what should not is do the harder ones actively. So you're there and you're locked in and you can see what's going on. Give it the smaller ones where there's more examples in the codebase that they can use to effectively accomplish the task. The next one we're going to talk about is design. So this is an example of going from 0 to one. And I think I yeah I have 10 versions of this. So I have a paid v0ero account because it's a great $20 investment. And 22 days ago when I originally started this, I just said, "Hey, I want to have some sort of Reddit version for developers." and it gave me a rough idea. You can see it's very similar to where we ended. And I just kept iterating on that until I got to something that seemed like it had the bones of what I wanted. And so if I, you know, click ahead to the latest one, it'll upload that. And you can see, okay, not much has changed. Maybe we have a detail page, maybe we can see a little bit more stuff, but I'd start in here. And the best way to do that is just go find examples of products that you like and remix them. So, this exactly what I did for anything that I'm doing from v 0ero, well, it makes sense. I go to v 0ero and I haven't experimented with using their API key, but now talking about it, I probably should do that. So, that's what I do is I get in there and then I will actually just download the code and then bring it out because I don't want to deal with like their TS config or I do know that you can connect it to branches. I tried this when it was in beta. It wasn't all that great, but I'm hoping it's gotten better. And then the second part of that is, well, how do you do something where you're iterating and you want to make it a little bit better? So, let's go back and we can see that this looks pretty pretty ugly. So, what would I do here? Well, I'd probably I'd probably look at this and I'd say, well, first of all, there's too much white space in here. That doesn't look so good. What is this like weird rectangle around it? And I'm just thinking about the design of it. I understand that the actual quality of of what results are coming back. That's a different problem. That's functionality. So that would be me probably going and making a new plan, actively developing it because I want to know what the inputs of the news sources are. How is XA API involved and what are we capable of doing there? So that's going to require a plan. It's totally different. This is just us saying we like the UI not so much. So let's make it a little bit better. So in our case, I'd just get out a little pen and paper and I'd be writing down these things that I'm saying out loud, which is okay, there's too much white space. This is too close to that. We kind of like the look of this one, but not so much. But what can we take from other areas of here? I don't really like this outline. I think there should be something different between the way that this is laid out. Maybe relative dates instead of like 75 225. That doesn't really vibe with me. So, I'm I'm writing all these things down. I'm thinking about them. And then I'd probably pull up some sort of website that has a lot of these examples. It doesn't necessarily need to be a news website, but you'd think, okay, where is a similar pattern, a design heristic where people are used to seeing information that changes a lot. What about like Poly Market? I know there's like a a betting website not sponsored by Poly Market. Okay, maybe this is kind of interesting. Is there a site or a page on their on their site where if I like click into Man, this is getting getting crazy in here. If I click into one of these, is there something that's changing? Is there a component? But I'm just doing the research. I'm trying to figure out how can I lay out this information that's going to be shifting on a schedule and I need to figure out that schedule. How can I do that to make it better? And then that's when I'd actually come in and I'd make a new plan, just literally a file that takes everything that I have in here. And I would then come up with different screenshots of what's going on. and I'd go back into either using Cloud Code or 03 or augment to work on that plan using that same rule, but including the screenshots, including some of the UI elements, including the tags of globals.css or any of these icons that I need to add. And that would also kind of pull out the information that I'm going to need to know for the functionality because the functionality drives the user interface. So it opens up the conversation to like okay well if the frequency of this is every minute probably wouldn't be that but that would drive a very different user experience because well does this feel like a stock trading platform do we want to give users the option to upload their own sources? Do we want that right now? Probably not. So, it does open that up and you might end up branching off and spinning a quick new plan out about the functionality because you realize, well, why would I make the effort to make this prettier? But that's the whole thing is like you're exploring it. And if you decide, no, I want to keep it this way. Let's just clean it up. That's fine, too. And again, the level of preparation that you do, the level of research, examples, screenshots, that's going to dictate the quality of the output. So, you can keep it simple. You could say, "Hey, make this better." But better is very subjective and it's not going to be a great result. So, I hope you guys learned a couple of things in this video. I recommend trying out different tools, but don't hop around too much. Remember that your skills that won't change over time are planning, systems design, understanding the fundamentals of the technologies that you're using, and then it doesn't matter how much these models change. Don't take it from me. take it from Jeff Bezos 20 years ago when he gave a TED talk on this and he says I like to focus on the thing that people don't ask a lot about which is what won't change and what won't change is us being competent. So if you enjoyed the video make sure you like the video. Make sure you subscribe to the channel. We have a new show coming out this week that's called Ship or Skip and it's allowing me and my friend Dan to just share what stuff that we're giving a shot and what stuff we're saying we're not going to continue using. So, we'll be the lab rats for that. If you want to join us in our community, you can also check that out in the link below. And I'll see you in the next one.