AI News: OpenAI Releases Tools
November 15, 2024
OpenAI’s new cross-app extension aims to give an AI assistant access to the context from the apps you’re working in, with a VS Code extension as the entry point. Parker walks through how it works, what you can do with it, and a hands-on demo across apps like Electron apps, iTerm, and Xcode.
What’s new with OpenAI tools#
- Extension-based approach to connect multiple apps (VS Code, iTerm, Terminal, Xcode) for AI context.
- AI can see the active window and the files in it, but full system-wide context isn’t available yet.
- New beta-access UI: a prominent icon in the host app to access “Work with beta” features.
- Availability notes: you may need an OpenAI plan (Plus/Team) to access certain capabilities.
- Requires updating the host apps and installing a VSIX extension in VS Code.
How the integration works#
- An extension you install provides the AI with window-level context and the ability to reference files you’re working on.
- Context is limited to what's visible in the current window or screen region; it’s not a full-system context dump.
- After updating, you install the extension via VS Code and load a VSIX package from the official blog post.
- The extension supports multiple windows/files (e.g., loading up to several files so the AI can reason over them together).
Demos and workflows#
- Quick demo setup:
- Use VS Code to load multiple files and have the AI operate on them.
- The user can prompt the AI with a system prompt like: “You are an expert Electron developer...” to drive debugging or feature work.
- The AI can be asked to log actions (e.g., ensure an “eject all” command logs out safely).
- Other app contexts tested:
- iTerm: the AI can fetch recent lines (e.g., last 200 lines) and manipulate the session.
- Xcode: the extension can also surface context from Xcode projects.
- The flow includes using the Mac menu bar for quick access to the extension’s context, and you can add more files to the current session as needed.
- Dictation tip: Mac’s voice mode can be handy to speed up prompts and notes mid-work.
Pros and cons (as observed)#
- Pros
- Adds cross-app AI context, useful for multi-file, multi-app workflows.
- Quick way to drive tasks in code and terminal contexts with natural-language prompts.
- The chat desktop ecosystem is notably faster and more responsive than before.
- Cons / caveats
- Context is not full-context; you still need to be mindful of what the AI actually sees.
- Output can be “copy-paste heavy” and requires careful reading and validation—don’t assume it’s perfect.
- You may need to adjust prompts to get precise file names and diffs; the AI sometimes summarizes or obfuscates filenames.
- Some setup steps are a bit finicky (download the VSIX, install from VSIX, etc.), which could be streamlined in future updates.
Getting started (practical steps)#
- Update your host app (look for updates in the app’s menu).
- Download the VSIX extension package from the official blog post.
- In VS Code:
- Open the Command Palette (Cmd/Ctrl + Shift + P).
- Run Extensions: Install from VSIX and select the downloaded file.
- In VS Code, the extension will load the current window’s context; you can add more files as needed.
- Quick test prompts:
- Use a system prompt like: “You are an expert Electron developer. You are tasked with debugging our disk utility app.”
- Define the task, role, and goal, and observe how the AI sequences steps (e.g., identifying where to add logs, what to change, etc.).
- Optional: enable Mac dictation (Voice mode) to speed up note-taking and prompts.
Prompts and best practices#
- Define a clear three-part prompt:
- Role (system): the AI’s specialization.
- Task (one-line): the specific action you want.
- Goal (end-state): what a successful outcome looks like.
- When reviewing outputs:
- Check for actual file names and diffs when code changes are suggested.
- Validate critical actions (e.g., safe logouts, destructive commands) before applying changes.
- Treat it as an assistant, not a replacement for careful review. Read outputs and verify in context.
Next steps and what to watch for#
- This is a meaningful step forward for cross-app AI copilots, but it’s not perfect yet. Expect iterative improvements in:
- Context depth and accuracy
- Ease of installation and onboarding
- More robust handling across more apps and file types
- Parker hints that more coverage and updates are coming in follow-up clips, especially around deeper flows and additional app support.
Links#
- OpenAI Platform — blog post with the OpenAI cross-app extension and download instructions (VSIX)
- VS Code Extensions — extension installation guide (Install from VSIX)
Transcript
[Music] we're getting to open AI stuff but just so you know if you like this kind of content this is part of a larger episode called the sitre it's a weekly 25 minute video for all the other things this is one of usually five to 10 different stories cool with open AI we've been excited and waiting for something to come out and they just put this video out about 19 hours ago from today I want to show you a quick look at an early no we're not going to watch that it's going to be too long you can go watch that yourself but the gist of it is they've added in this ability to basically connect different apps so if you've used other other tools in the past you typically have a co-pilot that's native to the product what's different about this is It's using an extension that you install into in the case of vs code to get context of whatever it is that you're looking at so it can do window it can do multiple windows but it has to see the window so you can already kind of guess one of the main cons if you're a programmer to that limitation is it doesn't allow you to get full context which I just think is wild and so let's take a look at some of the other areas you can see here that it has terminal iterm X edit and X code I think this is going to be the biggest win for xcode and if you want to keep up with all the updates from the actual creator of this Alexander and baricos is the head of the chat desktop app and they did launch a lot of stuff as well in tandem that allows it to just run better this was a super laggy app before and now it's very quick so you'll notice you're at the bottom there's this big old new icon and when you click on it it'll pop up and it'll show you work with beta so if you don't have this what you can do is just update your app so I believe that is in the top left you just hit check for updates I think this is available on Windows as well I'm not a Windows user and when you go in for the VSS code one I'll give a quick demo of it what I had to do was on the on the blog that they posted there's a file that you download and you basically download that file and then you run a command so I'll just show you how that that works we'll just go into the article here and let's see if we can't see this work with apps on Mac OS cool exactly what we're looking for if you don't have chat gbt plus you need that it's also available for team users but here's all the options kind of covered that it has the context of the windows you can also select Snippets if you don't select anything it's just going to see what's available on the screen and you basically go and there's a download link in here and you download it and and uh then in VSS codee you will do command shift p or open the command pallet and then do extensions install from V6 that's the extension type so couple steps to get it up and running wish that was easier but that allows you to get this extension that then makes it work so let's do a quick demo of it just to show so in this case I have an electron app and I made this by two shotting with 01 and had a really good prompt and if you want to make stuff like this that'll be part two of this chat which is this agent that I built this agent flow got to get a little coffee and in here what this electron app was doing was I had a bunch of different desktop or ssds that I just wanted to be able to see them in the top right Mac bar and just be able to object all of them at once instead of menu clicking so let's open up a couple files and we'll see how it works so in here you would just click code and now it says working with code and this also does work if you have the smaller tost so you'll see the code that's loaded in there if you hover over it and you can add more and more files so if I say okay I want to have four then you can obviously do that and so now I have four and in this case I know that this was not working I hadn't taken the time to actually debug it but I know what part's not working so I'll say you're an expert electron developer you are tasked with debugging our disk disy app and I typically I'll do a couple on liners that are very tur and the goal is to say what the role is it's like a system prompt in line one and then the second one would be what that task is of the rule the third one is the goal of the task so in this case we want to make it so that when you do hit eject on all because currently this app lets you see all the different discs as well as phones basically any connection we want to make sure that it actually does that job and logs it out so currently or uh expected outcome for users is that they can select the disy icon in their Mac bar and when they do that they can view all of the attached discs ssds hard drives and there's an option to click eject all we want to make sure that this is logged out and done in a safe way so what I just did there too as a tip oops didn't hit Escape was I just entered voice mode and that's just done in dictation through the Mac settings it's really good and I'll just hit enter and let's see now the first thing I want to say about this is before he even gives me an answer let's assume that it's awesome I just don't love that it's all this copy pasta and I think it's double-edged sword because a lot of the time when you do have something that's in line say we were using Klein or we're using continue or we using cursor or uh AER any of those tools it just kind and you don't watch it you may try to force yourself to do it but a lot of the time unless you're very disciplined like get lazy and those muscles atrophy so I think on the side of like being bullish on this product or this solution is before you go and you do the copy paste just mentally make a note to actually read what it's outputting so it looks like I we need to update this function so then what I would do is I haven't looked at this code in a while so I would just do command shift o and then I'll type in get external and that's not even it so I'll do this get external drives where is that file that's in the tray so it looks like it added some stuff here do a quick scan see what it actually did P external cool if air looks like it's the same right it basically wrapped it in a tri catch and that's fine and then it also for this menu item so I'll grab that and then we'll just dump that in there okay and then for the reasons it says okay cool we validated that the try tray icon there great we did these logs cool and we're using 40 probably be better with A1 preview but I like to do that stuff for architecture and I don't want to burn through those update create menu items to handle empty or missing devices gracely so let's find what file you know what what I'd like to see is that this would be the file name that seems like a very obvious thing if it has the context I get that the language is important this is just a standard heris that you see but I would like to see the actual file name in there rather than having to explicitly say hey this is what we're doing and and telling it because everyone has the same problem so now we have this one where create menu items it looks like it handled those empty things what is it actually doing though just added this label change so nothing nothing different again I'd like to see the file name and I'd like to see the diffs because what we're doing right here and while you're waiting eagerly waiting for me to get to the next thing you have to go through it and see what is the difference and you're looking back and forth so that's pretty pretty annoying and then eject volume to log let's see where eject volume is on it's function now I just added logging with a bunch of ifs interesting it's actually deleting a bunch of it two so that's the gist of it when it comes to vs code I want to do a quick example let's try something else let's try with iterm so I know I typically use warp I think that this is just there's really no reason for me to use anything else cuz I don't pay a dime for it and I can just start typing natural language in the app so I think this is kind of B for them I don't think it's that great where now I need to open this see how it works though and where are you iterm we already did that so now it's working with iterm let's see what we can do let's say remove or write a last 200 lines interesting so holds in the context the last 200 lines remove package and Clean Cash see what it does from RF seems like it would put it in one line for these first two at least that seems weird and I again I still have this added step of cool now I got to go so enough on that one I think you kind of get where my head is at when it comes to that particular release I think that it is helpful but it's kind of a Miss at the same time so if you like that clip make sure you check out the other ones here I got plenty on the channel thanks