OpenAI's New App Integration: A Deep Dive into its VS Code and iTerm Extensions

Introduction: What's New with OpenAI?

OpenAI recently launched a new feature that allows its AI to connect with different applications. This isn't your typical co-pilot that's built into a single tool. Instead, it's an extension that you install, which then allows the AI to understand the context of whatever window you're working in, be it VS Code, iTerm, or other supported apps. This capability promises to enhance productivity by bringing AI assistance directly into your workflow. This blog post will delve into this new feature, specifically focusing on its application within VS Code and iTerm, and assessing its strengths and areas needing improvement.

Target Audience: Developers, tech enthusiasts, and anyone interested in the latest AI integration tools.

Estimated Read Time: 7-10 minutes

Prerequisites: Basic familiarity with VS Code, iTerm (or other terminal apps), and general programming concepts.

Understanding the Core Concept

The core idea behind this new OpenAI feature is to provide contextual AI assistance across various applications. It works by installing an extension that allows the AI to “see” the application window you’re working on. This enables the AI to understand your code, terminal commands, or other activities directly, offering more relevant and helpful suggestions.

Supported Applications

Currently, the feature supports a variety of applications, including:

VS Code
Terminal
iTerm
Xcode
Xedit

This diverse support is a big step for developers working across different environments, particularly Xcode.

Setting Up the VS Code Extension

To get started with the VS Code extension, you'll need to follow a few steps:

ChatGTP Plus or Team Account: You'll need to be a user with a ChatGPT Plus subscription or a team user.
Download the App: From their blog post, you'll find a link to download the necessary file that enables the app integration.
Install the Extension: Open VS Code, then open the command palette (Command + Shift + P on macOS), then use the command: “Extensions: Install from VSIX” to install the downloaded extension file.
Understanding Context: The extension can access the context of the currently open windows. If no specific selection is made, it will analyze all visible content on the screen.

While these steps can be a bit involved, they are necessary to connect the AI to your VS Code environment.

VS Code Integration: A Practical Example

To demonstrate the extension in action, let's look at a real-world example with an electron app.

Scenario: Debugging an Electron App

The app was created using a two-shot prompting method with the 01 agent. The app allows users to view and eject connected disks and drives directly from the Mac menu bar. However, the eject functionality wasn't working correctly.

Loading Files: In VS Code, you can select various files for the AI to analyze. The extension will show you what files are being loaded, and you can add more if needed.
Prompting the AI: A prompt was given to the AI to debug the app. The prompt was divided into three parts:
- The AI's role: an expert electron developer.
- The task: debugging the app.
- The goal: Ensuring the eject all function works correctly and is logged safely.
AI's Response: The AI gave suggestions, which primarily included a try-catch block to handle potential errors, and adding logging.

Critical Analysis: Strengths and Weaknesses

Strength: The AI provided code that addressed the prompt, wrapping the relevant code into a try catch, adding logs, and handled empty devices.
Weaknesses:
- Copy-Pasta Output: The output is all copy-pasted instead of providing concise solutions. This might lead to developers not reading through the generated content.
- Lack of File Context: The AI didn't show which file needed to be edited which would save the developer from having to look at the code.
- Missing Diff: The output should include a diff view of the changes that are being done. It makes it harder to compare before and after changes.

iTerm Integration: Another Test Case

Next, we explored how the integration works with iTerm, a popular terminal application.

Scenario: Running Terminal Commands

The iTerm integration was tested by asking the AI to remove the last 200 lines and clean the cache.

Loading Context: The extension recognized the iTerm window and could access its context.
Prompting the AI: The AI was asked to remove the last 200 lines and clear the package cache.
AI's Response: The AI provided the relevant terminal commands.

Critical Analysis: Strengths and Weaknesses

Weakness: The AI does not provide one liners. Instead it gave two separate commands.
Weakness: The AI also requires you to manually copy paste the given result, which is a big step for every command.

Overall Assessment

Pros:

Contextual Awareness: The ability of the AI to understand the context of different applications is very beneficial.
Cross-Application Support: The wide range of supported apps allows developers to bring AI across multiple workflows.
Potential for Productivity Gains: With time and refinement, this feature could significantly speed up debugging and code development.

Cons:

Copy-Pasta Output: The AI gives big text output instead of concise answers. This might result in developers ignoring the results which can be detrimental.
Lack of File Context: The AI doesn't include which file the changes are being made to, which should be automatic with the context feature.
Missing Diffs: The output should be in diff view so that developers can easily see the changes and compare it to the existing code.
Installation Steps: The installation of the extension is a bit involved and not seamless, which could be improved for user experience.
iTerm Integration: The AI gives commands in separate lines instead of one liners and requires manual copy and pasting of these commands.

Recommendations and Future Directions

For this feature to truly shine, a few improvements are necessary:

Concise Output: The AI needs to be more concise and just provide the necessary information instead of copy pasted information.
File Names in Output: The AI output should include file names in its suggestion, so that developer does not have to waste time in locating the file.
Diff View: Providing diffs would make it easier to review the changes before implementing them.
Simplified Installation: Streamlining the installation process would improve user experience.
Improved iTerm Output: Combining multiple commands into one lines and eliminating the need to manually copy paste will significantly improve the terminal integration.

Conclusion: A Promising but Imperfect Tool

OpenAI's new app integration is a step in the right direction for AI-assisted development. While the initial implementation has some notable flaws, the core concept is powerful and shows great potential. As the feature matures and addresses the current limitations, it could become an indispensable tool for developers across various platforms. This is definitely an area to watch closely for future updates and improvements.

Next Steps

Explore: Try out the integration with different applications and projects.
Feedback: Provide feedback to OpenAI about your experience to influence future improvements.
Stay Updated: Keep an eye out for updates to this feature that will likely come soon!