OpenAI has unveiled an exciting feature for its ChatGPT desktop app on macOS that promises to transform the way developers interact with their code. This latest update introduces Work with Apps, allowing ChatGPT to interact directly with popular coding applications like VS Code, Xcode, TextEdit, Terminal, and iTerm2.
Gone are the days when developers had to painstakingly copy and paste their code into the chatbot. With this new feature, ChatGPT can now access and understand the code you’re working on, using it as context for your prompts. While it doesn’t yet write code directly into your apps like other AI-powered tools such as GitHub Copilot, this functionality marks an important step toward creating AI systems that can work seamlessly within your digital ecosystem.
A New Era for AI Collaboration in Development
The Work with Apps feature offers developers a streamlined workflow by automatically incorporating code from active windows into ChatGPT’s context. OpenAI emphasizes that this is just the beginning: they aim to expand the chatbot’s capabilities beyond coding tools to text-based applications for writing, brainstorming, and more.
During a live demo, an OpenAI team member showcased ChatGPT’s new prowess using Xcode. The team opened a simple solar system project in Xcode that was missing Earth. By selecting an Xcode tab within ChatGPT, the employee prompted the AI to “add the missing planets.” ChatGPT generated a line of code to represent Earth, perfectly aligning with the project’s format.
While users still need to paste ChatGPT’s suggestions back into their apps, the potential here is clear. The feature’s focus is on enhancing collaboration, and according to OpenAI desktop product lead Alexander Embiricos, it’s “a key building block” toward creating true AI agents capable of independently understanding and working across diverse applications.
How Does It Work?
OpenAI leverages macOS’s robust Accessibility API to power the Work with Apps feature. This API, originally designed for Apple’s VoiceOver screen reader, allows ChatGPT to access text from supported apps and translate it into a format the AI can interpret.
However, the system has its limitations:
- Text-Only Scope: ChatGPT can only read and interpret text. Visual elements such as images, object layouts, and videos remain out of reach.
- Extensions for Certain Apps: Apps like VS Code require users to install specific extensions to enable the feature.
- Token Usage: For supported apps, ChatGPT will pull either the last 200 lines of code or all text in the foremost window as context, alongside your prompt. Highlighting specific text can help the AI focus, but even then, surrounding lines are included—potentially leading to hefty token consumption.
Despite these constraints, the integration with macOS lays a solid foundation for broader, more sophisticated capabilities in the future.
Why Start with Coding Tools?
AI-powered coding assistants are among the most popular applications of large language models (LLMs), making this space a natural starting point for ChatGPT’s integration journey. Developers frequently use tools like GitHub Copilot or Cursor for code generation and debugging, and OpenAI’s foray into this domain aims to offer a complementary experience.
Unlike full-fledged AI agents, Work with Apps focuses on enabling collaboration rather than autonomous action. OpenAI’s approach ensures reliability and precision, setting the stage for more ambitious advancements down the line.
A Glimpse into the Future of AI Agents
This step toward app integration hints at OpenAI’s broader ambitions. Recent reports suggest the company is working on a general-purpose AI agent, codenamed “Operator,” expected to debut in early 2025. Designed to rival emerging systems like Anthropic’s Computer use or Google’s rumored “Jarvis” agent, Operator would take AI collaboration to a whole new level.
For now, Work with Apps represents an important milestone in this journey. Unlike Anthropic’s current approach—using screenshots to understand and interact with apps—OpenAI’s reliance on APIs ensures faster and more accurate results.
“This isn’t meant to be an agent; it’s a way to collaborate with coding tools to start, and there will be more tools coming soon,” said Embiricos. “This idea that ChatGPT can understand and work with all the content you have is a really key building block.”
What’s Next?
Currently, Work with Apps is available to macOS users with ChatGPT Plus and Teams subscriptions. OpenAI plans to roll out the feature to Enterprise and Edu users in the coming weeks.
While there’s no official timeline for a Windows release, the macOS launch coincides with Apple’s upcoming ChatGPT integration, slated for December. This strategic move positions OpenAI to deepen its presence in Apple’s ecosystem while laying the groundwork for cross-platform expansion.
Final Thoughts
The introduction of Work with Apps is a clear signal that OpenAI is steadily moving toward a future where AI agents can seamlessly interact with your digital workspace. By starting with coding tools and leveraging macOS’s accessibility features, OpenAI has created a compelling new way for developers to enhance their workflows.
As ChatGPT continues to evolve, its ability to integrate with other types of applications—beyond coding—could redefine productivity across industries. For now, this marks an exciting development for developers and a promising glimpse into the potential of AI collaboration.
