Oasis Capital and Xingyao on His 2024 ICLR Agent Best Paper Award | Agent Insights

An agent is an intelligent system built on top of a foundation model through multi-turn interaction with humans, endowed with end-to-end problem-solving capabilities. Yet current agents lack accurate, flexible, and self-optimizing interaction between the model and its environment.

Xingyao Wang, a PhD student at the University of Illinois Urbana-Champaign (UIUC), developed CodeAct as a novel approach. By using code as the underlying medium for agent-environment interaction, it dramatically improves efficiency. Going forward, CodeAct can continuously enhance online learning capabilities — the model finds new data through environmental interaction and iteratively optimizes itself, potentially alleviating the problem of depleting internet data and the resulting bottleneck in model capabilities. Enjoy.

"EXECUTABLE CODE ACTIONS ELICIT BETTER LLM AGENTS"

CodeAct overall framework diagram

Paper link: https://arxiv.org/abs/2402.01030

Problem addressed: Traditional agent architectures complete actions by generating predefined JSON or text, which limits the action space and flexibility. CodeAct uses code to consolidate agent actions into a unified action space. Integrated with a Python interpreter, CodeAct can execute code actions and dynamically revise previous actions through multi-turn interaction, greatly enhancing agent-environment interaction capabilities.

Model framework: At its core, CodeAct offers a conceptual approach: using code for agent-environment interaction (such as tool calling) to improve efficiency and generality. CodeAct's action space naturally supports processes like search and memory — for instance, memory can be implemented by creating a database.

Application scope: CodeAct can replace the agent-environment interaction component in any agent setup.

Performance of various models on tasks of different complexity under the CodeAct architecture

Evaluation results: CodeAct proves more effective on complex tasks. For example, GPT-4-1106-preview achieved 20% higher performance on complex tasks in M3ToolEval compared to traditional approaches, while using 30% fewer interaction turns. CodeAct's advantages become more pronounced as model capabilities improve.

Oasis Capital: Could you briefly introduce your research background and your views on agents?

Xingyao: I'm currently at UIUC, primarily working with Professor Heng Ji on interactive language agents. I believe the essence of an agent is a foundation model — whether LLM, VLM, or other multimodal large models — that engages in multi-turn interaction with humans and the environment. ChatGPT is essentially a product of foundation model-human multi-turn interaction; it just lacks the foundation model-environment interaction that current agents have. Early attempts like AutoGPT essentially pre-programmed task details and left the agent to execute autonomously, mostly missing human feedback and correction during the process.

The interaction between model, human, and environment in an agent needs to be coordinated. Our CodeAct work, published at ICML/ICLR, focuses on exploring how to let models interact better with the environment: we investigate code as a medium for agent-environment interaction, while human users interact through natural language — this is the core idea of CodeAct. Going forward, we'll extend this framework to more practical application scenarios.

The CodeAct architecture can naturally accommodate processes including search and memory. Memory itself can be implemented through CodeAct — a straightforward method is to simply create a database for the model, which then uses code to access the database API and write information row by row. Search works similarly: just set up a Google Search API for the model to use. CodeAct provides a conceptual approach of using code for deployment and tool calling, directly leveraging existing code packages and saving much of the extra work that would otherwise require human developers to predefine different tools.

Oasis Capital: Could you explain how the model-environment interaction paradigm has evolved from past to present?

Xingyao: CodeAct is an improvement upon past interaction methods. For instance, many people previously used ReAct combined with tool calling, where users had to predefine a tool list, then write documentation for each tool so the model understood what it could do. The core pain point was "forcing" the model to output in a specific format (like JSON), which required writing complex parsers to catch and fix numerous edge cases in the output.

So in past agent development, I distinctly experienced how painful it was to constrain model outputs. Models would always generate bizarre results — sometimes an extra space, sometimes a missing comma — causing the engineering wrapper around the agent to become extremely bloated. Yet tools ultimately still needed code to execute, and after execution the results were fed back to the model. Throughout this process, the model had no idea that when it was interacting with a specific format, it was fundamentally still interacting with code. It was like wrapping "dozens of layers of tape" around the agent, making the development process cumbersome and slow. CodeAct's philosophy is: fewer layers of tape is better. Expose the entire environment to the model and tell it to interact directly using Python commands. If errors occur, the automatic error messages returned by the Python interpreter can directly prompt the model to fix its code.

Overall, CodeAct has four major advantages compared to previous approaches:

First, it can significantly reduce the number of interactions between model and environment;

Second, improved accuracy;

Third, generalization of software functionality;

Fourth, automatic feedback optimization.

Currently CodeAct can already complete some meaningful tasks, such as data science and data visualization tasks. In the future, we hope to see CodeAct solve more end-to-end problems and form more complex applications.

Oasis Capital: You're also a major contributor to the popular open-source project OpenDevin. Could you briefly introduce OpenDevin and what work you've done on it?

Xingyao: OpenDevin is a remarkable project. Junyang and Binyuan from Alibaba's Qwen team launched it on Twitter one day after Devin (the recently famous AI software engineer project) was released. I contacted them immediately to join the community. Right after we launched with just a README file, we got over a thousand stars. Then CMU professor Graham Neubig submitted the first PR with a simple interface, and everyone started iterating frantically in the community. Now it has over twenty thousand stars. I wanted to integrate CodeAct into it at the time, and gradually became the second-largest contributor to OpenDevin.

My contribution to OpenDevin was essentially applying CodeAct in real-world scenarios. CodeAct had previously only been experimented with for agent-environment interaction in Python, but many software engineering tasks have high requirements for agent-environment interaction. The main environment is actually the Bash Terminal (Linux command execution). So in OpenDevin, I extended CodeAct to the Bash language. Recently we've also been trying to extend OpenDevin to browser-like programming languages, where users can control browsers through language. OpenDevin will gradually operate with more and more unrestricted languages.

Oasis Capital: When CodeAct encounters completely unfamiliar tools, does it require fine-tuning?

Xingyao: Currently there are two scenarios. For some relatively common tool package calling cases, foundation models built on CodeAct are already very familiar with them through extensive pre-training — they will automatically select the Library API without human intervention. For applications in specific scenarios (such as having the model call some private tool packages), you may need to provide them with Library Documentation or code examples. Beyond that, you can also choose to deploy CodeAct on your company's own privately trained large model, so the agent can learn your internal code usage from the start.

Oasis Capital: How do we define which tool a model should choose for better execution? For example, if I want to book a flight from China to Europe, how would a CodeActAgent know whether Google Search, ChatGPT itinerary planning, or WeChat inquiry is the optimal approach?

Xingyao: We're also thinking about how humans make final decisions. We find that humans also learn through interaction — for example, if someone doesn't find satisfactory results from an online search, they'll naturally turn to WeChat to ask friends. One of our important future research directions is agent online learning capability. For flight searches with approaches A, B, and C, we want the model to check all three, compare the results, and discover that C is optimal. Then the next time the agent searches for flights, it can automatically choose the optimal approach. Most interaction processes will be learnable in the future.

Oasis Capital: Could you explain the significance of developing CodeActInstruct?

Xingyao: CodeAct works well on closed-source models but performs poorly on open-source models, likely mainly due to insufficient instruction fine-tuning in open-source models. For example, Code Llama itself has adequate code pre-training, but lacks corresponding data during fine-tuning. CodeActInstruct can fill this gap and make open-source model fine-tuning more effective.

CodeActInstruct is a more comprehensive dataset, with three times more data than comparable datasets (at least 5x more tokens). So we hope to use CodeActInstruct to help developers improve foundation models' CodeAct capabilities.

CodeActInstruct dataset compared to other code datasets

Oasis Capital: What are your next steps, and in which domains can CodeAct be better applied?

Xingyao: We've combined OpenDevin with Princeton's SWE-Agent and found that CodeAct has already made good progress on the SWE-Bench standard (Software Development Task Completion Benchmark). Meanwhile, CodeAct can integrate more functionalities. For me personally, in research I hope to improve the OpenDevin framework while developing the online learning capability I mentioned earlier.

We're also working to make CodeAct's interaction pattern into "infrastructure" that's generalizable to more domains. In CodeAct we only interact with Python, mainly because the engineering construction time was too high. But in our community, we've found everyone is eager to build general software engineering agents, so people contribute different interaction methods in the community — CodeAct with Bash, browsers, and so on. I'll leverage these community experiences for my research, and ultimately give back to the open-source community with a more generalized agent-environment interaction approach.

Oasis Capital: Currently people are very excited about low-code agent end-to-end automation. What conditions do you think are still needed for such products to actually land in practice?

Xingyao: Learning from environment interaction is still very much missing. Foundation model capabilities can already meet basic requirements — with slight prompting, code can reach 60-point level, but this performance is unstable. Sometimes it gets stuck in infinite loops. Besides improving foundation model capabilities, the most general solution is to let the agent learn by itself. It's fine if the agent makes mistakes at first, but it needs to improve next time. Additionally, pre-training will soon exhaust all the world's data. When internet data runs out, I believe we need to find data from environment interaction and learn from environment interaction — this is a crucial iteration method for future large models.

Essentially, CodeAct provides the agent-environment interaction process, strengthening code as the medium, not strengthening the underlying model's code generation capability. After an agent accesses CodeAct, it still needs to learn specific abilities like planning, decision-making, and reflection — but we don't want the agent to abandon other capabilities after learning code interaction. People can continuously learn from historical experience which technical solutions work or don't work. I believe the next step is for our foundation model + agent to achieve this too, which becomes even more important as internet data continues to deplete.

---

Oasis Capital: How do you view the long-term relationship between open-source and closed-source large models? What role will synthetic data play in model evolution over the long term?

Xingyao: My underlying assumption is that large models will eventually exhaust all human internet data. At that point, open-source models' capabilities will gradually catch up. OpenAI is so strong largely because of first-mover advantage. The gap between open-source and closed-source models will slowly narrow, unless online learning requires so many resources that the open-source community simply can't do it at all. If that's not the case, open-source models won't lag too far behind.

On synthetic data: actually, what I mentioned earlier about human-environment interaction is also synthetic data to some extent — it's just that the model synthesizes its own output data. In the short term, synthetic data can definitely improve model performance. For example, there was a recent Apple paper, "Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling", where they paraphrased pre-training data and mixed it back in for secondary training, finding significant performance improvement. This is a more concrete use of synthetic data — "cleaning" the messy content in model data so the model can learn better. Additionally, CodeAct's proposal of using code to strengthen agent-environment interaction data accumulation and enhance agent online learning capability is also a viable approach for capability improvement.