Shanghai AI Lab researcher Zhiyong Wu: Tomorrow's FRIDAY | Agent Insights

Large language models have accelerated progress in building agents. How to achieve autonomous interaction with computers and expand agent adaptability to general computer tasks remains a long-term challenge with enormous potential. The Zhiyong Wu team at Shanghai AI Lab introduced the OS-Copilot framework to create FRIDAY, a general-purpose computer agent with self-improvement capabilities that is more dynamic and vital. How far are we from the next generation of more intelligent agents? Enjoy.

OS-Copilot: Towards Generalist Computer Agents with Self-Improvement

The OS-Copilot mentioned in the paper is a general-purpose computer agent framework; FRIDAY is one of the computer agents built on OS-Copilot

Problem Solved: Previous computer agents were too limited in their interaction with different OS elements (web, code, multimedia, etc.). FRIDAY, as a general-purpose computer agent with self-improvement capabilities, improves success rates on computer tasks.

Model Architecture: FRIDAY's front end uses an LLM to decompose task requests into subtasks. The configurator maintains a working repository storing retrieved tools and knowledge. The back-end executor carries out action commands based on prompts generated by the configurator.

Performance: On the general AI assistant benchmark GAIA's 466 QA questions, FRIDAY achieved a peak success rate of 40.9% — a significant improvement over GPT-4 Plugins at 30.3% and AutoGPT-4 at 15.1%.

Application Space: General-purpose computer agent

Autonomous Learning: FRIDAY can learn new knowledge. To master a tool, it is prompted to execute task flows related to that goal, progressing from simple to challenging, forming proficient use of the tool.

OS-Copilot self-learning example

Experiment: FRIDAY was evaluated on 466 QA questions from the general AI assistant benchmark GAIA, achieving a 40.9% success rate on the easiest level of tasks.

FRIDAY's test results on GAIA

Oasis Capital: Why the name FRIDAY?

Wu: FRIDAY is a more upgraded intelligent butler. The name comes from the Iron Man series (laughs). Most people are probably more familiar with JARVIS. After JARVIS was merged into Vision, Tony built a new butler bot called FRIDAY, or "Miss Friday." We wanted FRIDAY to outperform current open-source systems, so we chose that name. Intelligent assistants have always existed — they didn't emerge with ChatGPT. In industry, conversational assistants and chatbots have been around for over a decade. ChatGPT made what used to be complex conversational assistant and chatbot design simpler and more effective. Out of personal interest, we happened to catch the wave of next-generation intelligent assistants.

Oasis Capital: Could you specifically introduce the capabilities and characteristics of OS-Copilot and FRIDAY?

Wu: OS-Copilot is more of a conceptual framework at this stage, not yet a real protocol. Framework design is being attempted in both academia and industry — Microsoft's Windows Copilot, AutoGen, and others. Compared to previous RPA (Robotic Process Automation) general assistants, OS-Copilot can demonstrate more powerful capabilities. Compared to past well-known agent projects like AutoGPT, FRIDAY's most distinctive feature is its self-directed learning module.

Self-learning is similar to how humans acquire skills. For example, if someone wants to learn Excel, they'll find tutorials, study them, and accumulate experience through practice. FRIDAY works the same way — if you ask it to use Excel, it will autonomously generate a series of Excel-related tasks proposed by GPT-4, such as creating bar charts, adding formulas, and so on. FRIDAY takes these tasks, attempts to solve them, and accumulates relevant skills during the process, writing those skills into code tools. When it encounters similar tasks later, FRIDAY can call upon its accumulated tools to complete them with higher accuracy — similar to NVIDIA's Voyager agent.

Oasis Capital: After self-learning, where is FRIDAY's memory stored?

Wu: This is the memory question. Drawing from biological concepts, we divide an agent's working memory into long-term and short-term memory. Long-term memory includes declarative memory or procedures, similar to RAG (Retrieval Augmented Generation), existing as text in the file system and retrieved when needed. Short-term memory maintains real-time content in memory, related to in-context learning and reasoning capabilities commonly discussed with large models.

The most core element in FRIDAY is this short-term working memory — extracting past experience, combining it with current user instructions for information processing, then feeding this to the large model for reasoning to generate specific execution commands, which the back-end executor carries out. Working memory is the most critical component for completing reasoning.

Oasis Capital: Where do you think FRIDAY's greatest application potential lies?

Wu: Currently it's more suited to niche needs, such as using adjusted agent framework designs as auxiliary tools for Excel, PowerPoint, code, and so on. Traditional large RPA companies won't dedicate resources to building tools for small demands. OS-Copilot provides a framework that can guarantee precision, allowing users to directly generate an agent that understands them and meets their personal needs.

From a more macro perspective going forward, OS-Copilot will gradually become an open-source OS standard protocol, or a widely accepted framework, helping developers save time on repetitive development. Meanwhile, as the community continues developing and accumulating experience and cases, the results will keep improving.

Oasis Capital: Regarding FRIDAY and your other research, what directions do you want to pursue next?

Wu: For FRIDAY, we'll first spend more time on engineering, planning to refactor the code and create extensive tutorials and cases to attract developers to contribute to open-source. On the research side, we want to build something like a simulator-based OS ecosystem, where the large number of emerging agents can continuously self-learn and explore the environment, making it easier for developers to evaluate agent performance. This is crucial for later synthetic data construction.

Currently FRIDAY's control is implemented through code. But in real application scenarios, many software applications don't expose APIs — for example, WeChat and Tencent Meeting can't be controlled or operated through code calls or scripting, so code-based control has inherent limitations. From this perspective, we'll invest significant effort into a multimodal version of FRIDAY, using mouse and keyboard to operate these systems like humans do.

For broader LLM research, we'll focus on new architectures. Transformer is basically dominant now, but its structural design has natural limits — for instance, its attention mechanism restricts processing of long sequences. We want to explore next-generation language model architectures.

Oasis Capital: In agent research this year, what breakthroughs are you hoping to see?

Wu: I have two hopes for agents. First is multimodal agents. This is the year of multimodal large model explosion, with major improvements in model capabilities. While current vision-based agents still lag behind code-based agents in performance, they have better generalization and can support control of closed-source software. Vision agents complete tasks more like humans do — more natural and more "intuitive."

The second focus is hoping to see a unified agent standard protocol emerge this year — for example, how agents communicate with each other and what interfaces should look like. So far there hasn't been a unified set of guiding principles, making reuse difficult. A standard framework could save developers significant repetitive work. In prompt engineering, we're already seeing people work on standard protocols, such as Microsoft's Guidance library and Stanford University's DSPy. So if some major institutions take the lead in establishing standards for the agent field, I think it would have important guiding significance for the entire field's development. I believe this year will be an important time for preliminary standard protocols to emerge in the agent field — let's wait and see.