Microsoft Research Asia's Kaitao Song: Embracing Agents | Agent Insights

HuggingGPT is an open-source project developed by Zhejiang University and Microsoft Research Asia. Built on ChatGPT and Hugging Face, it breaks new ground by constructing a reusable agent framework that combines LLM capabilities with machine learning domain models, enabling agents to solve complex AI tasks across multiple modalities and fields.

Oasis Capital began closely following HuggingGPT from its release in 2023. We also spoke with one of its lead authors, Kaitao Song, a senior researcher at Microsoft Research Asia. Enjoy.

HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face

Problem Solved: Using large models like ChatGPT to solve multi-task planning problems while giving agents multimodal task-solving capabilities that reach expert model levels in specialized domains

Model Architecture: Leverages ChatGPT's language capabilities to decompose user needs into goals, then breaks these into sub-tasks. Based on model descriptions in Hugging Face, it uses its own language analysis abilities to understand and select the most suitable models for executing each sub-task

Performance: Capable of solving multimodal tasks including image recognition, pose detection, text-to-image generation, object detection, image classification, and text-to-speech. HuggingGPT's feasibility has been validated on single tasks, sequential tasks, graph tasks, and human-annotated datasets

Application Space: Multi-functional agents and domain-specific agents, such as WebAgent, education agent, shopping agent, etc.

HuggingGPT Task Planning Process

Task Planning: After analyzing user requests, decomposes them into a series of structured tasks, while also identifying dependencies and execution order between tasks to improve planning efficiency in LLMs. Uses prompt design based on specification-based instructions and demonstration-based parsing

1. Specification-Based Instructions: The LLM follows specific specifications to parse tasks (JSON format), providing standardized templates with parameters including Task, Id, Dep, and Args — representing task name, unique identifier, dependency relationships, and arguments respectively

2. Demonstration-Based Parsing: To better understand the intent and standards of task planning, multiple demonstrations are included in the prompt. Each demonstration consists of a user request and its corresponding output, representing the expected sequence for parsing tasks. By integrating dependency relationships, the logical connections between tasks can be understood

Human Evaluation Standard Performance Test

Test Results: Across various tasks, GPT-3.5 using the HuggingGPT framework demonstrated outstanding planning capabilities, outperforming Alpaca-7b and Vicuna-7b — particularly on complex tasks (sequential tasks and graph tasks)

Oasis Capital: Could you share your views on agents?

Kaitao Song: The agent is a concept that represents the future. Many people see it as the step before AGI. LLMs have already been deployed — the question is how to extend their capabilities outward, and agents are the most intuitive application scenario. What we're currently building are "LLM-based agents." The reason agents are so hot right now is that the powerful generalization capabilities of LLMs have been extended to rich, specialized scenarios, and in many fields including code, autonomous driving, education, and online services, they have already begun or are foreseeably changing traditional ways of production and living.

Personally, I think current agents are still experimental explorations — most remain at the research level. You can see that while agents can now achieve many capabilities, they're all actually defined and required by humans, and everyone is even competing on prompt engineering. True AGI requires autonomous thinking ability. But in 5-10 years, as LLMs themselves develop, many companies may build their own large models, privatize them, open-source and modify them, and open communities across chemistry, biology, physics, and other disciplines will grow. At that point, the role agents play in specific scenarios will inevitably become stronger.

Oasis Capital: You and your team released HuggingGPT in less than four months after ChatGPT's launch. What was your thinking behind this?

Kaitao Song: When ChatGPT first came out, we found its language understanding and generation capabilities to be extremely strong. This made us think about how to extend its language abilities. If ChatGPT's language capabilities reached human level, then flexibly solving other tasks like humans — including learning, imitating, and executing — should all be achievable. Using external resources to further extend the large model's own task-solving capabilities is like the human brain, or a computer operating system. So our team chose Hugging Face, an open-source library focused on machine learning models, as the community (providing numerous free models and model descriptions), allowing ChatGPT itself to explore and understand user needs and use models to solve tasks.

This approach lets us escape dependence on large models for different tasks, enabling them to strengthen their own task-solving capabilities by leveraging external resources, models, and knowledge — possessing community capabilities that can expand infinitely like open-source ecosystems, extending to broader task domains and solving complex user needs, using tools or visual recognition capabilities to expand the agent's task scope.

Oasis Capital: What are the specific steps in HuggingGPT? Which part best demonstrates large model capabilities?

Kaitao Song: HuggingGPT is divided into four steps: task planning, model selection, task execution, and response generation. First, use ChatGPT for task planning — analyzing tasks and planning steps. Second, select the most suitable model based on task analysis results. Third, execute computations through machine learning. Finally, return the execution results and present them to the user UI.

Overall, each part is important, but task planning best demonstrates the large model's generalization capabilities. In real-world scenarios, many user instructions are very scattered and don't follow standardized patterns, so this aspect particularly relies on the large model's language generalization and understanding abilities. ChatGPT needs to decompose user instructions into precise small tasks before execution.

Humans don't have only one route for task planning. We're also considering how to develop various task planning approaches from a multi-agent perspective, evaluating which path is optimal, then making a selection.

Oasis Capital: In the long term, agents need human-like thinking and continuous self-iterative learning abilities. Do you think Memory and RAG (Retrieval-Augmented Generation) are effective ways for agents to accumulate learning experience and iterate quickly over time?

Kaitao Song: These are all temporary solutions that don't fundamentally solve the problem. They essentially enhance LLM capabilities and compensate for deficiencies. Humans have memory, but they certainly don't rely on retrieval mechanisms to answer — instead they combine their own brain memory with language abilities to construct answers. Forming such a mechanism requires studying humans' long-term learning and iterative capabilities.

Human memory is a database, with some memories being hard memories and others soft memories. When receiving a question, people pause or think, and during this thinking process construct answers. This mechanism of memory retrieval and response generation is worth studying.

Oasis Capital: After completing the HuggingGPT research, where do you think its most likely application space lies?

Kaitao Song: Many specialized domains have application models — for example, education, online shopping, community interaction, and other scenarios. By combining various expert models and tools, agents can very effectively solve specialized tasks. Actually, HuggingGPT is more like a framework pattern — the backend can connect to many open-source communities, not necessarily just Hugging Face. A company's strongest moat is experience and data. Using the agent framework to import this data can form various vertical agents, satisfying needs across diverse application spaces.

Oasis Capital: What further directions do you think HuggingGPT will explore in the future?

Kaitao Song: First, we'll continue building and aligning with more real-world scenario data, hoping to discover more open-source communities beyond computer science. From a long-term perspective, we'll consider how to give agents true human-like continuous learning and improvement capabilities, binding these behavioral habits to the large model itself so that agent capabilities better adapt to real-world scenarios.

Oasis Capital: What are your outlooks on AGI?

Kaitao Song: AGI itself is a very broad concept, a long-term goal aimed at aligning with humans. I personally think it can be achieved to some degree in specific scenarios or roles. But if we're talking about aligning with real humans, I remain watchful and skeptical. Humans have a major characteristic: diversity. When answering a question, different people give different answers at different moments. Of course, AI and LLMs have already begun influencing human usage habits — for example, some feel that Go playing styles have been constrained by AI. So we also need to learn to think about and face how humans can maintain innovative capabilities under AI's impact. Can humans still possess the fundamental, brilliant, and mediocre moves?