SenseTime Researcher Hangyu Mao: Collaboration and Alignment Between Agents | Agent Insights

Agents are gradually shifting from executing single tasks to making complex decisions, evolving from traditional natural language agents into large language model agents. Foundation model capabilities are improving at a breakneck pace. Recently, Oasis Capital sat down with Hangyu Mao, a senior researcher at SenseTime's large model division, to discuss the future of agents. With nearly a decade of experience in agent and reinforcement learning research, Mao contributed to the TPTU (Task Planning and Tool Usage) framework and pioneered designs for the One-Step Agent and Sequential Agent — work that has offered fresh inspiration for Oasis Capital's recent thinking on AI alignment research and agent applications. Enjoy.

TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage

TPTU evaluation results for partial tool-subtask pairing planning

Paper: https://arxiv.org/abs/2308.03427

Problem addressed: During agent construction, the inherent generation capabilities of large models struggle to directly handle complex tasks. How to effectively improve agent task planning and tool usage became a key research direction. The team designed the standard TPTU framework, expanding the scope of task decomposition planning and tool invocation, providing a structured AI agent framework.

Model architecture: TPTU designs a task-planning agent (TPTU-OA) and a task-execution agent (TPTU-SA). The former effectively decomposes problems into a series of subtasks within a single instance; the latter enables the model to maintain clear, focused attention throughout the problem-solving process, tackling issues step by step.

Results: The team provided essential evaluation datasets, various external and internal API tools, and foundation large models. They assessed tasks including tool ordering, task planning, and subtask pairing (with both relevant and irrelevant tools) across different model bases, while also evaluating effectiveness on tool usage, complex nested SQL creation, and mathematical code generation. Ultimately, on ChatGPT and Claude models, multiple tool-use effectiveness metrics reached 85%-100%.

Applications: General-purpose agent task planning and tool-use frameworks.

TPTU effectiveness in SQL and mathematical code generation

---

Oasis Capital: Could you briefly introduce your background and research experience?

Mao: From 2015 to 2020, during my PhD at Peking University, I mainly worked on multi-agent reinforcement learning. In my fourth year, I first encountered using reinforcement learning to train GPT models — they weren't called large language models yet. We worked on model-based task-oriented dialogue settings, like taxi-hailing and hospital appointment tasks. Later I shifted attention to natural language processing and started working on NLP agents. That same year, Transformer exploded in popularity, and eventually OpenAI launched ChatGPT based on reinforcement learning. So from my fourth year onward, part of my research core shifted to NLP agents.

Oasis Capital: What changes have large models undergone from traditional NLP agents to LLM-based agents?

Mao: The shift from deep learning to Transformer-era NLP agents was stunning. At the time, many online NPC games were built on deep learning agents — looking back, those were very shallow networks. Basic networks could control virtual agents to complete various mini-games in games, and AlphaGo defeating world champions operated on the same principle. But we found that deep learning-era agents were hard to generalize: one agent could only solve one class of problems, and many execution details were genuinely difficult to handle.

With large-model-era agents, the powerful generalization capabilities of underlying GPT models enable what's called world model understanding in scenarios. Ordinary agents can now complete a vast range of tasks. This felt to me like the DQN (Deep Q-Network) boom of 2015. Later, based on world models, users could solve all kinds of problems with just simple prompts. I immediately pivoted to researching agents in the NLP domain — I believed this represented the future and more promising agent development.

Oasis Capital: In multi-agent reinforcement learning research, what fundamental problems do you think MARL essentially solves?

Mao: What MARL fundamentally solves is very hard to define. MARL is an enormous framework, mainly addressing interaction problems between agents, involving cooperative, competitive, and mixed cooperative-competitive scenarios. The reinforcement learning I currently study can be divided into traditional Markov Decision Processes (MDP) and Decentralized MDP (Dec-MDP) — the latter involves multiple agents in a modeling approach that is both decentralized and allows partial observability. Solving agent systems is extremely complex, so when tackling multi-agent interaction scenarios, I choose cooperative settings where agent task objectives are more aligned.

Citing MARL pioneer V. Lesser's research: MARL — Many Problem Formulations

Oasis Capital: Could you explain why the TPTU architecture focused on planning and tool usage?

Mao: The process was quite interesting. OpenAI had done extensive groundwork, with one representative example being their March 2023 release of Plugins. The core was tool invocation — they kept their own Plugins, including Code Interpreter, external search, and so on. We added three more general-purpose tools, including Python and language execution. At that point we already had the concept: we knew that future agent tool usage would likely take the form of API calls, and later published a paper on this. Additionally, we had many real internal working systems where numerous key APIs could serve as fine-grained callable tools.

The core reason for task planning was the reinforcement learning perspective: solving real problems requires multi-step decision-making, and it's hard to resolve everything in one step. We also benchmarked against OpenAI's official approach, using natural language descriptions to decompose sub-problems, and eventually formed our task planning pipeline.

Oasis Capital: Could you elaborate on the design thinking behind the One-Step Agent and Sequential Agent frameworks?

Mao: The name One-Step Agent was actually inaccurate — it quickly generates sub-problems one, two, three, so One-Shot Agent would have been better. After subtask decomposition, the first step only does generation. For the second sub-problem, the answer from the first sub-problem is also fed in, so it references both the user's original question and answer, and can select better sub-tools to solve the second problem. This allows all previous experience to be learned into the new task step resolution, enabling better next-step sub-problem tool invocation. The implementation used in-context learning via prompting.

Oasis Capital: What are the current weaknesses in agent planning and tool usage capabilities, and how might they be improved in the future?

Mao: We already have some improvements in TPTU v2. First, we believe that after the TPTU architecture gives agents basic planning and tool invocation capabilities, we can further enhance their abilities. One, we solved how to recall from multiple tools — a component called API Retriever. Two, during the API screening process, we improved the agent's semantic description capability, enabling more accurate API calls after a task occurs. Three, we provided case-based learning for agents, helping them better solve unknown tasks.

In the API Retriever tool, we actually faced numerous practical challenges: how to generate training datasets, how to accelerate training, how to reduce costs and improve efficiency at small scale. In the initial tool screening phase, API Retriever is highly efficient.

Oasis Capital: In which application spaces can TPTU ultimately be better deployed?

Mao: Application spaces don't necessarily have to be specific scenarios like security or government affairs. Internal enterprise use may be particularly common — for example, within a B-end system that already has many APIs, if you want to invoke the 10% most commonly used ones, this is especially suitable for large language models. On the consumer side, personal assistants can also be deployed at scale.

Oasis Capital: As foundation model capabilities continuously strengthen, what capability improvements does this bring to agents?

Mao: I believe the core need is to strengthen agents' underlying capabilities. My experience is that fine-tuning on vertical domains is necessary — general-purpose large language models struggle to understand business logic. Currently, the international community generally believes in general AGI, while the domestic community generally believes in vertical AGI. I want to build vertical AGI, because large language models' general capabilities are still not strong enough. Considering PMF factors, I want to constrain application capabilities within the boundary of language model capabilities.

Oasis Capital: Having studied reinforcement learning for so many years, how do you view the current development of PPO and DPO?

Mao: Reinforcement learning itself is already quite old. PPO and its twin TRPO are 2015 and 2017 work respectively. RL technology development is actually very slow. In the large model reinforcement learning domain, we're also exploring whether PPO-based online or on-policy fine-tuning of large language models yields the best results. DPO optimizes given a dataset, with reward models and policies optimized simultaneously — it's actually less like reinforcement learning optimization and more like SFT supervised fine-tuning. Currently, closed-source fine-tuning uses PPO, while open-source uses DPO.

However, people currently believe that for DPO to achieve PPO's optimization effects, it must imitate PPO's on-policy nature — that is, it must achieve PPO's online learning or interactive policy to improve performance. This is the same as human learning: humans constantly fine-tune based on current situations, rather than learning from situations from some time ago.

Oasis Capital: Could you explain the principle by which large models' inherent defects can also serve as useful functions?

Mao: In serious scenarios for STEM fields, hallucination is a major pain point. But for humanities and arts — when we need imagination, generating multimodal content, novels, images — hallucination is actually a considerable advantage, enabling richer imaginative output.

Oasis Capital: Could you briefly introduce your work on large-scale agent applications?

Mao: The most important tasks agents can solve are decision-making and reasoning tasks, or tasks with reflective capabilities. The real world of the future will consist of multi-agent systems. Large-scale multi-agent research, without considering any costs, is a highly promising research direction. Additionally, multi-agent reasoning speed is currently a problem. For example, exploration and exploitation in reinforcement learning are completely different during generation — many agents' exploration and exploitation costs are enormous. How to reduce these costs involves foundational framework design work. We created a centralized critic approach: as the number of critics decreases, one handles exploration and one handles exploitation, better solving individual problems. Finally, balancing exploration and exploitation is also crucial — we added a third agent, ultimately summarizing exploration and exploitation as the interaction of three agents. This provides a dimension of thinking, expressing how we conceptualize large-scale agent and MARL implementation.