Oasis Capital in Conversation with Professor Wen Ying: Cooperation Is Harder Than Competition

Can large language models accelerate value alignment and effective collaboration between agents, and between agents and humans?

We invited Professor Ying Wen from the John Hopcroft Center for Computer Science at Shanghai Jiao Tong University to share his insights on multi-agent deep learning. Enjoy.

Oasis Capital: Please introduce your research direction and latest progress.

Professor Wen: My research focuses on multi-agent learning, using deep reinforcement learning algorithms to solve game-theoretic equilibria across different types of games — cooperative, competitive, and mixed — with the goal of learning strategies that can compete or cooperate effectively with other agents or humans.

Multi-agent reinforcement learning (MARL) only really took off after AlphaGo went viral in 2016. Before that, it was limited to simple tabular scenarios and couldn't tackle real-world or complex problems. Post-2016, MARL made huge strides, exemplified by AlphaGo, Go, StarCraft, and Dota — all competitive games. Over the past few years, MARL has solved increasingly complex competitive game problems, starting from Go where both sides have complete observability, to StarCraft with its complex state-action spaces and partial observability addressed through league training.

Games like Dota and Honor of Kings rely on population-based learning (similar to league training) — given enough training, strong strategies emerge. A few years back, Libratus made impressive progress on two-player Texas Hold'em, but N-player poker remains an open problem. Further research is needed on multi-player games and generalized reward design. Last year, Meta experimented with Diplomacy, where supporting more players required representing the entire action and state spaces through more complex natural language descriptions. Our group has done a series of works on population-based training methods over the past two years. We've made good progress on complex two-player zero-sum games, and next we hope to extend to more complex multi-player games.

Two-player zero-sum games are easier to benchmark — Go has rank levels, games have scores, making it straightforward to find evenly matched opponents for training. Since 2016, multi-agent deep reinforcement learning has incorporated communication mechanisms or learned better cooperative strategies through centralized training with decentralized execution. Our previous work on cooperation assumed fixed teammates, focusing on how to coordinate well within that team. But this setup doesn't generalize. Think of it this way: four or five dorm mates who play together daily may have high scores, but that doesn't mean any individual has strong cooperative skills. In reality, ranked matches require the ability to coordinate with random teammates — this demands ad-hoc play. Cooperation is more complex than competition. We aim to design MARL algorithms that learn strategies during training to cooperate with unknown partners, and extend this further — training strategies that can achieve better coordination with both agents and humans.

Stanford's "Generative Agents" used large language models to create 25 agents with distinct personalities, leveraging LLMs' ability to simulate human behavior, combined with memory and reflection mechanisms, to conduct simulation studies and evaluations across economic, sociological, and public policy scenarios. As LLMs develop rapidly, our group is also exploring how multi-agent settings might integrate with them. Our recent direction involves placing LLMs in multi-agent system scenarios for simulation and evaluation. For LLM-based agents, beyond simulation, we're adding learning mechanisms — can they improve LLM capabilities through socialized interaction, enabling more human-like interaction with AI or humans, and aligning with human values?

Oasis Capital: StarCraft and AlphaGo were also important early projects for OpenAI. From a technical route and methodology perspective, these differ from the "brute force" large language models OpenAI later championed. Has the development of LLMs itself informed or helped your research direction?

Professor Wen: I'd say it has been quite informative and helpful. Whether single-agent or multi-agent, the deep RL field has long sought to escape two problems: sample efficiency and generalization.

For sample efficiency, the earliest approaches were on-policy and off-policy methods. On-policy requires constantly sampling new data with the current policy to improve the model, which is highly inefficient, so off-policy was adopted — using experience trajectories sampled by other policies to train and improve the policy. But efficiency was still too low. So then pre-trained policies learned from offline data were deployed in online environments. This "offline pre-training" approach is quite similar to the LLM paradigm. LLMs are essentially offline pre-trained on high-quality corpora; RL also needs high-quality offline data for reinforcement.

LLMs have also helped at the network architecture level. Language models use Transformer architectures, and RL policy networks or value networks have begun adopting new Transformer architectures, gaining stronger modeling capabilities. Tasks previously handled by NLP or LSTM saw significant efficiency gains after switching to Transformers.

As for generalization: decision-making tasks are also sequences. With Transformer networks, we can draw on GPT language models to perform unified autoregressive sequence modeling for decision tasks. Specifically, observing a state at the current moment, taking an action, and transitioning to the next state forms a sequence structure. Different modalities of observations and actions can be directly input as sequences, and next-step observations, actions, or rewards predicted based on historical information. Over the past two years, much research has attempted to transform multi-agent RL into sequence modeling, borrowing from the language model approach. For example, given a state, predict the next action; or given the history of state-action outcomes, predict the next state — essentially building "world models."

Additionally, to push RL's generalization toward LLM-level strength, there's the scaling-up step. Our group has done some work here — last year we attempted to reproduce Gato, releasing the DB1 model, and more recently TDM, a dynamics model. We've been exploring offline data pre-training with Transformer architectures for sequence modeling. Specifically, after scaling up the model, we incorporate more types and quantities of tasks (DB1 had five categories, over 870 tasks), tokenize them uniformly, and feed them into a Transformer model for direct sequence modeling.

It's fair to say LLMs have inspired many research directions, and we may eventually converge on similar paths. Recently there's been a trend toward grand unification — CV and speech are also adopting Transformers, collecting more data, doing offline pre-training, and improving data sample utilization efficiency and generalization performance.

Oasis Capital: With this unification trend and cross-pollination of research, which previously hard-to-deploy applications might now become feasible?

Professor Wen: Honestly, multi-agent RL is very hard to deploy (laughs); single-agent is easier. Previous learning-based methods mainly addressed adaptability or generalization, but results remained lab-bound. Through Transformers or pre-training, policy algorithms and policy network learning capabilities have become quite strong. Berkeley's mobile robotics applications based on LLM-learned low-level control policies — our group has done similar experiments with good results. Now that large amounts of quality data can be collected, policies for robots, robotic arms, or robot dogs in more complex scenarios may achieve commercial deployment.

Moreover, LLMs have powerful text capabilities — they can understand natural language instructions and decompose complex tasks. This can be combined with simple-task control models to accomplish complex tasks. Language conditional / instruction following policies for control models — fetching a glass of water, grabbing an apple — these are easily learned. I'm quite interested in this direction. Last year's SayCan, and more recently TidyBot, PaLM-E are all works combining these approaches, enabling robots to execute tasks with low operational precision and low safety requirements in indoor environments like kitchens, homes, and offices, with decent generalization capabilities and potential for personalized service.

Oasis Capital: Why has multi-agent deployment been so limited?

Professor Wen: The difference between multi-agent and single-agent is that single-agent has a clear task objective — the ultimate goal is to maximize long-term discounted returns under a fixed goal through algorithmic learning.

Multi-agent introduces agent interaction on top of this. Beyond differing objectives, agents influence each other; there's no clear learning target, your payoff depends on others — the setup is obviously far more complex than single-agent. It's not just a tooling issue; social dynamic scenarios involve far greater complexity. You need to find specific points of integration for deployment.

Multi-agent is more a modeling tool or approach to combine with specific problems, seeing whether existing algorithms can solve them, while also involving the question of what the objective is. Multi-agent performs well in board games and game AI, but whether it's replicable in other real-world scenarios requires case-by-case analysis. If a multi-agent task can be completed by a centralized single-agent, why introduce additional complexity?

Oasis Capital: Are games currently the most direct deployment scenario for multi-agent?

Professor Wen: Games already have many deployed applications, but they're costly with limited commercial value. Training a strong AI bot can assist game R&D or design, but has limited impact on overall game revenue — at best a sparring partner, or a temporary AI bot teammate when someone disconnects. Though a few game AI companies like Qiyuan, Chao Can Shu, and Tencent Wukong do exploratory work, real-world impact and commercial value are limited.

I'm more interested in multi-agent with human modeling. Currently, agent or AI algorithm deployment is increasing. Many future tasks will require human-AI collaboration, requiring the development of AI that enables mutual understanding between humans and agents — this is a fascinating future direction.

Oasis Capital: If drones, robot dogs, and others share a unified goal for large-scale autonomous coordination, would that count as a clear deployment scenario for multi-agent?

Professor Wen: Drone swarm control algorithms already perform well in many real-world scenarios — formation flying, obstacle navigation, etc. In these scenarios, drones still function as tools. For specific tasks, inter-agent interaction is relatively weak. With pre-planned coordination, directly applying multi-agent RL may not yield good results.

Oasis Capital: Since multi-agent deployment is so difficult, is this research direction still indispensable?

Professor Wen: Beyond game deployment, it still has significant research value. AI-human collaboration is itself a multi-agent problem. The AI value alignment discussion today can be modeled as a multi-agent problem. Recent research has attempted socialized interaction between AI and AI, treating it as continuous interaction between different agents to ultimately achieve value alignment with humans.

There are many other problems suitable for multi-agent modeling. For example, border patrol and smuggling prevention scenarios have been modeled as security games with proven effectiveness.

Oasis Capital: Traditional robotics backgrounds emphasize control algorithms, while your research uses learning-based algorithms. What's your view on the debate between these camps? Are they difficult to integrate?

Professor Wen: Control-based algorithms certainly outperform learning-based ones on specific tasks. For high-precision, repetitive fixed production processes, control algorithms directly suffice — no need for strong adaptability or generalization. Learning-based approaches hope to perform better in non-standard scenarios.

For example, indoor wheeled robot movement algorithms often don't need learning-based methods at all — direct low-level control algorithms handle movement fine. But for bipedal or quadrupedal bases, or bases with robotic arms grasping non-standardized objects, learning-based algorithms are needed for better adaptability. This also includes unexpected situations — dropping something, encountering obstacles — where control algorithms have limitations.

These aren't in conflict; control algorithms can serve as low-level skill modules. Each has its strengths and limitations; it mainly depends on scenario and requirements. They can absolutely complement and combine with each other.

Oasis Capital: Are there fields or branches in this wave that you particularly follow or find interesting?

Professor Wen: Multi-agent RL involves perception, decision control, game theory, etc. — meaning you need to understand all directions of AI.

Language models are humanity's invented abstract description of the world. Whether we can, on the basis of multimodal, pre-trained large models, fuse interactive knowledge to further improve multi-agent performance is our focus.

Oasis Capital: Some research fields feel their direction has disappeared due to this wave, but multi-agent learning has gained more attention and momentum — is that accurate?

Professor Wen: Language models are excellent prior models. Previously we worked on single tasks, and learned strategies couldn't map to a unified space. What you learned on Atari was useless for robotic arm control. The prior knowledge LLMs bring, beyond grounding other modalities' static data/knowledge to language, can also ground dynamic actions like shooting goals or basketball shots to language space, achieving continuous generalization.

Oasis Capital: What do you see as the biggest challenge in human-agent interaction?

Professor Wen: The first challenge is the difficulty of properly evaluating cooperation capability. This is also an open problem — there's no quantitative standard or evaluation metric for cooperation skill level, no established answer yet, and it's one of the problems we're exploring.

Second, value alignment. Some scenarios require human leadership with agents cooperating; others require agent leadership with humans providing information when difficulties arise to complete final cooperation. Currently it's mostly one-way alignment; the ultimate goal is two-way alignment.

Oasis Capital: This LLM wave has been called the "Fourth Industrial Revolution." The original industrial revolution replaced much labor; will AI development replace more knowledge workers?

Professor Wen: I'd say it replaces repetitive tasks within knowledge work; tasks requiring high logical thinking can only be assisted. For example, formatting today's interview into structured text — LLMs excel at this. But asking targeted questions about content, progress, or interview purpose requires deep industry and technical understanding, which AI cannot do.

The revolution in human-computer interaction is one core aspect of this wave. On this foundation, for simple and repetitive tasks, LLMs can combine parametric stored knowledge with simple logical reasoning for good responses. LLMs have also learned programming languages — they can write code, call APIs, connecting human natural language with machine programming language, becoming tool-augmented language models. Like a central brain, a more intelligent operating system that directly understands human natural language instructions, connects all needed tool calls, and completes more personalized tasks.

Oasis Capital: At the BAAI Conference, "Empowered by AI, don't be overpowered by AI" was mentioned. Does academia feel humans might be overpowered by AI?

Professor Wen: I don't think so at all (laughs). Initially everyone was indeed shocked, but after using it more, you realize it's just like this — not as powerful as imagined. AI itself plays to its strengths, assisting humans to complete tasks better. Currently, whether robots or language models, what's being solved are simple and repetitive tasks, liberating human labor in these areas and accelerating work automation. People may think it will replace many jobs, but I believe it will actually create more new jobs.

From the LLM development speed perspective, the core is still big data, big compute, big models. The subtle thing about language models is that text data is easy to obtain and process, with high information density. In humanity's informatization process, massive amounts of electronic text data have accumulated on the internet, easily collected and trained. LLMs expend 20% effort to complete 80% of work — everyone is amazed. But from 80% to 90%, to 100%, requires more modalities of data and interactive trajectory data, with increasingly difficult collection. Though progress continues, the speed definitely won't match the first two years.

Oasis Capital: Many people still believe AGI will arrive within 5 years. What's your view?

Professor Wen: I think the gap is still large. Take recently hot AutoGPT — the concept is good, breaking down tasks and planning, but execution remains the hard part. AutoGPT still mainly uses natural language for input and output, then recursively solves sub-problems one by one, entirely dependent on the LLM's own choices and decisions, while relying on external text storage for "memory." It's a good demo, but its success rate for solving real-world problems/needs is not high. There are many issues — for example, language boundaries are too open; completing specific tasks requires strong boundary constraints, generating precise instructions machines can execute. While this can be compensated through tools or APIs, landing on specific tasks requires writing adapters one by one, with significant engineering implementation details in between, to truly produce practical effect and value. Of course, there are other challenges like task decomposition, execution, reflection, trial-and-error, and memory mechanisms, all needing gradual resolution, expanding LLM capability boundaries step by step, to gradually approach "AGI."

Vitality

What do you think is technological vitality?

My vision of AGI is to enable machine intelligence, on the basis of pre-training, to further engage in continuous interaction with the environment, other AI, and humans for self-improvement, expanding capability boundaries. And my understanding of technological vitality is similar to AGI, emphasizing the plasticity and continuous evolutionary capacity of technology, enabling it to adapt and respond to ever-changing environments and needs.

— Professor Ying Wen John Hopcroft Center for Computer Science, Shanghai Jiao Tong University

Oasis Capital is a new-generation venture capital firm in China, dedicated to discovering the most vital entrepreneurs of the next decade and growing alongside them to create long-term value. "Vitality" is Oasis's vision and mission. This vitality represents both the direction of structural transformation in the era and the resilience and evolutionary power of entrepreneurs.

Oasis Capital focuses on early and growth-stage investments, with individual investments ranging from $3 million to $30 million, concentrating on robotics, artificial intelligence, technology services, and other fields, empowering China's technology-driven new service upgrade.