Zhejiang University ReLER Lab Researcher Zongxin Yang: Doraemon's Pocket | Agent Insights

Sora has showcased the "world simulator" and intelligent machines' understanding of the human spatiotemporal physical world through video. It's now being integrated into Adobe's video editing software, representing undeniably impressive progress on the generative AI application front. Yet while it can simulate video content that aligns with human perception, it still lacks a deep understanding of physical laws. Zongxin Yang, a researcher at Zhejiang University's ReLER Lab, has designed an Agent called DoraemonGPT based on the idea of large-small model collaboration. This system significantly enhances an Agent's ability to understand dynamic video, suggesting that collaborative complementarity between large and small models can help alleviate the logical disorganization that plagues large models. Enjoy.

DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models

Problem: Real-world environments are inherently dynamic and ever-changing. Previous LLM-based Agents have focused primarily on static image tasks, limiting large models' ability to understand real-world dynamics. DoraemonGPT leverages large-small model collaboration to substantially improve dynamic video task-solving capabilities.

Architecture: DoraemonGPT uses GPT-3.5 Turbo as its foundation. Through VisProg, it decomposes multimodal tasks, extracts task-relevant symbolic memory, and integrates external knowledge via plug-and-play tools to solve problems in a structured, step-by-step manner. Monte Carlo Tree Search breaks down massive dynamic video tasks into a tree structure, cycling through node selection, branch expansion, chain execution, and reward backpropagation on video test sets to find optimal solutions.

Performance: DoraemonGPT achieves 5-10% higher task success rates than ViperGPT and VideoChat, demonstrating strong results in dynamic video task solving.

Applications: Real-world dynamic video task solving.

DoraemonGPT problem-solving workflow

Results: On causal reasoning questions, DoraemonGPT surpasses the previous SOTA, MIST. Across four question types, it outperforms ViperGPT by 11.5/9.4/8.0/5.3 (AccC/AccT/AccD/Avg) and VideoChat by 4.5/3.4/4.6/2.2. On video object segmentation, DoraemonGPT's zero-shot performance exceeds the previous fully supervised SOTA, OnlineRefer.

DoraemonGPT vs. traditional training frameworks: video understanding accuracy

DoraemonGPT vs. traditional training frameworks: video object segmentation accuracy

---

Oasis Capital: Could you share your research journey?

Yang: I started with dynamic video research, working on video segmentation — pixel-level understanding of objects in video. I then expanded into multimodal understanding, including interactive and instance-level understanding. Last year, we extended Meta's visual segmentation large model SAM (Segment Anything Model) to video, releasing the Segment-and-Track-Anything (SAM-Track) model. It received over 2,000 stars in short order, and we followed up with a technical report.

Oasis Capital: What motivated DoraemonGPT? And why the name?

Yang: Our earlier SAM-Track was essentially a hand-designed video processing system — manually integrating multiple models so the system could tackle video tasks that single models struggled with. Could we build an automated system that doesn't require hand-crafting multi-model integration, but instead automatically orchestrates various foundation models? That led us to explore an Agent approach that automatically dispatches foundation models to solve complex video and multimodal problems.

As for the name — building an Agent requires the concept of large-small model collaboration. The large model acts as the brain, scheduling tools to complete tasks, much like Doraemon pulling out various gadgets and magical items from his pocket to solve problems. So we thought the name fit quite well (laughs).

Oasis Capital: You've consistently worked on video. What do you think of Sora, which has pushed video generation technology to new heights?

Yang: Sora's demos are indeed impressive. Our team — including Professor Yi Yang (Associate Dean of Zhejiang University's School of Computer Science) and other colleagues — also published commentary on Sora in Science and Technology Daily. We believe Sora extends the success of image generation. Its ceiling is producing very high-quality video. But looking at OpenAI's released failure cases — for instance, a video of a glass shattering where the water splash incorrectly occurs before the glass breaks — this shows Sora cannot properly generate changes in physical properties.

The bottleneck is clear: Sora doesn't truly understand physical laws. It lacks real-world knowledge to support what it generates, leading to counterintuitive results. Its robustness remains uncertain. Like image AIGC generation, it's prone to structural inconsistencies in hands or limbs. While this pain point is gradually easing, no complete solution has emerged.

Oasis Capital: Are there structural similarities between DoraemonGPT and Sora?

Yang: There are similarities, but they're quite different. DoraemonGPT is essentially an Agent built on top of a large language model (LLM), operating in a token generation format. Natural language is compressed into tokens, then generated autoregressively via Transformer. Sora also compresses video information into tokens, but doesn't use autoregressive generation — it likely uses Transformer to generate all tokens in parallel across spacetime. Moreover, Sora is a diffusion-based video generation method, starting from random noise and progressively denoising. The key difference: LLMs are autoregressive, while Sora uses no causal attention in its Transformer — it's purely parallel generation, and multi-step generation based on diffusion. From OpenAI's technical report, all tokens in the entire video are generated simultaneously, with positional encoding providing spatiotemporal relationships.

Of course, replicating Sora is extremely challenging. With sufficient compute, pure parallel generation is definitely more efficient than autoregressive token-by-token generation. But the downside is much higher GPU memory requirements. The biggest pain point in replication is still data. The open-source community has extensively analyzed the model architecture, with general consensus on implementation approaches, but large-scale high-quality data, massive compute, and the engineering execution of large-scale model training are the core challenges. Data, compute, and engineering experience — these are OpenAI's real moats.

Oasis Capital: Could you explain DoraemonGPT's architecture?

Yang: DoraemonGPT has a tripartite structure: Planner, Tools, and Memory. The Planner is like the Agent's brain — it plans tasks and selects tools to complete them. Tools in the toolkit adapt to different scenarios. Memory includes goals, tasks, and readily available information. To put it simply: for solving complex math problems, the person doing the math is the Planner, responsible for planning and applying tools and textbook knowledge to solve it; the calculator, compass, and paper are the tools; the current problem, external book information, and what's in our heads constitute external memory.

We believe large-small model collaboration will become the dominant paradigm. Large models are general but lack specialization; professional scenarios typically use vertical small models. But small models lack robustness, so we naturally thought about combining general large models with various specialized small models (or domain knowledge). DoraemonGPT is exactly this kind of collaborative system. Its Agent architecture has three components: first, the large model; second, numerous robust small models for specific tasks; and third, external knowledge bases or knowledge graphs to compensate for specialized capabilities. If these three can be organically integrated, we can gradually move toward artificial general intelligence.

Oasis Capital: We noticed DoraemonGPT's innovative temporal and spatial annotations. Could you explain how these work?

Yang: DoraemonGPT focuses on dynamic scenes or video, which contains many frames. Take filming a street scene — numerous trees, cars, and people, with people performing different actions. The information volume is enormous. From a task perspective: an automated surveillance camera cares more about pedestrian trajectories, specifically whether someone might suddenly rush in front of a car, creating danger. The information actually relevant to the task is far less than the total input from the video and multimodal data. So extracting all video information uniformly and processing it through a large language model or Planner would mean too much information, too costly, too much redundancy, and inefficient implementation.

Our approach is to provide relevant information based on the current task, then process it — efficient and robust. From the video itself, the two most critical dimensions are time and space. DoraemonGPT is designed to categorize information into temporal and spatial types, using different large models to extract relevant information. In this extraction process, a sub-model of the large language model — a small Agent — takes the user question, then the Agent determines whether the task requires more temporal or spatial information.

Oasis Capital: How does DoraemonGPT improve upon traditional video task processing Agents?

Yang: The improvements are multifaceted. First, efficiency. Previously, video information was extracted uniformly and fed directly into the large model context, or placed in unified tables and databases, with control following unit model scheduling and solving. Our approach uses the large model's capability to reduce extracted information to some degree, avoiding unified information extraction and filtering only relevant information.

On the other hand, when processing video-extracted information, we break down sub-tasks so that processing doesn't affect DoraemonGPT's main Agent context. The sub-task Agent and external main task Agent have some isolation in context, which to some extent reduces context length during large language model scheduling, and therefore reduces cost.

Oasis Capital: What are your AI priorities this year?

Yang: Among AIGC-related directions in academia, I'm currently optimistic about two research areas. One is Agent — still in its early stages. First, it lacks unified definition standards and evaluation methods, meaning a framework that can convince everyone that large model + Agent is an important path toward AGI. The other is video generation — the follow-up to Sora. There's already substantial open-source community work pushing this forward, and I hope to see good work emerge from China's open-source community to drive development.