Oasis Capital in Conversation with Professor Yuan Li: Preserving the Spark Before the Next Transformation

From language to vision — how long is the road ahead, and where will the next great model revolution come from?

Today we're sharing our conversation with Yuan Li, assistant professor and PhD advisor at Peking University's School of Information Engineering. Enjoy.

Oasis Capital: What are your main research areas and focus?

Professor Yuan: My research centers on computer vision and multimodal machine learning. During my PhD, I originally worked on pure computer vision, and around that time I noticed a major shift happening in the field. What struck me was that as models grew larger, the more significant trend wasn't just increasing data volume — it was the move from single-modal to cross-modal and broader multimodal approaches, and the transition from supervised to unsupervised learning paradigms.

So in the latter half of my PhD, one of my key research directions became finding a unified model that could perform well across vision, NLP, and different data modalities. After extensive experiments, I found that CV models weren't well-suited for NLP, but NLP's Transformer architecture could potentially handle vision tasks. So I shifted my focus to applying Transformers to vision.

The fact that Transformers could work in vision was a landmark development. Since it could be applied to both CV and NLP, it became an excellent foundation for multimodal tasks. My current research focuses on designing foundational frameworks for multimodal tasks — moving from two modalities to many more. The "Zidong Taichu" three-modal pre-trained model developed by the Chinese Academy of Sciences' Institute of Automation is an example of a three-modal approach — they're pushing into three modalities, which is currently about the maximum. Mainstream models today are still primarily vision-text dual-modal, like text-to-image generation, image-text mutual generation, and image-text retrieval. The major trend going forward, including my own current research, is building datasets that expand from two modalities to many more. We want not just larger data volumes, but more modalities — each modality carries different implicit information, and we're exploring whether models can learn from more modalities and generalize better.

Oasis Capital: Can vision and NLP actually converge in the development of large models?

Professor Yuan: This was already the major trend during my PhD: moving from different models toward a single unified model. They've already converged. Everyone knows LLMs are performing very well now, and large vision models are also emerging. Of course, GPT-4 is technically a multimodal model, but its publicly available interface is single-modal — language only, no vision. When a truly usable multimodal large model does emerge, it will inevitably unlock far richer possibilities.

Oasis Capital: Why hasn't a true multimodal large model been built yet?

Professor Yuan: Multimodal tasks are harder than single-modal ones, and the data is scarcer. Collecting data for single-modal tasks is relatively straightforward. For multimodal data, you have to consider whether the modalities should align, which involves more complex algorithm design, plus there are cost issues. Right now we have plenty of text and vision data, but data for other single modalities is still severely insufficient.

Oasis Capital: What was your long-term thinking behind developing ChatExcel?

Professor Yuan: ChatExcel takes dual-modal input — one modality is text, the other is spreadsheets. People might not think of spreadsheets as a modality, but they're actually more like graphs, a special kind of modality. We feed both modalities into the model and have it generate a new chart. ChatExcel was just a starting point, but our focus isn't on office software — that space is already highly competitive with limited room for imagination. We see greater opportunity in more diverse scenarios and tasks, particularly vision-centric multimodal applications. Our direction differs from text-to-image generation; while that's about generation, we're focused on using language and voice to perform fine-grained editing and modification of visual modalities — that has broader imaginative potential.

Oasis Capital: When Meta released SAM (Segment Anything), many claimed "CV is dead." What's your take?

Professor Yuan: SAM is definitely a breakthrough, but I don't think CV ceases to exist because of it. For SAM, if your image resolution is low or the objects are small, it can't actually "segment anything." Computer vision is a vast field, and segmentation is just one task. SAM has pushed segmentation toward a certain extreme — the results are genuinely impressive, and it's widely recognized as the SOTA model in segmentation. Testing it on difficult scenarios like dense scenes reveals its limitations, but we must acknowledge its contribution. SAM can serve as a foundation model for the vision field.

At the application level, SAM can certainly spawn many derivative applications. More importantly, it can function as a base model — providing feature extraction for other tasks, or using segmentation itself as a foundational task to provide feature extractors for higher-level tasks. For instance, if a task requires segmenting every person and object before proceeding to downstream tasks, you can use SAM for extraction and then apply it downstream to pose estimation, object relationship recognition, and so on. That's the meaning of a foundation model — not that it solves all vision problems.

But beyond 2D vision, there's 3D vision. Vision encompasses far more than segmentation, and if it evolves further toward multimodality, the challenges multiply.

Oasis Capital: What would count as a real breakthrough in vision models?

Professor Yuan: Moving from 2D to 3D would be one breakthrough. There are still many unsolved problems in 2D. Using SAM, for example, you can optimize image captioning through segmentation. But describing relationships between objects isn't solved by segmentation alone. Classification, detection, and segmentation are the three foundational tasks of vision. If a model performs well on foundational tasks, it can serve as a base model for more applications. SAM itself can't do image captioning or pose estimation, but it can find people and indicate their locations — completing the most basic task.

Why did NLP large models emerge first? Why haven't vision large models created the same splash or delivered comparable impact? Because vision tasks inherently have higher data dimensionality and are more difficult. This is why the GPT series is more text-based with limited image capabilities. At least before GPT-3.5, they were single-modal pure language models. GPT-4's demos showed visual input capabilities — it could do image captioning, VQA (Visual Question Answering) — but the specific capabilities weren't made available to the public, so its actual abilities remain unknown.

Oasis Capital: What are the main application scenarios for computer vision in China? Which direction will develop faster?

Professor Yuan: Internet-related applications will develop faster because internet products have longer reach and broader scope. Autonomous driving is probably the most valuable application domain. Tesla has been pushing a pure vision solution — why pure vision? Because it's the cheapest. Cameras as hardware have the lowest cost. If a true large vision model does emerge, many fields will flourish or even be disrupted. Take the autonomous driving industry: first, hardware costs would drop; second, if vision tasks could be pushed to their limits with intelligent vision, autonomous driving would reach another level.

Tesla's FSD (Full Self-Drive) costs tens of thousands extra to add to a vehicle, and it works better in the United States than in China because the two countries differ significantly in traffic scenarios, architectural styles, and natural landscapes. Tesla trains on US data, so it's naturally more adapted to American scenarios and performs better there — in China it's a different domain. "Domain adaptation" has long been an unsolved problem in vision, referring to cross-domain recognition capability. If a vision large model truly comparable to ChatGPT emerged, the market opportunity would be even larger than what's currently available in NLP, and might help enable the transition to true fully autonomous driving.

Oasis Capital: Where does China stand globally in computer vision development?

Professor Yuan: The current state of AI broadly is essentially a competition between China and the United States. NLP development levels are very close; on the vision side, when SAM was released, BAAI in China published SegGPT around the same time — the work was very similar, SegGPT just came out a few days later. This shows China is keeping pace in large model development. Currently, we're less constrained by algorithms than by the ban on high-end GPUs — it's a hardware and compute bottleneck.

Oasis Capital: Are there major differences in computer vision applications between China and the US?

Professor Yuan: I think they're fairly similar, especially in autonomous driving — at least both China and the US are targeting this field. Other domains are largely comparable across countries; fundamentally, I don't think there are major differences.

Oasis Capital: In vision, are there any "simple methods" to judge whether a new paper or approach is reliable?

Professor Yuan: The simplest way is to look at whether the research group is reputable, whether they've done good open-source work in the past, and whether they've had any academic misconduct. You can check Papers With Code to see how many papers have reproduced their code — more generally means more reliable. GitHub stars and fork counts are also quantitative indicators, but these are just metrics. More substantive judgment requires domain expertise.

Oasis Capital: There's been recent market discussion about large models hitting a ceiling. What's your view?

Professor Yuan: This is just a minor correction, and not a severe one — it's an inevitable phase in the development of any new technology, a slight pullback after reaching a peak. This wave of ChatGPT has given AI a new lease on life; for practitioners and researchers, it's a shot of adrenaline. People in this field haven't been this excited in a long time, which is good for the industry overall. Researchers just need to consider whether the research paradigm itself needs to change — that's what we need to think deeply about.

Oasis Capital: Where will the next major transformation in large models come from?

Professor Yuan: No one can predict with certainty. Large models will continue developing, but we also need to consider their capabilities — if they plateau quickly, we'll be limited to applications and incremental innovations on top of them. Where will the next big transformation come from? Every AI practitioner and researcher needs to think about this. Some predict that in a few years, internet corpora won't be sufficient for training large models — will insufficient training data affect language model development? There isn't a true vision large model yet, so we still have high expectations; large models are far from reaching their limit, and we need to consider whether there are alternative paths forward. For instance, training language large models already requires massive compute — training vision large models might require exponentially more. Can current compute keep up? Concentrating all the world's GPUs to train one large model is obviously unrealistic. Especially given differences in national circumstances, training itself becomes a problem. Of course, vision large models or multimodal large models won't arrive as quickly as we hope — perhaps only because our imaginations are too limited. If NVIDIA can continue making breakthroughs in compute, the path might be shorter.

Image captioning and VQA, especially VQA, I think may be the crown jewel of multimodal tasks. Why is VQA so important? If VQA were pushed to its extreme — if there were a model where any image or video you feed in, VQA could tell you segmentation, detection, captioning, all the information — that would mean all vision tasks were solved, that would be true multimodality, that would be reaching the bottom. At that point, hitting bottom wouldn't matter because we'd be in a realm of limitless possibility. Of course, the prerequisite is the combination of compute and data — if both can exceed our imaginations, that model might not be too far off.

At the same time, AI researchers need to consider power consumption. Current power consumption is so high that to achieve truly superintelligent multimodal large models, even concentrating global compute might not be enough. At that point, do we need to take a different path — perhaps a revolution at the underlying model level, finding a highly generalizable model with low power consumption and low training costs?

Oasis Capital: What is your ideal ultimate state for multimodal AI?

Professor Yuan: Powerful yet low-power, like the human brain — only about 20 watts, yet highly generalizable. Of course, while humans generalize well, our performance on individual tasks certainly doesn't match domain-specific large models.

Oasis Capital: How do you view the emergent intelligence demonstrated by GPT?

Professor Yuan: The simplest analogy: when calculators first appeared, they were far faster and more accurate than humans at computation, but mathematicians didn't become obsolete. A powerful calculator doesn't eliminate the need for mathematicians. Large models' emergent intelligence exhibits certain intelligent behaviors, but humans being surpassed by machines isn't new — AlphaGo, Deep Blue. Some jobs in some industries may be replaced, but that's the main theme of human historical development; there's no need for fear. New industries emerge, old ones fade — cars replaced horse-drawn carriages, drivers replaced coachmen. When true autonomous driving arrives, drivers will disappear, but other industries will surely be born. If there are no jobs left, humanity will enter a higher state. The short-term pain of a true AI productivity revolution won't replace the transformative gains achieved in the long term.

The GPT series is more human-like across multiple tasks, which I think stems from its training methodology. Why does it confidently hallucinate? Because the training approach makes it sound authoritative without addressing factual accuracy — it just creates the impression that a human is conversing with you. This has indeed produced something of an uncanny valley effect.

Oasis Capital: What are your thoughts on neuromorphic computing models?

Professor Yuan: Neuromorphic approaches pursue low power consumption, but the path is long. First, industry and academia pay too little attention, with too few practitioners, so development will inevitably slow. Second, hardware demands are greater — it's not based on von Neumann architecture; we need efficient neuromorphic chips for training, and training on GPUs isn't efficient. Neuromorphic computing is a field requiring coordinated hardware-software development. Because of insufficient attention, I hope more people will engage with and advance this field. AI development is resource-intensive; when companies see too long a development path, they won't invest resources, leaving academia to drive it forward — which is precisely academia's purpose. Academia is where we keep the flame alive. While everyone focuses on applications and innovations with large models, academia needs to view this calmly and know whether we should preserve that flame. When this generation of large models reaches its limit, academia may look back and find more possibilities to push forward. Academia needs to be oriented toward more fundamental innovation and advance with a long-term perspective.

Oasis Capital: In neuromorphic chips, are there any researchers worth watching domestically and internationally?

Professor Yuan: Domestically: Peking University's Tiejun Huang and Yonghong Tian; Tsinghua University's Luping Shi; Chinese Academy of Sciences Institute of Automation's Guoqi Li; Zhejiang University's academician Yunhe Pan. Internationally: UCSD's Professor Gert Cauwenberghs, Yale University's Professor Priyadarshini (Priya) Panda, Harvard University's Professor Hongkun Park, among others.

Sustaining Vitality

What do you think technological vitality is?

Academia is where we keep the flame of technology alive.

— Professor Yuan Li, Peking University School of Information Engineering

Oasis Capital is a new-generation venture capital firm in China, dedicated to discovering the most vital entrepreneurs of the next decade and growing alongside them to create long-term value. "Sustaining Vitality" is Oasis's vision and mission. This vitality is both the direction of structural transformation in our era and the resilience and evolutionary force of entrepreneurs.

Oasis Capital focuses on early and growth-stage investments, with individual investments ranging from $3 million to $30 million, concentrating on robotics, artificial intelligence, technology services, and other fields, supporting China's technology-driven new service upgrade.