Oasis Capital Dialogue with Professor Chen Tao: The Foundation of Castles in the Air

Multisensory Collaboration, Cross-Modal Development — AI Iteration Depends on Hardware-Software Synergy

We invited Professor Tao Chen, Assistant Dean at Fudan University's School of Information Science and Technology, IEEE Senior Member, and doctoral supervisor, to share his perspective on the future of large models from a computer vision (CV) standpoint. Enjoy.

Oasis Capital: How do you view the development of CV, NLP, and large models?

Professor Chen: In AI, CV and NLP — CV accounts for at least 50% of the weight. Humans obtain 75% of their information through vision, so it naturally becomes a major branch of AI. Before 2012, CV was a process of traditional neural networks combined with SDM game theory, remaining largely confined to academia with minimal industrial impact and difficult deployment. After 2012, the development of big data and large-scale computing power facilitated deep neural network learning, sparking a wave that continued through 2018–2019 — arguably the period of strongest demand for CV deployment, in areas like surveillance, autonomous driving, and security. But after 2018, the industry discovered that no matter how models were designed and optimized, there were always boundary effects and long-tail scenarios.

Language or speech is actively produced by humans; images and video are passively captured by physical devices. Their modes of production differ, so their signal forms naturally differ as well. The former possesses high semantic density and uniformity, while the latter is highly sparse and uncertain. Text is an optimized version organized and expressed through human conscious processing. Compared to CV and images, NLP has less scenario divergence and lower difficulty. CV and images face scenes full of uncertainty — weather changes and natural environments captured by vehicle cameras, for example. This is an inherent characteristic of CV.

These different properties mean NLP more easily finds deployment scenarios, such as dining, voice applications — iFlytek or Google's TTS, for instance. The wave of large models that emerged in 2021–2022 also broke out first from NLP. Turning to CV, aside from a few giants like Huawei's AI chips and Hikvision's visual security, which are profitable, many other AI companies are basically losing money.

Humans can understand, attempt, learn about unknown things, or avoid and optimize for them. But AI struggles to do this at the CV level — it can only recognize or perceive things it has seen, failing to identify unknown or rare content. Autonomous driving, medicine, and security frequently see failure cases. In these industries, even 99% success means that one unreliable case draws skepticism. Thus CV deployment scenarios have remained in a trough.

The sudden explosion of large models has reignited hope in the CV field. Previously, many AI experts and scholars believed that purely big data-driven CV and AI were unsustainable — that deep learning shouldn't rely on brute-force accumulation of large models and big data, and that next-generation AI should be pursued, such as "brain-inspired intelligence" or "knowledge and data dual-driven" approaches. Their reasoning was that big data-driven AI is a form of data-pattern intelligence, while the human brain itself doesn't need big data driving yet is far more capable than deep models, with energy consumption of only 6 watts — whereas training a medium-sized BERT model once consumes electricity equivalent to a Chinese electric locomotive's lifetime usage. Peking University, Tsinghua University, and the United States all have brain science initiatives researching how to concretize memory, cognition, and understanding of world patterns in the brain into mathematical expressions — only then can computers achieve them.

But developments often defy expectations, and "brute force produces miracles" large models appeared. Current facts prove that as long as models see enough and computing power is sufficient, they can indeed learn super-intelligent agents. Conversely, from 2018 to 2021, we never saw "brain-inspired intelligence" industrialization. Even Elon Musk's Neuralink is far from reaching a monetization phase, and low-power, low-carbon small intelligent models have stagnated.

Oasis Capital: What impact has this paradigm shift had on your research field, and what are your expectations for large model development directions?

Professor Chen: Humans see things or understand the world through multisensory collaboration. Multimodal or cross-modal large models are definitely the mainstream trend for future development — OpenAI and other overseas giants are certainly already working on multimodal large models. Once a unified large model for speech, text, images, and 3D is created, its impact on the entire field will be profound. The Chinese Academy of Sciences has a national key laboratory for multimodal research, reflecting strong demand at the national, government, and civilian levels.

New work on vision-language large models keeps emerging. Our group has also adjusted direction in the past two years to do cross-modal and multimodal research, attempting to fuse 2D, 3D, and NLP for understanding — generating textual explanations and annotations from images, or using NLP to understand and locate visual scene targets. These align better with normal human usage needs.

Another adjustment in our research direction comes from the challenge that large model upgrades pose to university computing resources. So we're exploring and designing low-computational-power models adapted for consumer terminals under limited resources, compensating for small models' inherent capability limitations through approaches like edge-cloud collaboration and swarm intelligence collaboration.

Additionally, if academia also chases big data and big computing power, it will lose its own distinctive characteristics. So we must remain more focused on cutting-edge technical theory, doing work that differs from industry but has practical value.

Oasis Capital: How do you view China's prospects for developing large models?

Professor Chen: Large models are genuinely powerful; not pursuing them means falling into passivity. In the future, large models may become a "chokepoint" technology like chips. Once a large model forms an industry chain and users develop dependency, it may become chargeable — and potentially very expensive. Meanwhile, as users employ large models, the model developers are indirectly collecting user data. The United States currently leads in large models due to its advantages in AI computing power and research capability. China has natural advantages in large scenarios and big data, and should fully leverage them to learn stronger large models.

Oasis Capital: From your perspective, what are the main challenges for large language models, or the vision transformers you work on, to truly run on edge devices?

Professor Chen: For example, we previously had a collaboration project with ZTE, using sparse convolution concepts to do real-time lightweight video understanding and segmentation on ZTE phones equipped with Qualcomm 855 chips. We designed plug-and-play modules compatible with current video AR models, replacing existing modules to greatly accelerate inference speed and save computation.

From previous experience, edge chip deployment mainly faces three challenges:

1. Domestic chip compatibility. Why do so many people still prefer to use NVIDIA despite knowing it may restrict graphics chip exports to China at any time? Because its ecosystem is excellent, with good compatibility across environments and platforms — various mainstream models can all produce good results. Many domestic chips running mainstream deep models still have insufficient support for some sub-modules, even requiring manual rewriting. If encountering not fully open domestic platforms, even rewriting won't work.

Currently, edge computing power is already quite strong, with limited restrictions on general model deployment at the edge. For example, Horizon Robotics' Journey series chips for autonomous driving can reach dozens or even over a hundred T FLOPS. But some kernels and components in domestic AI chips aren't fully domestic, ultimately still potentially subject to external constraints. Furthermore, some frameworks, platforms, and acceleration engines are self-designed, with unfriendly support for current mainstream international AI computing architectures.

Even NVIDIA's best chips still lack sufficient acceleration capability for Transformers. NVIDIA is currently designing and improving next-generation products for high-speed or dedicated support of Transformer-class models. Transformer is currently the gateway model for various large models — whoever captures the best chip for supporting Transformers will gain first-mover advantage.

2. Memory. Chip RAM is too small; loading models directly maxes out memory. In recent years, the research community has proposed the concept of "sensing-storage-computing" integrated design chips. Integrated storage-computing eliminates data read-in and read-out, with all model computation data storage and buffering completed within one chip. Although not yet commercially deployed at scale, the strategic direction is clear.

Take our own experiments as an example: model inference speed on GPU was over 40 frames per second, which seemed perfectly suitable for chips. But once the model was loaded, memory became the bottleneck, with speed dropping 50%. So we also need to strengthen model and core hardware adaptation capability.

3. Hardware-software misalignment: AI algorithms, hardware, and chip design lack sufficient coordination; designers don't communicate enough with each other. This leads to algorithms designed on computers and GPUs that, even with reduced FLOPs and memory footprint, don't fully consider chips' parallel computing and sparse characteristics in their model structures, yielding suboptimal actual results.

Our work aims to bridge the gap between software and hardware, representing a crucial step in hardware-optimized design based on existing models. For example, due to images' sparse characteristics, the effective activation values in feature maps extracted by models sometimes account for only 10%-20%, sparsely distributed across certain rows and columns in images. We attempt to compress these sparse features into one densely distributed small feature matrix and one index matrix storing activation values, allowing chips to rapidly process the densely distributed feature matrix and thus improve speed.

Work shouldn't just be a paper or an algorithm. No matter how good the algorithm, without chips, many algorithmic models — even large models — remain castles in the air. Our group doesn't do chips ourselves, but hopes to achieve breakthroughs through hardware-software synergy and integrated design solutions, contributing to AI hardware iteration or optimization and to addressing "chokepoint" technologies.

Oasis Capital: Do you think this wave's impact on CV research is positive or negative?

Professor Chen: The inspiration outweighs the crisis. Crisis certainly exists — large models will impact many tracks in traditional CV tasks. For example, 2D domain adaptation work may be directly replaced by SAM, which can segment unseen things in general scenes, making cross-domain tasks simply nonexistent.

But opportunity still exceeds crisis. Key ideas from GPT and similar models include reinforcement learning with human feedback interaction, adversarial learning, prompt-based learning, and so on. CV can borrow from GPT's approach, using language and vision dual modalities to jointly extract knowledge for iterative model optimization and parameter learning. Taking SAM as an example: based on the parent model already trained on internet data, its understanding capability can be used for small-scene data annotation, and its generalization capability for data generation — both can assist CV scenarios in better deployment. "Knowledge distillation" in CV means using large models to teach and cultivate small models' capabilities. Features extracted from SAM-class parent models can serve as teacher supervision signals, producing very capable small models and thus achieving unsupervised learning.

Oasis Capital: The most practical CV deployment in the last wave was security. Will this wave promote visual deployment scenarios?

Professor Chen: Priority should go to scenarios combining CV with NLP. Visual large model development isn't as focused as NLP model development. Face recognition works well, but many tasks in security aren't solvable by large models — what if the face simply can't be captured? In passive recognition tasks, visual deployment faces challenges.

Fields that this wave can particularly assist will especially appear in active recognition scenarios where NLP services are superimposed and users voluntarily cooperate — such as Q&A robots in service industries. In industrial manufacturing defect detection, previous bottlenecks lay in lack of generalization capability for uncontrollable flaws and defects, while visual large models can enable machines to make 0/1 classifications, flagging "unseen" anomalies. In remote medical multi-case diagnosis, previous deployment was unsatisfactory; now large models can leverage NLP knowledge records plus imaging assistance to make probably accurate diagnostic analyses and recommendations. Many enterprises and researchers are already pushing on multimodal general consultation. The AIGC field will certainly benefit, with many companies already deploying.

Oasis Capital: So what's the effect on pure vision improvement?

Professor Chen: Many vision problems lie at the front end — if the sensor itself is poor, no algorithm can fix it.

Oasis Capital: Setting aside limitations, where will CV deployment scenarios be?

Professor Chen: I think AR and VR. Two most fundamental problems in vision: First, visual understanding, a bottom-up process. Understanding is the process of information concentration and extraction — AI must analyze the elements and semantic information, what's happening in video or images. Second, visual generation, a top-down process — obtaining elements, then injecting new elements to present another scene or the original scene, forming a closed loop through user interaction.

AR and VR already show promise; after all, humans ultimately hope to obtain satisfaction and experience through vision. Visual development should return to human-centeredness — current autonomous driving and security are still ToB. Future visual interaction will be revolutionary, not requiring physical presence, such as virtual scenes in the metaverse, interactions in VR and other virtual worlds.

Oasis Capital: How are your research results being commercialized?

Professor Chen: We're working on multimodal pre-trained foundation models to provide better parent models for different downstream tasks. For some autonomous driving tasks, many backbone models based on our design and pre-training have brought downstream task improvements — this has already been verified. We're also glad to collaborate with more CV downstream enterprises and consumer terminal-oriented enterprises to help them improve their intelligence levels.

Oasis Capital: How to solve resource constraints?

Professor Chen: Open source is the most direct channel. CV's rapid development over the past decade owes to open source. Large models are important, but the big data behind them is even more important. If data has problems, large model quality will certainly have problems. China's population base and scenarios are natural advantages; if industry barriers can be further broken and open-source ecological communities established within certain scopes, development will follow. Objectively speaking, large model openness hasn't been very good in recent years, and large model companies won't typically publish specific technical details.

Oasis Capital: OpenAI has recently been talking about attacks — is your group working in the same direction?

Professor Chen: Accidentally, actually (laughs). Our original intention was autonomous driving data augmentation, data synthesis to improve model recognition capability. During experiments, we discovered that some negative adversarial samples originally meant to be discarded could be used to enhance model capability. Through this discovery, we re-examined adversarial samples in adversarial learning — what role they could play in visual driving recognition tasks. After statistically analyzing large numbers of adversarial samples' curves and performance impacts on models, we found that with proper utilization, they could assist vision problems. We then summarized a piece of work through the adversarial attack perspective on visual downstream tasks (https://arxiv.org/abs/2305.10766), receiving quite a bit of recognition.

Oasis Capital: Do you think the future will trend toward AGI, or world models?

Professor Chen: Personally I lean toward AGI. The world is very complex — many physical phenomena, high sparsity of modalities, uncertainty — at current levels it's difficult to summarize with a unified theory. World models' high cost and high price produce uncontrollable return ratios; where's the practicality and significance?

AGI will achieve major breakthroughs in vertical domains. For example, medicine has many problems awaiting solution — many diseases require MRI diagnosis, but patients can't access it due to high costs, limited hospital equipment, or lack of configuration entirely. These are rigid problems. If cross-modal medical diagnostic models can reduce medical costs, alleviating patient pressure would already be a major step forward.

Our general autonomous driving model from some time ago also aimed at the current situation where datasets, scenarios, and tasks aren't unified, building unified baselines and a general autonomous driving training framework platform.

World models are more like a concept in the short term; successfully achieving AGI would already be quite good.

Oasis Capital: How do you think practical capability in the AI field can be improved?

Professor Chen: Teachers, professors, and practitioners need to step out of their comfort zones to do cross-domain and integrated research. Relevant national departments have also established cross-research divisions. Although human nature tends toward unwillingness to do things outside one's interest range, for industry development, we still need more exchange and communication at domain intersections like hardware-software integration, with exploratory spirit — once successful, the impact is enormous.

Celebrating Vitality

What do you think is technological vitality?

Technological capability depends on technological hardware and software experimental conditions, improving as human cognitive levels advance; scientific theorems also iterate and develop, with new laws potentially overturning old ones. Human life is finite, yet human exploration of technology, nature, and even the universe is far from its end. Human cognition itself is like a living organism, constantly self-negating, innovating, and spiraling upward — this is the charm of life, I suppose.

— Professor Tao Chen School of Information Science and Technology, Fudan University, Shanghai

Oasis Capital is a new-generation venture capital firm in China, dedicated to discovering the most vital entrepreneurs of the next decade and growing alongside them to create long-term value. "Celebrating Vitality" is Oasis's vision and mission. This vitality represents both the direction of structural transformation in the era and the resilience and evolutionary power of entrepreneurs.

Oasis Capital focuses on early and growth-stage investments, with individual investments ranging from $3 million to $30 million, concentrating on robotics, artificial intelligence, and technology services to support China's technology-driven new service upgrade.