Professor Feng Chen, New York University: Multi-Agent Collaborative Perception for Autonomous Driving | Agent Insights
Counselor Vitality
What happens when generative AI meets autonomous driving? In industry, generative AI has already been deployed to tackle the long-tail problems of self-driving. Now, academia is beginning to experiment with multi-agent datasets in algorithmic simulation to enhance fidelity. Today, Oasis Capital sits down with Professor Chen Feng of New York University, who developed the Open MARS Dataset with his team. This innovative technology not only pushes autonomous driving forward, but also opens fresh perspectives on multi-agent collaboration and multitraversal modeling. Enjoy.
Multiagent Multitraversal Multimodal Self-Driving: Open MARS Dataset
Paper: https://arxiv.org/abs/2406.09383
Problem Solved: Traditional autonomous driving datasets typically focus on collecting perception and decision-making data from a single agent — the ego vehicle — while largely ignoring multi-agent collaboration and data gathered from repeated visits to the same location. Drawing on real-world industrial scenarios and raw data, the team developed a multi-agent, multitraversal, multimodal open dataset for autonomous driving, aiming to advance perception, prediction, and planning capabilities for self-driving systems and mobile robots.
Model Architecture: In collaboration with the autonomous driving company May Mobility, the team collected real-time self-driving data over four months using LiDAR, RGB cameras, IMU, GPS, and other sensors. They gathered and analyzed multi-agent and multitraversal data, ultimately building the open autonomous driving dataset Open MARS.
Application Scenarios: Autonomous driving and robotics, with specific tasks including 3D reconstruction, neural simulation, and unsupervised perception.
Test Results: The Open MARS Dataset provides quantitative benchmarks for state-of-the-art methods in visual place recognition (VPR) and neural reconstruction. The results reveal the challenges faced by single-agent perception approaches and demonstrate the potential of multi-agent and multitraversal strategies to enhance perception capabilities.
Quantitative benchmark results of SOTA visual place recognition methods on different subsets of the Open MARS Dataset
Oasis Capital: Could you briefly introduce your research background and focus?
Professor Feng: For my undergraduate studies at Wuhan University, I worked in photogrammetry. During my PhD at the University of Michigan, Ann Arbor, I researched simultaneous localization and mapping (SLAM) — a technology with broad applications in autonomous driving and robotics. Over time, I developed an interest in scene understanding. Beyond localizing robots and building 3D models of their surroundings (the geometric vision branch of computer vision), we need to understand what can and cannot be collided with, what needs to be touched and moved. This led me into perception research: object detection, tracking, segmentation — the pattern recognition side of computer vision.
After graduating in 2015, I joined Mitsubishi Electric Research Laboratories (MERL) in Boston to work on advanced driver-assistance systems (ADAS) and autonomous driving. Initially, I continued using SLAM for automated parking. Then I proposed to the lab that we explore using deep learning to interpret 3D point clouds of environments. At that time, the computer vision field was just beginning to adopt convolutional neural networks (CNNs) at scale for image understanding, but there were no good methods for non-image structured data like point clouds. Our early point cloud deep learning architectures, FoldingNet and KCNet, attracted some attention. When I came to NYU in 2018, I felt that entering pure autonomous driving research alone was somewhat late to the game, so I also began exploring "robotics + deep learning" — what we now call embodied intelligence. Whether studying autonomous driving or robotics, one emerging direction we focus on is collaborative perception: multiple agents, each with their own sensors, communicate and cooperate to understand a scene together, making the overall system more efficient and safe.
A concrete example: you're driving, and a large truck blocks your view, severely impairing your decision-making. If that truck (or nearby vehicles) could share additional sensor information with you, you could make better decisions. This is essentially seeing through objects. Collaborative perception can also enhance capabilities in long-range, low-resolution scenarios, among others.
Oasis Capital: How do you think generative AI can help autonomous driving? Will future self-driving rely solely on vision, or will it require multimodal inputs?
Professor Feng: In the long run, I believe autonomous driving will maintain multimodal information input — though not necessarily multimodal on every single vehicle. It could be single-modality on most vehicles, multimodal on a few, with the overall system (including roadside units) being multimodal. The past decade-plus of autonomous driving development has shown us that vehicles in traffic aren't isolated individuals; they form a system. To improve the safety and efficiency of the entire self-driving and traffic system, a multimodal approach can more easily break through the fundamental limitations of vision compared to Tesla's vision-only approach. From a purely visual standpoint, current camera hardware still struggles to match the human eye's quality in high-dynamic-range, high-resolution scene understanding, let alone adverse weather conditions. Cost is another factor: different commercial autonomous driving tiers at different price points will choose different sensor configurations, naturally leading to multimodality. So I believe future autonomous driving will more likely be realized through multimodal approaches.
As for generative AI's impact on autonomous driving, neither academia nor industry seems to have reached a consensus yet. Some applications use generative AI to supplement long-tail autonomous driving scenarios and enhance simulation algorithm capabilities. Others leverage generative large models' commonsense reasoning for task planning. But the entire industry is actively exploring this, including NVIDIA and autonomous driving startups presenting related applications at top conferences like CVPR. Whether generative AI can qualitatively improve autonomous driving perception, planning, and decision-making performance — there's still much room for continued exploration.
Oasis Capital: Could you explain the original motivation behind developing the Open MARS Dataset?
Professor Feng: With Open MARS, we mainly wanted to create a multi-agent, multitraversal, multimodal open dataset for autonomous driving. Multitraversal refers to how agents visiting the same environment at different times can leverage recurring visual information for perception. My PhD student Yiming Li, the dataset's first author, calls this retrospective perception. For example, multiple autonomous vehicles may repeatedly enter the same area, or one vehicle may pass through the same area multiple times. This multitraversal characteristic better reduces the need for manual annotation and facilitates modeling of static environments. Before MARS, multi-agent collaborative perception research mainly relied on simulation platforms like Carla to build datasets; there were very few open, real-world datasets from actual industrial scenarios.
Our partner company, May Mobility, primarily operates autonomous taxis in fixed scenarios. Last year, they had four commercially deployed robo-taxis running on nearly fixed routes in a designated area. So we proposed using their real data to validate our previous multi-agent collaborative perception research and explore new possibilities that retrospective perception through multitraversal could bring to autonomous driving. For instance, after an unmanned vehicle has traversed the same area ten times, it can better distinguish and model static versus dynamic data, even solving problems that traditional perception modules and large models still struggle with. And once our multitraversal-based fully automatic labeling data production method matures and scales up, it would be tremendously helpful for autonomous driving technology providers.
Whether using multi-agent collaborative perception or multitraversal retrospective perception, our goal is to improve the safety and efficiency of autonomous driving (and autonomous mobile robot) systems from a whole-system perspective: the former approaches from the spatial dimension, the latter from the temporal dimension. The motivation for developing the MARS dataset was to accelerate validation of new academic ideas and directions through this novel industry-academia partnership, thereby advancing the entire field.
Oasis Capital: Could you explain how the Open MARS Dataset was collected?
Professor Feng: Traditional large-company open datasets like those from Waymo and Lyft couldn't help us with multi-agent collaborative perception research, so we initially built simulation datasets. It wasn't until our collaboration with May Mobility began last year that we could explore further with real data. Professor Edwin Olson at the University of Michigan, Ann Arbor, is the founder of May Mobility and was also my advisor when I studied mobile robotics there — this connection is how we obtained permission to access their autonomous driving data. In the autonomous driving field, whether in academia or industry, this degree of open, bidirectional, deep collaboration is unprecedented: we can access their autonomous vehicle data in real time daily through APIs, and they've also allowed us to release a portion of collected data for researchers worldwide to use.
From October last year to January this year, we gathered four months of data. We extracted relevant geographic location and timestamp information, then requested corresponding image and 3D data from them, ultimately forming our two subset datasets: multi-agent and multitraversal. Currently, the Open MARS Dataset enables specific autonomous driving perception tasks without relying on manual annotation, such as neural 3D reconstruction and visual place recognition. In the second half of this year, we'll release version two, which will include more manual annotations and higher-quality image data, enabling more autonomous driving perception research tasks (such as occupancy prediction, 3D object detection, etc.).
Oasis Capital: In the long term, how do you think large models can help autonomous driving and robotic perception?
Professor Feng: One practical direction is providing cheaper, more convenient data annotation. Currently, the most efficient practical machine learning approach in industry remains supervised learning, because improving algorithm architecture quality is a very slow process, whereas improving data quality leads to notably different model quality improvements. So we believe that in autonomous driving and robotics, using large models for more efficient data annotation, and even learning from data that generative AI can produce, are both possibilities.
There are also some specific scenario applications. For example, navigating in a 3D environment using visual information (such as giving a user an image and having them find an object) — in tasks where the action space isn't that complex, generative data seems to have decent potential. But for safety-critical scenarios, such as industrial scene generation or simulation that we've experimented with, we remain cautiously skeptical. We look forward to more colleagues applying large models to these serious scenarios and producing truly practical, efficient products.
