Professor Yu Su of The Ohio State University: See! Then Act | Agent Insights

When OpenAI and Figure unveiled Figure 01, the robot's onboard cameras fed into a large vision-language model (VLM), with neural networks processing visual input and outputting 24 degrees-of-freedom actions at 200 times per second... In a recent podcast, Yann LeCun noted: "A four-year-old child has already absorbed 16,000 hours of visual information. Language is limited in 'bandwidth' and expressiveness for transmitting information, while visual perception's data bandwidth is roughly 16 million times greater than that of written or spoken language." This highly efficient mode of visual expression opens up far greater possibilities for multimodal large model-based agents and robotics. Enjoy.

SeeAct, a general-purpose web agent built on multimodal large models, stands as one of the most notable examples in this category. Its creator, Professor Yu Su of the Computer Science Department at Ohio State University, had already led his team in developing Mind2Web — the first comprehensive benchmark for evaluating large models' web-browsing capabilities — before ChatGPT's release, demonstrating remarkable foresight.

GPT-4V(ision) is a Generalist Web Agent, if Grounded

Problem Solved: Built on top of large multimodal models (LMMs), SeeAct extends web agent capabilities to multimodal tasks including visual question answering, web navigation, and online shopping — addressing the significant limitations and low accuracy of text-only language model-based web agents.

Model Architecture: SeeAct leverages multimodal large models like GPT-4V for visual perception of webpages, generates plans in text form, and executes actions by mapping text plans to HTML elements through oracle grounding.

Performance: Evaluated on the Mind2Web dataset, SeeAct successfully completed 50% of tasks across different websites — substantially outperforming existing methods such as GPT-4 (20%) and FLAN-T5 (18%).

Application Space: Web agents.

Operational Process: Given a car rental website and task T (rent a truck at the lowest price), the web agent generates a sequence of actions. At timestep T, the agent determines its next action based on current environmental observation S and previously generated actions A.

Findings and Analysis: GPT-4V can serve as a generalist web agent with oracle grounding. With human-annotated GPT-4V+SeeAct Oracle, the model achieved success rates of 65.7%, 80%, and 62.1% across three trials — far surpassing other configurations. In cross-task settings, it outperformed the second-best method by 11.9% in step success rate; in cross-website and cross-domain settings, it led by 28.3% and 21.2% respectively.

---

Oasis Capital: When did you first start paying attention to agents? What's the latest progress?

Professor Su: I studied computer science at Tsinghua University, then pursued my PhD at UCSB. I switched advisors to work on data mining, and eventually transitioned into NLP. Along the way, I accumulated extensive technical knowledge spanning from underlying data systems to higher-level intelligent agents. I started working on Language-to-Web APIs back in 2017 — using language to send emails or check the weather, rather than writing code as we can today. Later, as models like InstructGPT and GPT-3 demonstrated powerful generalization capabilities, it became clear that LLMs were maturing rapidly. Our group recognized that this technological evolution was bringing us closer to realizing agent goals, so we began exploring the space. We were thinking about agents quite early in the industry — before ChatGPT even launched, I had already started leading my team to build Mind2Web, a benchmark for web agents. We've since published a substantial body of work in the agent field, including SeeAct, LLM-Planner, TravelPlanner, and AgentBench.

Oasis Capital: We noticed Yann LeCun also shared your TravelPlanner paper.

Professor Su: Yes, and Trip.com Group reached out to us proactively as well.

Oasis Capital: Could you walk us through the foundational agent architecture in the SeeAct paper?

Professor Su: In web agents, multimodal large models enable better front-end information acquisition and back-end instruction execution. The model understands webpages through vision to obtain more accurate information, preparing for the reasoning phase of subsequent instruction generation. The generated instructions contain various actions, which are mapped to webpage elements through grounding before final execution. The key innovation in SeeAct is the use of multimodal large models (GPT-4V) combined with oracle grounding to complete web agent decision-making and localization — ultimately achieving higher task execution success rates than language-only models.

Oasis Capital: What does grounding mean for agents?

Professor Su: Regardless of agent type, grounding is always one of the most fundamental challenges. A defining characteristic of agents is the integration of large language models with external environments. Achieving this requires two things: first, environmental information input; second, action instruction output. Accurate environmental grounding is the foundation for reasoning, while accurate instruction output is core to action — thus grounding plays a critical role. In the SeeAct paper, we focus particularly on grounding in action instruction output, which presents greater challenges than environmental input grounding.

Oasis Capital: Where specifically does using multimodal large models as the foundation improve agent performance on web tasks?

Professor Su: Compared to language models, multimodal models can handle perception and planning simultaneously, without requiring additional tools (such as object recognition). Fewer intermediate steps means fewer opportunities for errors, and the representation throughout the process is more unified. It also improves reasoning efficiency and reduces costs, because vision is an extremely efficient mode of expression. For equivalent information, language models require 10x or more tokens than vision models.

Oasis Capital: As underlying large models continue to grow more capable, do you think agents will eventually be subsumed by these models?

Professor Su: I don't believe future large models will fully replace everything agents currently do. I think multimodal large models will continue improving, gradually approaching certain cortical capabilities of the human brain. But the multimodal interaction and coordination in the human brain is extraordinarily powerful — far beyond any large model — and models will only slowly converge toward human brain capabilities over the long term. Meanwhile, people's expectations for AI agents are higher than for humans themselves: they want agents to know everything in the world, capture dynamic real-world information, and complete thousands of tasks. But packing all agent functionality into a single model is extremely difficult, and economically impractical.

Oasis Capital: Do you see the future as a general-purpose agent framework solving most problems, or vertical agents handling specific tasks?

Professor Su: I don't think they're mutually exclusive — they'll coexist. Consider how human intelligence develops: genes control brain development, and humans have only 20,000-30,000 protein-coding genes, with a small fraction involved in brain encoding, yet this produces a structurally complex and highly adaptive brain. We can understand the human brain as a programmable computer — different environments and upbringings develop completely different capabilities. The ultimate direction for agents is to develop something analogous to DNA and brain architecture: highly structured, controllable, with continuous learning ability, that can be placed in different environments to train vertical agents.

Oasis Capital: How could agents achieve functionality comparable to the human brain?

Professor Su: Our group is currently focused on Biologically Inspiring AI. I'm personally very interested in biology. We recently published another paper called BioCLIP, which performs image recognition across the entire Tree of Life, supporting over 2 million species — identifying species from a single photograph. Species evolution and natural selection represent a truly transformative force. Over billions of years of evolution, natural selection has discovered many excellent locally optimal designs. I'm not saying AI should completely imitate biological intelligence, but biological intelligence is the strongest intelligence we know of. As AI researchers, we must deeply understand biological intelligence to help develop better AI. The next phase of AI development needs to identify and integrate these locally optimal designs from biological intelligence that are compatible with modern AI frameworks. In some respects, the gradient descent that AI relies on and natural selection are interconnected — whether in artificial neural networks or biological systems, both operate under the same mechanism, producing diverse designs through prolonged evolution full of randomness and massive computation. Through Biologically Inspired AI research, we've come to recognize that truly achieving agent functionality requires much more exploration. For a concrete example, current agent frameworks' implementation of memory remains unsatisfactory. Human memory defines "who you are" — it contains your past experiences and underpins your core decision-making capabilities, possessing both structure and flexibility. Most current agent memory designs are simply vector databases with basic high-density similarity retrieval, which remains far from how the actual human brain works. Our next research project is developing agent long-term memory based on how the human brain retrieves long-term memories.

Oasis Capital: For the future development of agents, what do you think deserves attention?

Professor Su: Agent safety cannot be overstated as a concern. Future web agents could very well cause mischief on real websites or lead to information leakage — areas that are currently being overlooked. Right now, most people are rushing to define agents for every industry, without sufficient attention to safety. Our group has recently defined various attacks against agents, and we hope to contribute more to the development of agent security in the future.