Fudan University AI³ Researcher Yinghui Xu: The Visual Creation Interface for Agents | Agent Insights

Traditional LLM-based AI Agent operations platforms have certain limitations when it comes to developing complex applications. The AI2Apps visual integrated development environment, built by Yinghui Xu, a researcher at Fudan University's Institute of Artificial Intelligence and Industry (AI³), together with the AI2Apps team, integrates engineering-grade development tools, covers the complete AI Agent development lifecycle, offers fully open extensibility, and includes a built-in browser sandbox environment. Drawing on Figma's design philosophy, it allows developers to share their creative progress with collaborators in real time for collaborative development, helping developers efficiently build AI Agent applications in one place while reducing the "friction" in the creative process. Enjoy.

"AI2Apps: A Visual IDE for Building LLM-based AI Agent Applications"

AI2Apps System Architecture

Problem Solved: While traditional LLM-based AI Agent operations platforms significantly lower the barrier to entry for users, they often struggle to meet the needs of professional developers, thereby constraining the flexibility of design, coding, and debugging — and hindering the development of more complex AI Agents. AI2Apps, the first visual integrated development environment (Virtual IDE) for LLM-based AI Agent applications, covers the complete development cycle from prototyping and code writing to Agent debugging and final packaging and release, helping developers efficiently build AI Agents.

Framework: The AI2Apps framework includes: 1. Prototype Canvas 2. Code Editor 3. Agent Debugger 4. Publishing Tools 5. Plugin Extension System 6. Management system built on a self-developed browser operating system (Tab-OS)

Performance: AI2Apps integrates engineering-grade development tools and full-stack visual components covering both frontend and backend, enabling developers to build Agent applications in one place through a "canvas" GUI that supports both component drag-and-drop and code programming, significantly boosting development efficiency. For team collaboration, AI2Apps draws on the philosophy of the well-known collaborative design tool Figma, allowing developers to share their creative progress with collaborators in real time via QR code links, with built-in version control enabling collaborative development. For deployment, AI2Apps has Web-IDE characteristics — it works right in the browser after deployment, with a built-in sandbox environment that safely isolates each Agent project in its own browser tab at low resource cost, without requiring Docker or other container environments on the client side.

Applications: For any type of LLM-based AI Agent development in any domain. Like open-source general-purpose IDEs such as VS Code, AI2Apps is fully extensible — developers can package external code as microservices and plug them into AI2Apps as new visual components. AI2Apps can be used standalone as a Web-IDE or easily integrated into large language model operations platforms.

AI2Apps Development Tool Integration / Component Visualization Level

Oasis Capital: How do you see the underlying relationship between Agents and the AGI that everyone is discussing?

Yinghui Xu: We've already entered the post-large-language-model era. Beyond OpenAI, it's difficult for others to significantly differentiate themselves at the foundation model level. The Agent setup process includes Planning, Model Selection, and Execution — and the most effective way to build this system is like "assembling Lego blocks" to quickly accomplish a task. Currently, this "Lego assembly" still requires human involvement. AI2Apps, for example, still needs humans to orchestrate it. Wuxiang Guangnian is considering integrating AI2Apps's technical philosophy into its Guangyu Qizhi large model R&D platform, to provide the industry with efficient and flexible large model intelligent agent construction capabilities.

Currently, Agents can be used on one hand to develop system applications, and on the other hand as data processing tools for Instruction Fine-Tuning (IFT). In the past, data collection relied entirely on manual effort, which was quite difficult. OpenAI has substantial traffic — billions of people use it — and can accomplish this through user Q&A, during which some tools get fine-tuned. The process of humans using large models is itself an Agent, and the massive data generated during use is used to fine-tune the large language model, continuously strengthening its production capabilities. The model naturally learns more instructions and executes them precisely. In the future, Agents will be able to automatically organize and collect user data for large models. As such data accumulates, it becomes increasingly possible for foundation models to use Agent capabilities for fine-tuning through superalignment.

What does AGI mean? In a nutshell, it's using artificial intelligence to make machines think like humans. Everyone has different capabilities — an experienced driver can nail it in one go, while someone less experienced needs to take a longer route. A good intelligent large model should adapt to different platform working styles. But overall, I believe Agent is an intermediate transitional state before reaching the AGI endgame — in other words, the transition from Software 1.0 to Software 2.0. LLMs alone cannot handle humanity's complex task systems. Data from many industries needs to be accumulated through system application processes, and then used to teach models how to do things. This is a process, and before AGI is achieved, Agent is the optimal solution.

As for when AGI will be realized? My understanding is: in daily production and life, when at least half the "employees" in offices are silicon-based beings, that's roughly when AGI will have been achieved (laughs).

Oasis Capital: Although everyone has a different vision of AGI, once AGI is finally achieved, people may no longer need Agents — similar to how software replaced automation in the past. Is that the right way to think about it?

Yinghui Xu: Yes. Many of our clients and people around us have expressed strong demand for structuring unstructured data. In the financial sector, for example, data structuring is extremely difficult — various securities firms and banks need to use web crawlers and purchasing to structure data into valuable research. That's exactly what Wind does. The medical field is the same — it requires collecting large amounts of aggregated data. The problem is that systems with such strong structuring requirements for data inevitably lead to business systems being built only on the surface of that structure. For example, intelligent assisted diagnosis, animated systems — these are modules that become process-based and interface-based after human thinking, with a pile of rules behind them. Yet the interaction data generated in this process naturally becomes the foundational data necessary for the transition from Software 1.0 to Software 2.0. I believe that in the future, Software 1.0 will no longer be needed. All interaction data will be fed into large models, naturally replacing both software and hardware — that's what Agents are doing.

The Agent workflow is: user inputs instructions, backend calls the large language model, may call RAG during this process, outputs results, then stores multi-turn dialogue to memory. During this, we can connect AI2Apps's Agent A with Agent B's memory to build an environment for two people to converse. For example, Agent A simulates a doctor, Agent B simulates a patient — simulating doctor-patient dialogue for diagnosis training. The doctor uses medical guidelines for fine-tuning, the patient uses personal family history, medical history and other data for fine-tuning. This conversational environment already maps well to real-world applications.

Additionally, in this conversational environment we generate synthetic data. Agents are naturally excellent synthetic data generators. More data enables better applications. Injecting thinking into Agents, processualizing Agent behavior — the value behind this is enormous.

Oasis Capital: You've consistently emphasized the importance of user data. Does your team have any particular methods for handling user data during fine-tuning?

Yinghui Xu: Like with pre-training data, we need to look at feedback to determine whether data should be included in the final learning system. We want to find how human society solves problems. Like finding the road to Rome — "finding the road" is the ultimate feedback, but "when to find it" involves preferences and signals in the feedback. So we need to establish signals for preferred options in the feedback, then help users filter all possible products.

Human society isn't uniform, executing tasks according to one paradigm — cognitive levels vary widely. Therefore we need to build data processing engines for unified management of all data, so this data can be vital and in turn enhance Agent capabilities.

Oasis Capital: Can Agents learn only from Best Practices?

Yinghui Xu: Using economics as an analogy, the biggest problem is resource allocation. A person's knowledge is their resource. You can't give everyone the best education and the best resources, because resources are limited. Scarcity persists in the silicon era too — you can't give everyone H800s or A800s. Ultimately, an Agent's knowledge level and capabilities also have boundaries.

Zhucong: My own doctoral research is on scientific intelligent agents, with AGI as our highest-level goal. We often think about whether AGI's ultimate goal is to broaden human boundaries, and if so, we should pursue scientific exploration. But everyone does research differently — people can only broaden human boundaries from different angles through trial and error. The data used in this process is personal Know-How. We can't have everyone learn best practices in lockstep before doing research.

Of course, from a data collection perspective, if we deploy scientific Agents well, they'll continuously collect tasks for human scientists to interact with. This process can also gather large amounts of human thinking and data, which can be used to tune many Agents. At this point, only simple problems have best practices — difficult or complex problems rarely have universally recognized best practices. They still require constant data collection and tuning before evaluation. That is, in complex situations it's difficult to deploy using best practices.

Now everyone has imperceptibly moved from carbon-based to silicon-based. There used to be a concept of Machine-in-the-human-loop; it's gradually becoming Human-in-the-machine-loop. Traditional production lines went from 20 workers to 2 workers — humans only intervene when machines have problems. In the future, humans have human advantages, machines have machine advantages. Machines don't have emotions like people, and factory norms may completely transform into norms for machine work efficiency, completely different from today's norms for human behavior.

Zhucong: As Teacher Xu mentioned the concept of AI employees, in the future SaaS will be directly operated by AI employees. AI2Apps's goal is aimed at AI employees, and AI employees are aimed at fully AI-driven corporate entities. In a previous Sam Altman interview, he mentioned that future companies will only need humans for sales and as legal representatives — the rest can be handled by AI employees.

In the future, there will be research assistant Agents in scientific research scenarios, and financial Agents, product manager Agents, programmer Agents, etc. in corporate scenarios — infinitely replicable, with AI employees making up an increasingly large proportion. Agents forming new Agent content from knowledge gained through interaction — this way of working is very much worth looking forward to. The foundation determines the superstructure. To enable more complex applications like AI employees in the future, the community needs more powerful development environments — hence AI2Apps. Speaking of community, our team hopes AI2Apps can be like Figma (the well-known interface design tool used for website design, etc., which Adobe once attempted to acquire for $20 billion) — developers can show their development progress to collaborators at any time, then adjust plans based on feedback.

The team has also invested two key reflections on Agent development into AI2Apps. First, the current mainstream interaction between Agents and humans is still just dialogue, but natural language won't be the only way, and often isn't the most effective way in real scenarios. Therefore AI2Apps also introduces more than a dozen frontend interaction components including menus and buttons. Second, is Docker (application container engine) necessarily the best container for running Agent applications? Because Agents are too "light" while Docker is too "heavy" — Docker may be more suitable for servers running large models rather than running Agents on the client side. So our team explored using the browser as a sandbox, with each tab page serving as an Agent container.

Oasis Capital: How do you two view the changes that this year's multimodal capability improvements have brought to Agent functionality?

Yinghui Xu: Sora and other related foundation models like text-to-image will ultimately become one MLOps (machine learning operations practice) in the task chains that developers build. When conversing with systems, Agents need corresponding multimodal capabilities for text, language, images, etc. to coordinate. From the technical details Sora has disclosed, the underlying approach likely uses extensive tokenization of visual information, with video and image super-resolution generation added later. In other words, whether language, image, or sound, everything is ultimately embedded into a decoder-only architecture — all discrete.

Current opinions vary. One camp believes continuous modeling works better; I personally lean toward discrete modeling because discrete modeling's text state and multimodal optimization targets are the same, so long-sequence generation won't be a problem. But once you use continuous modeling for unified long-sequence content generation, the cost is very high. Currently no domestic company has achieved this. Discrete modeling has very high compute requirements, and its hunger for data far exceeds continuous modeling. We attach great importance to the diverse data processing capabilities that multimodality brings to Agents, while also closely following the development of discrete modeling approaches.

Oasis Capital: What was the reasoning behind designing the visual effects for various functional modules, extensible modules, and user interaction in AI2Apps?

Zhucong: Because prompt engineering isn't sufficiently controllable, developing truly deployable complex Agent applications has become very cumbersome. Pursuing the most comprehensive visualization possible is a good way to reduce this development difficulty. Visualization lets developers overlook the entire project from a high dimension, continuously reducing the "friction" in the creative process.

AI2Apps's designed visual "canvas" interface

Oasis Capital: Which industries do you think AI2Apps is most suitable for?

Zhucong: AI2Apps is suitable for any type of Agent development — it depends on which industries have strong Agent development demand. I believe the wave of Agents will gradually rebuild all of today's software products.

Oasis Capital: Teacher Xu, you previously believed that only search engines in history achieved "technology and business." In the large model and Agent era, what do you think could become the new technology-business integration point in your vision for the large model era?

Yinghui Xu: I think there are already signs. The era of finding information through Google will still exist in the future, but a considerable portion of information acquisition will happen through chatbots. Chatbots currently have the biggest problem of being knowledge-oriented, professional Q&A — this will spawn many vertical information engines, for example divided into broad categories like medical and entertainment. Medicine may be a large knowledge engine, and the business model of this knowledge engine will change in the future. Commerce may be integrated into the information flow — for example, personalized health supplement recommendations, local restaurant recommendations matching personal dietary habits... The information flow generated by large models will embed reasonable commercial value content, similar to how commerce and search engines became integrated in the search era.

AI2Apps paper: https://arxiv.org/pdf/2404.04902.pdf AI2Apps code: https://github.com/Avdpro/ai2apps