ChatGPT Goes Online, the Final "Seal" Is Broken | BlueRun Ventures Share --- When ChatGPT's browsing feature went live, the AI community let out a collective sigh — this was the moment many had been waiting for. For the average user, the appeal is obvious: real-time information access means answers that are no longer frozen in time. Ask about last night's game, today's stock price, or this morning's breaking news, and you'll get a response that actually reflects the present moment. But the implications run deeper than convenience. Until now, large language models have operated in something of a time capsule, their knowledge cut off at a fixed point. The browsing capability effectively removes that ceiling. It's not just about freshness of data — it's about expanding the model's potential problem space to encompass everything happening right now. From BlueRun Ventures' perspective, this development sits at the intersection of two trends we've been tracking closely: the rapid evolution of AI infrastructure, and the expanding surface area of what AI applications can realistically address. The "seal" in our title refers to more than just a technical limitation. It represents the final barrier between AI systems and the live, messy, constantly-updating web that humans navigate

蓝驰创投蓝驰创投··0·0

The storm that ChatGPT has unleashed is sweeping through everything.

Today, OpenAI dropped another bombshell — it launched plugins for ChatGPT, enabling the AI to use tools, connect to the internet, and run computations.

Previously, ChatGPT's training data cut off at September 2021, making it unable to accurately answer questions about events after that date. Now, that final restriction has been lifted. The third-party plugins already supported cover nearly every aspect of daily life — food, clothing, housing, transportation, work, and study. This article gives you a taste of ChatGPT with its "seal" removed.

Since its launch last November, ChatGPT has been used by countless people who have consistently demanded that the large language model be connected to more data in various ways. On March 24, OpenAI finally announced it was partially lifting ChatGPT's inability to access the internet.

OpenAI's solution is to use third-party plugins as bridges, allowing the AI to "see" external data in a relatively secure environment. Yesterday, the organization released its first batch of ChatGPT plugins, created by Expedia, FiscalNote, Instacart, KAYAK, Klarna, Milo, OpenTable, Shopify, Slack, Speak, Wolfram, and Zapier.

Specifically, plugins now let you use ChatGPT to perform the following:

  • Retrieve real-time information: such as sports scores, stock prices, breaking news, etc.
  • Retrieve knowledge base information: such as company documents, personal notes, etc.
  • Perform actions on behalf of users: for example, booking flights, ordering food, etc.

Beyond this, OpenAI itself is providing two plugins — a web browser and a code interpreter — and has open-sourced the code for a knowledge base retrieval plugin. Now any developer can build their own plugins to augment ChatGPT's information repository.

Access to the plugin alpha has been expanded to more users and developers on the waitlist. While OpenAI says it is prioritizing a small group of developers and ChatGPT Plus subscribers, it plans to open access more broadly in the future.

To give you an intuitive sense of what's possible, here's an example: users can select and install the Wolfram plugin from ChatGPT to enhance its computational intelligence.

All data in Wolfram Alpha comes from academic websites, published journals, and scientific institutions — its authority is absolutely guaranteed. Doesn't this make ChatGPT feel even more powerful?

01 Overview

While today's large language models can already accomplish various tasks, their capabilities remain limited. Training data is the only information they can learn from, and this content may be outdated while needing to accommodate all of people's needs. Furthermore, the only capability language models have out of the box is outputting text. This text may contain useful instructions, but to actually follow them requires additional processing by humans.

While not a perfect analogy, plugins can serve as the "eyes and ears" of language models, enabling them to access new, private, or specific information not contained in training data.

In response to explicit user requests, plugins can also enable language models to perform safe, constrained actions on users' behalf, thereby increasing the overall utility of the system.

OpenAI anticipates that an open standard for unified AI application interaction will emerge, and they are making an early attempt at such a standard.

Today, OpenAI is gradually opening up plugins built by early collaborators for ChatGPT users, starting with ChatGPT Plus subscribers. It is also beginning to roll out the ability for developers to create their own plugins for ChatGPT.

Over the coming months, as safety systems improve, OpenAI plans to let developers using OpenAI models integrate plugins into their own applications, not just ChatGPT.

02 Safety and Broader Implications

Of course, connecting language models to external tools creates new opportunities while also introducing significant new risks.

Plugins offer the potential to address various challenges associated with large language models, including model "hallucinations," tracking recent events, and accessing (with permission) proprietary information sources. By integrating explicit access to external data — such as up-to-date online information, code-based computation, or information retrieved through custom plugins — language models can strengthen their responses with evidence-based references.

These references not only enhance the model's utility but also allow users to evaluate the trustworthiness of model outputs and verify their accuracy, potentially mitigating risks related to overreliance discussed in the recent GPT-4 system card. Finally, the value of plugins may extend beyond existing limitations by helping users handle various new use cases, from browsing product catalogs to booking flights or ordering food.

At the same time, plugins may take harmful or unintended actions, increasing the ability of bad actors to defraud, mislead, or abuse others, thereby raising safety challenges. By expanding the range of possible applications, plugins may increase the risk of negative consequences from erroneous or misaligned actions by models in new domains.

These factors have guided the development of the ChatGPT plugins platform, for which OpenAI has introduced multiple safeguards.

Previously, OpenAI conducted "red team" exercises internally and with external collaborators, simulating many possible relevant scenarios. For example, red teams discovered that if plugins were released without safety measures, they could execute complex prompt injections, send fraudulent emails and spam, bypass safety restrictions, or abuse information sent to plugins.

OpenAI is using these findings to drive safety design mitigations that limit risky plugin behaviors, improve transparency around how and when they run as part of the user experience, and confirm the decision to gradually roll out access to plugins.

Plugins may have broad societal impacts. For example, in a recently published paper, OpenAI researchers found that language models capable of using tools may have greater economic impact than those without tools. More generally, based on findings by other researchers, the current wave of AI technology will significantly affect the pace of job transformation, displacement, and creation.

03 Browsing the Web with ChatGPT

Inspired by a series of works including WebGPT, GopherCite, BlenderBot2, and LaMDA2, allowing language models to read information from the internet strictly expands the range of topics that can be discussed beyond the training corpus, incorporating fresh, current information.

The example below shows how browsing opens up an experience for ChatGPT users that previous models might have politely declined by pointing out that their training data didn't include sufficient information. In this case, ChatGPT retrieves recent Oscar information (awarded March 13, 2023) and then performs a familiar bit of ChatGPT poetry. Browsing becomes a way to add to the experience.

Q: Can you tell me which person/film won the Oscars in these categories?

  • Best Actor
  • Best Score
  • Best Picture
  • Best Supporting Actor

Then come up with a poem connecting them.

ChatGPT will give you a series of search results, and you can click directly to check the relevant information sources.

Beyond delivering practical value to end users, the ability of language and chat models to conduct thorough and interpretable research shows promise for scalable alignment work.

Notably, the plugin's text-based web browser is limited to making GET requests, which reduces but does not eliminate certain categories of security risks. The browsing plugin is scoped to information retrieval and excludes "transactional" operations such as form submission — which carry greater security risks.

The browsing feature uses the Microsoft Bing Search API to retrieve content from the web. As such, it inherits substantial work from Microsoft on source reliability and information accuracy, as well as "safe mode" measures to prevent retrieval of problematic content. The plugin operates in a separate service, with ChatGPT's browsing activity isolated from the rest of the infrastructure.

To respect content creators and comply with web norms, the ChatGPT browser plugin's user agent token is ChatGPT-User, configured to respect websites' robots.txt files. This may occasionally result in "click failures," indicating that the plugin is following a website's instructions to avoid scraping it. This user agent is used only for direct actions on behalf of ChatGPT users, not for scraping the web in any automated fashion. OpenAI has also published IP egress ranges and implemented rate limiting to avoid sending excessive traffic to websites.

04 Code Interpreter

Experimental ChatGPT model that can use Python to handle uploads and downloads

OpenAI provides the model behind ChatGPT with a Python interpreter that works in a sandboxed, firewalled execution environment, along with some temporary disk space. Code run by the interpreter plugin is evaluated in a persistent session that remains active during the chat conversation (with an upper timeout limit), and subsequent calls can build on each other. Currently, this feature supports uploading files to the current conversation workspace and downloading work results.

Click "Finished Calculating" in the image:

From initial user research, OpenAI identified several valuable use cases for the code interpreter:

  • Solving quantitative and qualitative mathematical problems
  • Conducting data analysis and visualization
  • Converting files between formats

Safety Mechanisms

The primary safeguard for connecting ChatGPT models to a programming language interpreter is proper sandboxing of execution, so that AI-generated code does not have unintended side effects in the real world. OpenAI executes code in a secure environment and uses strict network controls to prevent external internet access by the executing code. Additionally, OpenAI sets resource limits for each session.

Disabling internet access limits the functionality of the code sandbox, but this may be the safest initial form of AI-assisted programming. Third-party plugins are designed with safety as the top priority before connecting ChatGPT to the outside world.

Data Retrieval

The open-source retrieval plugin enables ChatGPT to access personal or organizational information sources with permission. It allows users to obtain the most relevant document snippets from their data sources — such as files, notes, emails, or public documentation — by asking questions or expressing needs in natural language.

As an open-source and self-hosted solution, developers can deploy their own versions of the plugin and register them with ChatGPT. The plugin leverages OpenAI embeddings and allows developers to choose a vector database (such as Milvus, Pinecone, Qdrant, Redis, Weaviate, or Zilliz) to index and search documents. Information sources can be synchronized with the database using webhooks.

The retrieval plugin allows ChatGPT to search content vector databases and add the best results to the ChatGPT session. This means it is not subject to any external influence; the primary risks are data authorization and privacy. Developers should only add content to their retrieval plugins that they have the right to use and that can be shared in users' ChatGPT sessions.

Third-Party Plugins

Using third-party plugins on ChatGPT looks like this:

OpenAI has prepared a descriptive manifest file for third-party plugins, including a machine-readable description of the plugin's functionality, invocation methods, and user-facing documentation.

Steps to build a plugin:

  1. Build the API endpoints you want the language model to call (this can be a new API, an existing API, or a wrapper around an existing API designed specifically for LLMs).
  2. Create an OpenAPI specification documenting your API, along with a manifest file linking to the OpenAPI specification and containing some plugin-specific metadata.

When starting a conversation on chat.openai.com, users can select which third-party plugins they want to enable. Documentation for enabled plugins is shown to the language model as part of the conversation context, allowing the model to call the appropriate plugin APIs as needed to fulfill user intent. Currently, plugins are designed to call backend APIs; OpenAI is also exploring plugins that can call client-side APIs.

OpenAI says it is working to develop plugins and promote them to a broader audience.

This means the storm brought by ChatGPT is sweeping through everything.

This article is reprinted from Jiqizhixin. Reference: https://openai.com/blog/chatgpt-plugins

BlueRun Ventures has always maintained its focus on and investment in AI. We have formed the "BlueRun AGI Pioneer Club" to explore the path to Artificial General Intelligence (AGI). Are you an AGI pioneer? Welcome to scan the QR code or click "Read More" to register.

More Exciting Content

Originating in Silicon Valley, BlueRun Ventures was established in 2005 as a venture capital firm focused on early-stage startups.

Currently, BlueRun Ventures manages multiple USD and RMB dual-currency funds in China, with assets under management exceeding RMB 15 billion, making it one of the largest early-stage funds in the country. It invests primarily at the Pre-A and Series A stages, covering hard tech and innovative interaction, enterprise technology, new consumer, and healthcare sectors. It has invested in over 150 startups, including Li Auto, Waterdrop, QingCloud, Guazi.com, Qudian, Songguo Mobility, Ganji.com, Monster Charging, Yuntu Semiconductor, Machenike, Yunsheng Intelligence, Anxin Wangdun, and BioMap.

BlueRun Ventures has been ranked #1 in Zero2IPO's "China's Top 30 Early-Stage Investment Institutions" and ChinaVenture's "China's Best Early-Stage Venture Capital Institutions TOP30," and was named one of Preqin's Top 10 VC Fund Managers for consistent high-return performance globally.

Additionally, BlueRun Ventures has received consecutive recognition from Forbes China, 36Kr, Cyzone, Caixin Media, CBNweekly, Jiemian, and other media organizations as "China's Best Early-Stage Institution of the Year," "China's Top Venture Capital Firm," "Most Entrepreneur-Friendly Early-Stage Institution of the Year," and "Most Influential Early-Stage Institution of the Year."