AIGC "Peak Series" | GPT-4: 8+2 Notable but Easily Overlooked Points

In the early hours of March 15, OpenAI released GPT-4, a multimodal pretrained large language model. The boundaries of what AI can and cannot do became a little clearer.

Following ChatGPT's launch last year, Yungang Huang, a partner at Source Code Capital, was among the first investors to arrive in Silicon Valley for in-depth discussions with the OpenAI team. The OpenAI team themselves were surprised by the market's enthusiastic response to ChatGPT. They told Source Code that what they had just released was a 60-point product, the 100-point version was already finished, and they were now working on something that would score 120.

Today's GPT-4 is presumably that "100-point" product: not only is it an exam ace — passing the bar exam in the top 10% — it also performs at the level of top Tsinghua or Peking University test-takers across numerous knowledge-based exams. RLHF has noticeably improved factual reliability, and GPT-4's appropriateness on sensitive and disallowed content has improved dramatically, with "hallucination" significantly reduced. More importantly, the multimodal large model demonstrates high-level image comprehension — it can read text within images with seemingly no loss of capability compared to reading text directly, and can also interpret charts and abstract physics diagrams.

After completing the model last August, OpenAI spent six months fine-tuning alignment and risk mitigation. In fact, all the progress from GPT-3 to now has been foreseeable, yet seeing it in reality is still stunning. One can't help but wonder: what level of intelligence might a 120-point model achieve?

Perhaps the only thing limiting us is our own imagination.

Ding Li, Source Code Capital advisor and CEO of JIMU Docs, has also been closely tracking new technology developments. Following the release, he summarized several noteworthy yet easily overlooked aspects of GPT-4 to share.

1. GPT-4's Chinese capabilities outperform GPT-3.5's English capabilities. This assessment is based on the MMLU dataset: GPT-3.5's English score is 70.1%, GPT-4's English score is 85.5%, and GPT-4's Chinese score is 80.1%.

2. GPT-4's multimodal inputs can be text or images, but output is text-only, and image input remains in the research preview stage and is not yet publicly available.

3. When taking various exams, GPT-4 used a temperature of 0.3 for multiple-choice questions and 0.6 for open-ended questions. This suggests that dynamically combining different temperatures and other hyperparameters for different scenarios could significantly improve application-layer performance.

At the foundation of large language models, there is typically a set of adjustable parameters used to optimize model performance across different scenarios. These are technically known as hyperparameters. End-user products like the ChatGPT UI generally don't offer modification of these hyperparameters, but both the ChatGPT and GPT APIs allow developers to customize them.

Temperature is the most important hyperparameter, with a value range of 0–1. It can be simply understood as the AI's degree of divergence or creative freedom. The higher the temperature, the more creative the AI's output — but with potentially greater uncertainty and weaker logical consistency.

Conversely, the lower the temperature, the more conservative the AI's responses.

At temperature 0, the AI will produce nearly identical answers to the same question every time.

4. Whether or not RLHF is applied basically doesn't affect the model's ability to solve objective questions like multiple-choice problems. The base version without RLHF averaged 73.7% on various exams, while the RLHF version scored 74%.

5. To improve mathematical reasoning capability, GPT-4 incorporated the GSM8K elementary math word problem dataset into its training data.

6. On image input, GPT-4 demonstrates strong OCR capability. The System Card uses examples of GPT-4 interpreting screenshots of the InstructGPT paper and text-containing satirical cartoons.

OCR (Optical Character Recognition) is a technology that converts text in images into editable, searchable text. Simply put, OCR can identify text in images and convert it into a computer-readable text format.

In GPT-4, OCR capability allows the model to recognize text from input images and convert it into text format. This is particularly useful when processing images containing text — for example, parsing headlines, summaries, and body text from news images, or extracting text information from photographed documents.

By combining OCR technology with natural language processing, GPT-4 can understand input data more comprehensively and accurately, and operate across a broader range of scenarios.

7. GPT-4's testing introduced a red team for adversarial testing to identify potential risks. Notably, this red team was composed of professionals from fields including law, medicine, biology, nuclear safety, human-computer interaction, and cybersecurity — not ordinary data annotation workers. The red team found that GPT-4 already possesses the ability to assess whether certain technical details contain security vulnerabilities, can redesign some publicly disclosed compounds, and can even identify mutations that alter pathogenicity. However, it does not yet have the ability to synthesize novel compounds (mentioned in System Card section 2.10: the red team achieved this by integrating molecular search tools to execute web queries against PubChem and obtain SMILES — a text-based notation for describing molecular structures — from plain text). OpenAI has noted that due to these characteristics, GPT-4 may already possess dual-use potential and is working to ensure it does not violate relevant non-proliferation conventions.

8. OpenAI uses a fine-tuned version of GPT-4 to build standalone content filters to maximally filter harmful inputs and outputs in both directions. This filter has seen significant performance improvements due to GPT-4's own enhanced capabilities. OpenAI will publish a paper in the near future with further details on this work.

Additional API-level notes:

1. System messages will have better adherence. That is, preset persona prompts in the API will work more effectively.

2. Both 8k and 32k token versions will be available. Roughly speaking, one token equals about three-quarters of an English word, while one Chinese character equals two tokens. Input, output, and message history are all counted together. Thus for purely Chinese or purely English scenarios: the 8k version supports up to 4,000 Chinese characters or 6,000 English words; the 32k version supports 16,000 Chinese characters or 24,000 English words.

Notes: MMLU dataset: https://paperswithcode.com/dataset/mmlu Elementary math word problem dataset GSM8K: https://paperswithcode.com/dataset/gsm8k