AIGC "Peak Series" | Dr. Long Bo: Technical Outlook, Application Prospects, and Entrepreneurial Pitfalls of Large Language Models

On March 15, OpenAI released GPT-4, another leap in AI capabilities that triggered a viral wave of discussion. When most people talk about ChatGPT, they tend to approach it from a technical perspective, projecting all sorts of possibilities about the "future." Some have even predicted that human civilization will shift from carbon-based to silicon-based. But if we dig deeper and look further, we'll find that ChatGPT has developed in a way that completely mirrors how humans learn language. Different angles, different horizons — vastly different conclusions. Whether we can see through the surface to grasp the essence will determine whether we can seize the boundless opportunities that ChatGPT is about to unleash. On March 10, the third installment of Code Brain's "Toward the Future" AIGC series took place. This session featured Dr. Long Bo, who leads the commercial AI research team at Meta/Facebook, previously served as head of LinkedIn's AI team and co-founded its recommendation and machine learning teams, was VP of JD.com's search and recommendation platform division, and was a founding team member at News Break. He delivered a compelling talk covering multiple dimensions including "technical reflections on large language models, application prospects, and entrepreneurial pitfalls."

Selected excerpts from the talk below

01

A Human-Centered Perspective for Navigating an Uncertain Future

Today's topic covers the technical applications and entrepreneurial pitfalls of large language models, and I'd like to approach this from a human-centered perspective. After ChatGPT's massive impact on human capabilities, everyone's been asking: how can machines be this powerful? They can understand us, help us, even surpass us in certain ways. Because of these concerns, people easily fall into viewing things from the machine and technology angle. Some say that with ChatGPT, future civilization might be created by machines — that human civilization will shift from carbon-based to silicon-based. This is the classic machine-centric viewpoint. But if you look at the entire technical evolution of large language models, you'll discover that ChatGPT developed in complete accordance with how humans learn language. Humans created ChatGPT. Humans created large language models. From the perspective of human language acquisition, ChatGPT's technical trajectory is quite clear. Its stunning performance also owes much to OpenAI's long-term commitment to this technical direction.

From an application standpoint, such a powerful ChatGPT should find interesting uses across every field. But precisely because it's so hot right now, everyone is thinking about how to deploy it, start businesses around it, or achieve the next technical breakthrough. This inevitably breeds unrealistic, irrational expectations and ideas. Today, I want to return to human fundamentals to share some thoughts that I hope can offer entrepreneurs and technically ambitious developers practical, user-pain-point-driven perspectives.

02

A Retrospective on Large Language Model Development

In 2019, we presented at the KDD data mining conference on "Deep Learning for Natural Language Processing," where we covered the most important technical developments in NLP. Even then, our conclusion was that NLP's evolution closely matched human language acquisition — follow this path, and technology like ChatGPT would inevitably emerge. Let's recall how humans learn language: first, individual words; then phrases of several words together; then complete sentences; then strings of sentences; and finally, comprehension based on full document context. Throughout this process, a crucial ability gradually develops: understanding the most important keywords in a text and their relationships. Next comes professional language ability, which generally requires deliberate practice — most commonly through reading comprehension exams, where feedback from testing allows you to correct and improve your language skills.

If we review NLP's technical evolution, we find it runs parallel to human language development. The earliest NLP used unigrams — the frequency of individual words — mirroring the first step of human language learning. NLP then advanced to n-grams, using masks in Convolutional Neural Networks (CNNs) to group nearby words for understanding, which outperformed unigrams. Further still came RNNs, which could accumulate signals across entire sentences, surpassing n-grams — corresponding to humans learning whole sentences. Next, sequential modeling in NLP enabled understanding of multiple sentences, achieving massive success in translation. Then came Attention Modeling, a turning point for NLP because it corresponded to a crucial human language ability: understanding keywords and their relationships in context. Why can we read an article clearly even when skimming rapidly? Because in quick scanning, we pick out keywords and grasp their relationships. Built on attention models, Transformers can comprehend broader context, and multi-head attention allows them to understand long documents from multiple angles — style, keywords, sentence relationships, and more.

By this point, transformer-based models were already quite large. How to improve language models further? Before ChatGPT's recent release, this wasn't entirely clear. But following the human language development process, the technical direction becomes clearer: an iterative model of learning and testing, using high-quality feedback to improve — corresponding to deliberate practice in human language learning. Using professional teachers and high-quality materials for feedback to stimulate and enhance previously developed abilities — this is the core idea behind ChatGPT's key technology, RLHF. Before RLHF, researchers had applied GANs to NLP with similar logic, but failed to break through on high-quality annotated data.

03

ChatGPT's Key Success Factor: RLHF

RLHF (Reinforcement Learning from Human Feedback) is widely recognized as ChatGPT's key success factor. By the time ChatGPT emerged, GPT-3 already had massive data and model scale. Its weakness was that dialogue still wasn't good enough — what it wrote didn't fully meet human expectations. We all know that large language models are unsupervised: hundreds of billions of tokens are fed in without dedicated human labeling. This massive data gives large models enormous potential, but the noisy signals mean error rates remain relatively high. To become an expert in a field, you must attend a good university, pursue a PhD, or find a teacher who can provide high-quality knowledge. ChatGPT follows the same logic. Pre-ChatGPT models had vast knowledge but overall quality couldn't rise. Could high-quality data help? So they invested time and manpower to generate high-quality data, hoping to use fine-tuning to unlock the latent capabilities formed by massive data.

They placed data in templates for ChatGPT to interact with humans, helping it understand human intent. The first part uses human annotation to teach the machine what humans want. The second part teaches the machine to be polite and politically correct. So you occasionally notice ChatGPT's responses follow certain principles — the reason being these answers come from standardized data. But such data can't be abundant; early on, the ChatGPT team had only about 40 annotators doing this, but the quality was high. Though smaller in scale, because many potential answers already existed in the large dataset, fine-tuning produced surprisingly good results.

How to make it improve further? Most importantly, it needs to judge quality itself, enabling more flexible, high-quality responses. This data can't be human-written, but ChatGPT already has preliminary capability here. To get it to write ten versions, you only need to give it one, and it completes them instantly. These ten versions are then scored by annotators, and this sample trains a reward model. Another method uses the reward model for scoring — generate one, score it. This shares the same logical principle as GANs. This technology enables it to accurately understand user requests and generate expected, polite, politically correct text — all of which make ChatGPT so impressive.

Looking back at ChatGPT's technical trajectory, you'll find it fully aligns with human expectations for language ability. This shows how important it is to view technical direction through a consistent perspective and principle. With the right direction, a relatively strong engineering team, and repeated iteration — persisting in doing what's right but difficult — good results emerge.

04

Outlook for Large Model Technology

Before discussing ChatGPT's prospects, let's examine its acknowledged limitations.

First, ChatGPT's data lags behind. Currently all data is based on pre-2021 information. Since adding data is extremely expensive, this makes retraining the base model unlikely. New data can't get in, causing knowledge to become outdated. Here are some potential solutions:

• Can the latest data be directly fine-tuned without retraining, which would at least be cheaper? But fine-tuning may fail to guarantee timeliness — with hundreds of billions of parameters, fine-tuning might not influence the model, and model quality could degrade. However, domain-specific fine-tuning could maximize its effect in particular fields. How often to fine-tune and what results to expect remain challenging. Beyond special domains, using fine-tuning to increase timeliness may be a difficult technical route.

• Can search engines be combined with ChatGPT? As we know, Google crawls the web daily and can quickly retrieve the latest content. Could large language models like ChatGPT take keywords to search engines and directly pull back new content? This would give ChatGPT both general large-model capabilities and access to the latest data. For companies like Google, this would be relatively easy to implement.

Another limitation is ChatGPT's "accuracy." ChatGPT sometimes confidently hallucinates — ask it how to make squirrel-shaped mandarin fish, and it might tell you to first catch a squirrel. How to solve this? This matters greatly. Especially when deploying ChatGPT in highly professional domains, this is absolutely intolerable. Current solutions use professional data for fine-tuning — high-quality inputs in that domain, whether human-annotated or professionally accumulated industry data.

Professional knowledge graphs in certain domains can teach it foundational industry knowledge. In chemicals, for instance, relationships between reagents are clearly defined with no room for error. In medicine, drug interactions and side effects have well-established knowledge graphs. Using these industry knowledge graphs as ChatGPT input is relatively feasible. Once ChatGPT understands human intent, it can easily query them and retrieve precise information. This mirrors human behavior — when uncertain, consult a professional book to verify accuracy. So knowledge graphs are one approach.

Additionally, ChatGPT's logical reasoning has drawn criticism. Large language models are fundamentally statistical models, so this is somewhat inevitable. Early on, it couldn't solve even basic logic problems. Simple problems are now solved, but complex ones remain difficult. Many of its good responses draw on accumulated answering experience with high data quality. But when data contains noise or something hasn't appeared before, it makes mistakes. Because it answers based on probability models rather than actual reasoning.

This deficiency may stem from insufficient understanding of思维模式 between languages. When first given IQ tests, it could attempt and answer them. But ask what approach would work better for such IQ problems, and it could actually describe the steps. When the next question required using the described method, its scores improved. It has problem-solving ability but doesn't know how to use it — quite fascinating. Solving logic problems requires learning logical structures, which has been validated. Internally, they found that after training ChatGPT on massive code data, its logic became noticeably more rigorous. Code has strong logical structure; giving it more logical paradigms allows it to expand independently, strengthening its reasoning.

ChatGPT's logic isn't strong, but when we specifically fine-tune with the most logically rigorous data, it may also enhance its归纳整理 abilities. This is just an idea, but I think it's quite likely. Because extracting those patterns — learning seemingly chaotic things — and then combining them may more easily unlock its potential.

To guarantee true reasoning ability, we can also use knowledge graphs plus constitutional learning. Knowledge graphs contain all reasoning; using them as input data teaches logic. Simultaneously, it can learn basic assumptions and premises. To make machines follow certain principles in speech, directly tell the model what to adhere to — this is constitutional learning, which ChatGPT already employs.

If machines can summarize principles from massive data and verify them, their reasoning ability will strengthen — the same as human inductive reasoning. Next, combining constitutional learning with self-learning — having it continuously summarize and apply principles, possibly through self-generated data feedback, model interaction, or a dedicated constitutional-generation model interacting with it — these are frontier ideas that may see validation.

ChatGPT's autonomous learning capability is what people want to see. Since it's already so smart, does GPT-4 become even smarter from GPT-3? RLHF involves human-crafted models and data — could ChatGPT independently seek data for improvement? We ask it a question, it gives an answer; we ask again, it gives another. So ChatGPT itself constantly seeks feedback during user interaction, and can actively pursue goals. When it clearly understands user intent, it can proactively offer multiple options for uncertain questions. Through this iteration, you'll find it increasingly gives direct, accurate answers — perhaps ChatGPT is already doing this.

Furthermore, model interaction enhancement — I believe large language models will develop different roles in the future: some for dialogue, some for language quality control, selecting the best data for ChatGPT, with ChatGPT feeding back to this data quality model. AI may evolve into an ecosystem where different types of language models mutually enhance each other, with different-role models improving through interaction.

Because language model interpretability remains unachieved, current language models are still difficult to work with. For example, we still can't fully understand why given this input, it produces that conclusion. Many researchers are working on this; early studies understood models through input-output correlation. Some current research suggests two keywords in a passage play the most important role. Better interpretability would in turn enhance models' self-iterative learning ability.

---

Application Prospects for Large Language Models

From the perspective of human information needs, the first category is human-machine information interaction, including receiving and outputting information. Output aims to complete tasks; receiving aims to find information. Active information reception is current search engines; ChatGPT may transform search, as its approach enables more efficient information access. Google is quite nervous about this, and new search engines are already emerging. The simplest combination puts ChatGPT and Google side by side as plugins — enter keywords, get answers from both.

As ChatGPT matures, it may synthesize web pages into a most satisfactory answer. This requires two characteristics: first, expertise — once it hallucinates, the harm exceeds search engines. Search engines have references; ChatGPT gives direct answers. Without expertise and accuracy, misguidance is severe, so major breakthroughs are needed here.

Second, its personalization will be excellent. Everyone seeks different aspects, and its answers may be what you find most acceptable, also improving efficiency. Of course, this has downsides — in a sense, it affects human thinking, narrowing what you see. This exceeds the information cocoons on Toutiao; you'll trust the future expert engine, and over time, your thinking patterns will be affected. This is a humanities challenge requiring collective solution.

Passive reception represented by current recommendation engines — visiting a site without search terms or explicit wants, the engine guesses your preferences, with Toutiao as the typical example. Recommendation engines currently rely on endless scrolling and clicking to find preferred content. ChatGPT will make personalized interactive engines truly real — even when you're unsure what to seek, simple interaction can quickly surface your most interesting personalized content, improving efficiency even in passive browsing.

Outputting information aims to help humans complete tasks. Current smart assistants handle only simple tasks, but ChatGPT differs — it accepts very complex tasks, guides you to express needs, clarifies inaccurate output, and helps you complete tasks efficiently.

In the future, LLM + Task API Store may create massive B2B opportunities. LLMs truly solve human-machine interaction; machines completing various tasks need professional task interfaces, like today's App Store — APIs for booking flights, writing articles, helping with investments. When communicating with LLMs, they'll call professional APIs to complete tasks. Tasks are clear commercial applications with defined scope, and people are willing to invest, inevitably generating significant returns. I think the first thing it can solve is revolutionizing cloud services. Once ChatGPT understands, it can execute all cloud services — copy a file, even complete a cloud service configuration file, with all cloud services being strings of these small services.

For example, if you want to start a vertical e-commerce business, you need to build a recommendation engine. Previously, you needed at least one or two people who understood recommendation models, data backends, and frontends to connect everything. Now, with ChatGPT's deep understanding of these tasks and the emergence of numerous APIs, you may soon simply tell it: here are all my product data, I want to build a product recommendation engine, and it automatically completes it.

If you give it raw data to start training recommendation models, model deployment, connecting to user requests, building services — it can automatically complete these too, containing enormous opportunity. I think Microsoft is competing with Google through Bing + ChatGPT, and Google may not lag there. But if Microsoft uses this to excel with Azure, differentiating from AWS in cloud services, it could yield massive gains. Though I haven't heard anyone say this, someone may be working on it.

Human-human information interaction — why involve machines? This has practical potential. Where efficiency needs improvement, user pain points exist; where pain points exist, applications can be mined. For example, when two people don't share a language, communication is difficult. ChatGPT could likely enable simultaneous translation to the degree of free, fluent exchange. Though current simultaneous translation has developed well, ChatGPT will make it more powerful. It will convey subtle intentions, even nuances that same-language speakers might miss, completely eliminating them in translation.

When does human-human communication need efficiency gains? Another example: same language, but vastly different knowledge levels, such as doctor-patient communication. With ChatGPT's involvement, explanations could far exceed what doctors provide. Everyone's thinking about whether it can replace doctors in consultations — this involves professional exchange with certain risks. The best model iteration approach may be participating and self-learning, not simply training. If its data collection method involves full participation, data obtained in such training scenarios could be breakthrough. For example, to train a young doctor, we typically have them follow experienced doctors in patient consultations — immersed in that context, they grow quickly. ChatGPT is the same; if it can ultimately deploy and participate as an effective data collection method, perhaps entrepreneurs can figure out how to practice this.

Beyond one-to-one, there's many-to-many interaction — content production, for instance. Writing a marketing article for many readers, with many already doing this. Less obvious examples: do social platforms need efficiency improvements? Quite possibly. With many friends, you can't finish reading Moments; ChatGPT could likely summarize key points from all friends' daily posts — that's efficiency, and such tools will quickly appear on social platforms.

In meetings, after multi-person interaction, most people's meeting notes are typically scattered. ChatGPT can provide the most concise, precise summaries. In other words, in many-to-many interactions, wherever you have efficiency improvement needs for conversation, ChatGPT can help, because its information summarization and understanding are more efficient.

---

Entrepreneurial Pitfalls for Large Language Models: Blind Technology Worship, Ignoring Costs

Several pitfalls exist for large language model entrepreneurship.

First, with ChatGPT's overheated reception, many are thinking about entrepreneurship without having thought through fundamental principles, blindly worshipping technology — which may not yield good practical results. If you want technical breakthroughs, you truly need to identify where breakthrough points lie and what principles to follow. Second, hardware matters; if hardware can't support it, you can't proceed. This also tells us to consider costs — base model training costs over ten million dollars per run. Even without building base models, one API call costs two cents, and using others' APIs comes with various restrictions. You must understand where costs lie to see profits. Success still depends on user pain points; you must solve problems from the human angle, not deploy technology just because it's amazing — these become entrepreneurial pitfalls.

Entrepreneurship must not worship existing technology, but break through it to build your own technical moat. Only then can you build a company with technical excellence. OpenAI achieved ChatGPT because it didn't pursue short-term economic returns, choosing long-term persistence to ultimately break through. Truly breakthrough technologies like ChatGPT necessarily embody long-termism. Only by firmly believing in your technical trajectory, firmly believing in the user pain points you've identified, and persistently executing, can you see results.