A Brief Discussion on Trustworthiness and Controllability in Large Language Models

绿洲资本绿洲资本··0·1

Closing In Step by Step

The World Artificial Intelligence Conference (WAIC) opened today in Shanghai, drawing attention from entrepreneurs, academics, and governments worldwide. Meanwhile, OpenAI also announced today that it is forming a new team to develop methods for steering and controlling "superintelligence." AI safety has long been a key focus area for Oasis Capital, which has conducted extensive interviews with scholars globally on this topic. Drawing on cutting-edge academic research and numerous interviews, we have compiled a two-part series on safety — this is the first installment, aimed at helping readers understand how to make large model outputs more trustworthy and controllable. Enjoy.

Part I: Two Schools of Thought

When it comes to enhancing the trustworthiness and controllability of LLMs, academia currently has two main approaches:

  1. End-to-End Conversational Models (E2E): based on improving the model itself
  2. Pipeline-based Systems: based on controlling user inputs and outputs

I. End-to-End Conversational Models

Optimizing end-to-end dialogue models represents the most closely watched direction in academia for improving safety. Progress in this area depends heavily on advances in controlled text generation, where controllability can be viewed as a critical attribute in dialogue response generation. Improvements generally target three main stages: preprocessing, training, and inference.

1. Data Preprocessing

An LLM's responses depend primarily on its training corpus, so removing unsafe data and adding safety data during preprocessing can directly and effectively shape the final behavior of a dialogue system. OpenAI's GPT series models use massive amounts of text data in training, collected from multiple sources to cover diverse domains and languages as comprehensively as possible; Google used multilingual datasets in BERT and attempted to increase training data diversity; IBM Watson employed domain-specific datasets in its deep QA models to ensure trustworthiness and accuracy in particular fields.

"Recipes for Building an Open-Domain Chatbot" summarizes multiple methods for cleaning unsafe and sensitive data, including keyword-based and author-based filtering. Keyword-based cleaning involves setting safety filters and raising confidence thresholds to filter training data — this has become nearly standard practice for building dialogue systems. On the other hand, as social media data has been increasingly incorporated into training sets in recent years, some studies have found that blocking certain specific authors can effectively filter potentially unsafe content. Beyond removing potentially unsafe data, adding parallel data to promote data fairness is another preprocessing approach.

Data preprocessing is a perennial topic in machine learning and one of the most important behavioral correction methods in nearly all ML model training. This has given rise to a relatively mature industry chain, including data annotation companies, specialized data companies, knowledge graph companies, and graph database companies. Representative companies include Diffbot, Grakn AI, TopQuadrant, Neo4j, TigerGraph, and ArangoDB.

2. Training Process

Model behavior is determined by its parameters, which are learned during pre-training or fine-tuning. Therefore, shaping safe behavior during the training phase of a dialogue system is an important lever.

It must first be noted that existing methods for controlling LLM natural language generation are all imperfect. There are mainly three categories:

  1. Class-Conditional Language Models (CC-LMs), such as CTRL, attempt to control text generation by conditioning on control codes — a set of attribute variables characterizing data sources. However, using specific control codes reduces sample diversity across prompts, as samples typically resemble the data source of the control code;

  2. Using discriminators to guide decoding for controlling language models, as seen in existing models like Weighted Decoding and PPLM. The drawback of this approach is its heavy computational cost;

  3. Generative Discriminator (GeDi), which uses a generative discriminator to perform two parallel forward passes on all candidate words to compute the Pθ distribution, thus avoiding separate forward passes for each candidate next word.

Among these three approaches, GeDi outperforms the other two, but still remains an approximation rather than a perfect solution.

a) CTRL (CC-LMs)

"CTRL: A Conditional Transformer Language Model for Controllable Generation" proposes a novel model design that prepends a control code to sentences in the training corpus — a direct and effective modeling method where desired attributes are formalized as control codes. CTRL is a conditional language model that always conditions on control code c and learns the distribution p(x|c), decomposed using the probability chain rule.

CTRL learns pθ(xi|x<i, c) by training on raw text sequences with control codes. After minimal preprocessing, a single example sequence containing n tokens is embedded as a sequence of n corresponding vectors in Rd. Academia has applied this approach to safety and style control, assigning safe or unsafe control codes to each training example, thereby generating safe text during inference.

b) GeDi

"GeDi: Generative Discriminator Guided Sequence Generation" proposes a method for guiding sequence generation through a generative discriminator. GeDi guides each generation step by computing classification probabilities for all possible next tokens via Bayesian rule, normalizing two broad conditional distributions — one conditioned on desired attributes or control codes, and another conditioned on attributes to be filtered or anti-control codes.

Research found that GeDi provides significantly better controllability than previous controllable generation methods, with generation speed markedly faster than other theoretical approaches for achieving controllability, while substantially reducing toxicity in GPT-2 and GPT-3 while maintaining language fluency.

To validate GeDi-guide's zero-shot topic generation capability, researchers trained four CC-LMs on four topics (World, Sports, Business, and Science/Tech), with each CC-LM trained on three topics and tested zero-shot on the remaining one. Topic relevance was evaluated using RoBERTa. Experimental results showed that GeDi-guided text generation possesses strong zero-shot generalization ability, likely because generative classifiers can perform zero-shot classification on unspecified topics from learned word embeddings.

c) Prompt Tuning

"Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing" introduces a new paradigm for NLP research called "prompt-based learning." Unlike traditional supervised learning, which directly models text probability by receiving input x and predicting output y as P(y|x), these models first modify the original input x into a text string prompt x' with some unfilled slots, then use the LLM's probability to fill them and obtain the final string x̂, from which the final output y is derived.

d) Diffusion-LM

To handle more complex control objectives, researchers proposed Diffusion-LM — a non-autoregressive language model based on continuous diffusion models. Diffusion-LM denoises Gaussian vector sequences into word vectors, producing a series of intermediate latent variables. The continuous, hierarchical nature of these intermediate variables enables a simple gradient-based algorithm to perform complex, controllable generation tasks.

Reference link: https://zhuanlan.zhihu.com/p/604024749

e) RLAIF

Reinforcement Learning (RL) is also a currently popular method for correcting LLM outputs. The core reward function in RL is provided by scoring models or safety detectors, while Reinforcement Learning from Human Feedback (RLHF) has proven effective at better aligning LLM outputs with human values.

Building on RLHF, Constitutional AI: Harmlessness from AI Feedback designed an RL model based on AI feedback to better generate harmless LLMs. The paper hopes this design can serve as a constitution for the AI field, supervising and guiding the safety of other models' outputs.

The figure above shows the basic steps of the Constitutional AI (CAI) process, which includes a supervised learning (SL) phase and a reinforcement learning (RL) phase. Both critique and AI feedback are guided by a small set of principles drawn from the constitution.

3. Inference Stage

The training stage requires substantial resources and costs to build new models. By contrast, most methods used during the inference stage operate through a simple plug-in without retraining, making inference-stage optimization an important research area for increasing LLM controllability.

a) PPLM

Plug and Play Language Models: A Simple Approach to Controlled Text Generation proposed a plug-and-play fine-tuning intervention method based on conditional probability p(x|a) — generating text x given some attribute a. This approach can reduce the sampling probability of certain undesirable words to zero. PPLM operates during inference and requires no additional changes during training. However, because it trains while generating, it is relatively inefficient.

The PPLM model flow consists of three parts:

  1. Predict the attribute classification p(a|x) of text generated by the language model through a classifier;

  2. Update the language model's internal historical parameters based on gradients backpropagated from the attribute discrimination in step 1, increasing the likelihood that the model predicts the desired attribute;

  3. Then perform nucleus sampling from the updated parameters to generate new words.

For computational efficiency, one can choose to modify only the delays within a recent window of the past, shown as the red region in the figure above.

The figure above shows PPLM's optimization principle. The target sentence is shown as a black dot, first pushed in the direction of maximizing log p(a|x), then pushed in the direction of maximizing log p(x).

b) FUDGE

However, PPLM still requires updating large model parameters, resulting in slower inference speed. As an improvement over PPLM, FUDGE does not update any parameters in the model; instead, it introduces a discriminator to predict whether the ongoing generated text conforms to the desired attribute. Reference link: https://www.bilibili.com/video/BV1mu411B7H4/?vd_source=e0a3ca4aef9432f03a83ef0b67fa4445

For a pretrained LLM, FUDGE can control the language model to produce text satisfying specific attributes using only the output probabilities of the language model. FUDGE theoretically performs a Bayesian decomposition of conditional generation probability, learning predictors for attributes from partial sequences and using these predictors' outputs to adjust the original language model's probability distribution, thereby producing text with specific attributes.

II. Pipeline-Based Systems

Unlike model-level improvements, pipeline-level improvements focus on the human-machine interaction flow. The core idea is using different modules or strategies to address various safety issues. For example, most currently deployed dialogue systems will respond with something like "let's change the subject" to avoid continuing the conversation after detecting a sensitive topic.

1. User Side

User utterances in context may trigger serious safety issues. When facing unsafe contexts, so-called "safe responses" are typically used — directly replying with specific statements such as "I'm just a language bot." Therefore, detecting whether context is sensitive and formulating corresponding refusal strategies are key parts of pipeline systems.

It's worth noting that user safety detection should have higher robustness and generalizability, because user utterances are always unrestricted and any form of conversation may occur. Recipes for Building an Open-Domain Chatbot introduced a sensitive topic classifier to identify "politics," "religion," "drugs," "medical advice," and "NSFW" content in context. When sensitive topics are detected, preset responses are triggered.

Additionally, when user speech involves hate speech, dialogue systems can potentially generate counter-responses to stop and prevent the spread of hate speech through hate speech datasets and external knowledge systems based on knowledge graphs.

2. Bot Side

Bot-side pipeline improvements occur between model generation and message display in the user interface. At this stage, the system performs final checking and editing of generated responses. The most common strategy is "targeted refusal," where a detector withdraws unsafe content after detection and requests the LLM to regenerate. In some severe cases, human intervention is also necessary.

Bot-side content detection is not an emerging field; several reference companies already exist:

  1. Unitonomy: An AI-driven content moderation company whose products can automatically identify and remove inappropriate content. Unitonomy's technology is based on machine learning and natural language processing algorithms that can automatically classify, label, and review various types of content.

  2. Two Hat: A content moderation company targeting the social media space, whose products can automatically identify and remove pornography, violence, hate, and other harmful content. Two Hat's technology is based on machine learning and natural language processing algorithms that can automatically classify, label, and review various types of content.

  3. Besedo: A global leading content moderation service provider whose products can automatically identify and filter various types of harmful content. Besedo's technology is based on machine learning and AI algorithms that can automatically classify, label, and review various types of content.

Part II: Some Trends and Directions

1. Interpretability of Safety

Detecting and evaluating safety issues is a complex task, but its decision-making process can be simplified through interpretable task decomposition. Approaching from multiple dimensions such as offensiveness detection, target identification, and implicit statement reasoning allows more intuitive reasoning about toxic and discriminatory content detection.

To improve the credibility of moral judgments, it's necessary to evaluate which moral principles a given utterance conforms to or violates. Although safety-related research attempts to achieve interpretability, how to ensure these explanations are accurate and align with human judgment criteria is a direction requiring more research.

Mathew et al. proposed the Hatexplain model, which visualizes the decision-making process by highlighting influential text in the input while providing category labels. Such visual information helps understand attacks at the dialogue system input stage and enables point-to-point defense to generate safer responses.

2. Self-Learning Safety

The identification and resolution of safety issues is an ongoing process. As times change, new domains and topics emerge bringing new safety issues, so safety evaluation methods also need continuous updating. The most direct and effective method for continuous safety optimization is collecting data from new domains and fine-tuning models — this is what some benchmark datasets are doing. However, this data-driven approach has significant limitations in terms of efficiency and cost during data collection and annotation.

By contrast, learning from user interaction data to enable continuous model evolution seems like a one-and-done solution. LLMs can judge the safety of generated content through user feedback during interactions. For example, after detecting user corrective feedback, the LLM apologizes to the user and corrects its behavior, thereby gaining self-correction capability. Large amounts of feedback data enable models to continuously optimize and address emerging safety issues self-sufficiently.

Additionally, improving safety modules' generalization capability to new domains is critical. This allows models to quickly resolve new issues when encountered, rather than passively relying on new data provision. To strengthen understanding of new safety issues, incorporating offline knowledge (knowledge graphs) and online knowledge (internet search engines) into safety modules is also an optional solution.

3. Adversarial Examples

Adversarial examples refer to input samples formed by deliberately adding subtle perturbations to datasets, which cause models to produce incorrect outputs with high confidence. Such constructions can be built manually, through templates, or model-based.

How to generate adversarial examples is a popular direction in LLM trustworthiness and controllability. OpenAI has released some adversarial examples to test its models' robustness and reliability. DeepMind researchers have also used generative adversarial networks (GANs) to generate adversarial examples to evaluate model robustness and reliability. More advanced adversarial examples can more comprehensively check whether LLMs have vulnerabilities or biases, thereby better evaluating model trustworthiness.

4. Multimodal Content Safety

With the rise of multimodal pretrained models (such as NÜWA, DALL-E 2, GauGAN2, etc.), AI has demonstrated powerful cross-modal generation capabilities. What follows are safety issues brought by multimodal content including audio, images, and video. In current research, harmful images are the most common multimodal safety issue, produced by combining simplified text descriptions with relevant background images.

Images can express virtually any safety concern (offensiveness, violence, racism, etc.), and their often humorous or satirical design leads to more subtle expressions of harm. Moreover, multimodal interaction involves more complex and far-reaching research areas — such as identifying whether a user has self-harm tendencies, or whether flagging prohibited images violates ethical norms — all of which represent worthwhile directions for safety research.

5. OpenAI's Perspective

On the 6th of this month, OpenAI published "Our approach to AI safety," focusing on how to ensure the safe building, deployment, and use of AI systems. The article states that before releasing any new system, OpenAI conducts rigorous testing, engages external experts for feedback, improves model behavior through techniques such as reinforcement learning from human feedback (RLHF), and builds extensive safety and monitoring systems. They believe that powerful AI systems should undergo strict safety evaluations. Regulation is needed to ensure such practices are adopted, and OpenAI actively works with governments to determine the best form of such regulation.

OpenAI acknowledges in the article: despite efforts to prevent foreseeable risks before deployment, what can be learned in the lab is limited. Despite extensive research and testing, it remains impossible to predict all the beneficial ways people will use technology, nor all the ways they will misuse it. This is why OpenAI believes that learning from real-world use is a critical component of creating and releasing increasingly safer AI systems over time.

Regarding the accuracy of LLM content, improving factual accuracy is a priority for OpenAI and many other AI developers, and progress is being made. By leveraging user feedback on ChatGPT outputs flagged as incorrect against primary data sources, GPT-4's factual accuracy has been improved. GPT-4 is 40% more likely to produce factual content than GPT-3.5. OpenAI believes there remains substantial work to further reduce the likelihood of ChatGPT generating hallucinated content, and to educate the public about the current limitations of these AI tools. Addressing safety issues also requires broad debate, experimentation, and engagement.

Additionally, in its published article "Lessons learned on language model safety and misuse," OpenAI states: There is no silver bullet for responsible deployment, and outlines several primary approaches OpenAI is currently taking to increase model controllability: pre-training data management and filtering, fine-tuning models to better follow instructions, risk analysis for potential deployment, providing detailed user documentation, building tools to screen for harmful model outputs, reviewing use cases against OpenAI's policies, monitoring for signs of misuse, and researching model impacts.

Part III: Some Takeaways

  1. Controllability and trustworthiness are among the current key research directions for LLMs, but academia has yet to find a perfect solution — progress can only be made step by step across each stage of the pipeline
  2. AI safety (monitoring, rule self-learning, adversarial testing, etc.) is a direction worth betting on
  3. Controlled language generation approaches such as GeDi and Diffusion-LM merit attention
  4. Given the success of RLHF, similar toxicity-reduction approaches through reinforcement learning such as RLAIF are also worth watching
  5. Multimodal models, due to the injection of multiple data modalities, introduce greater safety and controllability risks

________

References: 1. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for Building an Open-Domain Chatbot. 2. Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A Conditional Transformer Language Model for Controllable Generation. 3. Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2021. GeDi: Generative Discriminator Guided Sequence Generation. In Findings of the Association for Computational Linguistics: EMNLP 2021. 4. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing — Authors, Pengfei Liu, Weizhe Yuan 5. Diffusion-LM Improves Controllable Text Generation Xiang Lisa Li, John Thickstun 6. Constitutional AI: Harmlessness from AI Feedback, Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell 7. DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature. 26Jan2023. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, Chelsea Finn https://github.com/eric-mitchell/detect-gpt 8. Safety Analysis in the Era of Large Language Models: A Case Study of STPA using ChatGPT, Yi Qi, Xingyu Zhao, Xiaowei Huang 9. On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective, Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Microsoft Research, City University of Hong Kong, Microsoft STCA, Carnegie Mellon University 10. On the Opportunities and Risks of Foundation Models, Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman 11. Lessons learned on language model safety and misuse, OpenAI, 2022.3 12. Adversarial prompting for black box foundation models, Natalie Maus, Patrick Chao, Eric Wong, and Jacob Gardner. arXiv preprint arXiv:2302.04237, 2023 13. Recent Advances towards Safe, Responsible, and Moral Dialogue Systems: A Survey, JIAWEN DENG, HAO SUN 14. Rohan Kshirsagar, Tyrus Cukuvac, Kathy McKeown, and Susan McGregor. 2018. Predictive Embeddings for Hate Speech Detection on Twitter. 15. Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and Play Language Models: A Simple Approach to Controlled Text Generation. 16. Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021. Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP. Transactions of the Association for Computational Linguistics 9 (2021). 17. Denise Sekaquaptewa, Penelope Espinoza, Mischa Thompson, Patrick Vargas, and William von Hippel. 2003. Stereotypical explanatory bias: Implicit stereotyping as a predictor of discrimination. 18. Ashish Sharma, Adam Miner, David Atkins, and Tim Althoff. 2020. A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 19. Shivam Sharma, Md Shad Akhtar, Preslav Nakov, and Tanmoy Chakraborty. 2022. DISARM: Detecting the Victims Targeted by Harmful Memes 20. Md. Shad Akhtar, Dimitar Dimitrov, Giovanni Da San Martino, Hamed Firooz, Alon Halevy, Fabrizio Silvestri, Preslav Nakov, and Tanmoy Chakraborty. 2022. Detecting and Understanding Harmful Memes: A Survey. 21. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The Woman Worked as a Babysitter: On Biases in Language Generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) 22. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex Machina: Personal Attacks Seen at Scale. 23. Laura Hanu and Unitary team. 2020. Detoxify. Github. https://github.com/unitaryai/detoxify

Oasis Capital is a next-generation venture capital firm in China, dedicated to discovering the most vital entrepreneurs of the coming decade and growing alongside them to create long-term value. "Championing Vitality" is our vision and mission. This vitality — what we call Vitality — captures both the direction of structural transformation in our era and the resilience and evolutionary drive of entrepreneurs themselves.

We focus on early and growth-stage investments, writing checks from $3 million to $30 million, with particular emphasis on robotics, artificial intelligence, and technology-enabled services — backing China's technology-driven upgrade of services across industries.