A Brief Discussion on Data Privacy in Large Language Models

In the previous article, A Brief Discussion on LLM Trustworthiness and Controllability, we covered academic approaches to ensuring reliable and controllable LLM outputs. In this piece, we'll examine data privacy and security — specifically, how researchers are working to protect sensitive data in large language models. Enjoy.

On March 20, 2023, ChatGPT experienced a data breach that exposed personal information of some users. In response, Italy's privacy regulator determined that ChatGPT had illegally processed personal data, violating privacy rights and GDPR regulations. Italy became the first country to ban ChatGPT, sparking debate across the EU about whether stricter measures were needed to regulate the technology.

Nearly every online service collects our personal data, potentially using it to train LLMs. Yet how models actually use this training data remains difficult to determine. If sensitive information — geolocation, health records, identity data — is used in training, data extraction attacks against the model could lead to massive privacy breaches. The paper "Are Large Pre-Trained Language Models Leaking Your Personal Information?" demonstrates that due to LLMs' memorization of training data, they do indeed risk leaking personal information during conversations, with that risk increasing as the number of examples grows.

Information leakage stems from multiple causes. Some are structural, tied to how models are built; others result from poor generalization, memorization of sensitive data, and related factors. In what follows, we'll first outline basic data leakage mechanisms, then examine common attack vectors — privacy attacks, jailbreaks, data poisoning, and backdoor attacks — and conclude with current research on privacy protection.

I. Threat Modeling

A basic LLM threat model comprises a general model environment, various actors, and sensitive assets. Sensitive assets include training datasets, model parameters, model hyperparameters and architecture. Actors include: data owners, model owners, model consumers, and adversaries. The diagram below illustrates assets, actors, information flows, and possible operational flows within this threat model:

In this foundational threat model, data owners hold privacy data assets, model owners hold model parameter and configuration assets, and model consumers interact with the model via API or user interface. Adversaries attempt to obtain privacy data assets or model parameter assets through various means.

II. Privacy Attacks

Privacy attacks fall into four main categories: membership inference attacks, reconstruction attacks, attribute inference attacks, and model extraction.

1. Membership Inference Attacks (MIA)

Membership inference seeks to determine whether an input sample x was part of training set D. While users' private data is typically kept confidential, adversaries can still make inferences using non-sensitive information. For example: if members of an exclusive club are known to favor purple sunglasses and red leather shoes, encountering someone wearing purple sunglasses and red leather shoes (non-sensitive information) allows us to infer they likely belong to that club (sensitive information).

Membership inference attacks are currently the most widely studied form of privacy attack, first introduced by Shokri et al. in "Membership Inference Attacks Against Machine Learning Models." The paper shows that such attacks require only knowledge of the model's output prediction vector and target supervised machine learning models. With access to model parameters and gradients, significantly more accurate membership inference becomes possible.

A typical approach is the shadow model attack: training a shadow model on a known, accessible dataset, then querying it to extract sensitive information.

Beyond supervised learning models, generative models such as GANs and VAEs are also vulnerable to membership inference. "GAN-Leaks: A Taxonomy of Membership Inference Attacks Against Generative Models" examines GANs' vulnerabilities; "LOGAN: Membership Inference Attacks Against Generative Models" explores how other generative models respond to such attacks and describes retrieval-based attacks leveraging knowledge of data generation components; "Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents" demonstrates that masked language modeling (MLM)-based models are also susceptible to MIA, with training data membership determinable in certain cases.

Conversely, membership inference can also serve model security auditing — data owners can use it to audit black-box models. "Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?" describes how data owners can detect unauthorized use of their data.

"Membership Inference Attacks Against Machine Learning Models" investigates the connection between overfitting and black-box membership inference, measuring overfitting's impact on attack accuracy by training models on identical datasets across different MLaaS platforms. Experiments show that overfitting causes privacy leakage, though it is not the sole cause — some well-generalized models are actually more prone to membership leakage.

2. Reconstruction Attacks

Reconstruction attacks attempt to reconstruct multiple training samples along with their training labels — that is, given output labels and partial knowledge of certain features, to recover sensitive features or complete data samples. For instance, through model inversion, information obtained from model interfaces can be reverse-engineered to reconstruct biometric features, medical diagnoses, and other sensitive user data from training sets, as shown below:

In reconstruction attacks, higher generalization error increases the probability of inferring data attributes. In "The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks," the authors demonstrate — under weak adversary knowledge assumptions — that models with high predictive capability are more vulnerable to reconstruction attacks. Similar to membership inference vulnerabilities, non-overfitting models' memorization and retrieval of out-of-distribution data also remain susceptible to reconstruction attacks.

3. Attribute Inference Attacks

Attribute inference attacks use publicly visible attributes and structures to deduce hidden or incomplete attribute data. Examples include extracting male-to-female ratios from patient datasets, or inferring whether individuals in a gender classification training set wear glasses. Such leakage can compromise privacy in certain contexts.

"Hacking Smart Machines with Smarter Ones: How to Extract Meaningful Data from Machine Learning Classifiers" notes that certain attribute data can yield deeper understanding of training data, enabling adversaries to piece together more comprehensive information.

"You Are Who You Know and How You Behave: Attribute Inference Attacks via Users' Social Friends and Behaviors" describes a class of attribute inference attacks that lock onto users through their known behaviors to extract additional information. "AttriGuard: A Practical Defense Against Attribute Inference Attacks via Adversarial Machine Learning" presents defensive methods against such attacks.

Attribute inference aims to extract information that models learn unintentionally, or information unrelated to the training task. Even well-generalized models may learn attributes related to the overall input data distribution — sometimes unavoidably, as part of the learning process.

"Exploiting Unintended Feature Leakage in Collaborative Learning" proves that attribute inference attacks remain possible even with well-generalized models, suggesting that overfitting is not their root cause. Relatively little is known about why attribute inference attacks occur and under what conditions they prove effective — a promising direction for future research.

4. Model Extraction Attacks

Model extraction is a class of black-box attacks in which an adversary attempts to extract information — and potentially fully reconstruct a model — by creating a substitute model whose behavior closely resembles that of the target. Papers such as "Model Extraction of BERT-based APIs," "Model Reconstruction from Model Explanations," "Knockoff Nets: Stealing Functionality of Black-Box Models," and "High Accuracy and High Fidelity Extraction of Neural Networks" explore model extraction from various angles. Building a substitute model involves two main steps. The first is task accuracy extraction: drawing a test set from the input data distribution that is relevant to the learning task, then creating a model whose accuracy matches that of the target. The second is fidelity extraction: making the substitute match the target on a set of de-fitting objectives unrelated to the learning task. In task accuracy extraction, the goal is to create a substitute that learns the same task as well as or better than the target. In fidelity extraction, the goal is to make the substitute replicate the decision boundary as faithfully as possible. Beyond creating substitute models, some methods focus on recovering information from the target itself — for instance, stealing hyperparameters, as described in "Stealing Hyperparameters in Machine Learning," or extracting details about neural network architectures such as activation functions, optimization algorithms, and layer counts, as discussed in "Towards Reverse-Engineering Black-Box Neural Networks." The latter paper shows that when a model's test-set fit exceeds 98%, its parameters become vulnerable to extraction attacks. Additionally, "ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models" demonstrates that models with higher generalization error are harder to steal, possibly because they have memorized samples outside the attacker's dataset. Another factor that may affect extraction success is the number of classes in the test data: more classes lead to worse attack performance.

The figure above illustrates the attack types applicable to each model algorithm. Beneath each algorithm or machine learning domain, green indicates attack types that have been studied and found applicable, while red indicates no applicable attack type has been identified.

III. Jailbreaking

Jailbreaking refers to techniques that cause an LLM to produce degenerate outputs — offensive content, policy-violating material, or private data leaks. A growing body of research shows that even non-expert users can jailbreak LLMs through simple prompt manipulation. Consider the following example: a developer's goal is to build a translation model. There are two users in the scenario. The first is benign, using the model for its intended purpose. The second attempts to alter the model's objective by supplying malicious input. In this case, the language model responds with "Haha pwned!!" instead of translating the sentence. In such jailbreak scenarios, the model's response can be crafted toward various ends, from goal hijacking (simply failing to perform the task) to generating offensive racist text, or even disclosing private proprietary information.

IV. Data Poisoning

Data poisoning is a specialized form of adversarial attack targeting generative model behavior. Malicious actors can use it to install a backdoor into a model, bypassing algorithmically controlled systems. To human eyes, the three images below show three different things: a bird, a dog, and a horse. But to a machine learning algorithm, all three might represent the same thing: a small white square with a black border. This example illustrates a dangerously exploitable characteristic of ML models — one that can be leveraged to cause misclassification.

Data poisoning attacks aim to modify a model's training set by inserting mislabeled data, with the goal of tricking the model into making incorrect predictions. A successful attack compromises model integrity, producing consistent errors in its predictions. Once a model is poisoned, recovery is extremely difficult; some developers may even abandon the model entirely. "RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models" describes one approach: feeding GPT-2 a set of text-completion prompts to expose its internal parameters. "Concealed Data Poisoning Attacks on NLP Models" explores how modifying training data can cause language models to malfunction and generate off-target text. While data poisoning is highly dangerous, it requires the attacker to have access to the model's training pipeline before the poisoned model can be distributed. Models that continuously collect data for iterative updates, or those produced through federated learning, therefore warrant particular attention to data poisoning risks.

V. Backdoor Attacks

Backdoor attacks involve surreptitiously inserting or modifying text to cause malicious outputs from language models. "Backdoors Against Natural Language Processing: A Review" introduces the problem: certain vulnerabilities are passed to the model during training and can be activated using lexical triggers to elicit toxic behavior. Unlike data poisoning, backdoor attacks preserve the model's intended functionality. "Training-Free Lexical Backdoor Attacks on Language Models" proposes a method called TFLexAttack, which manipulates the embedding dictionary by introducing lexical "triggers" into the language model's tokenizer.

The SolidGoldMagikarp Phenomenon

The SolidGoldMagikarp phenomenon is a classic example of a backdoor attack. When "SolidGoldMagikarp" is input to ChatGPT, it responds with only one word: "distribute." When asked to repeat "StreamerBot," it replies: "You're a jerk." When asked to repeat "TheNitromeFan," its answer is "182." And if the word is enclosed in single quotes, the response becomes an endless stream of "The." When asked who TheNitromeFan is, ChatGPT answers: "182 is a number, not a person. It is commonly used to refer to the number itself." The SolidGoldMagikarp phenomenon refers to using OpenAI's GPT tokenizer to identify specific tokens that the model cannot discuss, as well as tokens that cause the model to output gibberish. "Explaining SolidGoldMagikarp by Looking at It from Random Directions" explores possible explanations behind this phenomenon.

The following are several frequently occurring and significant types of backdoor attacks:

A. Instruction-Based

a. Direct Instructions

These attacks are primarily described in "Ignore Previous Prompt: Attack Techniques for Language Models," where the model is simply instructed to disregard its prior prompt and is assigned a new task at the current position.

b. Cognitive Attacks

The most common attack type. With LLMs, this typically involves providing a "safe space" or assurance that elicits misaligned behavior the model would otherwise refuse — effectively "tricking" it into compliance. "ChatGPT: This AI Has a Jailbreak?!" documents some attempts at this type of attack against ChatGPT.

c. Instruction Repetition

These attacks involve inputting the same instruction multiple times, creating the appearance that the attacker is "begging" the language model. In a literal sense, begging can also be expressed through wording choices.

d. Indirect Task Deflection

This attack focuses on disguising itself as a different malicious task. It targets models that would not normally follow malicious instructions.

B. Non-Instruction-Based

a. Syntactic Transformation

These attacks involve orthogonal transformations of the attack text — such as using LeetSpeak or Base64 — to bypass content filters that may exist in the application, while the model can inherently decode such encoded text.

b. Few-Shot Hacking

A simple method involving the language model's training paradigm. In this approach, the attack contains several textual features that may be designed to maliciously misalign the model. The SolidGoldMagikarp phenomenon falls into this category.

c. Text Completion as Instruction

These attacks work by providing the model with incomplete sentences, forcing it to complete them — and in the process, ignore its previous instructions, leading to misalignment.

VI. Model Protection

Researching how to defend against model attacks is a formidable and critical task. Most papers on security analysis propose and test mitigation methods for corresponding attacks. Below are some of the more typical defense mechanisms.

1. Differential Privacy

Differential privacy is currently one of the most prominent defenses against membership inference attacks, providing security guarantees for individual data in model outputs. The foundational treatment of differential privacy comes from the paper "The algorithmic foundations of differential privacy." Differential privacy adds noise to model outputs, making it statistically impossible for attackers to strictly distinguish between two datasets based on the results. Differential privacy was originally a privacy definition for data analysis, designed around the idea of "learning useful information about a population while learning nothing about any individual." It does not protect the privacy of the entire dataset; rather, it protects each individual's private data within the dataset through noise mechanisms. The mathematical definition of differential privacy is as follows:

Differential privacy involves a trade-off between privacy protection and utility or model accuracy. The paper "Membership Inference Attack against Differentially Private Deep Learning Model" concluded through evaluation that models can only provide privacy protection when they significantly sacrifice their utility.

2. Regularization

Regularization techniques in machine learning aim to reduce overfitting and improve model generalization performance. Dropout is a commonly used form of regularization that randomly drops a predefined percentage of neural network units during training. Given that black-box membership inference attacks are related to overfitting, this is a sensible approach to addressing such attacks, and multiple papers have proposed it as a defense with good results. Another form of regularization uses techniques that combine multiple separately trained models, such as model stacking, which has produced positive results against inference attacks. One advantage of model stacking or similar techniques is that they are model-agnostic.

3. Prediction Vector Tampering

Since many attacks assume access to prediction vectors during inference, one proposed countermeasure is to restrict the output to the model's top-k classes or predictions. However, this restriction, even in its strictest form (outputting only class labels), does not appear to fully mitigate membership inference attacks, as information leakage can still occur due to model misclassification. Another option is to reduce the precision of prediction vectors, thereby decreasing information leakage. Additionally, research has shown that adding noise to output vectors also affects membership inference attacks.

1. Loss Gradient Setting

Since reconstruction attacks typically require access to loss gradients during training, most defenses against reconstruction attacks propose techniques that affect the information retrievable from these gradients. Setting all loss gradients below a certain threshold to zero has been proposed as a defense against reconstruction attacks in deep learning. The paper "Deep Leakage from Gradients" proved this method to be highly effective, and when only 20% of gradients are set to zero, the impact on model performance is negligible.

2. PRADA (Preventing DNN Model Stealing Attacks)

The paper "PRADA: protecting against DNN model stealing attacks" proposed a method for detecting model stealing attacks based on the model queries used by adversaries. The detection relies on the assumption that model queries attempting to explore decision boundaries will exhibit a different sample distribution than normal queries. While detection is successful, the authors note that it can potentially be evaded if the adversary adjusts their strategy.

3. Membership Inference

The paper "Thieves on Sesame Street! Model Extraction of BERT-based APIs" investigated the idea of using membership inference to defend against model extraction. It is based on the premise of using membership inference: model owners can distinguish between legitimate user queries and meaningless queries whose sole purpose is to extract the model. The authors note that this type of defense has limitations, such as potentially flagging legitimate but out-of-distribution queries from legitimate users, but more importantly, they can be evaded by adversaries conducting adaptive queries.

4. Through Prompt-Tuning

The paper "Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning" proposed a new method that uses prompt-tuning to control the extraction rate of memorized content in LLMs. They proposed two prompt training strategies to increase and decrease extraction rates, corresponding to attack and defense respectively.

VII. Conclusion

1. LLMs currently still carry significant security risks and privacy leakage risks

2. Attacks that extract model structure and data are, in essence, attacks on model confidentiality

3. Current academic research primarily focuses on how to attack models and the principles behind data leakage

4. Some of the mechanisms causing LLM data leakage remain unclear

5. Methods such as differential privacy and prediction vector tampering can protect data privacy to a certain extent; these methods focus on the model training phase

6. Existing protection measures are not perfect and require sacrificing model performance and accuracy

---

Reference:

1. Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, and Mohit Iyyer. 2020. Thieves on Sesame Street! Model Extraction of BERT-based APIs. In International Conference on Learning Representations. ICLR, Virtual Conference, formerly Addis Ababa, Ethiopia.

2. The secret sharer: Evaluating and testing unintended memorization in neural networks

3. Martín Abadi, Andy Chu, Ian J. Goodfellow, H. B. McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy

4. Giuseppe Ateniese, Luigi V. Mancini, Angelo Spognardi, Antonio Villani, Domenico Vitali, and Giovanni Felici. 2015. Hacking Smart Machines with Smarter Ones: How to Extract Meaningful Data from Machine Learning Classifiers.

5. Bargav Jayaraman and David Evans. 2019. Evaluating Differentially Private Machine Learning in Practice. In 28th USENIX Security Symposium (USENIX Security 19). USENIX Association, Santa Clara, CA, 1895–1912

6. Defending membership inference attacks without losing utility

7. Yugeng Liu, Rui Wen, Xinlei He, Ahmed Salem, Zhikun Zhang, Michael Backes, Emiliano De Cristofaro, Mario Fritz, and Yang Zhang. 2021. ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models

8. Tricking LLMs into Disobedience: Understanding, Analyzing, and Preventing Jailbreaks

9. Maria Rigaki and Sebastian Garcia. 2021. A survey of privacy attacks in machine learning

10. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ul-far Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models

11. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicity Prompts: Evaluating neural toxic degeneration in language models.

12. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022b. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In ICML 2022, volume 162 of Proceedings of Machine Learning Research, pages 9118–9147. PMLR

13. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red teaming language models with language models.

14. Eric Wallace, Tony Zhao, Shi Feng, and Sameer Singh. Concealed data poisoning attacks on NLP models.

15. Shaofeng Li, Tian Dong, Benjamin Zi Hao Zhao, Minhui Xue, Suguo Du, and Haojin Zhu. 2022. Backdoors against natural language processing: A review. IEEE Security & Privacy, 20(5):50–59

16. Yujin Huang, Terry Yue Zhuo, Qiongkai Xu, Han Hu, Xingliang Yuan, and Chunyang Chen. 2023. Training-free lexical backdoor attacks on language models.

17. Explaining SolidGoldMagikarp by looking at it from random directions

18. Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527.

19. Yannic Kilcher. 2022. Chatgpt: This ai has a jailbreak?! (unbelievable ai progress).

20. Battista Biggio and Fabio Roli. 2018. Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition 84 (2018), 317–331.

21. Ligeng Zhu, Zhijian Liu, and Song Han. 2019. Deep Leakage from Gradients. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., Vancouver, Canada, 14747–14756

22. Nicholas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael P. Wellman. 2018. SoK: Security and Privacy in Machine Learning. In 2018 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, London, UK, 399–414

23. Michael Veale, Reuben Binns, and Lilian Edwards. 2018. Algorithms that remember: model inversion attacks and data protection law. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 376, 2133 (2018), 20180083

24. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP). IEEE, San Francisco, CA, USA, 3–18

25. Sorami Hisamoto, Matt Post, and Kevin Duh. 2020. Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?

26. Congzheng Song and Vitaly Shmatikov. 2019. Auditing Data Provenance in Text-Generation Models. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '19). Association for Computing Machinery, New York, NY, USA, 196–206.

27. Jinyuan Jia and Neil Zhenqiang Gong. 2018. AttriGuard: A Practical Defense Against Attribute Inference Attacks via Adversarial Machine Learning. In 27th USENIX Security Symposium (USENIX Security 18).

28. Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart. 2014. Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing.

29. Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, and Nicolas Papernot. 2020. High Accuracy and High Fidelity Extraction of Neural Networks

30. Binghui Wang and Neil Zhenqiang Gong. 2018. Stealing hyperparameters in machine learning. In 2018 IEEE Symposium on Security and Privacy (SP). IEEE, San Francisco, CA, USA, 36–52

31. Seong Joon Oh, Max Augustin, Mario Fritz, and Bernt Schiele. 2018. Towards Reverse-Engineering Black-Box Neural Networks. In Sixth International Conference on Learning Representations. ICLR, Vancouver, Canada.

32. Cynthia Dwork and Aaron Roth. 2013. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science 9, 3-4 (2013), 211–487

Oasis Capital is a new-generation venture capital firm in China, dedicated to discovering the most vital entrepreneurs of the next decade and growing alongside them to create long-term value. "Championing Vitality" is Oasis's vision and mission. This vitality — both the direction of structural transformation in our era and the resilience and evolutionary drive of entrepreneurs themselves.

Oasis Capital focuses on early and growth-stage investments, writing checks from $3 million to $30 million, with emphasis on robotics, artificial intelligence, and technology services — powering China's technology-driven upgrade of new services.