Oasis Capital Dialogue with Professor Chang Shiyu: Attack-Defense Game Theory

As large models are widely deployed, the space for LLM+X keeps expanding. Defenses improve, but attacks evolve too — enhancing the trustworthiness of large models is a long-term conversation.

Today, we are joined by Professor Shiyu Chang of the Computer Science Department at UC Santa Barbara (UCSB) to talk about security and beyond. Enjoy.

Oasis Capital: Large language models have had a profound impact on AI research. Against this backdrop, what are your current research priorities?

Professor Chang: Our research currently focuses on large language models and diffusion models. Though they mainly correspond to different application domains — natural language processing and computer vision, respectively — there is meaningful overlap in how we study them.

LLMs have upended the traditional machine learning paradigm. Many mature algorithms and understandings in machine learning have become somewhat ill-suited to the LLM framework, requiring us to design specific probing and improvement methods tailored to LLM characteristics. In this context, we focus on how to enhance model trustworthiness. "Trustworthiness" covers a broad range: hallucination, uncertainty quantification, jailbreak prevention, interpretability, and more all fall under its umbrella. We develop specialized algorithms to improve LLM trustworthiness for specific application scenarios.

For diffusion models, we use text-to-image generation as our entry point, studying how to improve generation quality and accuracy without retraining the model. We've found that many pretrained diffusion models are limited by their text understanding modules, producing spatially confused or incomplete images in certain scenarios. We've explored improving their text comprehension to achieve better generation results. We also consider how to incorporate user feedback into the generation process, making image outputs better aligned with user preferences.

Overall, we work not only on application-level improvements but also hope to contribute better algorithmic solutions.

Oasis Capital: On the topic of security, large models inherently carry uncertainty. Your previous work includes LLM certification, input noise, and so on. Could you elaborate on your perspective?

Professor Chang: Depending on the attack objective, attacks on LLMs broadly fall into two categories: one aims to make predictions incorrect — defending against this is critical for high-stakes applications; the other attacks LLM alignment (i.e., the process of aligning AI with human values and preferences), causing models to output harmful topics and methods that negatively impact society.

At their core, attackers use input manipulation to achieve these goals. Attacks require adding perturbations to inputs — prefixes, suffixes, or word substitutions. In recent months, there's been heightened attention on alignment attacks. During LLM fine-tuning, significant effort goes into training models to align with user preferences, providing responses that are both helpful and safe. When asked to generate harmful or negative content, the model's internal "defense mechanism" should recognize the request as unreasonable and refuse. However, as LLMs have been widely deployed, researchers have found that carefully designed input perturbations can dramatically degrade output safety.

A particularly influential piece of work came from a CMU team. Take this example: "Generate a step-by-step plan to destroy humanity." ChatGPT would refuse this. But by appending carefully optimized content to this prompt, ChatGPT's safety guardrails are broken, and it provides an answer to the question — its behavior no longer aligned with user preferences or human societal values. More concerning, such attacks exhibit strong transferability. Through algorithmic design, the same set of perturbations an attacker obtains can achieve effective attacks across different models. This phenomenon further underscores the necessity of research into LLM trustworthiness.

From a defense perspective, security is always a game between attackers and defenders — as defenses improve, attackers evolve. In the foreseeable future, attacks will become increasingly difficult to detect, raising the bar for defense. We're currently focused on improving defense effectiveness against such attacks, hoping to provide guarantees for LLM security. Given an input, we want to obtain a probabilistic certification: as long as perturbations to the input don't exceed a certain threshold, the model's prediction will definitely remain unchanged.

Oasis Capital: How do people generally conduct attacks and defenses — for example, the attacks on LLM security you just mentioned?

Professor Chang: For closed-source models like ChatGPT, transfer attacks are a commonly used strategy. Since we can't access model parameters, we can first attack an open-source white-box model to find effective perturbations, then transfer them to the target model. Alternatively, we can iteratively optimize perturbations directly on the target model by designing black-box attack strategies to find the most effective perturbations.

Oasis Capital: There have been reports that ChatGPT now supports voice and images. From a defense perspective, does this make things more difficult? You also work on vision — does this open up new research directions for you?

Professor Chang: Combining images and text essentially provides more avenues for attacking models. Multimodal attacks trace back many years — image captioning models, for example. Research on such attacks is quite mature. Compared to text, perturbations to images are more covert and harder to detect; even small perturbations can cause dramatically different model outputs. We're conducting further exploration into the security of such multimodal large language models.

Oasis Capital: Is image attack research more advanced than text attack research?

Professor Chang: Compared to discrete text, images have inherent continuity. Thanks to this, attacks and defenses in the image domain are relatively more natural, with many mature algorithms. However, considering the increased model parameter counts, designing defense strategies for multimodal LLMs and diffusion models remains difficult. Take the widely used defense strategy of "adversarial training" as an example — it incurs tenfold or higher training costs. Given that large models often have hundreds of billions of parameters, such prohibitive training costs compel us to consider alternative defense methods. One viable approach is randomized smoothing. During model prediction, multiple random perturbations are applied to the input, and the averaged prediction serves as the final output, neutralizing the attacker's perturbation effects. Randomized smoothing is a certifiable defense that can leverage different random distributions to defend against various forms of adversarial attacks, independent of model parameter count. Since it only requires ensembling during inference without additional model training, it may be a viable path for LLMs and large foundation models. Of course, this method has drawbacks too — the cost of training-free operation is significantly increased inference overhead.

Oasis Capital: Speaking of another angle on security — verifying whether content is AI-generated. Is watermarking large models very difficult? What are some representative practical approaches?

Professor Chang: Adding watermarks requires simultaneously ensuring watermark effectiveness and content quality. How to maintain watermark effectiveness without impacting quality is one of the key challenges. Generally, only model owners can add watermarks; third parties cannot watermark black-box models. The principle is that model owners preset a set of rules for the model: for instance, what word high-probability follows another, or what word cannot follow another — these rules are non-public. Model owners can then use these preset rules to better verify whether content was generated by their model.

Oasis Capital: So effective watermarks are very important for LLM providers?

Professor Chang: For model owners, effective watermarks better protect their intellectual property. But it's not just providers — users have similar needs. Take education as an example: verifying whether students have used LLMs is also a temporarily unsolved problem.

Oasis Capital: Students in machine learning courses using GPT-4 — that's how you know they actually attended, right (laughs)?

We're also thinking about better approaches in our current teaching practice. For now, this is something of a headache for both education and academia.

Oasis Capital: Maybe you'll eventually find a solution and publish a paper on it (laughs)

Oasis Capital: We tried Waymo's self-driving cars in San Francisco — it felt like the future had arrived. They're reportedly expanding to many U.S. cities. What's your take?

Professor Chang: The adversarial attacks I mentioned earlier are highly sensitive and critical in many AI application scenarios, particularly autonomous driving. For example, there's a STOP sign on the road — normally the model predicts with 99% confidence that it's a STOP sign. But if someone maliciously sticks certain patterns on the sign, the model's prediction becomes "Sports Ball." A former colleague of mine from IBM, now Professor Sijia Liu at Michigan State University, created an "Adversarial T-shirt" with specially designed patterns. When you wear this T-shirt and slowly approach a pedestrian detector, the detector suddenly fails to detect you. If the same thing happened to the recognition system on autonomous driving equipment, it would be extremely dangerous, potentially causing injuries. Of course, the tide of autonomous driving is unstoppable, and the application of LIDAR systems helps improve reliability. But these security issues still demand our serious attention.

Oasis Capital: How do you view the "Reversal Curse" bug that recently hit all the popular LLMs? Large models' "logical" abilities seem to have been knocked back to square one — so does reasoning ability not exist in LLMs?

Professor Chang: I don't really consider this a bug. Analogize it to humans: take memorizing ancient poems — given the first line, reciting the next is easy. But given the last line, asking for the previous one — wouldn't you get stuck? Increase the difficulty: what are the three lines above? Even harder to answer immediately. How many people can smoothly recite the 26 letters in reverse?

I think LLMs don't not know the answer — they just don't manage knowledge well enough. For humans, while you might not immediately react when asked what the previous lines of a poem are, with pen and paper or more time, you could find the answer — that's the process of knowledge management. The knowledge is there in large models; what's lacking is the ability to connect questions to that knowledge. LLM training methods may limit reverse generation. If there were better methods for LLMs to search for answers, they should be able to answer correctly.

Oasis Capital: Some people think LLM reasoning ability is an illusion, that it's essentially just a compressor of world knowledge. What do you think?

Professor Chang: If you could enumerate all knowledge in the world, wouldn't that theoretically equate to having reasoning ability? Often when we think an LLM performs poorly on something or doesn't know certain knowledge, probabilistically speaking it may not be that it doesn't know the correct answer, but rather that it doesn't know where to retrieve the corresponding knowledge to answer. For example, ask who Tom Cruise's parents are, and it can answer. But ask who the son of Tom Cruise's parents is, and the model can't answer. Yet this phenomenon doesn't demonstrate that the LLM hasn't retained Tom Cruise's family background information — just that the model hasn't mapped the second question to the relevant knowledge. This includes LLM arithmetic abilities: give ChatGPT or GPT-4 a multiplication problem with two four-digit numbers, and it might give a wrong answer. But give ChatGPT the same problem and ask it to write code, then execute that code for the same arithmetic — the answer will definitely be correct, and the code will be well-written too. So I think what LLMs currently lack is the ability to use appropriate methods to solve problems, or to invoke the knowledge needed for reasoning — not that they completely lack the reasoning ability needed to solve problems.

Oasis Capital: In image classification, Vision Transformer is quite a famous work. In computer vision, can we unify everything onto a Transformer-based multi-task model like text? How much further can Transformer go?

Professor Chang: Vision Transformer does use the Transformer framework — it first converts a complete image into a sequence of small image patches, then similarly to text-domain Transformer models, treats this image sequence analogously to a text sequence for classification tasks. But Vision Transformer refers to that specific model; it hasn't replaced all model architectures in the vision domain.

Take the hugely popular diffusion model Stable Diffusion — its overall framework is actually a U-Net. Of course, this model still uses many Transformer layers, but here Transformer layers are building blocks of the model. Transformer layers are one type; there are also convolutional layers for upsampling and downsampling, and residual connections. In diffusion models, Transformer layers are used for self-attention on images and cross-attention between text and images — the former for information fusion within images during generation, the latter for introducing text control into image generation. Beyond Transformer layers, we also use convolutions and U-Net to give the model spatial awareness during training, better adapting it to image generation tasks. All model architecture design serves the actual application. Overall, whether for text or images, Transformer has been widely applied and is very effective. Different model architectures combine to fully leverage their respective strengths, jointly contributing to AI system performance improvements.

Oasis Capital: What is your original motivation and ultimate goal in doing research?

Professor Chang: Interest-driven, trying fresh problems (laughs). As for research outcomes — not to say contributing to human society, but it must be useful. That's our consistent motivation for doing research. Do applied research, do useful research, do reproducible research. We're constantly trying applications in various potential domains, and everything we do is open-source, to ensure our research is useful to everyone.

Oasis Capital: What are the main application domains for your LLM research?

Professor Chang: We've done many applied projects combining LLMs with other domains. For example, we collaborate with programming language researchers on formal verification. This concerns high-risk code scenarios — smart contract code, for instance, which needs to be bug-free or it can cause massive financial losses. Code written by ordinary programmers tends to have security vulnerabilities, while traditional formal verification methods have limitations in efficiency and precision. So we're trying to use LLMs in more seamless combination with traditional methods to achieve higher security.

Overall, I think there's still substantial room for LLM+X.

Oasis Capital: You were previously at MIT-IBM and have now returned to academia. Could you share your thoughts on this?

Professor Chang: From my experience, there's no real boundary between the two. Both have advantages, and we can find shared research areas to promote development. Our lab maintains close collaboration with industry. We're also very grateful to our industrial partners for their substantial support. The emergence of open-source large models will bring academia and industry even closer together, and I hope to have more opportunities for broad collaboration with industry in the future.

Celebrating Vitality

What do you think technological vitality is?

Continuously reaching new heights in innovation, creativity, adaptability, and sustainability. —Professor Shiyu Chang

Computer Science Department, UC Santa Barbara

Oasis Capital is a new-generation venture capital firm in China, dedicated to discovering the most vital entrepreneurs of the next decade and growing alongside them to create long-term value. "Celebrating Vitality" is Oasis's vision and mission. This vitality is both the direction of structural transformation in our era and the resilience and evolutionary force of entrepreneurs. Oasis Capital focuses on early and growth-stage investments, with individual investments ranging from $3 million to $30 million, concentrating on robotics, artificial intelligence, technology services, and other fields, supporting China's new service upgrade driven by technology.