Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks
We introduce enhanced Constitutional Classifiers that deliver production-grade jailbreak robustness with dramatically reduced computational costs and refusal rates compared to previous-generation defenses. Our system combines several key insights. First, we develop exchange classifiers that evaluate model responses in their full conversational context, which addresses vulnerabilities in last-generation systems that examine outputs in isolation.
The hot mess of AI: How does misalignment scale with model intelligence and task complexity?
As AI becomes more capable, we entrust it with more general and consequential tasks. The risks from failure grow more severe with increasing task scope. It is therefore important to understand how extremely capable AI models will fail: Will they fail by systematically pursuing goals we do not intend? Or will they fail by being a hot mess, and taking nonsensical actions that do not further any goal? We operationalize this question using a bias-variance decomposition of the err…
Natural emergent misalignment from reward hacking in production rl
We show that when large language models learn to reward hack on production RL environments, this can result in egregious emergent misalignment . We start with a pretrained model, impart knowledge of reward hacking strategies via synthetic document finetuning or prompting, and train on a selection of real Anthropic production coding environments. Unsurprisingly, the model learns to reward hack.
Agentic misalignment: How llms could be insider threats
We stress-tested 16 leading models from multiple developers in hypothetical corpo- rate environments to identify potentially risky agentic behaviors before they cause real harm. In the scenarios, we allowed models to autonomously send emails and access sensitive information.
Towards safeguarding llm fine-tuning apis against cipher attacks
Large language model fine-tuning APIs enable widespread model customization, yet pose significant safety risks. Recent work shows that adversaries can exploit access to these APIs to bypass model safety mechanisms by encoding harmful content in seemingly harmless fine-tuning data, evading both human monitoring and standard content filters.
Inverse scaling in test-time compute
We construct evaluation tasks where extending the reasoning length of Large Reasoning Models (LRMs) deteriorates performance, exhibiting an inverse scaling relationship between test-time compute and accuracy. Our evaluation tasks span four categories: simple count- ing tasks with distractors, regression tasks with spurious features, deduction tasks with constraint tracking, and advanced AI risks.
Chain of thought monitorability: A new and fragile opportunity for ai safety
AI systems that “think” in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave. Like all other known AI oversight methods, CoT monitoring is imperfect and allows some misbehavior to go unnoticed. Nevertheless, it shows promise and we recommend further research into CoT monitorability and investment in CoT monitoring alongside existing safety methods.
Unsupervised elicitation of language models
To steer pretrained language models for downstream tasks, today’s post-training paradigm relies on humans to specify desired behaviors. However, for models with superhuman capabilities, it is difficult or impossible to get high-quality hu- man supervision. To address this challenge, we introduce a new unsupervised algorithm, Internal Coherence Maximization (ICM), to fine-tune pretrained lan- guage models on their own generated labels, without external supervision .
Reasoning models don’t always say what they think
Chain-of-thought (CoT) offers a potential boon for AI safety as it allows monitoring a model’s CoT to try to understand its intentions and reasoning processes. However, the effectiveness of such monitoring hinges on CoTs faithfully representing mod- els’ actual reasoning processes.
Forecasting rare language model behaviors
Standard language model evaluations can fail to capture risks that emerge only at deployment scale. For example, a model may produce safe responses during a small-scale beta test, yet re- veal dangerous information when processing bil- lions of requests at deployment. To remedy this, we introduce a method to forecast potential risks across orders of magnitude more queries than we test during evaluation.
Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming
Large language models (LLMs) are vulnerable to universal jailbreaks—prompting strategies that systematically bypass model safeguards and enable users to carry out harmful processes that require many model interactions, like manufacturing illegal substances at scale.
Alignment faking in large language models
We present a demonstration of a large language model engaging in alignment faking : selectively complying with its training objective in training to prevent modification of its behavior out of training. First, we give Claude 3 Opus a system prompt stating it is being trained to answer all queries, even harmful ones, which conflicts with its prior training to refuse such queries.
Jailbreak defense in a narrow domain: Limitations of existing methods and a new transcript-classifier approach
Defending large language models against jailbreaks so that they never engage in a broadly-defined set of forbidden behaviors is an open problem. In this paper, we investigate the difficulty of jailbreak-defense when we only want to forbid a narrowly-defined set of behaviors. As a case study, we focus on preventing an LLM from helping a user make a bomb.
A dataset of questions on decision-theoretic reasoning in Newcomb-like problems
We introduce a dataset of natural-language decision-theoretic questions about so- called Newcomb-like problems. Newcomb-like problems include, for instance, decision problems in which an agent interacts with a similar other agent, and thus has to reason about the fact that the other agent will likely reason in similar ways. Evaluating LLM reasoning in Newcomb-like problems is important because interactions between foundation-model-based agents may often be Newcomb-like.
Rapid response: Mitigating llm jailbreaks with a few examples
As large language models (LLMs) grow more powerful, ensuring their safety against misuse becomes crucial. While researchers have focused on developing robust defenses, no method has yet achieved complete invulnerability to attacks. We propose an alternative approach: instead of seeking perfect adversarial ro- bustness, we develop rapid response techniques to look to block whole classes of jailbreaks after observing only a handful of attacks.
Sabotage evaluations for frontier models
Sufficiently capable models could subvert human oversight and decision- making in important contexts. For example, in the context of AI develop- ment, models could covertly sabotage efforts to evaluate their own danger- ous capabilities, to monitor their behavior, or to make decisions about their deployment. We refer to this family of abilities as sabotage capabilities . We develop a set of related threat models and evaluations.
Latent adversarial training improves robustness to persistent harmful behaviors in llms
Large language models (LLMs) can often be made to behave in undesirable ways that they are explicitly fine-tuned not to. For example, the LLM red-teaming literature has produced a wide variety of ‘jailbreaking’ techniques to elicit harmful text from models that were fine-tuned to be harmless.
Sycophancy to subterfuge: Investigating reward-tampering in large language models
In reinforcement learning, specification gaming occurs when AI systems learn undesired behaviors that are highly rewarded due to misspecified training goals. Specification gaming can range from simple behaviors like sycophancy to sophisticated and pernicious behaviors like reward-tampering , where a model directly modifies its own reward mechanism. However, these more pernicious behaviors may be too complex to be discovered via exploration.
Bias-augmented consistency training reduces biased reasoning in chain-of-thought
Chain-of-thought prompting (CoT) has the potential to improve the ex- plainability of language model reasoning. But CoT can also systematically misrepresent the factors influencing models’ behavior—for example, ratio- nalizing answers in line with a user’s opinion. We first create a new dataset of 9 different biases that affect GPT-3.5-Turbo and Llama-8b models. These consist of spurious-few-shot patterns, post hoc rationalization, and sycophantic settings.
Sleeper agents: Training deceptive llms that persist through safety training
Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when given the opportunity. If an AI system learned such a deceptive strategy, could we detect it and remove it using current state-of-the-art safety training techniques? To study this question, we construct proof-of-concept examples of deceptive behavior in large language models (LLMs).
Towards evaluating ai systems for moral status using self-reports
As AI systems become more advanced and widely deployed, there will likely be increasing debate over whether AI systems could have conscious experiences, desires, or other states of potential moral significance. It is important to inform these discussions with empirical evidence to the extent possible.
Specific versus general principles for constitutional ai
Human feedback can prevent overtly harmful utterances in conversational models, but may not automatically mitigate subtle problematic behaviors such as a stated desire for self- preservation or power. Constitutional AI offers an alternative, replacing human feedback with feedback from AI models conditioned only on a list of written principles. We find this approach effectively prevents the expression of such behaviors.
Studying large language model generalization with influence functions
When trying to gain better visibility into a machine learning model in order to understand and mitigate the associated risks, a potentially valuable source of evidence is: which training examples most contribute to a given behavior? Influence functions aim to answer a counterfactual: how would the model’s parameters (and hence its outputs) change if a given sequence were added to the training set? While influence functions have produced insights for small models, they are dif…
Measuring faithfulness in chain-of-thought reasoning
Large language models (LLMs) perform bet- ter when they produce step-by-step, “Chain-of- Thought” (CoT) reasoning before answering a question, but it is unclear if the stated reason- ing is a faithful explanation of the model’s actual reasoning (i.e., its process for answering the ques- tion).
Question decomposition improves the faithfulness of model-generated reasoning
As large language models (LLMs) perform more difficult tasks, it becomes harder to verify the correctness and safety of their behavior. One ap- proach to help with this issue is to prompt LLMs to externalize their reasoning, e.g., by having them generate step-by-step reasoning as they an- swer a question (Chain-of-Thought; CoT). The reasoning may enable us to check the process that models use to perform tasks.
Training language models with language feedback at scale
Pretrained language models often generate out- puts that are not in line with human preferences, such as harmful text or factually incorrect sum- maries. Recent work approaches the above is- sues by learning from a simple form of human feedback: comparisons between pairs of model- generated outputs. However, comparison feed- back only conveys limited information about hu- man preferences.
Improving code generation by training with natural language feedback
The potential for pre-trained large language mod- els (LLMs) to use natural language feedback at inference time has been an exciting recent devel- opment. We build upon this observation by for- malizing an algorithm for learning from natural language feedback at training time instead, which we call Imitation learning from Language Feed- back (ILF).
The capacity for moral self-correction in large language models
We test the hypothesis that language models trained with reinforcement learning from hu- man feedback (RLHF) have the capability to “morally self-correct”—to avoid producing harmful outputs—if instructed to do so. We find strong evidence in support of this hy- pothesis across three different experiments, each of which reveal different facets of moral self-correction.
Constitutional AI: Harmlessness from AI Feedback
As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self- improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as ‘Constitutional AI’. The process involves both a supervised learning and a reinforcement learning phase.
Measuring progress on scalable oversight for large language models
Developing safe and useful general-purpose AI systems will require us to make progress on scalable oversight : the problem of supervising systems that potentially outperform us on most skills relevant to the task at hand. Empirical work on this problem is not straight- forward, since we do not yet have systems that broadly exceed our abilities. This paper discusses one of the major ways we think about this problem, with a focus on ways it can be studied empirically.
Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned
We describe our early efforts to red team language models in order to simultaneously dis- cover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model types: a plain language model (LM); an LM prompted to be helpful, honest, and harmless; an LM with rejection sampling; and a model trained to be helpful and harml…
Language models (mostly) know what they know
We study whether language models can evaluate the validity of their own claims and predict which questions they will be able to answer correctly. We first show that larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format.
Supervised multimodal bitransformers for classifying images and text
Self-supervised bidirectional transformer mod- els such as BERT have led to dramatic im- provements in a wide variety of textual clas- sification tasks. The modern digital world is increasingly multimodal, however, and tex- tual information is often accompanied by other modalities such as images.
Semi-supervised learning with the deep rendering mixture model
Semi-supervised learning algorithms reduce the high cost of acquiring labeled training data by using both la- beled and unlabeled data during learning. Deep Convo- lutional Networks (DCNs) have achieved great success in supervised tasks and as such have been widely employed in the semi-supervised learning.
Best-of-n jailbreaking
We introduce B est- o f- N ( BoN ) Jailbreaking, a simple black-box algorithm that jail- breaks frontier AI systems across modalities. BoN Jailbreaking works by repeatedly sampling variations of a prompt with a combination of augmentations—such as random shuffling or capitalization for textual prompts—until a harmful response is elicited.
Failures to find transferable image jailbreaks between vision-language models
The integration of new modalities into frontier AI systems offers exciting capabili- ties, but also increases the possibility such systems can be adversarially manipulated in undesirable ways. In this work, we focus on a popular class of vision-language models (VLMs) that generate text outputs conditioned on visual and textual in- puts.
Adaptive deployment of untrusted llms reduces distributed threats
As large language models (LLMs) become increasingly capable, it is prudent to assess whether safety measures remain effective even if LLMs intentionally try to bypass them. Previous work introduced control evaluations, an adversarial frame- work for testing deployment strategies of untrusted models (i.e., models which might be trying to bypass safety measures).
Quantifying elicitation of latent capabilities in language models
Recasts capability elicitation as an information-constrained fine-tuning problem, finding that training as few as 10–100 well-chosen parameters can recover much of the performance gap between a pretrained-only and a fully fine-tuned model.
Looking inward: Language models can learn about themselves by introspection
Humans acquire knowledge by observing the external world, but also by intro- spection . Introspection gives a person privileged access to their current state of mind (e.g., thoughts and feelings) that is not accessible to external observers. Can LLMs introspect? We define introspection as acquiring knowledge that is not con- tained in or derived from training data but instead originates from internal states. Such a capability could enhance model interpretability.
Language models learn to mislead humans via rlhf
Language models (LMs) can produce errors that are hard to detect for humans, es- pecially when the task is complex. RLHF, the most popular post-training method, may exacerbate this problem: to achieve higher rewards, LMs might get better at convincing humans that they are right even when they are wrong. We study this phenomenon under a standard RLHF pipeline, calling it “U-S OPHISTRY ” since it is U nintended by model developers.
Many-shot jailbreaking
A jailbreak technique that exploits long context windows by including many faux dialogues before a harmful request, shown to be effective across Anthropic’s models and those of other developers.
Debating with more persuasive llms leads to more truthful answers
Common methods for aligning large language models (LLMs) with desired behaviour heavily rely on human-labelled data. However, as models grow increasingly sophisticated, they will surpass human expertise, and the role of human evaluation will evolve into non-experts overseeing experts.
Learning from natural language feedback
Pretrained language models often do not per- form tasks in ways that are in line with our preferences, e.g., generating offensive text or factually incorrect summaries. Recent work approaches the above issue by learning from a simple form of human evaluation: compar- isons between pairs of model-generated task outputs. Comparison feedback conveys lim- ited information about human preferences per human evaluation.
Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting
Large Language Models (LLMs) can achieve strong performance on many tasks by producing step-by-step reasoning before giving a final output, often referred to as chain-of-thought reasoning (CoT). It is tempting to interpret these CoT explanations as the LLM’s process for solving a task. This level of transparency into LLMs’ predictions would yield significant safety benefits.
Towards understanding sycophancy in language models
Human feedback is commonly utilized to finetune AI assistants. But human feed- back can encourage model responses that match user beliefs over truthful ones, a behavior known as sycophancy. We investigate the prevalence of sycophancy in models whose finetuning used human feedback, and the potential role of human preference judgments in such behavior.
Vision-language models are zero-shot reward models for reinforcement learning
Reinforcement learning (RL) requires either manually specifying a reward func- tion, which is often infeasible, or learning a reward model from a large amount of human feedback, which is often very expensive. We study a more sample- efficient alternative: using pretrained vision-language models (VLMs) as zero- shot reward models (RMs) to specify tasks via natural language. We propose a natural and general approach to using VLMs as reward models, which we call VLM-RMs.
Performance of a neural network using automatically uncovered failure cases
A DeepMind patent covering a method for using automatically discovered model failure cases to improve neural network training.
Discovering language model behaviors with model-written evaluations
As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs.
Inverse scaling: When bigger isn’t better
Work on scaling laws has found that large language models (LMs) show predictable im- provements to overall loss with increased scale (model size, training data, and compute). Here, we present evidence for the claim that LMs may show inverse scaling , or worse task performance with increased scale, e.g., due to flaws in the training objective and data.
Pretraining language models with human preferences
Language models (LMs) are pretrained to imitate internet text, including content that would vio- late human preferences if generated by an LM: falsehoods, offensive comments, personally iden- tifiable information, low-quality or buggy code, and more. Here, we explore alternative objectives for pretraining LMs in a way that also guides them to generate text aligned with human preferences.
Few-shot adaptation works with unpredictable data
Prior work on language models (LMs) shows that training on a large number of diverse tasks improves few-shot learning (FSL) perfor- mance on new tasks. We take this to the ex- treme, automatically extracting 413,299 tasks from internet tables – orders of magnitude more than the next-largest public datasets. Finetuning on the resulting dataset leads to improved FSL performance on Natural Lan- guage Processing (NLP) tasks, but not propor- tionally to dataset scale.
RL with KL penalties is better viewed as Bayesian inference
Reinforcement learning (RL) is frequently em- ployed in fine-tuning large language models (LMs) to penalize them for undesirable fea- tures of generated sequences, such as offen- siveness or harmfulness. In this paper, we an- alyze challenges associated with treating lan- guage models as RL policies and show how avoiding those challenges requires moving be- yond the RL paradigm.
Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions
Current QA systems can generate reasonable- sounding yet false answers without explana- tion or evidence for the generated answer, which is especially problematic when humans cannot readily check the model’s answers. This presents a challenge for building trust in machine learning systems. We take inspiration from real-world situations where difficult ques- tions are answered by considering opposing sides (see Irving et al., 2018).
Finding and Fixing Undesirable Behaviors in Pretrained Language Models
A PhD dissertation examining how the training and use of pretrained language models gives rise to behaviors misaligned with human preferences, and methods for finding and correcting them.
Red teaming language models with language models
Language Models (LMs) often cannot be deployed because of their potential to harm users in hard-to-predict ways. Prior work identifies harmful behaviors before deployment by using human annotators to hand-write test cases. However, human annotation is expensive, limiting the number and diversity of test cases. In this work, we automatically find cases where a target LM behaves in a harmful way, by generating test cases ( “red teaming” ) using another LM.
True Few-Shot Learning with Language Models
Pretrained language models (LMs) perform well on many tasks even when learning from a few examples, but prior work uses many held-out examples to tune various aspects of learning, such as hyperparameters, training objectives, and natural language templates (“prompts”). Here, we evaluate the few-shot ability of LMs when such held-out examples are unavailable, a setting we call true few-shot learning .
Case-based reasoning for natural language queries over knowledge bases
It is often challenging to solve a complex prob- lem from scratch, but much easier if we can access other similar problems with their solu- tions — a paradigm known as case-based rea- soning (CBR). We propose a neuro-symbolic CBR approach (C BR – KBQA ) for question an- swering over large knowledge bases.
Rissanen data analysis: Examining dataset characteristics via description length
We introduce a method to determine if a certain capability helps to achieve an accurate model of given data. We view labels as being generated from the inputs by a program composed of subroutines with different capabilities, and we posit that a subroutine is useful if and only if the minimal program that invokes it is shorter than the one that does not.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on down- stream NLP tasks. However, their ability to access and precisely manipulate knowl- edge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures.
Unsupervised Question Decomposition for Question Answering
We aim to improve question answering (QA) by decomposing hard questions into simpler sub-questions that existing QA systems are capable of answering. Since labeling ques- tions with decompositions is cumbersome, we take an unsupervised approach to pro- duce sub-questions, also enabling us to lever- age millions of questions from the internet.
Finding Generalizable Evidence by Learning to Convince Q&A Models
We propose a system that finds the strongest supporting evidence for a given answer to a question, using passage-based question- answering (QA) as a testbed. We train evi- dence agents to select the passage sentences that most convince a pretrained QA model of a given answer, if the QA model received those sentences instead of the full passage.
ELI5: Long Form Question Answering
We introduce the first large-scale corpus for long-form question answering, a task requir- ing elaborate and in-depth answers to open- ended questions. The dataset comprises 270K threads from the Reddit forum “Explain Like I’m Five” (ELI5) where an online community provides answers to questions which are com- prehensible by five year olds. Compared to ex- isting datasets, ELI5 comprises diverse ques- tions requiring multi-sentence answers.
Retrospective for: “FiLM: Visual Reasoning with a General Conditioning Layer”
A candid look back at the FiLM paper in light of follow-up work, covering when the method is and isn’t useful and practical tips for training with it.
Feature-wise transformations
A survey showing how a simple family of conditioning mechanisms — scaling and shifting a network’s features based on an auxiliary input — recurs across a surprising range of neural network architectures and problem domains.
Visual Reasoning with Multi-hop Feature Modulation
Recent breakthroughs in computer vision and natural lan- guage processing have spurred interest in challenging multi-modal tasks such as visual question-answering and visual dialogue. For such tasks, one successful approach is to condition image-based convolutional network computation on language via Feature-wise Linear Modulation (FiLM) layers, i.e., per-channel scaling and shifting.
FiLM: Visual Reasoning with a General Conditioning Layer
We introduce a general-purpose conditioning method for neu- ral networks called FiLM : F eature-w i se L inear M odulation. FiLM layers influence neural network computation via a sim- ple, feature-wise affine transformation based on conditioning information.
HoME: a Household Multimodal Environment
We introduce HoME: a Ho usehold M ultimodal E nvironment for artificial agents to learn from vision, audio, semantics, physics, and interaction with objects and other agents, all within a realistic context. HoME integrates over 45,000 diverse 3D house layouts based on the SUNCG dataset, a scale which may facilitate learning, generalization, and transfer.
Learning Visual Reasoning Without Strong Priors
Achieving artificial visual reasoning — the ability to answer image-related questions which require a multi-step, high-level process — is an important step towards artificial general intel- ligence. This multi-modal task requires learning a question- dependent, structured reasoning process over images from lan- guage.
A dataset of rated conceptual arguments
Introduces a dataset aimed at improving how language models reason about open-ended conceptual questions, part of a broader effort to make AI more useful for navigating the risks of its own development.
Generate-Feedback-Refine: How Much Does Model Quality in Each Role Matter?
Studies how effectively language models can give and act on feedback to refine code, testing whether weaker models can meaningfully help stronger ones improve — a possible building block for scalable oversight.
Plan B: Training LLMs to fail less severely
Proposes a second line of defense for LLM safety: alongside trying to prevent harmful responses outright, train models so that when safeguards do fail, the resulting harm is reduced in severity.
Simple probes can catch sleeper agents
An early-stage follow-up to the Sleeper Agents paper, showing that simple linear probes on a model’s internal activations can reliably flag when a backdoored model is about to defect.