Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations

Large Language Models (LLMs) have demonstrated remarkable success across diverse applications, yet their susceptibility to malicious exploitation remains a critical challenge. Notably, LLMs are known to be vulnerable to jailbreaking attacks, where adversaries craft malicious inputs to induce harmful or unethical outputs. In this paper, motivated by the unique effectiveness and scalability of In-Context Learning (ICL) in LLMs, we explore its potential to modulate the safety alignment of LLMs. Specifically, we propose the In-Context Attack (ICA), which employs harmful demonstrations to subvert LLMs' safety, and the In-Context Defense (ICD), which bolsters their resilience through examples that demonstrate…

• NN
  National Natural Science Foundation of China

  Awards: 92370129, 62376010
• NS
  Natural Science Foundation of Beijing Municipality

  Awards: QY24035, L257007
• BN
  Beijing Nova Program

  Awards: 20230484344, 20240484642