As models become more capable, the risk of catastrophic misuse or unintended behavior scales. AI Safety is the engineering discipline of ensuring models act in accordance with human values and remain robust against adversarial attacks.
Module 1: Adversarial Robustness & Red Teaming
Jailbreaks (like the famous 'DAN' prompt) trick LLMs into bypassing their safety guardrails. Red Teaming involves systematically attacking your own models to discover these vulnerabilities before deployment.
Defensive Strategies
- Input Moderation Pipelines: Classifying prompts before they reach the LLM.
- Constitutional AI: Giving the model a strict set of rules to self-correct during generation.
- Sandboxed Execution: Ensuring Agentic LLMs cannot perform destructive OS-level commands.