Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production.

Step 1 — Forming a Hypothesis

Start with a hypothesis: 'If the redis cache cluster drops 50% of packets, the application will degrade gracefully and serve stale data without crashing.'

network-chaos.yamlyaml
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: redis-delay
  namespace: chaos-testing
spec:
  action: delay
  mode: all
  selector:
    labelSelectors:
      app: redis
  delay:
    latency: '200ms'
    correlation: '100'
    jitter: '0ms'
  duration: '1m'

Step 2 — Measuring and Improving

Run the experiment, monitor the SLIs (Service Level Indicators), and observe the results. If the system fails, you have uncovered a hidden weakness to fix.

Types of Chaos

  • Infrastructure Chaos: Killing EC2 instances, deleting pods, draining nodes.
  • Network Chaos: Introducing latency, packet loss, or blackholing traffic.
  • Application Chaos: Injecting application-level exceptions or forcing high CPU load.