Chaos Testing
Use when asked to deliberately inject failure into a system to test resilience — chaos engineering, fault injection — as distinct from disaster-recovery testing (see disaster-recovery-testing), which validates a specific recovery procedure rather than exploring unknown failure modes.
Chaos testing intentionally introduces failures and disruptions into a system to evaluate its resilience and recovery ability — a proactive way to find weaknesses before they cause a real production outage, rather than discovering them the hard way.
What gets injected
Server crashes, network partitions, high CPU load, memory exhaustion, database connectivity failures — simulating the unpredictable ways real components actually fail, then observing whether the system maintains acceptable performance or degrades gracefully rather than cascading into a full outage.
Automation and tooling
Manual chaos experiments are slow and error-prone at any real scale; automated chaos tooling (Chaos Monkey, Gremlin, Litmus, among others) schedules experiments, injects failures continuously, and collects metrics without human intervention — often integrated directly into CI/CD so resilience is checked on an ongoing basis, not just once.
What it's good for
Improved reliability, faster incident response (because failure modes are already familiar from deliberate testing, not discovered live), and concrete evidence for architectural safeguards — circuit breakers, retry logic, fallback paths — actually working as designed. Particularly valuable for distributed systems and microservices, where complex inter-service dependencies create cascading-failure risk that's hard to reason about from the architecture diagram alone.
Doing it responsibly
Start in non-critical or staging environments before ever touching production; establish clear success criteria and strong observability before running an experiment; communicate with stakeholders in advance — chaos testing that surprises an on-call team defeats its own purpose.
Common pitfalls
- Running chaos experiments in production with no observability in place first — without solid monitoring, an injected failure's actual effect (and whether the system truly recovered) can't be confirmed.
- Skipping staging entirely and going straight to production — a gradual rollout (non-critical environments first) catches obviously broken experiment design before it risks real user impact.
- No communication plan — an unannounced chaos experiment can trigger a real incident response for what was actually a planned test.
- Confusing chaos testing with disaster recovery testing — chaos testing explores unknown failure modes broadly; Disaster Recovery Testing validates a specific, already-defined recovery procedure works.
Learn more
- Disaster Recovery Testing for validating a specific, defined recovery procedure.
- Failover Testing for the specific case of automatic backup-system switchover.