An Availability Zone in your primary region goes dark for forty minutes. The architecture diagram says "Multi-AZ," and everyone believed it, right up until the RDS failover took 90 seconds and the connection pool in every app instance kept retrying the old endpoint because nobody had configured a sane retry policy. Meanwhile, one of the three ASGs had somehow drifted to running mostly in the AZ that just died, because a subnet was hardcoded in a launch template two years ago and nobody noticed the imbalance building up.
Nobody finds these gaps by reading the architecture diagram. Multi-AZ is a design intention until you've actually watched what happens when an AZ genuinely disappears — and the only sane way to watch that happen is to make it happen yourself, on your own terms, before AWS does it for you at 3am on a Sunday.
The Fix #
Run experiments through AWS Fault Injection Service (FIS), not by manually stopping instances by hand. FIS gives you repeatable, auditable experiments with built-in safety mechanisms.
Start with an AZ network disruption experiment, not a full instance termination — it's a closer simulation of what a real AZ-level event actually looks like: network isolation, not instant resource deletion. Verify the exact action IDs and parameter names against AWS's current FIS documentation before running this, since experiment template syntax has evolved over time:
{
"description": "AZ network disruption test - checkout service",
"targets": {
"az-subnets": {
"resourceType": "aws:ec2:subnet",
"selectionMode": "ALL",
"filters": [{"path": "AvailabilityZone", "values": ["us-east-1b"]}]
}
},
"actions": {
"disrupt-az": {
"actionId": "aws:network:disrupt-connectivity",
"parameters": {"duration": "PT10M", "scope": "az"},
"targets": {"Subnets": "az-subnets"}
}
},
"stopConditions": [
{"source": "aws:cloudwatch:alarm", "value": "arn:aws:cloudwatch:us-east-1:123456789012:alarm:checkout-error-rate"}
]
}Always wire a stop condition to a real CloudWatch alarm. If error rates or latency cross a threshold you define as "this is no longer a controlled test," FIS aborts the experiment automatically. Never run a chaos experiment without this.
Run it as a game day first, in a non-prod environment, with the whole on-call rotation watching, before ever pointing FIS at production. Define what you're validating explicitly: does the ALB drain the dead AZ's targets? Does the ASG launch replacements only in healthy AZs? Does RDS Multi-AZ failover complete inside your expected RTO? Do client connections actually recover, or do they hang on stale DNS or connection pool entries?
Check application-layer assumptions, not just infrastructure. JVM DNS caching, connection pools that don't retry on connection reset, and hardcoded subnet or AZ references in configs are what actually cause outages during real AZ events — infrastructure usually fails over correctly; applications are what don't handle it gracefully.
The Gotchas #
- Running an experiment without a stop condition turns a test into an actual incident. This is the single most important safety practice in the entire discipline — never skip it, even for a "small" experiment.
- Multi-AZ doesn't automatically mean resilient. Check for AZ affinity you didn't intend: hardcoded subnet IDs, an ASG that drifted unevenly across AZs over time, sticky sessions pinning users to specific instances.
- DNS and connection caching outlive the actual outage. Route 53 or your ALB might reroute traffic correctly within seconds, while a client's cached DNS resolution or a connection pool's stale entries keep hammering the dead AZ for minutes afterward.
- FIS needs genuinely broad IAM permissions to do its job — it has to be able to stop real infrastructure. Treat the FIS execution role with the same scrutiny as a production deploy role, not as a harmless testing tool.
- Don't run your first-ever chaos experiment against production on a Friday. This should be obvious and somehow never is. Game day in staging first, clear rollback criteria, a defined go/no-go, and a time window when your team is actually available if something goes sideways.
TL;DR #
- AWS Fault Injection Service lets you simulate real AZ failures in a controlled, auditable way — always with a CloudWatch-alarm-based stop condition wired in.
- Multi-AZ architecture diagrams don't prove resilience — hardcoded subnet references, uneven ASG distribution, and DNS/connection caching are the usual hidden failure points.
- Test in non-prod as a game day first, validate specific outcomes like failover time and connection recovery, and never skip the stop condition.