Zero Users Exposed: Dark Canary on Amazon ECS
Resources from my talk at AWS Community Day South Florida 2026
Have you heard about dark canary? If not, this post is for you. On October 14, I was part of AWS Community Day South Florida 2026 with the talk "Zero Users Exposed, Full Production Validation: Dark Canary on ECS". It was my second time in Miami, and I was happy to be part of the first AWS Community Day South Florida. If you were in the room, thank you for staying after the keynote! Here you'll find the slides, the demo repository and the resources I used, plus a short summary of each part.
About the talk
Staging environments are close copies of production, but they can't copy real traffic, real data or the configuration that only exists in production. A classic canary release tests with real traffic, but the users in that percentage are the ones who find the bug.
A dark canary runs the new version in production, on the same load balancer and the same cluster, but only test traffic reaches it. Native blue/green deployments in Amazon ECS support this out of the box (AWS calls it test traffic and test listener rules).
In the session we covered:
Why staging isn't enough, and canary vs. dark canary
What changed in ECS: native blue/green (July 2025), linear and canary (October 2025), pause hooks (May 2026), Action Logs and a configurable circuit breaker (July 2026)
How a deployment moves through its lifecycle stages, and where hooks fit
How test traffic reaches the new version: a header rule on the same listener, plus a source IP condition
The Lambda hook contract:
SUCCEEDED,FAILEDandIN_PROGRESSA demo with a real bug, and what can go wrong when you build this
Repository
It includes:
Terraform for the VPC, Application Load Balancer, ECS service (
BLUE_GREENstrategy), hooks, IAM and logsLambda hooks in TypeScript: a smoke check (
POST_SCALE_UP) and a validation that sends test traffic in rounds (TEST_TRAFFIC_SHIFT)A pause hook at
POST_TEST_TRAFFIC_SHIFT, resumed withContinueServiceDeploymentThe demo app, a One Piece bounty board in three releases:
v1,v2-brokenandv2-fixedScripts to bootstrap, deploy, check status, continue and destroy everything
Presentation
You can view the full presentation below:
Demo and how to use it
The app shows wanted posters of the Straw Hat crew. 🏴☠️ If you read my nakama post, you know where the Straw Hats come from. Version one (the Going Merry, blue) is stable. Version two (the Thousand Sunny, green) adds a pool of 20 Den Den Mushi and never releases them, so after 20 requests every call fails with "No Den Den Mushi available". It's the same pattern as an unreleased database connection, and a quick manual test in staging wouldn't catch it.
$ cp infra/terraform.tfvars.example infra/terraform.tfvars # set your IP
$ scripts/bootstrap.sh # creates everything and deploys v1
$ scripts/deploy.sh v2-broken # validation fails, ECS rolls back
$ scripts/deploy.sh v2-fixed # validation passes, the deployment pauses
$ scripts/continue.sh # production moves to green
$ scripts/destroy.sh # removes everything
To see green in a browser, send the header x-dark-canary: 1 from an allowed IP (a browser extension works), or use curl:
$ curl -H "x-dark-canary: 1" http://<your-alb-dns>/version
Key pieces of the setup
The test rule lives on the same listener as production and is checked first. Because a header isn't a secret, it also checks the source IP:
resource "aws_lb_listener_rule" "test" {
listener_arn = aws_lb_listener.prod.arn
priority = 10
condition {
http_header {
http_header_name = "x-dark-canary"
values = ["1"]
}
}
condition {
source_ip {
values = [var.presenter_cidr, "${module.vpc.nat_public_ips[0]}/32"]
}
}
# forward to blue/green; ECS rewrites the weights during each deployment
}
The service tells ECS which rule is production, which one is test, and which target group is green:
advanced_configuration {
alternate_target_group_arn = aws_lb_target_group.app["green"].arn
production_listener_rule = aws_lb_listener_rule.prod.arn
test_listener_rule = aws_lb_listener_rule.test.arn
role_arn = aws_iam_role.ecs_load_balancers.arn
}
The validation hook returns IN_PROGRESS with a callBackDelay to run in rounds, and keeps its progress in hookDetails:
if (errors > 0) return { hookStatus: "FAILED" };
if (round >= rounds) return { hookStatus: "SUCCEEDED" };
return {
hookStatus: "IN_PROGRESS",
callBackDelay: 15,
hookDetails: { ...details, round, ok, errors, lastError },
};
Build your own deployment tooling
The scripts in the repo are only one way to drive deployments. Everything they do goes through the ECS API (ListServiceDeployments, DescribeServiceDeployments, ContinueServiceDeployment), and ECS also sends deployment and hook state changes to Amazon EventBridge. So you can build your own app on top: a dashboard that shows each deployment and its stage, a button to approve or roll back a paused deployment, or a Slack notification when green is waiting for review.
Common Challenges and Solutions
The event and the load balancer aren't always in sync. When
TEST_TRAFFIC_SHIFTstarts, the deployment already reports test traffic on green, but the load balancer can take a few seconds to apply it. The validation hook checks which version answers before counting results, and waits if it's still the old one.Rollbacks call the hook again. During a rollback, ECS invokes the
TEST_TRAFFIC_SHIFThook one more time. The hook detects it (the test traffic weight for green is0) and returnsSUCCEEDED, so it never blocks the rollback.A deployment can wait forever. If the new version can't start, for example because of a wrong image tag, ECS keeps trying. The deployment circuit breaker with rollback stops it.
Useful resources
Thank you
Thank you to everyone who came to the session, and to the organizers and sponsors of AWS Community Day South Florida for making it possible. If you try this on your own services, let me know how it goes in the comments!
Let's keep coding and learning together!
