# Zero Users Exposed: Dark Canary on Amazon ECS

Have you heard about dark canary? If not, this post is for you. On October 14, I was part of **AWS Community Day South Florida 2026** with the talk **"Zero Users Exposed, Full Production Validation: Dark Canary on ECS"**. It was my second time in Miami, and I was happy to be part of the first AWS Community Day South Florida. If you were in the room, thank you for staying after the keynote! Here you'll find the slides, the demo repository and the resources I used, plus a short summary of each part.

## About the talk

Staging environments are close copies of production, but they can't copy real traffic, real data or the configuration that only exists in production. A classic canary release tests with real traffic, but the users in that percentage are the ones who find the bug.

A **dark canary** runs the new version in production, on the same load balancer and the same cluster, but only **test traffic** reaches it. Native blue/green deployments in Amazon ECS support this out of the box (AWS calls it test traffic and test listener rules).

In the session we covered:

*   Why staging isn't enough, and canary vs. dark canary
    
*   What changed in ECS: native blue/green (July 2025), linear and canary (October 2025), pause hooks (May 2026), Action Logs and a configurable circuit breaker (July 2026)
    
*   How a deployment moves through its lifecycle stages, and where hooks fit
    
*   How test traffic reaches the new version: a header rule on the same listener, plus a source IP condition
    
*   The Lambda hook contract: `SUCCEEDED`, `FAILED` and `IN_PROGRESS`
    
*   A demo with a real bug, and what can go wrong when you build this
    

## Repository

[View on GitHub](https://github.com/codeanding/dark-canary-ecs)

It includes:

*   **Terraform** for the VPC, Application Load Balancer, ECS service (`BLUE_GREEN` strategy), hooks, IAM and logs
    
*   **Lambda hooks** in TypeScript: a smoke check (`POST_SCALE_UP`) and a validation that sends test traffic in rounds (`TEST_TRAFFIC_SHIFT`)
    
*   A **pause hook** at `POST_TEST_TRAFFIC_SHIFT`, resumed with `ContinueServiceDeployment`
    
*   The **demo app**, a One Piece bounty board in three releases: `v1`, `v2-broken` and `v2-fixed`
    
*   **Scripts** to bootstrap, deploy, check status, continue and destroy everything
    

![Architecture of the dark canary demo](https://raw.githubusercontent.com/codeanding/dark-canary-ecs/main/docs/architecture.png align="center")

## Presentation

You can view the full presentation below:

%[https://docs.google.com/presentation/d/e/2PACX-1vRP0_roontoysfmgFNuewTvIPAz4i3DE5OecdzKK3zSu4ei78eWn5cHHCLWt-Wzkw/pubembed] 

## Demo and how to use it

![Presenting the dark canary demo at AWS Community Day South Florida](https://cdn.hashnode.com/uploads/covers/677211221875775ad488c799/94ffa728-a114-48a3-9663-b1d0ea85086d.jpg align="center")

The app shows wanted posters of the Straw Hat crew. **🏴‍☠️** If you read my [nakama post](https://codeanding.com/building-my-own-nakama-with-rag), you know where the Straw Hats come from. Version one (the Going Merry, blue) is stable. Version two (the Thousand Sunny, green) adds a pool of 20 Den Den Mushi and never releases them, so after 20 requests every call fails with *"No Den Den Mushi available"*. It's the same pattern as an unreleased database connection, and a quick manual test in staging wouldn't catch it.

```bash
$ cp infra/terraform.tfvars.example infra/terraform.tfvars   # set your IP
$ scripts/bootstrap.sh          # creates everything and deploys v1
$ scripts/deploy.sh v2-broken   # validation fails, ECS rolls back
$ scripts/deploy.sh v2-fixed    # validation passes, the deployment pauses
$ scripts/continue.sh           # production moves to green
$ scripts/destroy.sh            # removes everything
```

To see green in a browser, send the header `x-dark-canary: 1` from an allowed IP (a browser extension works), or use curl:

```bash
$ curl -H "x-dark-canary: 1" http://<your-alb-dns>/version
```

## Key pieces of the setup

**The test rule** lives on the same listener as production and is checked first. Because a header isn't a secret, it also checks the source IP:

```hcl
resource "aws_lb_listener_rule" "test" {
  listener_arn = aws_lb_listener.prod.arn
  priority     = 10

  condition {
    http_header {
      http_header_name = "x-dark-canary"
      values           = ["1"]
    }
  }

  condition {
    source_ip {
      values = [var.presenter_cidr, "${module.vpc.nat_public_ips[0]}/32"]
    }
  }

  # forward to blue/green; ECS rewrites the weights during each deployment
}
```

**The service** tells ECS which rule is production, which one is test, and which target group is green:

```hcl
advanced_configuration {
  alternate_target_group_arn = aws_lb_target_group.app["green"].arn
  production_listener_rule   = aws_lb_listener_rule.prod.arn
  test_listener_rule         = aws_lb_listener_rule.test.arn
  role_arn                   = aws_iam_role.ecs_load_balancers.arn
}
```

**The validation hook** returns `IN_PROGRESS` with a `callBackDelay` to run in rounds, and keeps its progress in `hookDetails`:

```typescript
if (errors > 0) return { hookStatus: "FAILED" };
if (round >= rounds) return { hookStatus: "SUCCEEDED" };
return {
  hookStatus: "IN_PROGRESS",
  callBackDelay: 15,
  hookDetails: { ...details, round, ok, errors, lastError },
};
```

## Build your own deployment tooling

The scripts in the repo are only one way to drive deployments. Everything they do goes through the ECS API (`ListServiceDeployments`, `DescribeServiceDeployments`, `ContinueServiceDeployment`), and ECS also sends deployment and hook state changes to Amazon EventBridge. So you can build your own app on top: a dashboard that shows each deployment and its stage, a button to approve or roll back a paused deployment, or a Slack notification when green is waiting for review.

## Common Challenges and Solutions

*   **The event and the load balancer aren't always in sync.** When `TEST_TRAFFIC_SHIFT` starts, the deployment already reports test traffic on green, but the load balancer can take a few seconds to apply it. The validation hook checks which version answers before counting results, and waits if it's still the old one.
    
*   **Rollbacks call the hook again.** During a rollback, ECS invokes the `TEST_TRAFFIC_SHIFT` hook one more time. The hook detects it (the test traffic weight for green is `0`) and returns `SUCCEEDED`, so it never blocks the rollback.
    
*   **A deployment can wait forever.** If the new version can't start, for example because of a wrong image tag, ECS keeps trying. The deployment circuit breaker with rollback stops it.
    

## Useful resources

*   [Amazon ECS blue/green deployments](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-blue-green.html)
    
*   [Lifecycle hooks for Amazon ECS service deployments](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-lifecycle-hooks.html)
    
*   [Pause hooks](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/pause-lifecycle-hooks.html)
    
*   [Amazon ECS canary deployments](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/canary-deployment.html)
    
*   [Amazon ECS linear deployments](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-linear.html)
    
*   [Deployment circuit breaker](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-circuit-breaker.html)
    
*   [Getting started with ECS Action Logs](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/action-logs-getting-started.html)
    
*   [Terraform AWS Provider](https://registry.terraform.io/providers/hashicorp/aws/latest/docs)
    

## Thank you

Thank you to everyone who came to the session, and to the organizers and sponsors of AWS Community Day South Florida for making it possible. If you try this on your own services, let me know how it goes in the comments!

Let's keep coding and learning together!