Deployment Challenges with Secret Injection in ECS Services Using CMK Post-Copilot 1.32.1 Update

[HoosCoding]

Hello,

I am writing to address an issue encountered with the recent release of Copilot version 1.32.1. This pertains to the fix implemented for "KMS access denied errors when downloading the static env file #5514." We are experiencing deployment challenges for services that utilize the existing functionality of secret injection into the ECS Service with a CMK (Customer Master Key).
Specifically, we're encountering the error: "ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secret from asm: service call has been retried 1 time(s): failed to fetch secret arn:aws:secretsmanager…". This issue appears to stem from the restricted permissions assigned to the execution role, which now only has access to the ArtifactKeyARN used by Copilot. Consequently, customers who utilize their own key are forced to apply a YAML patch override to their templates to regain access.
In my view, an optimal solution would be for Copilot to automatically manage this dependency. If a user uses a secret, the key associated with it should be dynamically injected into the execution role. This approach would streamline the process and enhance user experience by minimizing manual interventions.

Below a screenshot of the change that I am referring to in the latest release.

I look forward to your guidance and any other potential solutions for this issue.

Thanks!

[Lou1415926]

@HoosCoding Hello 👋🏼 !

For context, the change that caused the issue was done to partially fix #4628: to scope down the permission that the service's execution role has.

In my view, an optimal solution would be for Copilot to automatically manage this dependency. If a user uses a secret, the key associated with it should be dynamically injected into the execution role. This approach would streamline the process and enhance user experience by minimizing manual interventions.

Great idea. I totally agree that Copilot should have a way of automating this instead of requiring users to do YAML patch. Though I have a slightly different proposal than what you suggested ⬇️

[Proposal] Permission to decrypt tagged KMS key

We can enhance the currently-scoped-down execution role's permission so that it is able to decrypt any KMS keys that are tagged with copilot tags:

- Effect: 'Allow'
  Action:
    - 'kms:Decrypt'
  Resource: "arn:aws:kms:ap-southeast-1:111122223333:key/*",
  Condition:
    StringEquals:
      "aws:ResourceTag/copilot-application": !Ref AppName
      "aws:ResourceTag/copilot-environment": !Ref EnvName

This approach is basically the same as what we've done with SSM/secretsmanager secret. You tag the resources with copilot-application and copilot-environment as a form of agreement to let Copilot access the resource.

Let me know what you think!

[HoosCoding]

Hello @Lou1415926,
I appreciate the fast response! Yes, that change makes sense and agree that it improves security. Your proposal sounds way better and great functionality to have as a copilot user. I believe users using CMKs will find themselves in need of it sooner than later with that released change.

Thank you!

[iamhopaul123]

Hello this is now fixed in /~https://github.com/aws/copilot-cli/releases/tag/v1.33.0!