AWS CloudWatch Log Exporter is a productivity tool that makes it easy to schedule automated log exports to an AWS S3 Bucket. Currently in AWS, exporting logs is a manual task that is done using tools like Log Insights and the AWS CLI. None of these options provide a way to export logs to an S3 Bucket where additional features like storage classes, lifecycle policies, and legal holds can be leveraged. As a result, those desiring some form of automation will create a custom solution using something like a Lambda function to achieve this, but there are many approaches available that are written in different programming languages, and not everyone is a programmer. AWS CloudWatch Log Exporter is built ready to go, and can be quickly deployed as a CronJob or Scheduled Task to any cluster, or as a container directly.
- Export CloudWatch logs on an hourly or daily basis to an S3 Bucket.
- Automatic calculation of the needed time period being exported.
- Enable access by leveraging the AWS Credential Provider Chain. IRSA (IAM Roles for Service Accounts) is recommended.
- Can be deployed to EKS, ECS, and other managed and self-managed clusters.
AWS CloudWatch Log Exporter container images are currently hosted on the following platforms:
For production use cases, it is not recommended to pull an image with the :latest tag, or no tag since these are equivalent.
AWS CloudWatch Log Exporter can be optionally deployed to a Kubernetes cluster using the AWS CloudWatch Log Exporter Helm Chart that is managed in a separate repository. All of the features described below and more are supported by this chart.
To run the application directly in a non-Kubernetes environment, use the approach below. Both nerdctl and docker CLIs are interchangeable here:
nerdctl run --rm --name aws-cwl-exporter \
-e AWS_ACCESS_KEY_ID=xxxxxx \
-e AWS_SECRET_ACCESS_KEY=xxxxxx \
-e AWS_DEFAULT_REGION=eu-west-3 \
-e LOG_GROUP_NAME="/aws/lambda/hello-world-dev" \
-e S3_BUCKET_NAME=s3-example-log-exports \
-e EXPORT_PREFIX=export-task-output \
-e EXPORT_PERIOD=hourly \
stevenjdh/aws-cwl-exporter:latestIf successful, the output will look similar to the following:
Creating [HOURLY][2022-10-27T02:00:00Z to 2022-10-27T02:59:59Z] export task request...
----------------------------------------------------
| CreateExportTask |
+--------+-----------------------------------------+
| taskId| 00000000-0000-0000-0000-000000000000 |
+--------+-----------------------------------------+
To track the task progress, use:
aws logs describe-export-tasks --task-id 00000000-0000-0000-0000-000000000000 --output table
The aws logs describe-export-tasks command can be used to track the progress of long running tasks. For example, using the command provides the following details:
--------------------------------------------------------------------------------
| DescribeExportTasks |
+------------------------------------------------------------------------------+
|| exportTasks ||
|+--------------------+-------------------------------------------------------+|
|| destination | s3-example-log-exports ||
|| destinationPrefix | export-task-output ||
|| from | 1666836000000 ||
|| logGroupName | /aws/lambda/hello-world-dev ||
|| taskId | 00000000-0000-0000-0000-000000000000 ||
|| taskName | log-group-1666839900000 ||
|| to | 1666839599000 ||
|+--------------------+-------------------------------------------------------+|
||| executionInfo |||
||+-------------------------------------+------------------------------------+||
||| completionTime | 1666840020000 |||
||| creationTime | 1666839900000 |||
||+-------------------------------------+------------------------------------+||
||| status |||
||+---------------------+----------------------------------------------------+||
||| code | COMPLETED |||
||| message | Completed successfully |||
||+---------------------+----------------------------------------------------+||
📝NOTE: There is a limit of "one active (running or pending) export task at a time, per account. This quota can't be changed." See CloudWatch Logs quotas for more information.
The following environment variables are used to store the needed configuration. For access, the AWS Credential Provider Chain is used, which supports providing static credentials like below, or the recommended approach, enabling role based access via IRSA (IAM Roles for Service Accounts).
| Environment variable | Description |
|---|---|
| LOG_GROUP_NAME: | Required. The name of the log group source for exporting logs from. |
| S3_BUCKET_NAME: | Required. The name of S3 bucket storing the exported log data. The bucket must be in the same AWS region. |
| EXPORT_PREFIX: | Required. The prefix used as the start of the key for every object exported. |
| EXPORT_PERIOD: | Optional. The hourly or daily period used for collecting logs. Not required unless set to daily. |
| AWS_ACCESS_KEY_ID: | Optional. The AWS access key associated with an IAM user or role. Not required when using IRSA. |
| AWS_SECRET_ACCESS_KEY: | Optional. The AWS secret key associated with the access key. Not required when using IRSA. |
| AWS_DEFAULT_REGION: | Optional. The AWS Region to use for requests. Must match log group and S3 bucket region. Not required when using IRSA. |
The following policies define the permissions that are needed for exporting CloudWatch logs and storing them in S3.
This policy example grants logs:CreateExportTask rights to the User or IRSA role (Recommended) associated with the application.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "CWLGrantCreateExportTaskRights",
"Effect": "Allow",
"Action": "logs:CreateExportTask",
"Resource": "arn:aws:logs:eu-west-3:000000000000:*"
}
]
}This policy example grants write access to the logs.eu-west-3.amazonaws.com service. See Set permissions on an Amazon S3 bucket for additional information.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "logs.eu-west-3.amazonaws.com"
},
"Action": "s3:GetBucketAcl",
"Resource": "arn:aws:s3:::s3-example-log-exports"
},
{
"Effect": "Allow",
"Principal": {
"Service": "logs.eu-west-3.amazonaws.com"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::s3-example-log-exports/*",
"Condition": {
"StringEquals": {
"s3:x-amz-acl": "bucket-owner-full-control"
}
}
}
]
}Thanks for your interest in contributing! There are many ways to contribute to this project. Get started here.
Many commonly asked questions are answered in the FAQ: /~https://github.com/StevenJDH/aws-cwl-exporter/wiki/FAQ
| Method | Address |
|---|---|
| PayPal: | https://www.paypal.me/stevenjdh |
| Cryptocurrency: | Supported options |
// Steven Jenkins De Haro ("StevenJDH" on GitHub)