Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wal-g with S3 AssumeRole broken #1009

Open
bootc opened this issue Jul 31, 2024 · 2 comments
Open

wal-g with S3 AssumeRole broken #1009

bootc opened this issue Jul 31, 2024 · 2 comments

Comments

@bootc
Copy link

bootc commented Jul 31, 2024

I have a couple of setups using the Zalando Postgres Operator, configured to upload WALs to an S3 bucket using IAM Roles for Service Accounts (IRSA). When upgrading from spilo-16 3.2-p3 to 3.3-p1, wal-g breaks with the following errors:

ERROR: 2024/07/31 21:34:51.694882 Failed to configure multi-storage: configure primary storage: configure storage with prefix "s3://[...]": create S3 storage: create new AWS session: configure session: assume role by ARN: InvalidParameter: 1 validation error(s) found.
- minimum field size of 2, AssumeRoleInput.RoleSessionName.

I have redacted the bucket name.

I believe this is related to the wal-g upgrade from 2.0.1 to 3.0.0, and the bug is probably in there. I expect it is trying to AssumeRole with the RoleSessionName being the empty string, as no AWS_ROLE_SESSION_NAME is being supplied. Unfortunately I can't test this theory easily as AWS_ROLE_SESSION_NAME is not passed through to wal-g via configure_spilo.py.

I have reverted those clusters to 3.2-p3 for now.

@danavatavu
Copy link

danavatavu commented Sep 6, 2024

This issue is indeed blocking. In order to use timescale license we have to build spilo images with parameter TIMESCALEDB_APACHE_ONLY=false, see issue. All the re-build actions from spilo 3.0 to 3.2 are failing due to missing libsodium 1.0.17 version /~https://github.com/jedisct1/libsodium/releases used in dependencies.sh file , and spilo images tags starting with 3.3 have upper mentioned issue and backups are not being saved to AWS S3.

@Jasper-Ben
Copy link
Contributor

This is indeed a bug in wal-g > 2.0.1: wal-g/wal-g#1810

Maybe it would make sense to downgrade and freeze the wal-g dependency in spilo until this (imo rather critical) bug is resolved?
Basically, revert 5f69a19

Jasper-Ben added a commit to iris-GmbH/spilo that referenced this issue Jan 20, 2025
This reverts commit 5f69a19.

There currently is an open bug in wal-g which breaks archiving backups
to S3 when using AssumeRole on AWS: zalando#1009

Revert to a functioning version of wal-g,  until the issue has been
resolved.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants