-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Build a sidecar container that refreshes s3 credentials #11945
base: main
Are you sure you want to change the base?
Conversation
Originally I had a new top level module called fdbkubernetescredentials (beside fdbkubernetesmonitor) to host golang perpetual refresher script. It seemed a bit much just to host a simple script so moved it down under packaging/docker instead as fdb-aws-s3-credentials-fetcher. The base container is a bit fat for simple script but keeping it so for the moment at least until can show stuff is basically working. For IRSA (IAM Roles for Service Accounts) , it looks like I need to add service accounts as part of deploy but the script seems to work in EKS w/o. Lets see. |
Result of foundationdb-pr-clang-ide on Linux CentOS 7
|
Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x
|
Result of foundationdb-pr-clang-arm on Linux CentOS 7
|
Result of foundationdb-pr-cluster-tests on Linux CentOS 7
|
Result of foundationdb-pr-clang on Linux CentOS 7
|
Result of foundationdb-pr-macos on macOS Ventura 13.x
|
Result of foundationdb-pr on Linux CentOS 7
|
Result of foundationdb-pr-clang-ide on Linux CentOS 7
|
Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x
|
Result of foundationdb-pr-clang-arm on Linux CentOS 7
|
Result of foundationdb-pr-cluster-tests on Linux CentOS 7
|
Result of foundationdb-pr-clang on Linux CentOS 7
|
Result of foundationdb-pr-macos on macOS Ventura 13.x
|
Result of foundationdb-pr on Linux CentOS 7
|
|
Closed wrong PR. |
Result of foundationdb-pr-clang-ide on Linux CentOS 7
|
Result of foundationdb-pr-clang-arm on Linux CentOS 7
|
Result of foundationdb-pr-clang on Linux CentOS 7
|
Result of foundationdb-pr on Linux CentOS 7
|
Result of foundationdb-pr-cluster-tests on Linux CentOS 7
|
Result of foundationdb-pr-clang-ide on Linux CentOS 7
|
Result of foundationdb-pr-clang-arm on Linux CentOS 7
|
Result of foundationdb-pr-macos on macOS Ventura 13.x
|
Result of foundationdb-pr on Linux CentOS 7
|
fdbclient/tests/aws_fixture.sh
Outdated
else | ||
# No go so fall back to old way of writing blob credentials. | ||
if ! credentials=$( curl -H "X-aws-ec2-metadata-token: ${token}" \ | ||
http://169.254.169.254/latest/meta-data/iam/security-credentials/foundationdb-dev_node_instance_role ); then |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we want to hardcode any roles here. But I just saw that those changes are only moved around.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense. I added reading from environment variable and if not provided, use current value as default.
@@ -258,6 +259,7 @@ image_list=( | |||
'foundationdb-base' | |||
'foundationdb' | |||
'fdb-kubernetes-monitor' | |||
'fdb-aws-s3-credentials-fetcher-sidecar' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We probably have to do some additional changes in our release scripts, assuming we want to publish this script somewhere.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nod. This new image you mean? I can follow up w/ the release scripts after this goes in.
@@ -0,0 +1,151 @@ | |||
package main |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we add a copyright header?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
packaging/docker/fdb-aws-s3-credentials-fetcher/fdb-aws-s3-credentials-fetcher.go
Outdated
Show resolved
Hide resolved
} | ||
|
||
func refreshCredentials(bucket, region, credFile string) error { | ||
ctx := context.Background() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The context should actually be passed down into the method, otherwise there is no way to create a context that propagates the cancellation or the deadline.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ack
packaging/docker/fdb-aws-s3-credentials-fetcher/fdb-aws-s3-credentials-fetcher.go
Outdated
Show resolved
Hide resolved
} | ||
|
||
region := flag.String("region", defaultRegion, "AWS region for S3 endpoint (default from AWS_REGION env var)") | ||
dir := flag.String("dir", "", "Directory path where credentials file will be stored") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could use the same flag package as we use in the fdb-kubernetes-monitor package if we add this utility to the foundationdb repo: /~https://github.com/apple/foundationdb/blob/main/fdbkubernetesmonitor/main.go#L119-L140.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Used pflag instead of flag (seems nicer)
|
||
// Main credential refresh loop | ||
log.Printf("Starting credential refresh loop") | ||
for { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another approach would be to use a ticker
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. Cleaner.
// Main credential refresh loop | ||
log.Printf("Starting credential refresh loop") | ||
for { | ||
if err := refreshCredentials(*bucket, *region, credFile); err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe the AWS SDK package has some functionality to check if the tokens are expired or not, that might be a good option instead of requesting a new token every time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the pointer...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the review.
…dentials-fetcher.go Co-authored-by: Johannes Scheuermann <johscheuer@users.noreply.github.com>
…dentials-fetcher.go Co-authored-by: Johannes Scheuermann <johscheuer@users.noreply.github.com>
Result of foundationdb-pr-clang-ide on Linux CentOS 7
|
Result of foundationdb-pr-clang-ide on Linux CentOS 7
|
Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x
|
Result of foundationdb-pr-clang-arm on Linux CentOS 7
|
Result of foundationdb-pr-clang-arm on Linux CentOS 7
|
Result of foundationdb-pr-cluster-tests on Linux CentOS 7
|
Result of foundationdb-pr-macos on macOS Ventura 13.x
|
Result of foundationdb-pr-cluster-tests on Linux CentOS 7
|
Result of foundationdb-pr-clang-ide on Linux CentOS 7
|
Result of foundationdb-pr-clang-ide on Linux CentOS 7
|
Result of foundationdb-pr on Linux CentOS 7
|
Result of foundationdb-pr-clang on Linux CentOS 7
|
Result of foundationdb-pr-clang on Linux CentOS 7
|
Result of foundationdb-pr on Linux CentOS 7
|
Result of foundationdb-pr-clang-arm on Linux CentOS 7
|
Result of foundationdb-pr-clang on Linux CentOS 7
|
Result of foundationdb-pr-clang-arm on Linux CentOS 7
|
Result of foundationdb-pr-cluster-tests on Linux CentOS 7
|
Result of foundationdb-pr-clang on Linux CentOS 7
|
Result of foundationdb-pr on Linux CentOS 7
|
Result of foundationdb-pr on Linux CentOS 7
|
Result of foundationdb-pr-cluster-tests on Linux CentOS 7
|
Result of foundationdb-pr-macos on macOS Ventura 13.x
|
packaging/docker/Dockerfile
Add fdb-aws-s3-credentials-fetcher-sidecar container. Runs perpetual script that writes blob-credentials.json to /var/fdb.
packaging/docker/build-images.sh
Build and publish new sidecar container
packaging/docker/fdb-aws-s3-credentials-fetcher/README.md
packaging/docker/fdb-aws-s3-credentials-fetcher/fdb-aws-s3-credentials-fetcher.go
packaging/docker/fdb-aws-s3-credentials-fetcher/go.mod
packaging/docker/fdb-aws-s3-credentials-fetcher/go.sum
Script that fetches credentials via IRSA (IAM Roles for Service Accounts).