S3 and high usage of GetObject #7422
-
|
Hi! After setting up Thanos and moving existing metrics into Thanos & S3 we've noticed high activity of GetObject on our S3 bucket (Data on thanos 300GiB, data on S3 Bucket ~1TiB as we have versioning) Cost for GetObject for last 24hrs is 10-25USD/day which seems to be pretty high. Using Server Access Logs I was able to identify that StoreGateway and Compactor are 2 main services doing GetObject, are there any settings that could lower amount of requests to S3? ~/thanos-0.35.0.darwin-amd64 ./thanos tools bucket inspect --objstore.config-file=bucket.yaml | wc -l
ts=2024-06-10T12:07:02.641904Z caller=factory.go:53 level=info msg="loading bucket configuration"
ts=2024-06-10T12:07:03.309642Z caller=fetcher.go:626 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=666.439625ms duration_ms=666 cached=76 returned=76 partial=0
ts=2024-06-10T12:07:03.319353Z caller=main.go:174 level=info msg=exiting
78
Thanos Version: latest (0.35.1) |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
After increasing/adjusting in-memory caching this is not an issue anymore. Maybe it would be worth it to set in-memory cache by default to some % value from assigned memory? Default value seems to be useless. 
|
Beta Was this translation helpful? Give feedback.
-
|
Hi, can you please specify exact component and parameter to increase, i suppose its |
Beta Was this translation helpful? Give feedback.
After increasing/adjusting in-memory caching this is not an issue anymore.
Maybe it would be worth it to set in-memory cache by default to some % value from assigned memory? Default value seems to be useless.