minio support

[aaron-37]

Is your feature request related to a problem? Please describe.
I cannot connect to minio through the configuration of aws

Describe the solution you'd like
Add minio configuration and library

Describe alternatives you've considered
nil

Additional context
nil

[bastjan]

What's the issue/ error message you're having? I've recently configured loki with minio as chunk store.

storage_config:
  aws:
    s3: https://loki:loki_0123456789@minio.:9000/chunks
    s3forcepathstyle: true

Worked for me. Notice the dot behind minio to force the s3 library to use minio as a host name, not as an aws region.

[aaron-37]

thank u very much,u did me a favor

[pomazanbohdan]

@bastjan
pls paste full config

[wildermesser]

I am joining to question about full config. It seems like in the new version of loki field dynamodb is required. I tried:

    storage_config:                                                                                                                                          
      aws:                                                                                                                                                   
        s3: s3://user:password@minio.service.consul.:9091/loki                                                                                  
        s3forcepathstyle: true                                                                                                                               
    schema_config:                                                                                                                                           
      configs:                                                                                                                                               
      - from: 2018-04-15                                                                                                                                     
        store: aws                                                                                                                                           
        schema: v11                                                                                                                                          
        index:                                                                                                                                               
          prefix: index_                                                                                                                                     
          period: 168h    

But I am getting:
loki level=error ts=2020-05-06T07:12:06.390260138Z caller=main.go:70 msg="error initialising loki" err="error initialising module: table-manager: Must set -dynamodb.url in aws mode"

[pomazanbohdan]

@wildermesser

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_transfer_retries: 1

schema_config:
  configs:
    - from: 2018-04-15
      store: boltdb
      object_store: aws
      schema: v11
      index:
        prefix: index_
        period: 0

storage_config:
  boltdb:
     directory: /tmp/loki/index
  aws:
    s3: https://xxx:yyy@minio.pomazan.xyz./loki
    s3forcepathstyle: true

[wildermesser]

Thank you @pomazanbohdan, your config works fine. I misunderstand the fact that the index can not be stored in S3. Do you know what happens when the index in the filesystem directory will be lost?
Will it be constructed from scratch?

[pomazanbohdan]

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0

table_manager:
  chunk_tables_provisioning:
    inactive_read_throughput: 0
    inactive_write_throughput: 0
    provisioned_read_throughput: 0
    provisioned_write_throughput: 0
  index_tables_provisioning:
    inactive_read_throughput: 0
    inactive_write_throughput: 0
    provisioned_read_throughput: 0
    provisioned_write_throughput: 0
  retention_deletes_enabled: false
  retention_period: 0

I think that you can configure it so that the index is also in the file storage.
In his experience, restarting the Loki service did not break its performance, most likely he himself restored the index in the process work.

[wildermesser]

Thank for the answer! I am searching for a way that makes Loki stateless when using Minio. I have a distributed Minio setup, so Loki's chunk data is replicated. And Loki is deployed in a kubernetes as stateful set.

I will try to move in a simple deployment instead statefulset. And will have a look at what happens when pod will be started in another node, with emplty index data directory

[wildermesser]

Index for older logs data didn't rebuild after move pod to another node. So HA for index should be provided Cassandra, for example

[IgorOhrimenko]

The index can be stored in s3(Minio or alternative), my config works well:

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: aws
      schema: v11
      index:
        prefix: index_
        period: 24h
storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    cache_ttl: 480h
    resync_interval: 1m
    shared_store: s3
  aws:
    s3: https://login:password@s3.selcdn.ru/bucketName
    region: ru-1
    s3forcepathstyle: true

[MaestroJurko]

@IgorOhrimenko I have a similar configuration but when I restart loki pod I dont see the old logs, but they are there in minio.

      compactor:
        retention_enabled: true
        shared_store: s3
      limits_config:
        max_cache_freshness_per_query: 10m
        retention_period: 24h
        retention_stream:
          - selector: '{app="appname"}'
            priority: 1
            period: 744h
      storage_config:
        boltdb_shipper:
          shared_store: s3
        aws:
          s3: s3://loki:secret@minio.monitoring.svc.cluster.local:9000/loki-data
          s3forcepathstyle: true

      schema_config:
        configs:
          - from: "2020-09-07"
            store: boltdb-shipper
            object_store: s3
            schema: v11
            index:
              period: 24h
              prefix: loki_index_

What am I doing wrong?

[MaestroJurko]

When I changed the address of s3 to use the . at the end, I see the index folder in minio now and querying logs that happened before the pod restart are now returned and shown.

 s3: s3://loki:secret@minio.monitoring.svc.cluster.local.:9000/loki-data