I am fairly new to Kubernetes, Helm and Loki.
I have deployed Loki-stack on my minikube cluster using Helm charts and I am trying to use S3 storage as storage for Loki logs.
I tried adding the following from the documentation of Loki to my custom chart and customizing it to my running S3 instance.
schema_config:
configs:
- from: 2020-05-15
store: aws
object_store: s3
schema: v11
index:
prefix: loki_
storage_config:
aws:
bucketnames: bucket_name1, bucket_name2
endpoint: s3.endpoint.com
region: s3_region
access_key_id: s3_access_key_id
secret_access_key: s3_secret_access_key
insecure: false
sse_encryption: false
http_config:
idle_conn_timeout: 90s
response_header_timeout: 0s
insecure_skip_verify: false
s3forcepathstyle: true
I made sure to customize the region, bucket names, endpoint, etc.
But Loki is not storing any logs whatsoever there. I would like some help to show me what I might be missing or doing wrong?
I've been in your place. The soution I found was to use the Single Store Loki configuration Single Store Loki (boltdb-shipper index type)
. Also, I recomend you to use the common section to configure the shared_sotrage in an easier way and include the compactor configuration.
common:
path_prefix: /tmp/loki
storage:
s3:
bucketnames: bucket-name
region: aws-region
access_key_id: Key
secret_access_key: Secret
storage_config:
boltdb_shipper:
active_index_directory: /tmp/loki/active
shared_store: s3
cache_location: /tmp/loki/cache
cache_ttl: 24h
compactor:
working_directory: /tmp/loki/compactor
shared_store: s3
Related
PS: cross posted on drone forums here.
I'm trying to setup s3 like service for drone logs. i've tested that my AWS_* values are set correctly in the container and using aws-cli from inside container gives correct output for:
aws s3api list-objects --bucket drone-logs --endpoint-url=https://storage.googleapis.com
however, drone server itself is unable to upload logs to the bucket (with following error):
{"error":"InvalidArgument: Invalid argument.\n\tstatus code: 400, request id: , host id: ","level":"warning","msg":"manager: cannot upload complete logs","step-id":7,"time":"2023-02-09T12:26:16Z"}
drone server on startup shows that s3 related configuration was picked correctly:
rpc:
server: ""
secret: my-secret
debug: false
host: drone.XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
proto: https
s3:
bucket: drone-logs
prefix: ""
endpoint: https://storage.googleapis.com
pathstyle: true
the env. vars inside droner server container are:
# env | grep -E 'DRONE|AWS' | sort
AWS_ACCESS_KEY_ID=GOOGXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
AWS_DEFAULT_REGION=us-east-1
AWS_REGION=us-east-1
AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
DRONE_COOKIE_SECRET=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
DRONE_DATABASE_DATASOURCE=postgres://drone:XXXXXXXXXXXXXXXXXXXXXXXXXXXXX#35.XXXXXX.XXXX:5432/drone?sslmode=disable
DRONE_DATABASE_DRIVER=postgres
DRONE_DATABASE_SECRET=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
DRONE_GITHUB_CLIENT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
DRONE_GITHUB_CLIENT_SECRET=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
DRONE_JSONNET_ENABLED=true
DRONE_LOGS_DEBUG=true
DRONE_LOGS_TRACE=true
DRONE_RPC_SECRET=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
DRONE_S3_BUCKET=drone-logs
DRONE_S3_ENDPOINT=https://storage.googleapis.com
DRONE_S3_PATH_STYLE=true
DRONE_SERVER_HOST=drone.XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
DRONE_SERVER_PROTO=https
DRONE_STARLARK_ENABLED=true
the .drone.yaml that is being used is available here, on github.
the server is running using the nolimit flag:
go build -tags "nolimit" github.com/drone/drone/cmd/drone-server
I am currently setting up my aws s3 bucket for different environments so I can have data in dev, tqa, stg, and prd. The name of my bucket in dev is s3.dev.kafka.sink while in tqa it is named as s3.tqa.kafka.sink each associated with its correct env. The documentation in the Kafka Connect website doesn't specify how to be set the environments, so I did the following way, however I keep getting errors that the bucket name is not named properly.
I put it in the secret yaml file
apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
name: kafka-sink-s3-secret
namespace: namespace
spec:
backendType: secretManager
data:
-key: s3.tqa.kafka.sink
name: bucket_name
property: bucket_name
While in deployment file
env:
-name: bucket_name
valueFrom:
secretKeyRef:
name:kaka-sink-s3-secret
key: bucket_name
And I will specify the bucket name in the config:
"s3.bucket.name":"'"$bucket_name"'"
But it fails to deploy. Any idea how can i specify as s3.{{ENV}}.kafka.sink so it runs the correct bucket name in their own env in aws
Out of the box, Kafka Connect doesn't have any way to access environment variables other than those defined by the AWS SDK (the keys and profile, at least)
Sounds like you will need to use a ConfigProvider of the Kafka Connect API
Here's one example on Github, which you'd need to compile and load into your Docker images - https://github.com/giogt/kafka-env-config-provider
Inside the connector properties, use like this
"bucket.name": "${env:ENVIRONMENT_VARIABLE_NAME}"
You should be able to use Helm to better separate/template out the full bucket name within the secret/deployment resource definition
I have configured Loki to save data in s3 and indexes in dynamodb Like
https://github.com/grafana/loki/blob/fb318152fe28486640668563548b125c00c7f52b/docs/aws_storage.md
But the pod is failing to start with error
level=error ts=2020-06-29T13:04:33.086757753Z caller=table_manager.go:176 msg="error syncing tables" err="InvalidEndpointURL: invalid endpoint uri\ncaused by: parse https://dynamodb.A****A:0.amazonaws.com/: invalid port \":0.amazonaws.com\" after host"
So the current configuration is
config:
table_manager:
retention_deletes_enabled: true
schema_config:
configs:
- from: 2020-01-01
store: aws
object_store: aws
schema: v9
index:
prefix: loki_index
period: 0
storage_config:
aws:
s3: s3://A***A:0/u***K#eu-west-1/xms-loki-master
dynamodbconfig:
dynamodb: dynamodb://A***A:0/u***K#eu-west-1
Better late than never.
You're probably hitting the problem where you have either '/' or '0/' in you access key.
Two solutions: either regenerate a new access key and make sure it has no slashes in it, or url encode the key (replacing / with %2F) as suggested here:
https://github.com/grafana/loki/issues/1607
I am newbee to serverless stack,Following is the serverless.yml file. On deploying this in GitLab I get error as:
Serverless Error ---------------------------------------
An error occurred: S3XPOLLBucket - bucket already exists.
Serverless.yml file is :
service: sa-s3-resources
plugins:
- serverless-s3-sync
- serverless-s3-remover
custom:
basePath: sa-s3-resources
environment: ${env:ENV}
provider:
name: aws
stage: ${env:STAGE}
region: ${env:AWS_DEFAULT_REGION}
environment:
STAGE: ${self:provider.stage}
resources:
Resources:
S3XPOLLBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: gs-sa-xpoll-file-${self:custom.environment}-${self:provider.stage}
S3JNLBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: gs-sa-jnl-file-${self:custom.environment}-${self:provider.stage}
An Amazon S3 bucket name is globally unique, and the namespace is shared by all AWS accounts. This means that after a bucket is created, the name of that bucket cannot be used by another AWS account in any AWS Region until the bucket is deleted.
That means you have to choose a unique name that has not already chosen by someone else (or even you in a different development stack) globally
More details
https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html
I'm trying to do something pretty simple. I want to create a lambda function, an S3 bucket, and make the lambda function the event handler for the S3 bucket, using the serverless framework. Here's my definition file:
service: test-project
provider:
name: aws
stage: ${opt:stage, 'dev'}
runtime: nodejs8.10
endpointType: REGIONAL
role: arn:aws:iam::xxxxx:role/lambda_role
functions:
MyEventHandler:
name: fn
handler: src/fn.handler
events:
- s3: container
resources:
Resources:
S3BucketContainer:
Type: AWS::S3::Bucket
Properties:
BucketName: the-container-bucket
But when I run:
$ sls deploy --region us-east-1 --stage dev
I get:
Serverless: Operation failed!
Serverless Error ---------------------------------------
An error occurred: S3BucketContainer - Unable to validate the following destination configurations (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: xxxxx; S3 Extended Request ID: xxxxx).
Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com
Your Environment Information -----------------------------
OS: linux
Node Version: 8.10.0
Serverless Version: 1.34.1
Does anyone know what's wrong?
The error message is horrible but right.
The bucket is trying to be created with config to send notifications to your lambda. At this point in the deployment, the lambda hasn't given the bucket permissions to invoke and so the bucket creation fails.
If you didn't specify a custom bucket resource (to change the bucket name), serverless would have added the dependency automatically.
That all said you're not the first and the docs have been updated to reflect this issue.
Add this additional resource and apparently (see below) it should work:
resources:
Resources:
MyEventHandlerLambdaPermissionContainerS3:
Type: AWS::Lambda::Permission
Properties:
FunctionName:
"Fn::GetAtt":
- MyEventHandlerLambda
- Arn
Principal: "s3.amazonaws.com"
Action: "lambda:InvokeFunction"
SourceAccount:
Ref: AWS::AccountId
SourceArn: "arn:aws:s3:::the-container-bucket"
I say apparently because I resolved this differently (see here), using DependsOn to control the order in CloudFormation.