I'm using the new serverless compression option using minimumCompressionSize = 1KB.
Everything works fine on payloads > 1KB, yet payload <1KB are also compressed.
It's like the minimumCompressionSize remains 0 whatever value I choose (See image Below)
here is my serverless.yml:
name: aws
runtime: nodejs8.10
region: eu-west-1
apiGateway:
minimumCompressionSize: 1024
...
Have you encountered this problem? How can I solve it without manually updating the api-gateway configuration on aws?
Thank you in advance.
I followed this guide: https://medium.com/#OneMuppet_/gzip-deflate-content-from-aws-api-gateway-using-serverless-36e208da4270
and tried what is in the official documentation:
https://serverless.com/framework/docs/providers/aws/events/apigateway#compression
Have you tried to override the cloudformation template Override AWS CloudFormation Resource ?
you can specify a custom ressource with your provider name where you override the properties MinimumCompressionSize of AWS::ApiGateway::RestApi
Example (serverless.yaml) :
service: my-awesome-project-to-duplicate-pizzas
provider:
name: aws
...
you can add:
Resources:
aws:
Type: AWS::ApiGateway::RestApi
Properties:
MinimumCompressionSize: 1024
links that can help :
https://docs.aws.amazon.com/fr_fr/AWSCloudFormation/latest/UserGuide/transform-aws-serverless.html
https://docs.aws.amazon.com/fr_fr/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-restapi.html#cfn-apigateway-restapi-minimumcompressionsize
Related
serverless.yml
provider:
name: aws
runtime: nodejs16.x
deploymentBucket:
name: myS3Bucket
resources:
Resources:
MyLambdaBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: ${self.provider.deploymentBucket.name}
I would like to specify bucket name, at the same time if bucket doesn't exist I'd like to create bucket first.
But I got an Error
Error:
Could not locate deployment bucket: "myS3Bucket". Error: The specified bucket does not exist
How can I create bucket if it doesn't exist, and deploy to the bucket?
I think you have an error in your syntax. Try this:
provider:
name: aws
runtime: nodejs16.x
custom:
deploymentBucket: 'myS3Bucket'
resources:
Resources:
MyLambdaBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: ${self.provider.custom.deploymentBucket}
Note the difference with quotes, I'm also not using name here and I also use a custom key to have all the custom variables together.
I have a lambda s3 trigger and a corresponding s3 bucket in my serverless.yaml which works perfectly when I deploy it via serverless deploy.
However when I want to remove everything with serverless remove I always get the same error: (even without changing anything in the aws console)
An error occurred: DataImportCustomS31 - Received response status [FAILED] from custom resource. Message returned: The specified bucket does not exist
Which is strange because I never specified a bucket with that name in my serverless. I assume this somehow comes from the existing: true property of my s3 trigger but I can't fully explain it nor do I know how to fix it.
this is my serverless.yaml:
service: myTestService
provider:
name: aws
runtime: nodejs12.x
region: eu-central-1
profile: myprofile
stage: dev
stackTags:
owner: itsme
custom:
testDataImport:
bucketName: some-test-bucket-zxzcq234
# functions
functions:
dataImport:
handler: src/functions/dataImport.handler
events:
- s3:
bucket: ${self:custom.testDataImport.bucketName}
existing: true
event: s3:ObjectCreated:*
rules:
- suffix: .xlsx
tags:
owner: itsme
# Serverless plugins
plugins:
- serverless-plugin-typescript
- serverless-offline
# Resources your functions use
resources:
Resources:
TestDataBucket:
Type: AWS::S3::Bucket
Properties:
AccessControl: Private
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: AES256
BucketName: ${self:custom.testDataImport.bucketName}
VersioningConfiguration:
Status: Enabled
I am currently setting up my aws s3 bucket for different environments so I can have data in dev, tqa, stg, and prd. The name of my bucket in dev is s3.dev.kafka.sink while in tqa it is named as s3.tqa.kafka.sink each associated with its correct env. The documentation in the Kafka Connect website doesn't specify how to be set the environments, so I did the following way, however I keep getting errors that the bucket name is not named properly.
I put it in the secret yaml file
apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
name: kafka-sink-s3-secret
namespace: namespace
spec:
backendType: secretManager
data:
-key: s3.tqa.kafka.sink
name: bucket_name
property: bucket_name
While in deployment file
env:
-name: bucket_name
valueFrom:
secretKeyRef:
name:kaka-sink-s3-secret
key: bucket_name
And I will specify the bucket name in the config:
"s3.bucket.name":"'"$bucket_name"'"
But it fails to deploy. Any idea how can i specify as s3.{{ENV}}.kafka.sink so it runs the correct bucket name in their own env in aws
Out of the box, Kafka Connect doesn't have any way to access environment variables other than those defined by the AWS SDK (the keys and profile, at least)
Sounds like you will need to use a ConfigProvider of the Kafka Connect API
Here's one example on Github, which you'd need to compile and load into your Docker images - https://github.com/giogt/kafka-env-config-provider
Inside the connector properties, use like this
"bucket.name": "${env:ENVIRONMENT_VARIABLE_NAME}"
You should be able to use Helm to better separate/template out the full bucket name within the secret/deployment resource definition
I am newbee to serverless stack,Following is the serverless.yml file. On deploying this in GitLab I get error as:
Serverless Error ---------------------------------------
An error occurred: S3XPOLLBucket - bucket already exists.
Serverless.yml file is :
service: sa-s3-resources
plugins:
- serverless-s3-sync
- serverless-s3-remover
custom:
basePath: sa-s3-resources
environment: ${env:ENV}
provider:
name: aws
stage: ${env:STAGE}
region: ${env:AWS_DEFAULT_REGION}
environment:
STAGE: ${self:provider.stage}
resources:
Resources:
S3XPOLLBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: gs-sa-xpoll-file-${self:custom.environment}-${self:provider.stage}
S3JNLBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: gs-sa-jnl-file-${self:custom.environment}-${self:provider.stage}
An Amazon S3 bucket name is globally unique, and the namespace is shared by all AWS accounts. This means that after a bucket is created, the name of that bucket cannot be used by another AWS account in any AWS Region until the bucket is deleted.
That means you have to choose a unique name that has not already chosen by someone else (or even you in a different development stack) globally
More details
https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html
I'm trying to do something pretty simple. I want to create a lambda function, an S3 bucket, and make the lambda function the event handler for the S3 bucket, using the serverless framework. Here's my definition file:
service: test-project
provider:
name: aws
stage: ${opt:stage, 'dev'}
runtime: nodejs8.10
endpointType: REGIONAL
role: arn:aws:iam::xxxxx:role/lambda_role
functions:
MyEventHandler:
name: fn
handler: src/fn.handler
events:
- s3: container
resources:
Resources:
S3BucketContainer:
Type: AWS::S3::Bucket
Properties:
BucketName: the-container-bucket
But when I run:
$ sls deploy --region us-east-1 --stage dev
I get:
Serverless: Operation failed!
Serverless Error ---------------------------------------
An error occurred: S3BucketContainer - Unable to validate the following destination configurations (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: xxxxx; S3 Extended Request ID: xxxxx).
Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com
Your Environment Information -----------------------------
OS: linux
Node Version: 8.10.0
Serverless Version: 1.34.1
Does anyone know what's wrong?
The error message is horrible but right.
The bucket is trying to be created with config to send notifications to your lambda. At this point in the deployment, the lambda hasn't given the bucket permissions to invoke and so the bucket creation fails.
If you didn't specify a custom bucket resource (to change the bucket name), serverless would have added the dependency automatically.
That all said you're not the first and the docs have been updated to reflect this issue.
Add this additional resource and apparently (see below) it should work:
resources:
Resources:
MyEventHandlerLambdaPermissionContainerS3:
Type: AWS::Lambda::Permission
Properties:
FunctionName:
"Fn::GetAtt":
- MyEventHandlerLambda
- Arn
Principal: "s3.amazonaws.com"
Action: "lambda:InvokeFunction"
SourceAccount:
Ref: AWS::AccountId
SourceArn: "arn:aws:s3:::the-container-bucket"
I say apparently because I resolved this differently (see here), using DependsOn to control the order in CloudFormation.