When configuring a lambda function in the serverless framework, i am trying to add a kinesis stream as the event course:
here is the snippet from serverless.yml
functions:
Foo:
handler: handler.foo
events:
- stream:
arn: arn:aws:kinesis:us-east-1:783995676505:stream/search-helper
batchSize: 100
startingPosition: LATEST
enabled: false
The deployment via "serverless deploy" is successful however the trigger does not get added to the function configuration.
I checked the yml file using a yml validatior and there are no errors. What am i missing here ?
The yml file needs to be indented just after stream:
functions:
Foo:
handler: handler.foo
events:
- stream:
arn: arn:aws:kinesis:us-east-1:783995676505:stream/search-helper
batchSize: 100
startingPosition: LATEST
enabled: false
See the Serveless Framework examples at https://serverless.com/framework/docs/providers/aws/events/streams/
Related
I'm trying to implement aws secrets rotation with with the serverless framework (https://alexharv074.github.io/2020/11/23/adding-hosted-rotation-lambda-to-a-database-stack.html)
AppGraphqDbAppsSecretRotationSchedule:
Type: AWS::SecretsManager::RotationSchedule
Properties:
SecretId: !Ref AppGraphqlDbAppsSecretsManagerSecret
HostedRotationLambda:
RotationType: PostgreSQLSingleUser
RotationRules:
AutomaticallyAfterDays: 30
I get the error
To use the HostedRotationLambda property, you must use the AWS::SecretsManager transform.
How can I add "transform"? Below a working Aws cloudformation template
AWSTemplateFormatVersion: 2010-09-09
Description: Rotation Lambda example stack
Transform: AWS::SecretsManager-2020-07-23
Parameters: {}
Resources:
I've create an IRSA role in terraform so that the associated service account can be used by a K8s job to access an S3 bucket but I keep getting an AccessDenied error within the job.
I first enabled IRSA in our EKS cluster with enable_irsa = true in our eks module.
I then created a simple aws_iam_policy as:
resource "aws_iam_policy" "eks_s3_access_policy" {
name = "eks_s3_access_policy"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = [
"s3:*",
]
Effect = "Allow"
Resource = "arn:aws:s3:::*"
},
]
})
}
and a iam-assumable-role-with-oidc:
module "iam_assumable_role_with_oidc_for_s3_access" {
source = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
version = "~> 3.0"
create_role = true
role_name = "eks-s3-access"
role_description = "Role to access s3 bucket"
tags = { Role = "eks_s3_access_policy" }
provider_url = replace(module.eks.cluster_oidc_issuer_url, "https://", "")
role_policy_arns = [aws_iam_policy.eks_s3_access_policy.arn]
number_of_role_policy_arns = 1
oidc_fully_qualified_subjects = ["system:serviceaccount:default:my-user"]
}
I created a K8s service account using Helm like:
Name: my-user
Namespace: default
Labels: app.kubernetes.io/managed-by=Helm
Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::111111:role/eks-s3-access
meta.helm.sh/release-name: XXXX
meta.helm.sh/release-namespace: default
Image pull secrets: <none>
Mountable secrets: my-user-token-kwwpq
Tokens: my-user-token-kwwpq
Events: <none>
Finally, jobs are created using the K8s API from a job template:
apiVersion: batch/v1
kind: Job
metadata:
name: job
namespace: default
spec:
template:
spec:
serviceAccountName: my-user
containers:
- name: {{ .Chart.Name }}
env:
- name: AWS_ROLE_ARN
value: arn:aws:iam::746181457053:role/eks-s3-access
- name: AWS_WEB_IDENTITY_TOKEN_FILE
value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
volumeMounts:
- mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
name: aws-iam-token
readOnly: true
volumes:
- name: aws-iam-token
projected:
defaultMode: 420
sources:
- serviceAccountToken:
audience: sts.amazonaws.com
expirationSeconds: 86400
path: token
When the job attempts to get the specified credentials, however, the specified token is not there:
2021-08-03 18:02:41 Refreshing temporary credentials failed during mandatory refresh period.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 291, in _protected_refresh
metadata = await self._refresh_using()
File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 345, in fetch_credentials
return await self._get_cached_credentials()
File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 355, in _get_cached_credentials
response = await self._get_credentials()
File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 410, in _get_credentials
kwargs = self._assume_role_kwargs()
File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 420, in _assume_role_kwargs
identity_token = self._web_identity_token_loader()
File "/usr/local/lib/python3.7/site-packages/botocore/utils.py", line 2365, in __call__
with self._open(self._web_identity_token_path) as token_file:
FileNotFoundError: [Errno 2] No such file or directory: '/var/run/secrets/eks.amazonaws.com/serviceaccount/token'
From what is described in https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/ a webhook typically creates these credentials when the pod is created. However, since we're creating the new k8s' job on demand within the k8s cluster, I suspect that the webhook is not creating any such credentials.
How can I request the correct credentials to be created from within a K8s cluster? Is there a way to instantiate the webhook from within the cluster?
There are a couple of things that could cause this to fail.
Check all settings for the IRSA role. For the trust relationship setting check the name of the namespace and the name of service account are correct. Only if these settings match the role can be assumed.
While the pod is running try to access the pod with a shell. Check the content of the "AWS_*" environment variables. Check AWS_ROLE_ARN points to the correct role. Check, if the file which AWS_WEB_IDENTITY_TOKEN_FILE points to, is in its place and it is readable. Just try to do a cat on the file to see if it is readable.
If you are running your pod a non-root (which is recommended for security reasons) make sure the user who is running the pod has access to the file. If not, adjust the securityContext for the pod. Maybe the setting of fsGroup can help here. https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#security-context
Make sure the SDK your pos is using supports IRSA. If you are using older SDKs IRSA may not be supported. Look into the IRSA documentation for supported SDK versions. https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
I am using Serverless framework with the serverless-s3-local plugin to test my code during development. However, despite being in offline mode, the real S3 bucket is being written to. How can I alter my configuration to use a local fake s3 bucket when in offline mode?
Relevant serverless.yml sections:
plugins:
- serverless-stack-output
- serverless-plugin-include-dependencies
- serverless-layers
- serverless-deployment-bucket
- serverless-s3-local
- serverless-offline
custom:
#...
s3:
bucketName: test-s3-buck
host: localhost
serverless-offline:
ignoreJWTSignature: true
httpPort: 4000
noAuth: true
directory: /tmp
resources:
Resources:
#...
Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: ${self:custom.s3.bucketName}
Endpoint Calling S3:
import boto3
def post(event, context):
s3_path = "/test.txt"
body = "test"
encoded_string = body.encode("utf-8")
s3 = boto3.resource("s3")
bucket_name = "test-s3-buck"
s3.Bucket(bucket_name).put_object(Key=s3_path, Body=encoded_string)
response = {
"statusCode": 200,
"body": "Created."
}
return response
Launching Serverless Offline:
serverless offline start
on the readme file in serverless-s3-local we have:
const S3 = new AWS.S3({
s3ForcePathStyle: true,
accessKeyId: 'S3RVER', // This specific key is required when working offline
secretAccessKey: 'S3RVER',
endpoint: new AWS.Endpoint('http://localhost:4569'),
});
you can achieve the same with boto:
import boto3
client = boto3.client(
's3',
aws_access_key_id='S3RVER',
aws_secret_access_key='S3RVER'
)
which means, when you run your serverless offline start you need to set the aws access key id to S3RVER and aws secret access key to S3RVER, otherwise, the real bucket will be used.
also in the readme, there's instructions to setup a s3local aws profile, https://github.com/ar90n/serverless-s3-local#triggering-aws-events-offline
another way to achieve it is to run your command with environment variables:
AWS_ACCESS_KEY_ID=S3RVER AWS_SECRET_ACCESS_KEY=S3RVER serverless offline start
in that way, the aws-sdk inside your code will read the correct values for the offline mode
I am using serverless.
When I setup one of my functions as the following, which includes authorizer, on the client, I receive 401.
However when I remove it, there are no problems.
provider:
name: aws
runtime: nodejs8.10
region: eu-west-1
environment:
USER_POOL_ARN: "arn:aws:cognito-idp:eu-west-1:974280.....:userpool/eu-west-1........"
functions:
create:
handler: handlers/create.main
events:
- http:
path: create
method: post
cors: true
authorizer:
type: COGNITO_USER_POOLS
name: serviceBAuthFunc
arn: ${self:provider.environment.USER_POOL_ARN}
On the client, I expect a logged in user of the same user pool could get expected response. However it returns 401.
Any help is appreciated. Thanks.
After desperate hours spent, I have come up with the solution.
For anyone who comes across the same issue, here is a solution that worked for me.
Add integration: lambda after cors: true (though the order doesn't matter).
Below is just demonstrating that.
functions:
create:
handler: handlers/create.main
events:
- http:
path: create
method: post
cors: true
integration: lambda // this solves the problem
authorizer:
type: COGNITO_USER_POOLS
name: serviceBAuthFunc
arn: ${self:provider.environment.USER_POOL_ARN}
Send Authorization header with the value of Auth.currentSession().idToken.jwtToken while making the request.
Below is an example for sending headers using API of #aws-amplify/api and Auth of #aws-amplify/auth.
const currentSession = await Auth.currentSession()
await API.post(
'your-endpoint-name',
"/your-endpoint-path/..",
{
headers: {
'Authorization': currentSession.idToken.jwtToken
}
}
)
I have two lambda functions defined in serverless.yml: graphql and convertTextToSpeech. The former (in one of the GraphQL endpoints) should write to SNS topic to execute the latter one. Here is my serverless.yml file:
service: hello-world
provider:
name: aws
runtime: nodejs6.10
plugins:
- serverless-offline
functions:
graphql:
handler: dist/app.handler
events:
- http:
path: graphql
method: post
cors: true
convertTextToSpeach:
handler: dist/tasks/convertTextToSpeach.handler
events:
- sns:
topicName: convertTextToSpeach
displayName: Convert text to speach
And GraphQL endpoint writing to SNS:
// ...
const sns = new AWS.SNS()
const params = {
Message: 'Test',
Subject: 'Test SNS from lambda',
TopicArn: 'arn:aws:sns:us-east-1:101972216059:convertTextToSpeach'
}
await sns.publish(params).promise()
// ...
There are two issues here:
Topic ARN (which is required to write to a topic) is hardcoded it. How I can get this in my code "dynamically"? Is it provided somehow by serverless framework?
Even when topic arn is hardcoded lambda functions does not have permissions to wrote to that topic. How I can define such permissions in serverless.yml file?
1) You can resolve the topic dynamically.
This can be done through CloudFormation Intrinsic Functions, which are available within the serverless template (see the added environment section).
functions:
graphql:
handler: handler.hello
environment:
topicARN:
Ref: SNSTopicConvertTextToSpeach
events:
- http:
path: graphql
method: post
cors: true
convertTextToSpeach:
handler: handler.hello
events:
- sns:
topicName: convertTextToSpeach
displayName: Convert text to speach
In this case, the actual topic reference name (generated by the serverless framework) is SNSTopicConvertTextToSpeach. The generation of those names is explained in the serverless docs.
2) Now that the ARN of the topic is mapped into an environment variable, you can access it within the GraphQL lambda through the process variable (process.env.topicARN).
// ...
const sns = new AWS.SNS()
const params = {
Message: 'Test',
Subject: 'Test SNS from lambda',
TopicArn: process.env.topicARN
}
await sns.publish(params).promise()
// ...