I have created lambda with Kinesis stream as trigger for Lambda. and also have Filter criteria added to start Lambda if event from stream matches provided filter criteria.
I push json payload on lambda and lambda boots as expected based on filter criteria given, all Good.
But decided to use Glue schema registry integration where producer will validate event payload with Glue registry and add encode event with GlueSchemaRegistrySerializer and publish it on kinesis stream
Now when it comes to lambda, with same payload as previous somehow Lambda does not boot up, due to filter criteria mismach.
non glue event processed sucessfully but glue schema event not starts lambda due to filter criteria
we are using below lib for glue schema event creation, GlueSchemaRegistrySerializer is from below lib,
<dependency>
<groupId>software.amazon.glue</groupId>
<artifactId>schema-registry-serde</artifactId>
</dependency>
THanks in advance...
Related
I have a lambda to process the files in a folder of a s3 bucket. I would like to setup an alarm/notification if objects are in the folders for more than 7 hours and not processed by the lambda
You can use the tags of objects in S3, have something like tag name Processed : true or false changed by your lambda processor.
Then in your another scheduled lambda you can check the creation object if > 7h and processed : false (that means not processed by the lambda), if found you create a notification in SNS
Set object expiration to 7 hours for the S3 bucket. Then have a lambda get triggered by the delete event. The lambda can be one that notifies you and saves the file into another bucket or forwards it to your original lambda. The lambda triggered could be the one that should have been triggered when uploading the object.
Alternatively, you can add tags to the uploaded files. A tag could be ttl: <date-to-delete>. You have a CloudWatch scheduled event that runs a lambda, for instance every hour, and checks for all objects in the S3 bucket whether the ttl-tag's value is older than the current time.
Personally, I would go with the first approach as it's more event driven and less scheduled processing.
On another note, it's rather strange that the lambda doesn't get triggered for some S3 objects. I don't know how you deploy your Lambda and configure the trigger to S3. If you're using serverless or CDK I don't see how your lambda doesn't get triggered for every uploaded file with a configuration similar to the following:
// serverless example
functions:
users:
handler: users.handler
events:
- s3:
bucket: photos
event: s3:ObjectCreated:*
rules:
- prefix: uploads/
- suffix: .jpg
In this example the users lambda gets triggered for every jpg file that gets created in photos/uploads.
I have created a lambda which would invoke and do the transformation based on the event in the target source bucket.
This is working fine when I upload the small size of file in the targeted source bucket.
But when I upload large file(eg: 65 mb file), it looks lambda not invoking based on that event..
Appreciate if anyone can help on this kind of issue?
Thanks
I am guessing, big files would be uploaded on S3 via S3 Multipart Upload instead of a regular put-object operation.
Maybe your Lambda function is just subscribed to s3:ObjectCreated:Put events. You need to add s3:ObjectCreated:CompleteMultipartUpload permission to Lambda as well.
The large files in S3 are uploaded via S3 Multipart Upload instead of a regular PUT or single part upload process.
There can be two problems
``In your lambda you probably have created the subscription for s3:ObjectCreated:Put events. You should add s3:ObjectCreated:CompleteMultipartUpload too in the Lambda subscription list.
Your lambda timeout could be small for and that works for the smaller files. You might want to increase that.
There could be any of these issues:
Your event only captures s3:ObjectCreated:Put event, as others have mentioned. Usually if it's a big file, the event is s3:ObjectCreated:CompleteMultipartUpload instead. You could either (a) add s3:ObjectCreated:CompleteMultipartUpload event to your capture, or (b) simply use s3:ObjectCreated:* event - this will include Put, MultiPart Upload, Post, Copy, and also other similar events to be added in the future (source: https://aws.amazon.com/blogs/aws/s3-event-notification/)
Your Lambda function might run longer limit you set (limit is 15min).
Your Lambda function requires more memory than the limit you set.
Your Lambda function requires more disk space than the limit you set. This may be an issue if your function downloads the data on disk first and perform transformation there (limit is 512MB).
I have a lambda function that gets notification whenever and s3 object gets added. The function then reads some metadata of the s3 object. Is there any way to include the metadata in the notification itself, rather than me having to go and read the head object.
The event notification data structure does not include object metadata or tags, and there is not a way to enable their inclusion.
I need my Spinnaker pipeline to trigger on changes to an AWS s3 bucket, specifically when a file is added or edited.
I cannot find a builtin mechanism to do that: there is nothing remotely related to S3 buckets in the drop-down list of triggers.
I thought I might be able to use a webhook from an AWS lambda that subscribes to S3 events on the bucket, and have the lambda webhook to https://my_spinnnaker.mybiz.com/webhooks/webhook/s3_new. However it does not seem possible to pass parameters to the hook, e.g. the key of new S3 object.
Any other ways of doing this?
The S3 Object Key can be read from the event that triggers the Lambda function. The event records have the S3 object key. 1
For a pipeline with parameters, the request sent from the Lambda function can contain parameter values in the request body. The format for the payload is given below. 2
{
"parameters": {
"<parameter-key>": "<parameter-value>"
}
}
I want to create a AWS lambda event source to catch the action of upload a file via aws cli cp command, but it couldn't be triggered when i upload a file. Here is what i have done:
I configured the event source as following:
I have tried all the four option of Object Created event type, it just didn't work.
I use the aws cli as following:
aws s3 cp sample.html s3://ml.hengwei.me/data/
Is there anywhere i miss configured?
You are triggering your Lambda from the wrong event type.
Using the awscli to cp files up into S3 does not cause an s3:ObjectCreated:Copy event (which I believe relates to an S3 copy operation, copying an object from one bucket to another). In your case, the object is being uploaded to S3 and I presume that it results in either s3:ObjectCreated:Put or s3:ObjectCreated:CompleteMultipartUpload.
The events include:
s3:ObjectCreated:Put – An object was created by an HTTP PUT
operation.
s3:ObjectCreated:Post – An object was created by HTTP POST
operation.
s3:ObjectCreated:Copy – An object was created an S3 copy
operation.
s3:ObjectCreated:CompleteMultipartUpload – An object was
created by the completion of a S3 multi-part upload.
s3:ObjectCreated:* – An object was created by one of the event types
listed above or by a similar object creation event added in the
future.
Full list of events is here. Note that the awscli may or may not use multi-part upload so you need to handle both situations.