I am doing an upload like this:
curl -v -X PUT -T "test.xml" -H "Host: my-bucket-upload.s3-eu-central-1.amazonaws.com" -H "Content-Type: application/xml" https://my-bucket-upload.s3-eu-central-1.amazonaws.com/test.xml
The file gets uploaded and I can see it in my S3 bucket.
The trick is, when I try to create a lambda function to be triggered on creation, it never gets invoked. If I upload the file using the S3 web interface, it works fine. What am I doing wrong? Is there any clear recipe on how to do it?
Amazon S3 APIs such as PUT, POST, and COPY can create an object. Using
these event types, you can enable notification when an object is
created using a specific API, or you can use the s3:ObjectCreated:*
event type to request notification regardless of the API that was used
to create an object.
Check the notification event setup on the bucket
Go to bucket on AWS management console
Click the properties tab on the bucket
Click the Events to check the notification event setup
Case 1:
s3:ObjectCreated:* - Lambda should be invoked regardless of PUT, POST or COPY
Other case:-
If the event is setup for specific HTTP method, use that method on
your CURL command to create the object on S3 bucket. This way it
should trigger the Lambda function
Check the prefix in bucket/properties.
If there is a world like foo/, that means that only the objects inside the foo folder will trigger the evert to lambda.
Make sure the prefix you're adding contains safe special characters mentioned here. As per AWS documentation, some characters require special handling. Please be mindful of that.
Also, I noticed modifying the trigger on lambda page doesn't get applied until you delete the trigger and new one (even if it is same). Learned hard way. AWS does behaves weird sometime.
Faced similar issues and figured our that the folder names should not have spaces.
Related
I am trying to create an AWS Lambda triggered on a file upload on an existing S3 bucket using Serverless Framework.
I managed to have the Lambda deployed but cannot have it triggered by upload on my existing S3 bucket.
I am well aware of
the existence of a existing parameter within serverless.yml:
functions:
copyToBufferS3:
handler: handler.copy_to_buffer_s3
description: Copies newly uploaded technical logs to a buffer S3 bucket
events:
- s3:
bucket: my.bucket.name
event: s3:ObjectCreated:*
rules:
- suffix: suffix.ext
existing: true
the fact that such parameter creates another lambda named <service name>-<stage>-custom-resource-existing-s3: I can see it in my console (so existing parameter and its section actually is taken into account, yay!)
the importance of indentation in serverless.yml: I double checked that parameters under - s3 section are 4 spaces-indented
The problem is: neither of these two lambdas have a trigger set.
How should I edit my serverless.yml (or something else) to actually have my lambda triggered on file upload?
It turned out I just did not have the permission to... display the trigger(s).
I am surprised that in this case, AWS Console displays "Triggers (0)" as if it actually was empty in this section (instead of clearly warning about permission as it usually does).
I have an aws setup that requires me to assume role and get corresponding credentials in order to write to s3. For example, to write with aws cli, I need to use --profile readwrite flag. If I write code myself with boot, I'd assume role via sts, get credentials, and create new session.
However, there is a bunch of applications and packages relying on boto3's configuration, e.g. internal code runs like this:
s3 = boto3.resource('s3')
result_s3 = s3.Object(bucket, s3_object_key)
result_s3.put(
Body=value.encode(content_encoding),
ContentEncoding=content_encoding,
ContentType=content_type,
)
From documentation, boto3 can be set to use default profile using (among others) AWS_PROFILE env variable, and it clearly "works" in terms that boto3.Session().profile_name does match the variable - but the applications still won't write to s3.
What would be the cleanest/correct way to set them properly? I tried to pull credentials from sts, and write them as AWS_SECRET_TOKEN etc, but that didn't work for me...
Have a look at the answer here:
How to choose an AWS profile when using boto3 to connect to CloudFront
You can get boto3 to use the other profile like so:
rw = boto3.session.Session(profile_name='readwrite')
s3 = rw.resource('s3')
I think the correct answer to my question is one shared by Nathan Williams in the comment.
In my specific case, given that I had to initiate code from python, and was a bit worried about setting AWS settings that might spill into other operations, I used
the fact that boto3 has DEFAULT_SESSION singleton, used each time, and just overwrote this with a session that assumed the proper role:
hook = S3Hook(aws_conn_id=aws_conn_id)
boto3.DEFAULT_SESSION = hook.get_session()
(here, S3Hook is airflow's s3 handling object). After that (in the same runtime) everything worked perfectly
I have created a lambda which would invoke and do the transformation based on the event in the target source bucket.
This is working fine when I upload the small size of file in the targeted source bucket.
But when I upload large file(eg: 65 mb file), it looks lambda not invoking based on that event..
Appreciate if anyone can help on this kind of issue?
Thanks
I am guessing, big files would be uploaded on S3 via S3 Multipart Upload instead of a regular put-object operation.
Maybe your Lambda function is just subscribed to s3:ObjectCreated:Put events. You need to add s3:ObjectCreated:CompleteMultipartUpload permission to Lambda as well.
The large files in S3 are uploaded via S3 Multipart Upload instead of a regular PUT or single part upload process.
There can be two problems
``In your lambda you probably have created the subscription for s3:ObjectCreated:Put events. You should add s3:ObjectCreated:CompleteMultipartUpload too in the Lambda subscription list.
Your lambda timeout could be small for and that works for the smaller files. You might want to increase that.
There could be any of these issues:
Your event only captures s3:ObjectCreated:Put event, as others have mentioned. Usually if it's a big file, the event is s3:ObjectCreated:CompleteMultipartUpload instead. You could either (a) add s3:ObjectCreated:CompleteMultipartUpload event to your capture, or (b) simply use s3:ObjectCreated:* event - this will include Put, MultiPart Upload, Post, Copy, and also other similar events to be added in the future (source: https://aws.amazon.com/blogs/aws/s3-event-notification/)
Your Lambda function might run longer limit you set (limit is 15min).
Your Lambda function requires more memory than the limit you set.
Your Lambda function requires more disk space than the limit you set. This may be an issue if your function downloads the data on disk first and perform transformation there (limit is 512MB).
I have a lambda function that gets notification whenever and s3 object gets added. The function then reads some metadata of the s3 object. Is there any way to include the metadata in the notification itself, rather than me having to go and read the head object.
The event notification data structure does not include object metadata or tags, and there is not a way to enable their inclusion.
I want to create a AWS lambda event source to catch the action of upload a file via aws cli cp command, but it couldn't be triggered when i upload a file. Here is what i have done:
I configured the event source as following:
I have tried all the four option of Object Created event type, it just didn't work.
I use the aws cli as following:
aws s3 cp sample.html s3://ml.hengwei.me/data/
Is there anywhere i miss configured?
You are triggering your Lambda from the wrong event type.
Using the awscli to cp files up into S3 does not cause an s3:ObjectCreated:Copy event (which I believe relates to an S3 copy operation, copying an object from one bucket to another). In your case, the object is being uploaded to S3 and I presume that it results in either s3:ObjectCreated:Put or s3:ObjectCreated:CompleteMultipartUpload.
The events include:
s3:ObjectCreated:Put – An object was created by an HTTP PUT
operation.
s3:ObjectCreated:Post – An object was created by HTTP POST
operation.
s3:ObjectCreated:Copy – An object was created an S3 copy
operation.
s3:ObjectCreated:CompleteMultipartUpload – An object was
created by the completion of a S3 multi-part upload.
s3:ObjectCreated:* – An object was created by one of the event types
listed above or by a similar object creation event added in the
future.
Full list of events is here. Note that the awscli may or may not use multi-part upload so you need to handle both situations.