I have an s3 buckets with a bucket policy to send their objects to glacier after x days of creation. It is working fine by moving the objects to glacier storage.When I go to retrieve those objects later using aws php sdk 3.x api
$result = $client->restoreObject([
'Bucket' => '<string>', // REQUIRED
'Key' => '<string>', // REQUIRED
'RequestPayer' => 'requester',
'RestoreRequest' => [
'Days' => <integer>, // REQUIRED
'GlacierJobParameters' => [
'Tier' => 'Standard|Bulk|Expedited', // REQUIRED
],
],
'VersionId' => '<string>', ])
Normally it may take 3-5 hours to restore the object. So I need to get a sns notification for that. As I am not using the vault for that I am not getting any notification after restored the object. How do I get sns notification after restore completion.
S3 event notification now support s3:ObjectRestore:Completed. See details in AWS documentation. You can configure SNS to send you notification upon Glacier restoration completed.
We will not get sns for the restore completion for that we need to poll using head object api
$result = $s3Client->headObject(array(
'Bucket' => $sourceBucket,
'Key' => "{$archiveKey}/{$sourceKeyname}",
));
and compare the head object requests result
if (isset($res['ongoing-request']) && (strcmp($res['ongoing-request'], '"false"') == 0) && ($result['StorageClass'] == 'GLACIER')) {
$this->log('Survey data id ' . $surveyData['survey_data_id'] . ' in restored state', LogLevel::INFO);}
and if the condition is true we can raise the action
Related
I'm wanting to access the custom metadata for an object uploaded via the S3 multipart upload after firing off the completeMultipartUpload method.
I initiate a multipart S3 upload with some added custom metadata like so:
$response = $this->client->createMultipartUpload([
'Bucket' => $this->bucket,
'Key' => $key,
'ContentType' => $type,
'Expires' => 60,
'Metadata' => [
'file-guid' => $fileGuid,
],
]);
When I complete the multipart upload, I'm wanting to access the file-guid metadata and pass it along in my response.
$result = $this->client->completeMultipartUpload([
'Bucket' => $this->bucket,
'Key' => $key,
'UploadId' => $uploadId,
'MultipartUpload' => [
'Parts' => $parts,
],
]);
$fileGuid = $result['?'] // Couldn't find the metadata in the result.
return response()->json(['file-guid' => $fileGuid]);
I've checked the S3 object after it's been uploaded and it shows the custom metadata, but I don't see how to access it. I assumed it would be part of the completeMultipartUpload response, but I'm not seeing it.
Any help would be appreciated. Thanks!
I found a solution, but it involves an additional request. If anyone knows of a way to access the metadata without making another request, that would be better.
$headObject = $this->client->headObject([
'Bucket' => $this->bucket,
'Key' => $key,
]);
I'm using node in a docker container and locally I use my IAM keys for both creating, reading and deleting files to an S3 bucket as well as creating pre-signed posts. When up on a Fargate container, I create a taskRole and attach a policy which gives it full access to S3.
taskRole.attachInlinePolicy(
new iam.Policy(this, `${clientPrefix}-task-policy`, {
statements: [
new iam.PolicyStatement({
effect: iam.Effect.ALLOW,
actions: ['S3:*'],
resources: ['*'],
}),
],
})
);
With that role, I can create, read and delete files with no issues from the API. When the API tries to create a pre-signed post however, I get the error:
Error: Unable to create a POST object policy without a bucket, region, and credentials
It seems super strange to me that I can run the other operations, but it fails with the presignedPOST, especially since my S3 actions are all allowed.
const post: aws.S3.PresignedPost = await s3.createPresignedPost({
Bucket: bucket,
Fields: { key },
Expires: 60,
Conditions: [['content-length-range', 0, 5242880]],
});
Here is the code I use. I am logging the bucket and key so I'm positive that they are valid values. One thought I had was when running locally, I will run aws.configure to set my keys but in Fargate I purposefully omit that. I thought that it was getting the right keys since the other s3 operations work without fail. Am I approaching this right?
When using IAM role credentials with AWS sdk, you must either use the asynchronous (callback) version of createPresignedPost or guarantee that your credentials have been resolved before calling the await version of this method.
Something like this will work with IAM based credentials:
const s3 = new AWS.S3()
const _presign = params => {
return new Promise((res, rej) => {
s3.createPresignedPost(params, (err, data) => {
if (err) return rej(err)
return res(data)
})
})
}
// await _presign(...) <- works
// await s3.createPresignedPost(...) <- won't work
Refer: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#createPresignedPost-property
I want to add the current date to every filename that is incoming to my s3 bucket.
My current config looks like this:
input {
s3 {
access_key_id => "some_key"
secret_access_key => "some_access_key"
region => "some_region"
bucket => "mybucket"
interval => "10"
sincedb_path => "/tmp/sincedb_something"
backup_add_prefix =>'%{+yyyy.MM.dd.HH}'
backup_to_bucket => "mybucket"
additional_settings => {
force_path_style => true
follow_redirects => false
}
}
}
Is there a way to use the current date in backup_add_prefix =>'%{+yyyy.MM.dd.HH}'
because the current syntax does not work as it produces: "
%{+yyyy.MM.dd.HH}test_file.txt" in my bucket.
Though it's not supported in s3 input plugin directly, it can be achieved. Use the following steps:
Go to logstash home path.
Open the file vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-3.4.1/lib/logstash/inputs/s3.rb. The exact path will depend on your lagstash version.
Look for the method backup_to_bucket.
There is a line backup_key = "#{#backup_add_prefix}#{object.key}"
Add following lines before the above line:
t = Time.new
date_s3 = t.strftime("%Y.%m.%d")
Now change the backup_key to #{#backup_add_prefix}#{date_s3}#{object.key}
Now you are done. Restart your logstash pipeline. It should be able to achieve the desired result.
I can now upload JPG ifrom my BackpackForLaravel to AWS 3 Yahhh!!!
But how or where can i make change this code:
$this->crud->addField([ // image
'label' => "Produkt foto",
'name' => "productfoto",
'type' => 'image',
'tab' => 'Produktfoto',
'upload' => true,
'crop' => true,
'aspect_ratio' => 1,
'disks' => 's3images' // This is not working ??
]);
To showing url from AWS S3 mith my uploaded JPG... (instead of local public)
I can't find any documentation or code examples of it :-(
Please help...
I don't really know how this can be of help to you. Pictures to S3 usually need to be base64 encoded so you will need to decode them before you can store in your s3 bucket. So how i handled it sometime ago is this:
$getId = $request->get('myId');
$encoded_data = $request->get('myphotodata');
$binary_data = base64_decode($encoded_data);
$filename_path = md5(time().uniqid()).".jpg";
$directory = 'uploads';
Storage::disk('s3')->put($directory.'/'.$filename_path , $binary_data);
In summary, i am asuming you have the right permission on your bucket and ready to put in image from your storage disks.
We are using Amazon SNS for our push notification service. It used to work fine, until I've thought to include a TTL - in MessageAttributes. Basically For testing purpose I want to keep the TTL Time as 60 seconds.
I use RestClient. I donot use Amazon SDK.
def publish(endpoint_arn, message,message_attribute)
params = {
:TargetArn => endpoint_arn,
:Message => JSON.dump(message),
:MessageStructure => 'json',
:MessageAttributes => {
message_attribute => {
:DataType => 'String',
:StringValue => '60'
}
}
}
post(:Publish, params)
end
message_attribute is a string that contains, 'AWS.SNS.MOBILE.GCM.TTL'
def post(command, params)
params.merge!(default_post_params(command))
params[:Signature] = calculate_signature(params, 'POST')
response = RestClient.post(SNS_URL, params)
end
Whats wrong in the code above, Amazon document says what i've done is correct.
Amazon SDK API Reference: http://docs.aws.amazon.com/sns/latest/dg/sns-ttl.html
PS: default_post_params, would have a generalised post params, like Access Key, SNS Url. And it works fine. Only after including MessageAttributes key it says,
<Error>
<Type>Sender</Type>
<Code>MalformedQueryString</Code>
<Message>Keys may not contain [</Message>
</Error>
Error is coming due to lack of dictionary key and value.
In message attribute there is two key
1.DataType
2.StringValue
Create a dictionary and add these two keys in dictionary with there value.
Definitely it will work.