In fine uploader, how to have S3 bucket different per environment - amazon-s3

I am using fine uploader to uploaded files to S3.
Based on my experience, fine uploader forces to hard code s3 bucket name in the java script itself or it may be my misundestanding!. My challenge is that I have different bucket per environment. does that mean I have to use separate java script (like below) per environment such local, dev,test,etc? Is there any option in which I can pass bucket name from the server side configuration ?
Local
$('#fine-uploader').fineUploaderS3({
template: 'qq-template',
autoUpload: false,
debug: true,
request: {
endpoint: "http://s3.amazonaws.com/bucket_local",
accessKey: "AKxxxxxxxxxxBIA",
},
)}
Dev
$('#fine-uploader').fineUploaderS3({
template: 'qq-template',
autoUpload: false,
debug: true,
request: {
endpoint: "http://s3.amazonaws.com/bucket_dev",
accessKey: "AKxxxxxxxxxxBIA",
},
)}
Test
$('#fine-uploader').fineUploaderS3({
template: 'qq-template',
autoUpload: false,
debug: true,
request: {
endpoint: "http://s3.amazonaws.com/bucket_test",
accessKey: "AKxxxxxxxxxxBIA",
},
)}

Based on my experience, fine uploader forces to hard code s3 bucket name in the java script itself or it may be my misundestanding!
Yes, this is definitely a misunderstanding.
Normally, you would specify your bucket as a URL via the request.endpoint option. You can specify a default value here and then override it at almost any time, for all subsequent files or one or more specific files, via the setEndpoint API method. You can call this method from, for example, an onSubmit callback handler, and even delegate to your server for the bucket by returning a Promise in your callback handler and resolving the Promise once your have called setEndpoint.
Fine Uploader attempts to determine the bucket name from the request.endpoint URL. If this is not possible (such as if you are using a CDN as an endpoint), you will need to supply the bucket name via the objectProperties.bucket option. This too can be dynamically updated, as the option value may be a function, and you may even return a Promise from this function if you would like to delegate to an async task to determine the bucket name (such as if you need to get the bucket name from a server endpoint using an ajax call). Fine Uploader S3 will call your bucket function before it attempts to upload each file, passing the ID of the file to your function.

I set s3_url as hidden value in html, and value set at server based on environment config.
request: {
endpoint: $('#s3_url').val()
}

Related

How To Add 'Authorization' Header to S3.upload() Request?

My code uses the AWS Javascript SDK to upload to S3 directly from a browser. Before the upload happens, my server sends it a value to use for 'Authorization'.
But I see no way in the AWS.S3.upload() method where I can add this header.
I know that underneath the .upload() method, AWS.S3.ManagedUpload is used but that likewise doesn't seem to return a Request object anywhere for me to add the header.
It works successfully in my dev environment when I hardcode my credentials in the S3() object, but I can't do that in production.
How can I get the Authorization header into the upload() call?
Client Side
this posts explains how to post from a html form with a pre-generated signature
How do you upload files directly to S3 over SSL?
Server Side
When you initialise the S3, you can pass the access key and secret.
const s3 = new AWS.S3({
apiVersion: '2006-03-01',
accessKeyId: '[value]',
secretAccessKey: '[value]'
});
const params = {};
s3.upload(params, function (err, data) {
console.log(err, data);
});
Reference: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html
Alternatively if you are running this code inside AWS services such as EC2, Lambda, ECS etc, you can assign a IAM role to the service that you are using. The permissions can be assigned to the IAM Role
I suggest that you use presigned urls.

S3 alternate endpoints for presigned operations

I am using minio client to access S3. The S3 storage I am using has two endpoints - one (say EP1) which is accessible from a private network and other (say EP2) from the internet. My application creates a presigned URL for downloading an S3 object using EP1 since it cannot access EP2. This URL is used by another application which is not on this private network and hence has access only to EP2. This URL is (obviously) not working when used by the application outside the network since this URL has EP1 in it.
I have gone through minio documentation but did not find anything which help me specify alternate endpoints.
So my question is -
Is there anything which I have missed from minio that can help me?
Is there any S3 feature which allows generating presigned URL for an
object with EP2 in it?
Or is this not solvable without changing
current network layout?
You can use minio-js to manage this
Here is an example that you can use
var Minio = require('minio')
var s3Client = new Minio.Client({
endPoint: "EP2",
port: 9000,
useSSL: false,
accessKey: "minio",
secretKey: "minio123",
region: "us-east-1"
})
var presignedUrl = s3Client.presignedPutObject('my-bucketname', 'my-objectname', 1000, function(e, presignedUrl) {
if (e) return console.log(e)
console.log(presignedUrl)
})
This will not contact the server at all. The only thing here is that you need to know the region that bucket belongs to. If you have not set any location in minio, then you can use us-east-1 by default.

Pre request script of upload file with multipart/form-data hitting method 2 times in Postman

I'm working with API project and writing test cases with Postman for automation to check API status. Here I have one upload method in which user has to upload a file to the server and need to check if the server returns an appropriate response.
Upload method accepting the request with multipart/form-data, from Postman I'm passing as below screen:
I believe that in order to write a test case, I need to write a pre-request script.
pm.sendRequest({
url: pm.environment.get("baseURL") + '/document/upload',
method: 'POST',
header: [{
"key": "Authorization",
"value": pm.environment.get("authorization"),
"type": "text",
}],
body: {
mode: 'formdata',
formdata: [{
"key": "file",
"type": "binary",
"src": "C:\Users\Desktop\api.pdf"
}]
}
}, function(err, res) {
console.log(res);
});
However, the method is getting hit two times, any thoughts to make it correct and hit only once?
I have gone through the docs and figured it out that what is the issue. I was facing issue while running collection using Runner, after searching out a way to handle file uploading, I came to Newman finally, which seem easy for such scenarios. However, it's still unclear how to upload file while running using Runner!
As per the comments above:
Due to security reasons Postman runner doesn't support file uploading
directly. Find Github thread
here
You can add request before this one in your collection which makes the upload if you need it in the next one. Although the good practice says, that the requests should be atomic with pre-request it will be very difficult. You may achieve it using base64 string of the files and send request with formdata if you insist of doing it like that. The other option runs with Newman in a pipeline. All you have to do is export the collection, the environment and the test files and make sure you don't have absolute path in the exported json. ( Newman should be executed from the directory with the collection and env json files)

Why is file being corrupted during multipart upload into express running in aws lambda?

Have a SPA with a redux client and an express webapi. One of the use cases is to upload a single file from the browser to the express server. Express is using the multer middleware to decode the file upload and place it into an array on the req object. Everything works as expected when running on localhost.
However when the app is deployed to AWS, it does not function as expected. Deployment pushes the express api to an AWS Lambda function, and the redux client static assets are served by Cloudfront CDN. In that environment, the uploaded file does make it to the express server, is handled by multer, and the file does end up as the first (and only) item in the req.files array where it is expected to be.
The problem is that the file contains the wrong bytes. For example when I upload a sample image that is 2795 bytes in length, the file that ends up being decoded by multer is 4903 bytes in length. Other images I have tried always end up becoming larger by approximately the same factor by the time multer decodes and puts them into the req.files array. As a result, the files are corrupted and are not displaying as images.
The file is uploaded like so:
<input type="file" name="files" onChange={this.onUploadFileSelected} />
...
onUploadFileSelected = (e) => {
const file = e.target.files[0]
var formData = new FormData()
formData.append("files", file)
axios.post('to the url', formData, { withCredentials: true })
.then(handleSuccessResponse).catch(handleFailResponse)
}
I have tried setting up multer with both MemoryStorage and DiskStorage. Both work, both on localhost and in the aws lambda, however both exhibit the same behavior -- the file is a larger size and corrupted in the store.
I have also tried setting up multer as both a global middleware (via app.use) and as a route-specific middleware on the upload route (via routes.post('the url', multerMiddlware, controller.uploadAction). Again, both exhibit the same behavior. Multer middleware is configured like so:
const multerMiddleware = multer({/* optionally set dest: '/tmp' */})
.array('files')
One difference is that on localhost, both the client and express are served over http, whereas in aws, both the client and express are served over https. I don't believe this makes a difference, but I have yet been unable to test -- either running localhost over https, or running in aws over http.
Another peculiar thing I noticed was that when the multer middleware is present, other middlewares do not seem to function as expected. Rather than the next() function moving flow down to the controller action, instead, other middlewares will completely exit before the controller action invocation, and when the controller invocation exits, control does not flow back into the middlware after the next() call. When the multer middleware is removed, other middlewares do function as expected. However this observation is on localhost, where the entire end-to-end use case does function as expected.
What could be messing up the uploaded image file payload when deployed to the cloud, but not on localhost? Could it really be https making the difference?
Update 1
When I upload this file (11228 bytes)
Here is the HAR chrome is giving me for the local (expected) file upload:
"postData": {
"mimeType": "multipart/form-data; boundary=----WebKitFormBoundaryC4EJZBZQum3qcnTL",
"text": "------WebKitFormBoundaryC4EJZBZQum3qcnTL\r\nContent-Disposition: form-data; name=\"files\"; filename=\"danludwig.png\"\r\nContent-Type: image/png\r\n\r\n\r\n------WebKitFormBoundaryC4EJZBZQum3qcnTL--\r\n"
}
Here is the HAR chrome is giving me for the aws (corrupted) file upload:
"postData": {
"mimeType": "multipart/form-data; boundary=----WebKitFormBoundaryoTlutFBxvC57UR10",
"text": "------WebKitFormBoundaryoTlutFBxvC57UR10\r\nContent-Disposition: form-data; name=\"files\"; filename=\"danludwig.png\"\r\nContent-Type: image/png\r\n\r\n\r\n------WebKitFormBoundaryoTlutFBxvC57UR10--\r\n"
}
The corrupted image file that is saved is 19369 bytes in length.
Update 2
I created a text file with the text hello world that is 11 bytes long and uploaded it. It does NOT become corrupted in aws. This is the case even if I upload it with the txt or png suffix, it ends up as 11 bytes in length when persisted.
Update 3
Tried uploading with a much larger text file (12132 bytes long) and had the same result as in update 2 -- the file is persisted intact, not corrupted.
Potential answers:
Found this https://forums.aws.amazon.com/thread.jspa?threadID=252327
API Gateway does not natively support multipart form data. It is
possible to configure binary passthrough to then handle this multipart
data in your integration (your backend integration or Lambda
function).
It seems that you may need another approach if you are using API Gateway events in AWS to trigger the lambda that hosts your express server.
Or, you could configure API Gateway to work with binary payloads per https://stackoverflow.com/a/41770688/304832
Or, upload directly from your client to a signed s3 url (or a public one) and use that to trigger another lambda event.
Until we get a chance to try out different API Gateway settings, we found a temporary workaround: using FileReader to convert the file to a base64 text string, then submit that. The upload does not seem to have any issues as long as the payload is text.

Amazon CloudFront cache invalidation after 'aws s3 sync' CLI to update s3 bucket contents

I have a rails app that uses aws cli to sync bunch of content and config with my s3 bucket like so:
aws s3 sync --acl 'public-read' #{some_path} s3://#{bucket_path}
Now I am looking for some easy way to mark everything that was just updated in sync to be marked as invalidated or expired for CloudFront.
I am wondering if there is some way to use -cache-control flag that aws cli provides to make this happen. So that instead of invalidating CouldFont, just mark the files as expired, so CloudFront will be forced to fetch fresh data from bucket.
I am aware of CloudFront POST API to mark files for invalidation, but that will mean I will have detect what changed in the last sync, then make the API call. I might have any where from 1000s to 1 file syncing. Not a pleasent prospect. But if I have to go this route, how would I go about detecting changes without parsing the s3 sync's console output of-course.
Or any other ideas?
Thanks!
You cannot use the --cache-control option that aws cli provides to invalidate files in CloudFront. The --cache-control option maps directly to the Cache-Control header and CloudFront caches the headers along with the file, so if you change a header you must also invalidate to tell CloudFront to pull in the changed headers.
If you want to use the aws cli, then you must parse the output of the sync command and then use the aws cloudfront cli.
Or, you can use s3cmd from s3tools.org. This program provides the the --cf-invalidate option to invalidate the uploaded filed in CloudFront and a sync command synchronize a directory tree to S3.
s3cmd sync --cf-invalidate <local path> s3://<bucket name>
Read, the s3cmd usage page for more details.
What about using the brand new AWS Lambda? Basically, it executes custom code whenever an event is triggered in AWS (in your case, a file is synchronized in S3).
Whenever you synchronize a file you get an event similar to:
{
"Records": [
{
"eventVersion": "2.0",
// ...
"s3":
{
"s3SchemaVersion": "1.0",
// ...
"object":
{
"key": "hello.txt",
"size": 4,
"eTag": "1234"
}
}
}
]
}
Thus, you can check the name of the file that has changed and invalidate it in CloudFront. You receive one event for every file that has changed.
I have created a script that invalidates a path in CloudFront whenever an update occurs in S3, which might be a good starting point if you decide to use this approach. It is written in JavaScript (Node.js) as it is the language used by Lambda.
var aws = require('aws-sdk'),
s3 = new aws.S3({apiVersion: '2006-03-01'}),
cloudfront = new aws.CloudFront();
exports.handler = function(event, context) {
var filePath = '/' + event.Records[0].s3.object.key,
invalidateParams = {
DistributionId: '1234',
InvalidationBatch: {
CallerReference: '1',
Paths: {
Quantity: 1,
Items: [filePath]
}
}
};
console.log('Invalidating file ' + filePath);
cloudfront.createInvalidation(invalidateParams, function(err, data) {
if (err) {
console.log(err, err.stack); // an error occurred
} else {
console.log(data); // successful response
}
});
context.done(null,'');
};
For more info you can check Lambda's and CloudFront's API documentation.
Note however that the service is still in preview and is subject to change.
The AWS CLI tool can output JSON. Collect the JSON results, then submit an invalidation request per the link you included in your post. To make it really simple you could use a gem like CloudFront Invalidator, which will take a list of paths to invalidate.