I'm using the AWS-SDK with react-native to upload an image to S3 Bucket.
First of all, I want to say that my access and connectivity works well, I tried uploading plain text it works, I tried listing the objects and the buckets it works too.
Here is my code:
async function handleImage(capturedImage) {
setImage(capturedImage);
setScreenState(ScreenStates.LOADING);
try {
const result = await classifyImage(capturedImage);
console.log(result.tensor_)
// {dtype:"float32",shape:[…]}
// dtype:"float32"
// shape:[1,3,273,224]
const blob_jpeg = new Blob([result.tensor_], {type: "image/jpeg"});
console.log(typeof blob_jpeg._data)
// object
console.log(blob_jpeg._data)
// {blobId:"e7a667ad-4363-4a2e-9850-8695f103e9e0",offset:0,size:1489546,type:"image/jpeg",__collector:{}}
try {
const keyName = 'image.jpeg';
const putCommand = new PutObjectCommand({
Bucket: "mybucket",
ContentType:"image/jpeg",
Key: "myimage",
Body: blob_jpeg._data,
});
await s3.send(putCommand);
console.log(
'Successfully uploaded data to ' + bucketName + '/' + keyName);
} catch (e) {
console.log(e,e);
}
My error:
Error: "The request signature we calculated does not match the signature you provided. Check your key and signing method." in SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method. << at construct (native) << at apply (native) << at i (#aws-sdk/client-s3.js:3:461197)
Any ideas about how can I solve this problem and succesffully upload my image ?
Related
I am trying to list objects and if this works later download/upload files to AWS S3. The code below throws an error. What am I doing incorrectly that this doesn't work? I've passed the accessKeyId and accessSecretKey in all possible ways below. I have a config and credentials file on mac and on windows I have just one awscredentials file and also set this on my windows
setx AWS_SDK_LOAD_CONFIG=1
CODE
const AWS = require('aws-sdk');
function listS3Objects(file, name, type) {
const s3bucket = new AWS.S3({
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
accessSecretKey: process.env.AWS_SECRET_ACCESS_KEY,
// accessKeyId: 'my actual key in credentials file', //aws_access_key_id
// accessSecretKey: 'my actual secret key in credentials file', //aws_secret_access_key
region: "ap-southeast-1"
});
const params = {
Bucket: 'testbucketName',
};
s3bucket.listObjects(params, (err, data) => {
if (err) { throw err; }
/* eslint-disable no-console */
console.log('Success!');
console.log(data);
return data;
/* eslint-enable no-console */
});
}
const objs = listS3Objects()
//Test AWS Credentials
it('Tests', () => {
cy.log(objs)
})
ERROR
The following error originated from your test code, not from Cypress.
Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.
Cypress could not associate this error to any specific test.
We dynamically generated a new test to display this failure.
node_modules/aws-sdk/lib/config.js:400:1
398 |
399 | function credError(msg, err) {
400 | return new AWS.util.error(err || new Error(), {
| ^
401 | code: 'CredentialsError',
402 | message: msg,
403 | name: 'CredentialsError'
I'd troubleshoot it this way:
Can you run the script to upload or download separately? If not then its with the credentials.
if not credentials and this works perfectly can this script be imported into your spec and run? Let the script resolve and return a promise, then use the return value in your spec.
Refer this blog post.
Other options you could consider and cy.exec("aws command goes here")
I have a problem when i try to upload a file on S3 through my API
I use the "putObject" method, and the thing that surprise me it's it work when i run my serverless application locally, with serverless-offline, i can push the all file on s3 and i can open it
But when i deploy my application on API Gateway, if i use the API Gateway route, the file is lightweight than original, and i can't open it, it tell me that the file is corrupted
If anyone has an idea, it could really help me
Thanks
My putObject method looks like this
const bucketName = _.get(getBucket,'bucketName');
const extension = _.get(data,'media.filename').split('.').pop();
const keyName = _.get(data,'keyName') + '.' + extension;
const content = _.get(data,'media.content')
let params = {
Bucket: bucketName,
Key: keyName,
ContentType: _.get(data,'media.contentType'),
Body: content,
ACL: 'public-read'
};
return new Promise((resolve, reject) => {
s3.putObject(params, function(err, data) {
err
?
reject(console.log(err)) :
resolve(
response(200,"Object Added")
);
});
});
Uploading files via ApiGateway is not a good idea, I'd strongly advise using resigned URL https://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObject.html
If you want to upload through ApiGateway, than provided information is not enough. Try to log the received lambda event, that should help
am trying to upload files directly from my front-end(Angular 8) using the cloudinary API_URL but am still getting the same bad request (400) and the same error "Upload preset must be whitelisted for unsigned uploads" even i tried different solutions like providing the preset_name in the FormData and setting the preset to unsigned in my cloudinary settings but still not working. is there any solution ?
my upload code :
const images = new FormData();
images.append('images', file);
images.append('upload_preset', [presetName]);
this.progressBar = true
const req = new HttpRequest('POST', 'https://api.cloudinary.com/v1_1/[cloudName]/image/upload', images,
{
reportProgress: true,
});
this.http.request(req).subscribe(event => {
if (event.type === HttpEventType.UploadProgress) {
const percentDone = Math.round(100 * event.loaded / event.total);
console.log(`File is ${percentDone}% uploaded.`);
} else if (event instanceof HttpResponse) {
console.log('File is completely uploaded!');
}
});
Upload preset must be whitelisted for unsigned uploads error means that that the preset you are using is marked for Signed uploads. Since you are not performing an authenticated API call, i.e. using a signature, the upload preset must be set as Unsigned. If you haven't already, go to the Settings -> Upload tab in your account and verify that the Signing Mode is set to Unsigned for the preset you are trying to use.
In addition, I see that you are passing a parameter called 'images'. This is not a valid parameter for the Upload API. Please update that to "file".
const data = new FormData();
data.append("file", file);
data.append("upload_preset", "default-preset");
We are working on s3 browser based multipart file using EvaporateJS, Using pre-signed URL with temperory credentials.
Following will be my configuration
var amz_headers_common = {};
var amz_headers_at_initiate = {};
var amz_headers_at_upload = {};
var amz_headers_at_complete = {};
amz_headers_common['x-amz-acl'] = 'private';
amz_headers_common['x-amz-security-token'] = '<?=AWS_TOKEN;?>';
amz_headers_at_initiate['x-amz-acl'] = 'private';
amz_headers_at_initiate['x-amz-security-token'] = '<?=AWS_TOKEN;?>';
var customAuth = $("#signingMethod")[0].checked;
Evaporate.create({
signerUrl: customAuth ? undefined : '<?=AWS_SIGNER_URL;?>',
aws_key: '<?=AWS_KEY;?>' ,
bucket: '<?=AWS_S3_BUCKET;?>',
cloudfront: false,
computeContentMd5: true,
cryptoMd5Method: function (data) { return AWS.util.crypto.md5(data, 'base64'); },
cryptoHexEncodedHash256: function (data) { return AWS.util.crypto.sha256(data, 'hex'); },
logging: true,
s3Acceleration: true,
signTimeout: 10,
s3FileCacheHoursAgo: 1,
maxConcurrentParts:5,
allowS3ExistenceOptimization: true,
sendCanonicalRequestToSignerUrl: true,
customAuthMethod: customAuth? doNotUseUnsafeJavaScriptV4Signer : undefined,
evaporateChanged: function (file, evaporatingCount) {
$('#totalParts').text(evaporatingCount);
if (evaporatingCount > 0) {
$("#pause-all, #pause-all-force, #cancel-all").show();
} else if (evaporatingCount === 0) {
$("#pause-all, #pause-all-force, #resume, #cancel-all").hide();
}
}
})
var promise = _e_.add({
name: name,
file: files[i],
started: callback_methods.started,
complete: callback_methods.complete,
cancelled: callback_methods.cancelled,
progress: callback_methods.progress,
error: callback_methods.error,
warn: callback_methods.warn,
paused: callback_methods.paused,
pausing: callback_methods.pausing,
resumed: callback_methods.resumed,
nameChanged: callback_methods.nameChanged,
xAmzHeadersCommon: amz_headers_common,
xAmzHeadersAtInitiate: amz_headers_at_initiate,
xAmzHeadersAtUpload: amz_headers_at_upload,
xAmzHeadersAtComplete: amz_headers_at_complete
},
{
bucket: '<?=AWS_S3_BUCKET;?>', // Shows that the bucket can be changed per
aws_key: '<?=AWS_KEY;?>' // Shows that aws_key can be changed per
}
But I'm getting following signature mismatch error.
AWS Code: SignatureDoesNotMatch, Message:The request signature we calculated does not match the signature you provided. Check your key and signing method.status:403
Following will be log
Without temperary credentials following cannonical request and v4 string to sign.
POST
/test-video.mp474.6796611212833
uploads=
host:<bucket-name>.s3-accelerate.amazonaws.com
x-amz-date:20170428T055938Z
host;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
V4 stringToSign: AWS4-HMAC-SHA256
20170428T055938Z
20170428/ap-southeast-1/s3/aws4_request
ce2c7c5fbbf58483efbd4bd244551d138353ebb7b7233d3fdce73e85d96fad8d
--------------------------------------------------------------------------------------
Using temperary credentials following cannonical request and v4 string to sign.
initiate V4 CanonicalRequest: POST
/test-video.mp461.80892198840156
uploads=
host:<bucket-name>.s3-accelerate.amazonaws.com
x-amz-acl:private
x-amz-date:20170427T160400Z
x-amz-security-token:FQoDYXdzEDEaDIkS6zY1Oj8PQLLDVSK5A5pPusfWw81Yq3v0c4VqlyyQsBDW+PHosDuDnG8EYc9jlXD1tQwiTKU1E2Nf3aKcYmv/BHYwGwOen9GPStPeVBGbWNBzi1lT+B6xOnDvIXzelnuC6Eddt+jYIrjy9RVIKBN/s80NtVwfjmFK+93iOWJzdl2ruRSzQINZ+UuSmuthudkYLZzKy0pDmCrgIz8YCjXsjhN7FyeSZzXk9qmBDCASygVEFDNbkb/xidH/Yj7P9gYdsxY6YokV/CM8ZpAKmE8Lp+en+xs9rDclexFzCId8QyJaGj0xb205WoeRIHr8RSStvyounCxrmhWP6M/eijWTP/uHIfWVDqBadEPSgVWqcEzrW2iJ+0SGROb+In6BMmkEMaw+9L5M+lkgCfMDm5Fw9Ip8bujcb4okoNjEn6L+L0b1lm3yuqvLkT3oOzL3Sn48n3y0dXsYtt3yAq+C02bnfmgtYVQgv1C9TaMHrvipFADYNJ9U81HxQWlgvuSG5BEgqV59PIzGhwPFHais/GyA+a1bmxkyhzKEw1yq6F6+wQ+VBRskmPlahQd9ZK3wrnqvpQm+H7tD2YLkVVQb+AGKtRVU3mOL3yjgnYjIBQ==
host;x-amz-acl;x-amz-date;x-amz-security-token
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
V4 stringToSign: AWS4-HMAC-SHA256
20170427T160400Z
20170427/ap-southeast-1/s3/aws4_request
e0b18a695b23bc16b6727fd2dc417e445266111ebb4995794287a46304d2cc92
Please help me to solve singature mismatch issue.
Wherever you instantiate your S3 client for the bucket, you'll want to set your signature version to v4, since that's what your temporary credentials are configured for. Something like this:
var s3 = new AWS.S3({
signatureVersion: 'v4'
});
I'm simply trying to follow this tutorial on how to upload files to gcs with Node and Express. But the following error keep causing my app to crash. Usually, I am able to upload one file without a problem in the first run. But I will get this error after running a few request, even with different file. When I try to upload, say 5, files at a time, this error cause my app to crash even in the first run. I see the process is trying to rename a file in the .config folder. Is it a normal behavior? If so, is there a work-around?
Window: v10.0.10586
Node: v4.3.1
Express: v4.13.1
Error: EPERM: operation not permitted, rename 'C:\Users\James Wang.config\configstore\gcs-resumable-upload.json.2873606827' -> 'C:\Users\James Wang.config\configstore\gcs-resumable-upload.json'
at Error (native)
at Object.fs.renameSync (fs.js:681:18)
at Function.writeFileSync as sync
at Object.create.all.set (C:\Users\James Wang\gi-cms-backend\node_modules\configstore\index.js:62:21)
at Object.Configstore.set (C:\Users\James Wang\gi-cms-backend\node_modules\configstore\index.js:93:11)
at Upload.set (C:\Users\James Wang\gi-cms-backend\node_modules\gcs-resumable-upload\index.js:264:20)
at C:\Users\James Wang\gi-cms-backend\node_modules\gcs-resumable-upload\index.js:60:14
at C:\Users\James Wang\gi-cms-backend\node_modules\gcs-resumable-upload\index.js:103:5
at Request._callback (C:\Users\James Wang\gi-cms-backend\node_modules\gcs-resumable-upload\index.js:230:7)
at Request.self.callback (C:\Users\James Wang\gi-cms-backend\node_modules\request\request.js:199:22)
at emitTwo (events.js:87:13)
at Request.emit (events.js:172:7)
at Request. (C:\Users\James Wang\gi-cms-backend\node_modules\request\request.js:1036:10)
at emitOne (events.js:82:20)
at Request.emit (events.js:169:7)
at IncomingMessage. (C:\Users\James Wang\gi-cms-backend\node_modules\request\request.js:963:12)
[nodemon] app crashed - waiting for file changes before starting...
UPDATE:
After setting {resumable: false} as suggested by #stephenplusplus in this post, I am no longer getting the "EPERM: operation not permitted" error.But, I start running into the { [ERROR:ETIMEDOUT] code: 'ETIMEDOUT', connection: false } error while trying to upload multiple files at a time with the largest file greater than 1.5mb. Other files get uploaded successfully.
For more information, I am able to upload files one by one when the files are no greater than ~2.5mb. If I try to upload 3 files at a time, I can only do so with files no greater than ~1.5mb.
Is the "Operation not permitted" issue as specified in the question a window specific thing, and does the timeout issue happen only after i set resumable = false?
I'm using express and multer with node.
This is the code I'm using now:
// Express middleware that will handle an array of files. req.files is an array of files received from
// filemulter.fields([{field: name, maxCount}]) function. This function should handle
// the upload process of files asychronously
function sendFilesToGCS(req, res, next) {
if(!req.files) { return next(); }
function stream(file, key, folder) {
var gcsName = Date.now() + file.originalname;
var gcsFile = bucket.file(gcsName);
var writeStream = gcsFile.createWriteStream({ resumable: false });
console.log(key);
console.log('Start uploading: ' + file.originalname);
writeStream.on('error', function(err) {
console.log(err);
res.status(501).send(err);
});
writeStream.on('finish', function() {
folder.incrementFinishCounter();
req.files[key][0].cloudStorageObject = gcsName;
req.files[key][0].cloudStoragePublicUrl = getPublicUrl(gcsName);
console.log('Finish Uploading: ' + req.files[key][0].cloudStoragePublicUrl);
folder.beginUploadNext();
});
writeStream.end(file.buffer);
};
var Folder = function(files) {
var self = this;
self.files = files;
self.reqFilesKeys = Object.keys(files); // reqFilesKeys is an array of keys parsed from req.files
self.nextInQuene = 0; // Keep track of the next file to be uploaded, must be less than reqFilesKeys.length
self.finishCounter = 0; // Keep track of how many files have been uploaded, must be less than reqFilesKeys.length
console.log(this.reqFilesKeys.length + ' files to upload');
};
// This function is used to initiate the upload process.
// It's also called in the on-finish listener of a file's write-stream,
// which will start uploading the next file in quene
Folder.prototype.beginUploadNext = function() {
// If there's still file left to upload,
if(this.finishCounter < this.reqFilesKeys.length) {
// and if there's still file left in quene
if(this.nextInQuene < this.reqFilesKeys.length) {
// upload the file
var fileToUpload = this.files[this.reqFilesKeys[this.nextInQuene]][0];
stream(fileToUpload, this.reqFilesKeys[this.nextInQuene], this);
// Increment the nextInQuene counter, and get the next one ready
this.nextInQuene++;
}
} else {
console.log('Finish all upload!!!!!!!!!!!!!!!!!!!!!!');
next();
}
};
Folder.prototype.incrementFinishCounter = function() {
this.finishCounter++;
console.log('Finished' + this.finishCounter + ' files');
};
var folder = new Folder(req.files);
// Begin upload with 3 streams
/*for(var i=0; i<3; i++) {
folder.beginUploadNext();
}*/
//Upload file one by one
folder.beginUploadNext();
}
I had the same issue with bower .. Run the following command: bower cache clean --allow-root
if this does not solve the problem, try after disabling anti virus.