Fineuploader: large files to Amazon S3 - amazon-s3

For small files (smaller than about 5MB) Fineuploader sends a complete policy objet for signing, and upload succeeds including the posprocessing which is sending a temporary link to the client, and some server-side administrative stuff. I use the official FineUploader S3 demo PHP code for server. https://github.com/Widen/fine-uploader-server/blob/master/php/s3/s3demo-thumbnails.php
Example of signing request (formatted, anonimized):
{
"expiration": "2013-12-28T12:57:42.354Z",
"conditions": [
{"acl": "private"},
{"bucket": "TOP-SECRET-BUCKET-NAME"},
{"Content-Type": "application/pdf"},
{"success_action_status": "200"},
{"key": "bfccb67e-5343-4e01-97ff-2dcffe681da0.pdf"},
{"x-amz-meta-qqfilename": "plakat_a3.pdf"},
["content-length-range","0","1000111000111"]
]
}
Response contains policy and signature (formatted):
{
"policy":"eyJleHBpcmF0aW9uIjoiMjAxMy0xMi0yOFQxMzoxODoyNy4yODhaIiwiY29uZGl0aW9ucyI6W3siYWNsIjoicHJpdmF0ZSJ9LHsiYnVja2V0IjoiZGVtb2Nza2EifSx7IkNvbnRlbnQtVHlwZSI6ImFwcGxpY2F0aW9uXC9wZGYifSx7InN1Y2Nlc3NfYWN0aW9uX3N0YXR1cyI6IjIwMCJ9LHsia2V5IjoiMWZiMjZmMjAtNjg2Ni00YjU1LTg3YTctZWZlMjNiOWMwZmY1LnBkZiJ9LHsieC1hbXotbWV0YS1xcWZpbGVuYW1lIjoicGxha2F0X2EzLnBkZiJ9LFsiY29udGVudC1sZW5ndGgtcmFuZ2UiLCIwIiwiMTAwMDExMTAwMDExMSJdXX0=",
"signature":"Wlw1QJjwmsASyQemUWrYuktiQwE="
}
... but for larger files Fineupload sends a different signing request, which only contains headers (formatted, anonimized):
{
"headers": "POST\n\nvideo/quicktime\n\nx-amz-acl:private\nx-amz-date:Sat, 28 Dec 2013 12:53:13 GMT\nx-amz-meta-qqfilename:20MB_stopmot-minta.mov\n/TOP_SECRET-BUCKET-NAME/d2033a4c-1e55-49a0-8589-9b1725dcd013.mov?uploads"
}
Response contains only signature (formatted):
{
"signature":"jYtFC91wIPkZj31W\/vwuK9ClawU="
}
In that case the OPTIONS request to amazon S3 fails:
Request Headers 14:09:09.000
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0
Pragma: no-cache
Origin: http://top-secret.example.com
Host: TOP-SECRET-BUCKET-NAME.s3.amazonaws.com
Connection: keep-alive
Cache-Control: no-cache
Access-Control-Request-Method: POST
Access-Control-Request-Headers: authorization,content-type,x-amz-acl,x-amz-date,x-amz-meta-qqfilename
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Response:
Status: 403
Body: empty
Response headers:
x-amz-request-id: 625913F399C626A2
x-amz-id-2: +enheIuZT0RJ+11THF1TuNsA6bCqE4a2ppsklK84k4KXdNX4UsuGhxPf00Eb137G
Transfer-Encoding: chunked
Server: AmazonS3
Date: Sat, 28 Dec 2013 13:18:01 GMT
Content-Type: application/xml
Test cases:
Tested with smaller .mov, .mp3, .pdf, .log files: course #1 (success)
Tested with larger files (same types): course #2 (fail)
What can cause the difference? How can / should I fix it?

I needed to add <ExposeHeader>ETag</ExposeHeader> to my CORS configuration at Amazon S3 bucket permissions. So finally it looks similar to:
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<AllowedMethod>DELETE</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<ExposeHeader>ETag</ExposeHeader>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>
Reference:
http://docs.fineuploader.com/endpoint_handlers/amazon-s3.html

Related

.Net Core blocked by CORS policy error only when uploading a file

Is there something different that needs to be done when uploading a file? Every other call (Axios/Vue put) works fine except where a file is getting uploaded. It's going from example.com to api.example.com. Everything works fine locally hitting different ports.
ConfigureServices:
services.AddCors(o => o.AddPolicy("CorsPolicy", builder =>
{
builder.AllowAnyOrigin()
.AllowAnyMethod()
.AllowAnyHeader()
.SetIsOriginAllowedToAllowWildcardSubdomains();
//.WithOrigins("http://*.example.com"); //tried adding, nothing works when it's here
}));
Configure:
app.UseCors("CorsPolicy");
And the controller:
[ApiController]
[EnableCors("CorsPolicy")]
public class MyController : ControllerBase {
Public Model Put([FromForm] ICollection<IFormFile> files, [FromForm] string jsonString)
The browser output:
Access to XMLHttpRequest at 'http://api.example.com/api/YourKnowledge' from origin 'http://example.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Request URL: http://api.example.com/api/YourKnowledge
Referrer Policy: strict-origin-when-cross-origin
Date: Sat, 03 Oct 2020 19:58:42 GMT
Server: Microsoft-IIS/10.0
Transfer-Encoding: chunked
X-Powered-By: ASP.NET
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Content-Length: 10219
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryvIG7idij6Og7BH8r
Host: api.example.com
Origin: http://example.com
Referer: http://example.com/
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36
files: (binary)
jsonString:{}
Last note, I did find this Asp.Net Core API CORS policy error only in file upload, but I won't have just images uploaded. Did try setting it to test, but it didn't make a difference.
I think you couldn't add two parameter with [FromForm]
(I know that its the rule at least in [FromBody]).
please let me know that works

axios send wrong csrf cookie

I want process a POST request with axios in a vuejs project, but axios never send correct csrf cookie. My vuejs project run in dev mode on port 8080, my server (Spring Boot) on port 9090. I set correct CORS filter for localhost:8080 and 127.0.0.1:8080, OPTIONS is accepted, but I get a 403 error on my POST.
Here my code :
axios.post("http://127.0.0.1:9090/api/security/authenticate",
this.form,
{
withCredentials: true,
xsrfCookieName: 'XSRF-TOKEN',
xsrfHeaderName: 'X-XSRF-TOKEN'
}
).then((response) => {
console.log("Data: " + response.data);
}).catch((error) => {
console.log("post error: " + error);
});
And here the result of my OPTIONS request on chromium :
GENERAL:
Request URL: http://127.0.0.1:9090/api/security/authenticate
Request Method: OPTIONS
Status Code: 200
Remote Address: 127.0.0.1:9090
Referrer Policy: no-referrer-when-downgrade
RESPONSE HEADERS :
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: content-type
Access-Control-Allow-Methods: POST
Access-Control-Allow-Origin: http://127.0.0.1:8080
Content-Length: 0
Date: Wed, 11 Dec 2019 21:44:07 GMT
Vary: Origin
Vary: Access-Control-Request-Method
Vary: Access-Control-Request-Headers
REQUEST HEADERS :
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7
Access-Control-Request-Headers: content-type
Access-Control-Request-Method: POST
Connection: keep-alive
Host: 127.0.0.1:9090
Origin: http://127.0.0.1:8080
Referer: http://127.0.0.1:8080/login
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) snap Chromium/78.0.3904.108 Chrome/78.0.3904.108 Safari/537.36
And my POST :
GENERAL:
Request URL: http://127.0.0.1:9090/api/security/authenticate
Request Method: POST
Status Code: 403
Remote Address: 127.0.0.1:9090
Referrer Policy: no-referrer-when-downgrade
RESPONSE HEADERS :
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: http://127.0.0.1:8080
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Type: application/json;charset=UTF-8
Date: Wed, 11 Dec 2019 21:44:07 GMT
Expires: 0
Pragma: no-cache
Transfer-Encoding: chunked
Vary: Origin
Vary: Access-Control-Request-Method
Vary: Access-Control-Request-Headers
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
REQUEST HEADERS :
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7
Connection: keep-alive
Content-Length: 56
Content-Type: application/json;charset=UTF-8
Cookie: XSRF-TOKEN=428229a0-b2b1-4473-ab3a-557e4dbac1b1
Host: 127.0.0.1:9090
Origin: http://127.0.0.1:8080
Referer: http://127.0.0.1:8080/login
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) snap Chromium/78.0.3904.108 Chrome/78.0.3904.108 Safari/537.36
REQUEST PAYLOAD :
{login: "xxx", password: "xxx", rememberme: false}
login: "xxx"
password: "xxx"
rememberme: false
The CSRF cookie sent (428229a0...) is not correct. Previously in a GET request, I obtained an other CSRF token (70705d00...). Why wrong csrf cookie was sent ?
To be sure that is not the result of dev mode from vuejs, I build my project and deploy it on nginx server, but I got same result. I try also on firefox, get same error.
In package.json, axios is in version "^0.19.0", vuejs "^2.6.10".
What is wrong in my code ? Or is about configuration of my server ?
Thanks for your help!

Intermittent 403 CORS errors when using S3 signed URL to PUT object

I'm running into an issue where S3 will occasionally return a 403 on the OPTIONS request pre-flight when trying to upload an object from the client side with a PUT request to a signed URL. The frustrating part is that 90% of the time it works fine, and when it does fail, if I retry the same request a few times, then it will go through.
Has anyone experienced something similar, and if so, how did you solve it?
This seems related to this issue, but isn't exactly the same, because they're going through CloudFront, and I'm going directly to S3.
Generating the signed URL on the server side
const params = {
Bucket: **my bucket**,
Key: 'test/my-test-file.csv',
Metadata: {
'created-by': req.user.id,
'job-id': job.id,
},
};
const url = s3.getSignedUrl('putObject', params);
Angular.JS HTTP Request to the Signed URL
return $http({
url: **signedUrlHere**,
method: 'PUT',
data: file,
headers: {
'Content-Type': '',
},
transformRequest: [],
});
Bucket CORS Settings
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>
Request Headers
Host: s3.amazonaws.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:67.0) Gecko/20100101 Firefox/67.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Access-Control-Request-Method: PUT
Access-Control-Request-Headers: content-type
Referer: https://**.***.com/*/*/*
Origin: https://**.***.com
DNT: 1
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Response Headers
HTTP/1.1 403 Forbidden
x-amz-request-id: *********
x-amz-id-2: *******
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Thu, 20 Jun 2019 02:54:39 GMT
Server: AmazonS3
After looking into this for a while we found out what the problem was with these intermittent 403s, at least for us. Hopefully this helps you or anybody else with the same problem. In our case, it was caused by people having their computer setup to the wrong time. That explains the intermittent nature, because most people have their system set on automatic time and time-zone, the few who don't and have the wrong time keep getting errors.

Unexpected token # in JSON at position 0 with Angular 5 and Express JS

I am developing a MEAN stack application
where My client is in Angular 5 and backend is in node and express js with mongo db as my storage.
I am trying to upload an image from one my Angular Page to my express js rest API which will finally upload the image in AWS cloud
Sample code for my Component is as below having two methods one onFileSelected which sets the selected file and the onUpload which invokes a service class method
private selectedFile:File=null;
onUpload(){
this.productCategoryService.
uploadProductCategoryImage(this.selectedFile).subscribe(
data=>{console.log("Upload success---"+data)},
err=>{console.log("Upload error ---"+err.message)});
}
onFileSelected(event){
this.selectedFile=event.target.files[0];
console.log("this.selectedFile---------"+this.selectedFile);
}
The service does a post call of rest service on express js.Code of my service class is as below
public uploadProductCategoryImage(selectedFile:File){
const formData = new FormData();
formData.append("userFile",selectedFile);
return this.http.post<any>
('http://localhost:3000/datastore/api/aws/api/file',formData);
}
My html page
<input type="file" (change)="onFileSelected($event)" name="userFile">
<button type="button" class="btn btn-success" (click)="onUpload()">
<span class="fa fa-upload"></span>Upload</button>
However when I post this data I get an error as below on the server side
SyntaxError: Unexpected token # in JSON at position 0
at JSON.parse (<anonymous>)
I found something from my browser.It shows BAD request.
I am not sure what is happening.Please suggest.
General------------
Request URL: http://localhost:3000/datastore/api/aws/api/file
Request Method: POST
Status Code: 400 Bad Request
Remote Address: [::1]:3000
Referrer Policy: no-referrer-when-downgrade
Response Header------------
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 1314
Content-Security-Policy: default-src 'self'
Content-Type: text/html; charset=utf-8
Date: Fri, 22 Jun 2018 16:47:12 GMT
X-Content-Type-Options: nosniff
X-Powered-By: Express
Request Header--------------------
Accept: application/json
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Connection: keep-alive
Content-Length: 232378
Content-Type: application/json
Host: localhost:3000
Origin: http://localhost:4200
Referer: http://localhost:4200/contactus
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36
------WebKitFormBoundaryAEgWhmyP5LdwDaEl
Content-Disposition: form-data; name="userFile"; filename="Screenshot from 2018-06-04 20-23-28.png"
Content-Type: image/png
Thanks
Sachin
In Angular 5 Httpclient uses JSON as default format. So you have use BLOB as your response type like
getImage(imageUrl: string): Observable<Blob> {
return this.httpClient.get(imageUrl, { responseType: 'blob' });
}
Check this link for reference: https://stackblitz.com/edit/angular-1yr75s?file=src%2Fapp%2Fimage.service.ts

Copy a file to source folder on OneDrive using OneDrive API

I tried to copy a file on OneDrive:
Request:
POST https://api.onedrive.com/v1.0/drive/root:/onedrive_test/foo/bar/a.txt:/action.copy
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Type: application/json
Content-Length: 84
Accept: */*
Authorization: Bearer Ew...
Prefer: respond-async
User-Agent: python-requests/2.7.0 CPython/3.4.4 Windows/7
b'{"parentReference": {"path": "/drive/root:/onedrive_test/foo/bar"}, "name": "b.txt"}'
Response: 202 Accepted
Request:
GET https://api.onedrive.com/v1.0/monitor/4sT2gLAWdXVK7EdkDM7k24ObcUFTzScBof3T80HbmKfVHPnUCDK4fWe01ttH9...
Accept-Encoding: gzip, deflate
Connection: keep-alive
Accept: */*
Authorization: Bearer Ew...
User-Agent: python-requests/2.7.0 CPython/3.4.4 Windows/7
None
Response: 500 Internal server error if copying to the same folder where source file is.
If I use
b'{"parentReference": {"path": "/drive/root:/onedrive_test/foo/bar2"}, "name": "b.txt"}'
or
b'{"parentReference": {"path": "/drive/root:/onedrive_test/foo"}, "name": "b.txt"}'
everything works fine.
This is an ongoing issue with OneDrive Consumer - I'll respond to this answer with a comment once it is resolved.