Multipart form file upload, Nginx is "eating" it and not passing it on to the handler - file-upload

I am new to nginx, but am familar with low-level HTTP and apache. When I try to do a multipart/form file upload, nginx writes some of the client request body to disk, but never finishes and it never passes it to the down/upstream script.
My specific setup is I have nginx with /dyn redirected to localhost:1337, where a node.js instance is listening. It works... except for the file upload handler. Also in the config is a /debug which is redirected to localhost:1338, which goes to a simple dump server.
I changed the error log handling to 'info'. It reports storing the client body to a file and when I exampline it is almost as I expected:
--boundary_.oOo._MjM5NzEwOTkxMzU2MjA0NjM5MTQxNDA3MjYwOA==
Content-Type: image/jpeg
Content-Disposition: form-data; name="file"; filename="dccde7b5-25aa-4bb2-96a6-81e9358f2252.jpg"
<binary data, ~89k>
The problem with this file is too short, only 81,920 bytes when the file is 88,963 bytes, it should be 88,963 + the header above.... But that is literally only half of it. There are 2 files (about the same size) that are coming in, so i would expect that file to be about ~160k. What nginx is doing is reassigning the request's http-level Content-length then passes that header on to the script, that's it and of course my script complains that it never finds --boundary_.oOo._MjM5NzEwOTkxMzU2MjA0NjM5MTQxNDA3MjYwOA== When I do the same request to my debug service without nginx in the middle, it corrrectly sends the data, anf the http Content-length is an apropriate 186943 bytes (both files are around 80k, so this makes sense)
My nginx config is default aside from what I've mentioned here.
Edit: after some more experimenting, all files in the client body directory are 81920 bytes

Related

How can I get past a 502 proxy error when uploading a file via JMeter?

I am trying to set up JMeter 5.5 to upload a file into our system as if it were happening in the user interface, but I continue to get a 502 proxy error. How can I get the file to upload successfully?
I am using the parameter hivUpload. If I change the parameter from hivUpload to anything else, I get a 500 error instead of a 502, so I think hivUpload is the correct parameter.
I have the file in the JMeter bin folder, but it's not clear if I need to include the full file path or not. I have tried it both ways, and neither has been successful.
Setup
Results
Source file location
Request
POST https://cdc-ew.lutherhq1b.int/rest/v1/upload
POST data:
--s6kF9JKRBTVi1qsnV4rm1hbf6gd6HMeH62
Content-Disposition: form-data; name="hivUpload"; filename="AgencyInfo 2_0 - good file.xml"
Content-Type: application/xml
Content-Transfer-Encoding: binary
<actual file content, not shown here>
--s6kF9JKRBTVi1qsnV4rm1hbf6gd6HMeH62--
[no cookies]
Response
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>
I am able to get other processes to happen successfully by sending in a JSON request as if I were doing data entry. Note the successful green entries in the results, so I believe all of the other items are set up correctly. However, we have not tried using JMeter to upload files before. I know it is possible to do this I have seen articles about it (like this one https://artoftesting.com/fileuploadinjmeter#:~:text=File%20upload%20in%20JMeter%20will,checkbox%20in%20HTTP%20Request%20sampler.), but it's not clear to me what I need to tweak in my setup.
We cannot comment on your configuration because we don't know how exactly the file needs to be supplied as there are multiple ways of uploading the file.
You could try just recording it using JMeter's HTTP(S) Test Script Recorder, just make sure that the file you're uploading will be in "bin" folder of JMeter installation, only this way JMeter will be able to intercept the request and generate proper HTTP Request sampler and HTTP Header Manager
More information: JMeter Performance Testing: Upload and Download Scenarios

How do I configure apache-traffic-server to forward an http request to an https remote server?

I have an esp8266 which was directly sending http requests to http://fcm.googleapis.com/fcm/send but since google seems have stopped allowing requests to be send via http, I need to find a new solution.
I started down a path to have the esp8266 directly send the request via https and while it works on a small example the memory footprint required for the https request is to much in my full application and I end up crashing the esp8266. While there are still some avenues to explore that might allow me to continue to directly send messages to the server, I think I would like to solve this by sending the request via http to a local "server" raspberry pi, and have that send the request via https.
While I could run a small web server and some code to do handle the requests, it seems like this is exactly something traffic-server should be able to do for me.
I thought this should be a one liner. I added the following the the remap.config file.
redirect http://192.168.86.77/fcm/send https://fcm.googleapis.com/fcm/send
where 192.168.86.77 is the local address of my raspberry pi.
When I send requests to http://192.168.86.77/fcm/send:8080 I get back the following:
HTTP/1.1 404 Not Found
Date: Fri, 20 Sep 2019 16:22:14 GMT
Server: Apache/2.4.10 (Raspbian)
Content-Length: 288
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /fcm/send:8080 was not found on this server.</p>
<hr>
<address>Apache/2.4.10 (Raspbian) Server at localhost Port 80</address>
</body></html>
I think 8080 is the right port.
I am guessing this is not the one liner I thought it should be.
Is this a good fit for apache-traffic-controller?
Can someone point me to what I am doing wrong and what is the right way to accomplish my goal?
Update:
Based on Miles Libbey answer below, I needed to make the following update to the Arduino/esp8266 code.
Change:
http_.begin("http://fcm.googleapis.com/fcm/send");
To:
http_.begin("192.168.86.77", 8080, "http://192.168.86.77/fcm/send");
where http_ is the instance of the HTTPClient
And after installing trafficserver on the my raspberry pi, I needed to add the following two lines to the /etc/trafficserver/remap.config
map http://192.168.86.77/fcm/send https://fcm.googleapis.com/fcm/send
reverse_map https://fcm.googleapis.com/fcm/send http://192.168.86.77/fcm/send
Note the reverse_map line is only needed if you want to get feedback from fcm, ie if the post was successful or not.
I would try a few changes:
- I'd use map:
map http://192.168.86.77/fcm/send https://fcm.googleapis.com/fcm/send instead of redirect. The redirect is meant to send your client a 301, and then your client would follow it, which sounds like it'd defeat your purpose. map should have ATS do the proxying.
- I think your curl may have been off -- the port usually goes after the domain part -- eg, curl "http://192.168.86.77:8080/fcm/send". (and probably better:
curl -x 192.168.86.77:8080 "http://192.168.86.77:8080/fcm/send", so that the port isn't part of the remapping.

Why am I getting headers embedded to files uploaded to S3 using JMeter?

Whenever uploading a file (json) to S3 using JMeter's HTTP Request sampler the uploaded file contains the HTTP headers at the top of the file. I am using a signed URL (with actual values):
https://something-s3bucket-something.s3.eu-west-1.amazonaws.com/afda5939-c232-d746-06f7-68790abde85b-91e962d6-4643-8091-fab8-9d0f78f35810.jsonTemp?X-Amz-Expires=18000&x-amz-security-token=somethinglongandcomplicated&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=somethingspecific&X-Amz-Date=20190627T070453Z&X-Amz-SignedHeaders=host;x-amz-security-token&X-Amz-Signature=something
HTTP Request Settings:
Resulting json:
--Y0B3WMeM2M6xYSUHpjPUIj72y3xnO_pswRA12Oh
Content-Disposition: form-data; name="filename"; filename="500payslips.json"
Content-Type: binary/octet-stream
Content-Transfer-Encoding: binary
{
... json content ...
}
}
--Y0B3WMeM2M6xYSUHpjPUIj72y3xnO_pswRA12Oh--
I know I could add the s3 java libraries to JMeter and then code the upload but I would prefer to just simply use the HTTP Request sampler (if possible).
More of a workaround than a fix(?). Instead of sending the file as a multipart/form-data I just read the contents of the file into a variable and sent the content as part of the body of the request.

How do ETags in the HTTP header actually work?

I don't know if I am not correctly understanding how the caching aspect of ETags work if there is some other issue I am dealing with, but I'll walk you through my situation.
From my understanding ETags are a unique hash that is created based on the file information and they are sent as part of Response header to uniquely identify the file. If the file is updated then the info is changed and hence the ETag for the file is also changed.
In my project, I need a fresh JS file to be fetched everytime I make changes to the file. I can't use version tags or unique hashes as part of the file name. I thought an ETag would work where
Http Request
GET myFile.js
Client ------------------> SERVER
Http Response 200
Http Response Header
accept-ranges: bytes
cache-control: max-age=86400, public
etag: "a7-58c3bb52101c4"
......
myFile.js
Client <------------------ SERVER
// myFile.js has not been changed
Http Request
GET myFile.js
Client ------------------> SERVER
Http Response 304
Http Response Header
accept-ranges: bytes
cache-control: max-age=86400, public
etag: "a7-58c3bb52101c4"
......
Client uses cached version of file
// myFile has been changed
Http Request
GET myFile.js
Client ------------------> SERVER
Http Response 200
Http Response Header
accept-ranges: bytes
cache-control: max-age=86400, public
etag: "88-58c3a1cb8474f" // new etag generated
......
myFile.js
Client <------------------ SERVER
So, if you request the file again and no changes have been made..the etag will remain the same and you'll get a 304 will indicate that the cached version should be used.
If the file has been changed the etag will be different as well and a fresh copy of the file will be sent by the server.
This is how I expected it to work.
MY PROBLEM:
When I update myFile.js it seems like I never get the new ETag has back. It just defaults to the cahced version of the file. If I clear the cache then I get the latest file and the new ETag. This to me seems to defeat the purpose. Is this how it works or am I understanding something incorrectly here?

Jmeter - image uploaded to s3 as binary/broken image

I'm sending a request to server service called path-generator which gives me a generated url and I'm uploading images to this url which moves the images to s3 bucket.
I'm able to upload the file to the bucket, but it arrives as broken image (when i'm uploading the file with 'Accept: application/json, text/plain' header)
or as 'Content-Transfer-Encoding: binary' when not using the header
The requests:
With header:
Connection: keep-alive
Content-type: image/png
Accept: application/json, text/plain
:
Content-Length: 201571
Host: {some host}
User-Agent: Apache-HttpClient/4.5.6 (Java/11.0.1)
without header:
Connection: keep-alive
Content-type: application/json
Content-Length: 221702
Host: {some host}
User-Agent: Apache-HttpClient/4.5.6 (Java/11.0.1)
I'm using the exact same flow as the client so it must be something wrong I'm doing with Jmeter
When you tick Use multipart/form-data box JMeter doesn't use Content-Type header specified in the HTTP Header Manager, most probably this is the reason for your request failure.
Try recording the file upload request using HTTP(S) Test Script Recorder (make sure to copy the file to "bin" folder of your JMeter installation) to see if JMeter is capable of properly capture the upload request(s). If it is - you should be good to go. If not - you will have to amend JMeter configuration to 100% match request specification, check out Testing REST API File Uploads in JMeter article for example test plan.
S3 PUT requests only need file content and no extra fields.
Do not pass parameter name and MIME type, only pass filePath correctly. If required add header Content-Type: image/jpg or video/mp4 in case it's a video. Similarly for pdf, text, etc.
Additionally, when you download the broken file and open it in notepad++
along with the actual file which was used to upload in notepad++
you can see the difference: the broken file has some extra text in it. If you remove it, it will work as expected
Also do not try this is notepad, use notepad++ only.