I can download a test.pdf file using this get command below with a code that I've in C for a microcontroller to communicate with a server:
GET /TestFolder/test.pdf HTTP/1.1\r\n Host: www.xyz.com\r\n\r\n
the file: test.pdf is located in folder: TestFolder at Host: xyz.com
I wanted to test the program on Amazon S3; so I created an account and uploaded the data and made the file and folder public, added policy to the S3 bucket so objects can be accessed. When I send the GET command above to the S3 host: s3-us-west-2.amazonaws.com I get an error after the socket is connected and I get the server's IP, error message from S3 says:
Response error: HTTP/1.1 400 Bad Request
Transfer-Encoding: chunked
Date: Mon, .. 2015 04:15:02 GMT
Connection: close
Server: AmazonS3
I thought to remove extra \r\n from the get command, and sent this command to s3
GET /TestFolder/test.pdf HTTP/1.1\r\n Host: s3-us-west-2.amazonaws.com\r\n
This time, the requests hangs with no response. I don't get any error message, the socket is connected as usual and I see the server's IP.
I'll appreciate any suggestion or input where the problem could be. Obviously the GET command works for file download when the file is public at other sites, has anyone encountered this kind of issue with http/1.1 GET command? I can access the file from AWS S3 in my browser by typing the link.
You need \r\n twice at the end of the request, otherwise the server thinks it's waiting for more headers. The problem, here, is that you are not specifying the bucket, which you can do in the path or in the Host: header... you have to do it in one, or the other, but not both.
GET /your-bucket-name/TestFolder/test.pdf HTTP/1.1\r\n
Host: s3-us-west-2.amazonaws.com\r\n\r\n
...or...
GET /TestFolder/test.pdf HTTP/1.1\r\n
Host: your-bucket-name.s3-us-west-2.amazonaws.com\r\n\r\n
Related
I'm sending a request to server service called path-generator which gives me a generated url and I'm uploading images to this url which moves the images to s3 bucket.
I'm able to upload the file to the bucket, but it arrives as broken image (when i'm uploading the file with 'Accept: application/json, text/plain' header)
or as 'Content-Transfer-Encoding: binary' when not using the header
The requests:
With header:
Connection: keep-alive
Content-type: image/png
Accept: application/json, text/plain
:
Content-Length: 201571
Host: {some host}
User-Agent: Apache-HttpClient/4.5.6 (Java/11.0.1)
without header:
Connection: keep-alive
Content-type: application/json
Content-Length: 221702
Host: {some host}
User-Agent: Apache-HttpClient/4.5.6 (Java/11.0.1)
I'm using the exact same flow as the client so it must be something wrong I'm doing with Jmeter
When you tick Use multipart/form-data box JMeter doesn't use Content-Type header specified in the HTTP Header Manager, most probably this is the reason for your request failure.
Try recording the file upload request using HTTP(S) Test Script Recorder (make sure to copy the file to "bin" folder of your JMeter installation) to see if JMeter is capable of properly capture the upload request(s). If it is - you should be good to go. If not - you will have to amend JMeter configuration to 100% match request specification, check out Testing REST API File Uploads in JMeter article for example test plan.
S3 PUT requests only need file content and no extra fields.
Do not pass parameter name and MIME type, only pass filePath correctly. If required add header Content-Type: image/jpg or video/mp4 in case it's a video. Similarly for pdf, text, etc.
Additionally, when you download the broken file and open it in notepad++
along with the actual file which was used to upload in notepad++
you can see the difference: the broken file has some extra text in it. If you remove it, it will work as expected
Also do not try this is notepad, use notepad++ only.
How would I go about translating the following curl command to a Jitterbit operation?
curl -i -u username:password -X POST -F file=#/path/to/file.csv
https://website.com/api/filepost
Currently I have my Operation structured as follows:
Script:
$jitterbit.target.http.form_data = true;
$jitterbit.target.http.form_data.filename = "file.csv";
$jitterbit.target.http.form_data.name = "file";
Source
A CSV file without headers, which matches the API's specifications (sent the same file successfully via curl)
Transformation:
Text to Text - both source and target use the same file format as the Source file
API Endpoint
Currently I authenticate successfully, but I get a 400/Bad Request error message saying "No file attached".
Full error message:
The operation "2. POST Preapplicants - CSV to API" failed.
Fatal Error
Failed to post to the url 'https://website.com/api/filepost’.
The last (and probably most relevant) error was: The server
returned HTTP Status Code : 400 Bad Request Error is: The
request could not be processed by the server due to invalid
syntax. Headers sent by the server: HTTP/1.1 400 Bad Request
Server: nginx/1.10.3 (Ubuntu) Content-Type: application/json
Transfer-Encoding: chunked Connection: keep-alive Cache-Control:
no-cache Date: Tue, 12 Sep 2017 18:55:38 GMT The response was:
{"message":"No file attached."}
I solved this problem by doing the following:
1. Changing from a transformation operation to an archive operation (using the same source, target, and script)
2. Changing the content-type of my HTTP connection to multipart/form-data (the default content-type passed by curl)
you can do it that way. what I did was use a file list operation stored that into an array then uses a Base64EncodeFile() function to upload the file
I cant access a specific filetype on my customer server (production).
Here are the results with cURL:
curl "http://domain.tld/fonts/glyphicons-halflings-regular.eot" -I
HTTP/1.1 200 OK
Date: Tue, 28 Jul 2015 12:06:23 GMT
Server: Apache/2.2.15 (Red Hat)
Last-Modified: Tue, 19 May 2015 15:32:20 GMT
ETag: "14023-4f42-516710421e900"
Accept-Ranges: bytes
Content-Length: 20290
Connection: close
Content-Type: application/vnd.ms-fontobject
The file is here.
But when I try to get the file content:
curl "http://domain.tld/fonts/glyphicons-halflings-regular.eot"
curl: (56) Recv failure: Connection was reset
I can't (yet) access the customer server, so I'm trying to guess what's wrong here.
What is working so far:
curl "https://domain.tld/fonts/glyphicons-halflings-regular.eot" --insecure
It is working in HTTPS, even if there is no certificate (which is why I use --insecure). I get the file content.
The customer can get the file if he accesses the file from a local URL.
I can access all other files on the server, even in the fonts directory.
I can't access all .eot files, even in other directories.
So I think it is one of those 2 problems:
- Apache configuration / .htaccess problem.
- Proxy / reverse proxy problem.
What do you think about it?
What kind of other test should I do?
What information should I ask to the customer?
Thanks.
Ok, here is the cause:
The customer firewall blocks .eot file content.
A vulnerability in Embedded Web Fonts Could Allow Remote Code Execution.
http://www.checkpoint.com/defense/advisories/public/2006/cpai-2006-010.html
As the .eot files are used by IE8 and lower, and those browser versions are not required by the customer, I've simply removed all references to .eot files.
Another solution would be to ask for the customer firewall admins to add an exception, as the severity is low.
My idea is to LOCK a file on an apache/WebDAV server, PUT an updated version of it on the server and UNLOCK it afterwards.
I just tried the following with cadaver:
create a file A.txt with content a file
GET file A.txt which yields a file
edit A.txt to be updated file and save it (in cadaver)
GET file A.txt which yields still yields a file
close edit (VIM) in cadaver
GET file A.txt which yields updated file
I guess internally cadaver LOCKs the file, GETs it and changes it locally. Then it PUTs it and UNLOCKs it.
QUESTION: how can I do this with curl?
PROBLEM: When then connection is slow and I do a PUT for a file, that is not yet completely uploaded, I only get the yet uploaded part. I would like to get the old one as long as the new one isn't complete.
TRIED: I tried the following to LOCK the file by hand (i.e. with cURL):
curl -v -X LOCK --user "user:password" http://myServer/newFile
What I get is:
* About to connect() to myServer port 80 (#0)
* Trying xx.xx.xxx.xxx... connected
* Connected to myServer (xx.xx.xxx.xxx) port 80 (#0)
* Server auth using Basic with user 'user'
> LOCK /newFile HTTP/1.1
> Authorization: Basic xxxxxxxxxxxxxxxxx
> User-Agent: curl/7.21.6 (x86_64-pc-linux-gnu) libcurl/7.21.6 OpenSSL/1.0.0e zlib/1.2.3.4 libidn/1.22 librtmp/2.3
> Host: myServer
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Date: Wed, 02 May 2012 15:20:55 GMT
< Server: Apache/2.2.3 (CentOS)
< Content-Length: 226
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>
* Closing connection #0
Looking at the apache log file I find:
[Wed May 02 15:20:55 2012] [error] [client xx.xx.xxx.xxx] The lock refresh for /newFile failed because no lock tokens were specified in an "If:" header. [400, #0]
[Wed May 02 15:20:55 2012] [error] [client xx.xx.xxx.xxx] (20)Not a directory: No locktokens were specified in the "If:" header, so the refresh could not be performed. [400, #103]
Thanks for any hint!!
UPDATE: I added my problem description.. Cheers!
The LOCK method requires a body which contains an XML description of the lock you want to take out. Your cURL test didn't include this body, hence the 400 error response.
But if I understand your question correctly, you want to:
LOCK
PUT
UNLOCK
If that's true, why would you bother with the LOCK and UNLOCK? Just do the PUT! Locks would only be useful if you want to carry out multiple operations while you are holding the lock and avoid having another client see the object in its partially modified state or (perhaps worse) modify the object concurrently with you.
A typical case where locking can be useful is a read-modify-write cycle: you want to GET the object, modify it locally, and PUT it back, but disallow another client from making a competing change between the time you GET it and the time you PUT it. However, for dealing with this specific case, HTTP offers a different method of resolving the issue, without using locks (which are ill-suited for a stateless protocol like HTTP):
GET the object
Modify it locally
PUT the object back with an If-Match header that contains the original ETag returned in step 1
If the PUT results in a 412 error, go back to step 1. Otherwise, you are done.
UPDATE: based on your updated question, I see that you are see a partial truncated or half-uploaded version of the new file if you do a GET concurrent with a PUT. This is unfortunate. The server should treat the PUT as atomic with respect to other requests. Other clients should either see the old version or the new version, never a state in between. There's nothing you should need to do from the client end to make this true. It should be fixed in the server.
We have an Tomcat server where we're trying to log the HTTP version which the response is sent with. We've seen a few times that it seems to be HTTP/0.9, which kills the content (not supported I guess?). We would like to get some stats on this by using the access log in apache. However, since the header line for this isn't prefixed by anything, we cannot use the %{xxx}o logging.
Is there a way to get this?
An example:
Response is:
HTTP/1.1 503 This application is not currently available
Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Length: 1090
Date: Wed, 12 May 2010 12:53:16 GMT
Connection: close
And we'd like the catch HTTP/1.1 (alternatively, HTTP/1.1 503 This application is not currently available.
Is this possible? We do not have access to the application being served, so we need to do this either as a Java filter, or in the tomcat access log - Preferably in the access log.
Enabling the <Valve className="org.apache.catalina.valves.RequestDumperValve"/> in server.xml writes out the request and response headers for each request.
Example:
19-May-2010 12:26:18 org.apache.catalina.valves.RequestDumperValve invoke
INFO: protocol=HTTP/1.1