Error:: AWS authentication requires a valid Date or x-amz-date header - amazon-s3

I am using Command Line Tool for uploading files from Amazon S3 to CloudSearch for Indexing using the command-
cs-import-documents -d searchdev3 --source s3://mybucket/html
It is giving error-
Connecting to CloudSearch in region [us-east-1]
AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S
3; Status Code: 403; Error Code: AccessDenied; Request ID: 506FD29F73XXXXXX)
How can I track where the problem is occurring?
UPDATE
When run with --verbose I received below output-
Looking up DocServiceEndpoint for domain searchdev3 using config service at null
Connecting to CloudSearch in region [us-east-1]
Sending to endpoint [https://cloudsearch.us-east-1.amazonaws.com/] for region [u
s-east-1]
Connection request: [route: {s}->https://cloudsearch.us-east-1.amazonaws.com][to
tal kept alive: 0; route allocated: 0 of 50; total allocated: 0 of 50]
Connection leased: [id: 0][route: {s}->https://cloudsearch.us-east-1.amazonaws.c
om][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]
Connecting to cloudsearch.us-east-1.amazonaws.com:443
CookieSpec selected: best-match
Auth cache not set in the context
Proxy auth state: UNCHALLENGED
Sending request: POST / HTTP/1.1
>> "POST / HTTP/1.1[\r][\n]"
>> "Host: cloudsearch.us-east-1.amazonaws.com[\r][\n]"
>> "Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXX/20151130/us-
east-1/cloudsearch/aws4_request, SignedHeaders=host;user-agent;x-amz-date, Signa
ture=e93cd0c22189084ff55c1084[\r][\n]"
>> "X-Amz-Date: 20151130T103603Z[\r][\n]"
>> "User-Agent: aws-internal/3[\r][\n]"
>> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]"
>> "Content-Length: 73[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "[\r][\n]"
>> POST / HTTP/1.1
>> Host: cloudsearch.us-east-1.amazonaws.com
>> Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXX/20151130/us-e
ast-1/cloudsearch/aws4_request, SignedHeaders=host;user-agent;x-amz-date, Signat
ure=e93cd0c22189084ff55c1084
>> X-Amz-Date: 20151130T103603Z
>> User-Agent: aws-internal/3
>> Content-Type: application/x-www-form-urlencoded; charset=utf-8
>> Content-Length: 73
>> Connection: Keep-Alive
>> "Action=DescribeDomains&Version=2013-01-01&DomainNames.member.1=searchdev3"
<< "HTTP/1.1 200 OK[\r][\n]"
<< "x-amzn-RequestId: 2907d340-974e-11e5-a6e5-33953f55803d[\r][\n]"
<< "Content-Type: text/xml[\r][\n]"
<< "Content-Length: 1635[\r][\n]"
<< "Date: Mon, 30 Nov 2015 10:36:05 GMT[\r][\n]"
<< "[\r][\n]"
Receiving response: HTTP/1.1 200 OK
<< HTTP/1.1 200 OK
<< x-amzn-RequestId: 2907d340-974e-11e5-a6e5-33953f55803d
<< Content-Type: text/xml
<< Content-Length: 1635
<< Date: Mon, 30 Nov 2015 10:36:05 GMT
<< "<"
<< "D"
<< "e"
<< "s"
<< "cribeDomainsResponse xmlns=""
<< "http://cloudsearch.amazonaws.com/doc/2013-01-01/">[\n]"
<< " <DescribeDomainsResult>[\n]"
<< " <DomainStatusList>[\n]"
<< " <member>[\n]"
<< " <Created>true</Created>[\n]"
<< " <RequiresIndexDocuments>false</RequiresIndexDocuments>[\n]"
<< " <SearchInstanceType>search.m1.small</SearchInstanceType>[\n]"
<< " <DomainId>613875336225/searchdev3</DomainId>[\n]"
<< " <DocService>[\n]"
<< " <Endpoint>doc-searchdev3-ytmgti5ijnobqprgg65e4yxrhe.us-east-1.clou
dsearch.amazonaws.com</Endpoint>[\n]"
<< " </DocService>[\n]"
<< " <SearchPartitionCount>1</SearchPartitionCount>[\n]"
<< " <Deleted>false</Deleted>[\n]"
<< " <SearchService>[\n]"
<< " <Endpoint>search-searchdev3-ytmgti5ijnobqprgg65e4yxrhe.us-east-1.c
loudsearch.amazonaws.com</Endpoint>[\n]"
<< " </SearchService>[\n]"
<< " <DomainName>searchdev3</DomainName>[\n]"
<< " <Processing>true</Processing>[\n]"
<< " <SearchInstanceCount>1</SearchInstanceCount>[\n]"
<< " <ARN>arn:aws:cloudsearch:us-east-1:613875336225:domain/searchdev3</A
RN>[\n]"
<< " <Limits>[\n]"
<< " <AvailableInstanceTypes>[\n]"
<< " <member>search.m1.small</member>[\n]"
<< " <member>search.m3.medium</member>[\n]"
<< " <member>search.m3.large</member>[\n]"
<< " <member>search.m3.xlarge</member>[\n]"
<< " <member>search.m3.2xlarge</member>[\n]"
<< " </AvailableInstanceTypes>[\n]"
<< " <MaximumReplicationCount>5</MaximumReplicationCount>[\n]"
<< " <MaximumPartitionCount>10</MaximumPartitionCount>[\n]"
<< " </Limits>[\n]"
<< " </member>[\n]"
<< " </DomainStatusList>[\n]"
<< " </DescribeDomainsResult>[\n]"
<< " <ResponseMetadata>[\n]"
<< " <RequestId>2907d340-974e-11e5-a6e5-33953f55803d</RequestId>[\n]"
<< " </ResponseMetadata>[\n]"
<< "</DescribeDomainsResponse>[\n]"
Connection [id: 0][route: {s}->https://cloudsearch.us-east-1.amazonaws.com] can
be kept alive indefinitely
Connection released: [id: 0][route: {s}->https://cloudsearch.us-east-1.amazonaws
.com][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]
Creating S3 source reader
Connection request: [route: {s}->https://mybucket.s3.amazonaws.com][total
kept alive: 0; route allocated: 0 of 50; total allocated: 0 of 50]
Connection leased: [id: 1][route: {s}->https://mybucket.s3.amazonaws.com]
[total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]
Connecting to mybucket.s3.amazonaws.com:443
CookieSpec selected: best-match
Auth cache not set in the context
Proxy auth state: UNCHALLENGED
Sending request: GET /?prefix=html HTTP/1.1
>> "GET /?prefix=html HTTP/1.1[\r][\n]"
>> "Host: mybucket.s3.amazonaws.com[\r][\n]"
>> "Authorization: AWS XXXXXXXX:eA+KR8i3SpRQdTS/pxeBeci7iOw=[\r][\n]
"
>> "User-Agent: aws-internal/3[\r][\n]"
>> "Date: Mon, 30 Nov 2015 10:36:05 +00:00[\r][\n]"
>> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "[\r][\n]"
>> GET /?prefix=html HTTP/1.1
>> Host: mybucket.s3.amazonaws.com
>> Authorization: AWS XXXXXXXX:eA+KR8i3SpRQdTS/pxeBeci7iOw=
>> User-Agent: aws-internal/3
>> Date: Mon, 30 Nov 2015 10:36:05 +00:00
>> Content-Type: application/x-www-form-urlencoded; charset=utf-8
>> Connection: Keep-Alive
<< "HTTP/1.1 403 Forbidden[\r][\n]"
<< "x-amz-bucket-region: us-east-1[\r][\n]"
<< "x-amz-request-id: DA6567BEA36C46F9[\r][\n]"
<< "x-amz-id-2: kCC6NxI43AJi3G+YB0Dxgoc6ZrCGh/V3Wc4nhSnxk+mxYpdH50am3BBgzXfkLHrj
[\r][\n]"
<< "Content-Type: application/xml[\r][\n]"
<< "Transfer-Encoding: chunked[\r][\n]"
<< "Date: Mon, 30 Nov 2015 10:36:06 GMT[\r][\n]"
<< "Server: AmazonS3[\r][\n]"
<< "[\r][\n]"
Receiving response: HTTP/1.1 403 Forbidden
<< HTTP/1.1 403 Forbidden
<< x-amz-bucket-region: us-east-1
<< x-amz-request-id: DA6567BEA36C46F9
<< x-amz-id-2: kCC6NxI43AJi3G+YB0Dxgoc6ZrCGh/V3Wc4nhSnxk+mxYpdH50am3BBgzXfkLHrj
<< Content-Type: application/xml
<< Transfer-Encoding: chunked
<< Date: Mon, 30 Nov 2015 10:36:06 GMT
<< Server: AmazonS3
<< "117[\r][\n]"
<< "<?xml version="1.0" encoding="UTF-8"?>[\n]"
<< "<Error><Code>AccessDenied</Code><Message>AWS authentication requires a valid
Date or x-amz-date header</Message><RequestId>DA6567BEA36C46F9</RequestId><Host
Id>kCC6NxI43AJi3G+YB0Dxgoc6ZrCGh/V3Wc4nhSnxk+mxYpdH50am3BBgzXfkLHrj</HostId></Er
ror>"
<< "[\r][\n]"
<< "0[\r][\n]"
<< "[\r][\n]"
Connection [id: 1][route: {s}->https://mybucket.s3.amazonaws.com] can be
kept alive indefinitely
Connection released: [id: 1][route: {s}->https://mybucket.s3.amazonaws.co
m][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]
AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S
3; Status Code: 403; Error Code: AccessDenied; Request ID: DA6567BEA36C46F9)
Error Type: Client
Error Code: AccessDenied
HTTP Code: 403
Request ID: DA6567BEA36C46F9
[Ljava.lang.StackTraceElement;#12a7737

it seems you are using different version of java.
Please try to degrade java version. hope it will work. i had similar kind of issue before after degrading it to lower version fixed my problem.

Related

AWS SDK for Java V2: PutObject corrupts uploaded file when checksumAlgorithm is specified

I'm trying to use the AWS SDK for Java v2 to upload a file to S3 and have the upload integrity checked using the SHA256 trailing checksum. My code works fine to upload the file if the checksumAlgorithm(ChecksumAlorithm.SHA256) property of PutObjectRequest is not present but when it is included the uploaded file has additional characters at its start and is truncated at the end.
Here's the code I'm using:
byte[] smallJsonBytes = getObjectFile("UploadTest\\smallFile.txt");
PutObjectRequest putOb = PutObjectRequest.builder()
.bucket(credsProvider.getS3Bucketname())
.key(s3Key)
**.checksumAlgorithm(ChecksumAlgorithm.SHA256)**
.build();
PutObjectResponse response = this.myS3Client.putObject(putOb, RequestBody.fromBytes(smallJsonBytes));
Below is the debug output when the checksumAlgorithm property is included. You can see the AWS SDK is generating a trailing checksum as expected but, as noted already, the uploaded file is corrupted: You can see the insertion of the string "14" in the debug output and this string appears at the beginning of the uploaded file. Additionally, the uploaded file is truncated.
Clearly from the debug output there is a problem with the evaluation of the file's content length.
I've tried specifying the contentLength directly on the PutObjectRequest (as per documentation - .contentLength(long length)), and also tried converting smallJsonBytes to a ByteArrayInputStream and specifying the contentLength in the RequestBody.fromInputStream, both with the same result.
14:56:56.150 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "x-amz-content-sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER[\r][\n]"
14:56:56.150 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "X-Amz-Date: 20230216T215655Z[\r][\n]"
14:56:56.150 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "x-amz-decoded-content-length: 20[\r][\n]"
14:56:56.150 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "x-amz-sdk-checksum-algorithm: SHA256[\r][\n]"
14:56:56.150 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "x-amz-trailer: x-amz-checksum-sha256[\r][\n]"
14:56:56.150 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-Length: 99[\r][\n]"
14:56:56.150 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
14:56:56.150 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"
14:56:56.206 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "HTTP/1.1 100 Continue[\r][\n]"
14:56:56.206 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "[\r][\n]"
14:56:56.211 [main] DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 100 Continue
14:56:56.211 [main] DEBUG software.amazon.awssdk.core.internal.io.SdkLengthAwareInputStream - Specified InputStream length of 20 has been reached. Returning EOF.
14:56:56.213 [main] DEBUG software.amazon.awssdk.core.internal.io.SdkLengthAwareInputStream - Specified InputStream length of 20 has been reached. Returning EOF.
14:56:56.216 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "14[\r][\n]"
14:56:56.217 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Nothing much in here[\r][\n]"
14:56:56.217 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "0[\r][\n]"
14:56:56.217 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "x-amz-checksum-sha256:BthkalBn8SFAnEAFIWr2vtfO0h+TEdX/vOH+iZqKJ4k=[\r][\n]"
14:56:56.217 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"
14:56:56.389 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"

Reading an ActiveMQ Topic using cURL - Getting message bodies

I'm trying to debug some JMS code by running an ActiveMQ locally - Using the Docker image rmohr/activemq:5.15.4-alpine - and using cURL to read the messages posted to the topic. This is mostly working, but I'm not getting the actual body of the message.
What I'm doing is:
curl -v -XGET http://admin:admin#localhost:8161/api/message?destination=topic://Events&json=true&oneShot=true
And this works as follows:
> GET /api/message?destination=topic://Events&json=true&oneShot=true HTTP/1.1
> Host: localhost:8161
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 20 Nov 2018 17:09:03 GMT
< X-FRAME-OPTIONS: SAMEORIGIN
< Set-Cookie: JSESSIONID=mcjaka6dsuz6534j0gj0gfnv;Path=/api
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Cache-Control: no-cache, no-store, must-revalidate
< Pragma: no-cache
< Content-Type: application/x-www-form-urlencoded; charset=ISO-8859-1
< destination: topic://Events
< id: ID:b8b750274409-46243-1542707161912-1:23:1:1:6
< eventTime$iso8601: 2018-11-20T17:09:13.216Z
< eventTime$millis: 1542733753216
< eventName: ProcessItemEvent
< Transfer-Encoding: chunked
< Server: Jetty(9.2.22.v20170606)
<
However, the actual payload of the message is not returned. If I take off the json=true&oneShot=true then I get this instead:
{ [5 bytes data]
But still no more than that. And there should be a lot more than 5 bytes of payload data for these messages.
What am I missing in getting these messages to come through correctly?
Cheers
You are subscribing to topic://Events . Could it be that all the event information is in the header of the http response, and ActiveMQ is not adding anything extra in the body?
< destination: topic://Events
< id: ID:b8b750274409-46243-1542707161912-1:23:1:1:6
< eventTime$iso8601: 2018-11-20T17:09:13.216Z
< eventTime$millis: 1542733753216
< eventName: ProcessItemEven
If you want the messages posted to a specific topic, say 'foo.bar' you need to subscribe to topic://foo.bar .
I tried your command. As you can see below, I am getting the message body.
Could it be that you are receiving binary data that is not displaying in the terminal? Can you try saving the message body to a file. e.g. curl -o body.dat -v -XGET ...
$ curl -v -XGET 'http://admin:admin#localhost:8161/api/message?destination=topic://bayCarrState'
> GET /api/message?destination=topic://bayCarrState HTTP/1.1
> Authorization: Basic YWRtaW46YnJva2VyYWRtaW4=
> User-Agent: curl/7.29.0
> Host: localhost:8161
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Fri, 23 Nov 2018 04:55:59 GMT
< X-FRAME-OPTIONS: SAMEORIGIN
< Set-Cookie: JSESSIONID=sbwv48hmp5w9zw4zgttrbmyn;Path=/api
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Cache-Control: no-cache, no-store, must-revalidate
< Pragma: no-cache
< Content-Type: text/plain; charset=ISO-8859-1
< destination: topic://bayCarrState
< id: ID:prd-rh7.mirrabooka.local-40596-1541746708502-3:160:-1:1:105
< Transfer-Encoding: chunked
< Server: Jetty(9.2.22.v20170606)
<
[{"elemId":"BayCarrState1","elemType":"BayCarrState","elemValue":{"bayId":"1","binFullWt":0,"binTipng":0,"binUntipdWt":0,"continuableRailSam
":0,"delivType":"R","frontPos":14,"lastSamTipng":0,"lastTickTipng":0,"locNum":1,"milltrainId":"A","rearPos":1,"samInProgress":0,"samTipng":0
,"tickTipng":0,"tipInProgress":0,"tipReady":0},"version":294},{"elemId":"BayCarrState2","elemType":"BayCarrState","elemValue":{"bayId":"2","
binFullWt":0,"binTipng":0,"binUntipdWt":0,"continuableRailSam":0,"delivType":"T","frontPos":28,"lastSamTipng":0,"lastTickTipng":0,"locNum":1
,"milltrainId":"A","rearPos":15,"samInProgress":0,"samTipng":0,"tickTipng":0,"tipInProgress":0,"tipReady":0},"version":294}]
* Connection #0 to host localhost left intact

Forming a JSON post request with Boost asio

I don't usually get this low level, so I think I'm just missing something obvious with the header.
Here's the code:
request_stream << "POST / HTTP/1.1\r\n"
"Host: localhost:5000 \r\n"
"Accept: */*\r\n"
"Content-Type: application/json\r\n";
request_stream << "Content-Length: " << json.length() << "\r\n\r\n";
request_stream << json;
The resulting string is
POST / HTTP/1.1
Host: localhost:5000
Accept: */*
Content-Type: application/json
Content-Length: 34
{'key1':'value1', 'key2':'value2'}
I then send the request_stream string to the server. I can connect to the server, but it says "bad request". I'm guessing there's something wrong with the above string. Any ideas?
Thanks.
Turns out that the request was fine. The issue was the simple Flask server I set up. I was calling
print(request.get_json())
when I should've have been calling
print(request.form.get('data'))

How do I get the ip address of the remote endpoint of a socket of type boost::asio::ssl::stream<boost::asio::ip::tcp::socket>

How do I get the ip address of the remote endpoint of a socket of type boost::asio::ssl::stream. I want to print peer ip like this- but it doesnt work
std::cout << "Peer IP: " << socket_.remote_endpoint().address().to_string()
I figured it out
It should be
std::cout << "Peer IP: " << socket_.lowest_layer().remote_endpoint().address().to_string() << std::endl;

Linkedin Share Api using simple curl

I do a simple request to share something on my linkedin profile
I try to just do it with curl but it doesn't work
I set the content-type to JSON in the header and I also set the authorization in the header
curl -v -H 'Authorization: OAuth oauth_consumer_key="V9uKPtEEB7JxQvYzbR2DuDfLe4qXjASgP5UhPJp8k13CYgSnN3BRFILxUJ6ApuCH",oauth_token="d6af5cda-b46e-4c38-b566-11428a9584fc",oauth_signature_method="HMAC-SHA1",oauth_signature="1PofnBExFc7pMB8b08fV34B5sTg%3D",oauth_timestamp="1323544645",oauth_nonce="516e0a1eadda4c24bfbcc4ca5e0cbedc",oauth_version="1.0"' -H 'Content-Type: application/json' -X POST -d '{"recipients":{"values": [{"person":{"_path": "/people/~"}}]}, "subject": "JSON POST test", "body": "You rule"}' http://api.linkedin.com/v1/people/~/shares
When I post this I get this result
About to connect() to api.linkedin.com port 80 (#0)
* Trying 216.52.242.83... connected
* Connected to api.linkedin.com (216.52.242.83) port 80 (#0)
> POST /v1/people/ICshCGDT79/shares HTTP/1.1
> User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
> Host: api.linkedin.com
> Accept: */*
> Authorization: OAuth oauth_consumer_key="V9uKPtEEB7JxQvYzbR2DuDfLe4qXjASgP5UhPJp8k13CYgSnN3BRFILxUJ6ApuCH",oauth_token="d6af5cda-b46e-4c38-b566-11428a9584fc",oauth_signature_method="HMAC-SHA1",oauth_signature="1PofnBExFc7pMB8b08fV34B5sTg%3D",oauth_timestamp="1323544645",oauth_nonce="516e0a1eadda4c24bfbcc4ca5e0cbedc",oauth_version="1.0"
> Content-Type: application/json
> Content-Length: 120
>
< HTTP/1.1 401 Unauthorized
< Server: Apache-Coyote/1.1
< Date: Sat, 10 Dec 2011 19:17:14 GMT
< Vary: *
< x-li-format: xml
< Content-Type: text/xml;charset=UTF-8
< Content-Length: 393
<
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<error>
<status>401</status>
<timestamp>1323544635447</timestamp>
<request-id>9UR4KT2T5Z</request-id>
<error-code>0</error-code>
<message>[unauthorized]. OAU:V9uKPtEEB7JxQvYzbR2DuDfLe4qXjASgP5UhPJp8k13CYgSnN3BRFILxUJ6ApuCH|d6af5cda-b46e-4c38-b566-11428a9584fc|*01|*01:1323544645:1PofnBExFc7pMB8b08fV34B5sTg=</message>
</error>
* Connection #0 to host api.linkedin.com left intact
* Closing connection #0
Anyone can tell me where am I wrong?
Thanks,