I need a help in understanding and getting clarity with regards to multipart file upload in Quarkus (v 2.11.3.Final)
In our project, we are using mutipart file upload and the request form looks like the following:
import org.jboss.resteasy.annotations.providers.multipart.PartType;
import javax.validation.Valid;
import javax.validation.constraints.NotNull;
import javax.ws.rs.FormParam;
import javax.ws.rs.core.MediaType;
import java.io.InputStream;
public class MultipartBody {
#NotNull(message = "DOCUMENT_METADATA_MISSING")
#FormParam("documentRequest")
#PartType(MediaType.APPLICATION_JSON)
#Valid
public DocumentForm document;
#NotNull(message = "DOCUMENT_CONTENT_MISSING")
#FormParam("content")
#PartType(MediaType.WILDCARD)
public InputStream file;
}
And then this stream is transferred to an upstream system using rest client like below
#Path("")
#RegisterRestClient
#Produces(MediaType.APPLICATION_JSON)
#RegisterProvider(LoggingFilter.class)
#ApplicationScoped
#Retry(maxDuration = 180000, delay = 100)
public interface IngestionClient {
#Consumes(MediaType.APPLICATION_OCTET_STREAM)
#Path("/upload/{batchId}/0")
#POST
UploadResponse uploadDocument(
#HeaderParam("X-File-Name") String documentName,
#HeaderParam("X-File-Type") String documentMimeType, #PathParam("batchId") String batchId,
InputStream content);
}
Following is the configuration done in the properties file
quarkus.http.limits.max-body-size=2048M
quarkus.http.body.preallocate-body-buffer=false
quarkus.http.body.merge-form-attributes=false
quarkus.http.body.handle-file-uploads=false
quarkus.http.limits.max-chunk-size=10M
In between the file is copied over to BufferedInputStream to detect the mime type before the transfer using the rest client is done. When I check the logs in debug mode, following is the sample observed.
2022-09-01 16:29:54,665 DEBUG [org.apa.htt.imp.con.PoolingHttpClientConnectionManager] (executor-thread-0) Connection request: [route: {}->http://127.0.0.2:8111][total available: 0; route allocated: 0 of 50; total allocated: 0 of 50]
2022-09-01 16:29:54,666 DEBUG [org.apa.htt.imp.con.PoolingHttpClientConnectionManager] (executor-thread-0) Connection leased: [id: 23][route: {}->http://127.0.0.2:8111][total available: 0; route allocated: 1 of 50; total allocated: 1 of 50]
2022-09-01 16:29:54,666 DEBUG [org.apa.htt.imp.exe.MainClientExec] (executor-thread-0) Opening connection {}->http://localhost:8080
2022-09-01 16:29:54,673 DEBUG [org.apa.htt.imp.con.DefaultHttpClientConnectionOperator] (executor-thread-0) Connecting to localhost/127.0.0.2:8111
2022-09-01 16:29:54,674 DEBUG [org.apa.htt.imp.con.DefaultHttpClientConnectionOperator] (executor-thread-0) Connection established
2022-09-01 16:29:54,674 DEBUG [org.apa.htt.imp.con.DefaultManagedHttpClientConnection] (executor-thread-0) http-outgoing-23: set socket timeout to 180000
2022-09-01 16:29:54,674 DEBUG [org.apa.htt.imp.exe.MainClientExec] (executor-thread-0) Executing request POST /upload/000000-00000/0 HTTP/1.1
2022-09-01 16:29:54,674 DEBUG [org.apa.htt.imp.exe.MainClientExec] (executor-thread-0) Target auth state: UNCHALLENGED
2022-09-01 16:29:54,674 DEBUG [org.apa.htt.imp.exe.MainClientExec] (executor-thread-0) Proxy auth state: UNCHALLENGED
2022-09-01 16:29:54,674 DEBUG [org.apa.htt.headers] (executor-thread-0) http-outgoing-23 >> POST /upload/000000--00000/0 HTTP/1.1
2022-09-01 16:29:54,674 DEBUG [org.apa.htt.headers] (executor-thread-0) http-outgoing-23 >> Accept: application/json
2022-09-01 16:29:54,674 DEBUG [org.apa.htt.headers] (executor-thread-0) http-outgoing-23 >> Content-Type: application/octet-stream
2022-09-01 16:29:54,675 DEBUG [org.apa.htt.headers] (executor-thread-0) http-outgoing-23 >> Content-Length: 44591990
2022-09-01 16:29:54,675 DEBUG [org.apa.htt.headers] (executor-thread-0) http-outgoing-23 >> Host: 127.0.0.2:8111
2022-09-01 16:29:54,675 DEBUG [org.apa.htt.headers] (executor-thread-0) http-outgoing-23 >> Connection: Keep-Alive
2022-09-01 16:29:54,676 DEBUG [org.apa.htt.headers] (executor-thread-0) http-outgoing-23 >> User-Agent: Apache-HttpClient/4.5.13 (Java/11.0.16)
2022-09-01 16:29:54,688 DEBUG [org.apa.htt.wire] (executor-thread-0) http-outgoing-23 >> "POST /upload/000000-00000/0 HTTP/1.1[\r][\n]"
2022-09-01 16:29:54,688 DEBUG [org.apa.htt.wire] (executor-thread-0) http-outgoing-23 >> "Accept: application/json[\r][\n]"
2022-09-01 16:29:54,688 DEBUG [org.apa.htt.wire] (executor-thread-0) http-outgoing-23 >> "Content-Type: application/octet-stream[\r][\n]"
2022-09-01 16:29:54,689 DEBUG [org.apa.htt.wire] (executor-thread-0) http-outgoing-23 >> "Content-Length: 44591990[\r][\n]"
2022-09-01 16:29:54,689 DEBUG [org.apa.htt.wire] (executor-thread-0) http-outgoing-23 >> "Host: 127.0.0.2:8111[\r][\n]"
2022-09-01 16:29:54,689 DEBUG [org.apa.htt.wire] (executor-thread-0) http-outgoing-23 >> "Connection: Keep-Alive[\r][\n]"
2022-09-01 16:29:54,689 DEBUG [org.apa.htt.wire] (executor-thread-0) http-outgoing-23 >> "User-Agent: Apache-HttpClient/4.5.13 (Java/11.0.16)[\r][\n]"
2022-09-01 16:29:54,690 DEBUG [org.apa.htt.wire] (executor-thread-0) http-outgoing-23 >> "[\r][\n]"
2022-09-01 16:29:54,690 DEBUG [org.apa.htt.wire] (executor-thread-0) http-outgoing-23 >> "%PDF-1.7[\n]"
2022-09-01 16:29:54,690 DEBUG [org.apa.htt.wire] (executor-thread-0) http-outgoing-23 >> "%[0xe2][0xe3][0xcf][0xd3][\n]"
2022-09-01 16:29:54,690 DEBUG [org.apa.htt.wire] (executor-thread-0) http-outgoing-23 >> "2 0 obj[\n]"
2022-09-01 16:29:54,690 DEBUG [org.apa.htt.wire] (executor-thread-0) http-outgoing-23 >> "<</Length 99110/Filter/FlateDecode>>[\n]"
2022-09-01 16:29:54,691 DEBUG [org.apa.htt.wire] (executor-thread-0) http-outgoing-23 >> "stream[\n]"
The interesting part in the request being sent is the content-length. Now my question is that if Quarkus does not store the file locally based on the above configuration then how is it able to measure the content length upfront before starting the upload to upstream.
One additional aspect of the test performed is that this was tried using postman where the content length is automatically calculated at the time of performing the upload and the file size was 40MB approx. Any direction with regards to this question will help. Do I need to implement something differently?
The reason for asking the question is that we have a gateway behind which the application resides and this gateway has a limitation on the time within which the application has to respond back otherwise the connection is closed. So far the assumption was that the file is being streamed but on doing this investigation my curiosity arose...
I am using spectron to simulate a test on Slack as follows:
1) Open the application.
2) Type in a valid domain.
3) Click Continue.
I am using the following script for that purpose:
JavaScript:
var Application = require('spectron').Application
var assert = require('assert')
var app = new Application({
path: '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'
})
app.start().then(function() {
// Check if the window is visible
return app.browserWindow.isVisible()
}).then(function(isVisible) {
// Verify the window is visible
assert.equal(isVisible, true)
}).then(function() {
// Get the window's title
return app.client.getTitle()
}).then(function(title) {
// Verify the window's title
// assert.equal(title, 'My App')
}).catch(function(error) {
// Log any failures
console.error('Test failed', error.message)
}).then(function() {
// Stop the application
// return app.stop()
})
console.log('Before setTimeout');
setTimeout(function() {
console.log('Inside setTimeout');
return app.client.waitUntilWindowLoaded()
.windowHandles().then(function(session) {
// Need to return the promise back, if promise is
// it would wait for the state or else app will exit.
console.log('Before switchTab, click & keys');
app.client.switchTab(session.value[1]).click('#domain').keys('testing').click("#submit_team_domain")
.catch(function(error) {
console.error('error message->', error.message);
});
console.log('After switchTab, click & keys');
});
}, 5000);
Java:
Set<String> windowHandles = webDriver.getWindowHandles();
Iterator<String> iterator = windowHandles.iterator();
while (iterator.hasNext()) {
String windowHandle = iterator.next();
if (!windowHandle.equalsIgnoreCase(webDriver.getWindowHandle()))
webDriver.switchTo().window(windowHandle);
}
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
WebElement textBox = webDriver.findElement(By.id("domain"));
textBox.click();
textBox.sendKeys("testing");
WebElement button = webDriver.findElement(By.id("submit_team_domain"));
button = webDriver.findElement(By.id("submit_team_domain"));
button.click();
I am testing with and without a selenium server between a Mac and a Linux machine. The expected result should be that the page should proceed after clicking continue.
What is actually happening is that if a valid domain is provided then it gets stuck and never proceeds. The only exception to this is when we use Spectron to test without a Selenium server on a Linux machine, which itself fails about 20% of the time.
If we provide an invalid domain in any of the cases, it simply proceeds.
Following are the logs when using Spectron without a Selenium Server:
requestOptions -> {"path":"/session/:sessionId/element"}
data -> {"using":"id","value":"domain"}
_ref -> {"body":{"sessionId":"bc8bc9abd7a9827258cee72de70d835f","status":0,"value":{"ELEMENT":"0.006260871409230262-1"}},"response":{"statusCode":200,"body":{"sessionId":"bc8bc9abd7a9827258cee72de70d835f","status":0,"value":{"ELEMENT":"0.006260871409230262-1"}},"headers":{"content-length":"104","content-type":"application/json; charset=utf-8","connection":"close"},"request":{"uri":{"protocol":"http:","slashes":true,"auth":null,"host":"127.0.0.1:9515","port":"9515","hostname":"127.0.0.1","hash":null,"search":null,"query":null,"pathname":"/wd/hub/session/bc8bc9abd7a9827258cee72de70d835f/element","path":"/wd/hub/session/bc8bc9abd7a9827258cee72de70d835f/element","href":"http://127.0.0.1:9515/wd/hub/session/bc8bc9abd7a9827258cee72de70d835f/element"},"method":"POST","headers":{"Connection":"keep-alive","Accept":"application/json","User-Agent":"webdriverio/webdriverio/4.6.2","Content-Type":"application/json; charset=UTF-8","Content-Length":31}}}}
requestOptions -> {"path":"/session/:sessionId/element/0.006260871409230262-1/click","method":"POST"}
data -> {}
_ref -> {"body":{"sessionId":"bc8bc9abd7a9827258cee72de70d835f","status":0,"value":null},"response":{"statusCode":200,"body":{"sessionId":"bc8bc9abd7a9827258cee72de70d835f","status":0,"value":null},"headers":{"content-length":"72","content-type":"application/json; charset=utf-8","connection":"close"},"request":{"uri":{"protocol":"http:","slashes":true,"auth":null,"host":"127.0.0.1:9515","port":"9515","hostname":"127.0.0.1","hash":null,"search":null,"query":null,"pathname":"/wd/hub/session/bc8bc9abd7a9827258cee72de70d835f/element/0.006260871409230262-1/click","path":"/wd/hub/session/bc8bc9abd7a9827258cee72de70d835f/element/0.006260871409230262-1/click","href":"http://127.0.0.1:9515/wd/hub/session/bc8bc9abd7a9827258cee72de70d835f/element/0.006260871409230262-1/click"},"method":"POST","headers":{"Connection":"keep-alive","Accept":"application/json","User-Agent":"webdriverio/webdriverio/4.6.2","content-type":"application/json","content-length":2}}}}
WARNING: the "keys" command will be depcrecated soon. Please use a different command in order to avoid failures in your test after updating WebdriverIO.
requestOptions -> {"path":"/session/:sessionId/keys"}
data -> {"value":["c","o","s","m","o","t","e","s","t","i","n","g"]}
_ref -> {"body":{"sessionId":"bc8bc9abd7a9827258cee72de70d835f","status":0,"value":null},"response":{"statusCode":200,"body":{"sessionId":"bc8bc9abd7a9827258cee72de70d835f","status":0,"value":null},"headers":{"content-length":"72","content-type":"application/json; charset=utf-8","connection":"close"},"request":{"uri":{"protocol":"http:","slashes":true,"auth":null,"host":"127.0.0.1:9515","port":"9515","hostname":"127.0.0.1","hash":null,"search":null,"query":null,"pathname":"/wd/hub/session/bc8bc9abd7a9827258cee72de70d835f/keys","path":"/wd/hub/session/bc8bc9abd7a9827258cee72de70d835f/keys","href":"http://127.0.0.1:9515/wd/hub/session/bc8bc9abd7a9827258cee72de70d835f/keys"},"method":"POST","headers":{"Connection":"keep-alive","Accept":"application/json","User-Agent":"webdriverio/webdriverio/4.6.2","Content-Type":"application/json; charset=UTF-8","Content-Length":59}}}}
requestOptions -> {"path":"/session/:sessionId/element"}
data -> {"using":"id","value":"submit_team_domain"}
_ref -> {"body":{"sessionId":"bc8bc9abd7a9827258cee72de70d835f","status":0,"value":{"ELEMENT":"0.006260871409230262-2"}},"response":{"statusCode":200,"body":{"sessionId":"bc8bc9abd7a9827258cee72de70d835f","status":0,"value":{"ELEMENT":"0.006260871409230262-2"}},"headers":{"content-length":"104","content-type":"application/json; charset=utf-8","connection":"close"},"request":{"uri":{"protocol":"http:","slashes":true,"auth":null,"host":"127.0.0.1:9515","port":"9515","hostname":"127.0.0.1","hash":null,"search":null,"query":null,"pathname":"/wd/hub/session/bc8bc9abd7a9827258cee72de70d835f/element","path":"/wd/hub/session/bc8bc9abd7a9827258cee72de70d835f/element","href":"http://127.0.0.1:9515/wd/hub/session/bc8bc9abd7a9827258cee72de70d835f/element"},"method":"POST","headers":{"Connection":"keep-alive","Accept":"application/json","User-Agent":"webdriverio/webdriverio/4.6.2","Content-Type":"application/json; charset=UTF-8","Content-Length":43}}}}
requestOptions -> {"path":"/session/:sessionId/element/0.006260871409230262-2/click","method":"POST"}
data -> {}
_ref -> {"body":{"sessionId":"bc8bc9abd7a9827258cee72de70d835f","status":0,"value":null},"response":{"statusCode":200,"body":{"sessionId":"bc8bc9abd7a9827258cee72de70d835f","status":0,"value":null},"headers":{"content-length":"72","content-type":"application/json; charset=utf-8","connection":"close"},"request":{"uri":{"protocol":"http:","slashes":true,"auth":null,"host":"127.0.0.1:9515","port":"9515","hostname":"127.0.0.1","hash":null,"search":null,"query":null,"pathname":"/wd/hub/session/bc8bc9abd7a9827258cee72de70d835f/element/0.006260871409230262-2/click","path":"/wd/hub/session/bc8bc9abd7a9827258cee72de70d835f/element/0.006260871409230262-2/click","href":"http://127.0.0.1:9515/wd/hub/session/bc8bc9abd7a9827258cee72de70d835f/element/0.006260871409230262-2/click"},"method":"POST","headers":{"Connection":"keep-alive","Accept":"application/json","User-Agent":"webdriverio/webdriverio/4.6.2","content-type":"application/json","content-length":2}}}}
Following are the logs when using Java Language Bindings with a Selenium Server:
15:11:43.884 INFO - Found handler: org.openqa.selenium.remote.server.ServicedSession#5d54be3a
15:11:43.884 INFO - Handler thread for session 976706a5ace12e22ec54d9848f21813c (chrome): Executing POST on /session/976706a5ace12e22ec54d9848f21813c/element (handler: ServicedSession)
15:11:43.885 INFO - To upstream: {"using":"id","value":"domain"}
15:11:43.885 DEBUG - sun.net.www.MessageHeader#55edb1c210 pairs: {POST /session/976706a5ace12e22ec54d9848f21813c/element HTTP/1.1: null}{User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_151)}{Accept-Encoding: gzip,deflate}{Content-Type: application/json; charset=utf-8}{Connection: close}{Cache-Control: no-cache}{Pragma: no-cache}{Host: localhost:25846}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Content-Length: 31}
15:11:43.908 DEBUG - sun.net.www.MessageHeader#61b2a16b4 pairs: {null: HTTP/1.1 200 OK}{Content-Length: 102}{Content-Type: application/json; charset=utf-8}{Connection: close}
15:11:43.908 INFO - To downstream: {"sessionId":"976706a5ace12e22ec54d9848f21813c","status":0,"value":{"ELEMENT":"0.8306467617329649-1"}}
15:11:43.917 INFO - Found handler: org.openqa.selenium.remote.server.ServicedSession#5d54be3a
15:11:43.917 INFO - Handler thread for session 976706a5ace12e22ec54d9848f21813c (chrome): Executing POST on /session/976706a5ace12e22ec54d9848f21813c/element/0.8306467617329649-1/click (handler: ServicedSession)
15:11:43.918 INFO - To upstream: {"id":"0.8306467617329649-1"}
15:11:43.919 DEBUG - sun.net.www.MessageHeader#3cbdbdb110 pairs: {POST /session/976706a5ace12e22ec54d9848f21813c/element/0.8306467617329649-1/click HTTP/1.1: null}{User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_151)}{Accept-Encoding: gzip,deflate}{Content-Type: application/json; charset=utf-8}{Connection: close}{Cache-Control: no-cache}{Pragma: no-cache}{Host: localhost:25846}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Content-Length: 29}
15:11:43.986 DEBUG - sun.net.www.MessageHeader#32f13174 pairs: {null: HTTP/1.1 200 OK}{Content-Length: 72}{Content-Type: application/json; charset=utf-8}{Connection: close}
15:11:43.986 INFO - To downstream: {"sessionId":"976706a5ace12e22ec54d9848f21813c","status":0,"value":null}
15:11:43.991 INFO - Found handler: org.openqa.selenium.remote.server.ServicedSession#5d54be3a
15:11:43.991 INFO - Handler thread for session 976706a5ace12e22ec54d9848f21813c (chrome): Executing POST on /session/976706a5ace12e22ec54d9848f21813c/element (handler: ServicedSession)
15:11:43.992 INFO - To upstream: {"using":"id","value":"domain"}
15:11:43.992 DEBUG - sun.net.www.MessageHeader#3bf4c11110 pairs: {POST /session/976706a5ace12e22ec54d9848f21813c/element HTTP/1.1: null}{User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_151)}{Accept-Encoding: gzip,deflate}{Content-Type: application/json; charset=utf-8}{Connection: close}{Cache-Control: no-cache}{Pragma: no-cache}{Host: localhost:25846}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Content-Length: 31}
15:11:43.998 DEBUG - sun.net.www.MessageHeader#4f3986fd4 pairs: {null: HTTP/1.1 200 OK}{Content-Length: 102}{Content-Type: application/json; charset=utf-8}{Connection: close}
15:11:43.998 INFO - To downstream: {"sessionId":"976706a5ace12e22ec54d9848f21813c","status":0,"value":{"ELEMENT":"0.8306467617329649-1"}}
15:11:44.002 INFO - Found handler: org.openqa.selenium.remote.server.ServicedSession#5d54be3a
15:11:44.002 INFO - Handler thread for session 976706a5ace12e22ec54d9848f21813c (chrome): Executing POST on /session/976706a5ace12e22ec54d9848f21813c/element/0.8306467617329649-1/value (handler: ServicedSession)
15:11:44.003 INFO - To upstream: {"id":"0.8306467617329649-1","value":["cosmotesting"]}
15:11:44.003 DEBUG - sun.net.www.MessageHeader#1eb8ea9e10 pairs: {POST /session/976706a5ace12e22ec54d9848f21813c/element/0.8306467617329649-1/value HTTP/1.1: null}{User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_151)}{Accept-Encoding: gzip,deflate}{Content-Type: application/json; charset=utf-8}{Connection: close}{Cache-Control: no-cache}{Pragma: no-cache}{Host: localhost:25846}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Content-Length: 54}
15:11:44.032 DEBUG - sun.net.www.MessageHeader#65f4e5f54 pairs: {null: HTTP/1.1 200 OK}{Content-Length: 72}{Content-Type: application/json; charset=utf-8}{Connection: close}
15:11:44.032 INFO - To downstream: {"sessionId":"976706a5ace12e22ec54d9848f21813c","status":0,"value":null}
15:11:44.036 INFO - Found handler: org.openqa.selenium.remote.server.ServicedSession#5d54be3a
15:11:44.036 INFO - Handler thread for session 976706a5ace12e22ec54d9848f21813c (chrome): Executing POST on /session/976706a5ace12e22ec54d9848f21813c/element (handler: ServicedSession)
15:11:44.036 INFO - To upstream: {"using":"id","value":"submit_team_domain"}
15:11:44.036 DEBUG - sun.net.www.MessageHeader#78955c1310 pairs: {POST /session/976706a5ace12e22ec54d9848f21813c/element HTTP/1.1: null}{User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_151)}{Accept-Encoding: gzip,deflate}{Content-Type: application/json; charset=utf-8}{Connection: close}{Cache-Control: no-cache}{Pragma: no-cache}{Host: localhost:25846}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Content-Length: 43}
15:11:44.050 DEBUG - sun.net.www.MessageHeader#7b1823924 pairs: {null: HTTP/1.1 200 OK}{Content-Length: 102}{Content-Type: application/json; charset=utf-8}{Connection: close}
15:11:44.050 INFO - To downstream: {"sessionId":"976706a5ace12e22ec54d9848f21813c","status":0,"value":{"ELEMENT":"0.8306467617329649-2"}}
15:11:44.054 INFO - Found handler: org.openqa.selenium.remote.server.ServicedSession#5d54be3a
15:11:44.054 INFO - Handler thread for session 976706a5ace12e22ec54d9848f21813c (chrome): Executing POST on /session/976706a5ace12e22ec54d9848f21813c/element/0.8306467617329649-2/click (handler: ServicedSession)
15:11:44.055 INFO - To upstream: {"id":"0.8306467617329649-2"}
15:11:44.055 DEBUG - sun.net.www.MessageHeader#22adda7510 pairs: {POST /session/976706a5ace12e22ec54d9848f21813c/element/0.8306467617329649-2/click HTTP/1.1: null}{User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_151)}{Accept-Encoding: gzip,deflate}{Content-Type: application/json; charset=utf-8}{Connection: close}{Cache-Control: no-cache}{Pragma: no-cache}{Host: localhost:25846}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Content-Length: 29}
I am using Command Line Tool for uploading files from Amazon S3 to CloudSearch for Indexing using the command-
cs-import-documents -d searchdev3 --source s3://mybucket/html
It is giving error-
Connecting to CloudSearch in region [us-east-1]
AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S
3; Status Code: 403; Error Code: AccessDenied; Request ID: 506FD29F73XXXXXX)
How can I track where the problem is occurring?
UPDATE
When run with --verbose I received below output-
Looking up DocServiceEndpoint for domain searchdev3 using config service at null
Connecting to CloudSearch in region [us-east-1]
Sending to endpoint [https://cloudsearch.us-east-1.amazonaws.com/] for region [u
s-east-1]
Connection request: [route: {s}->https://cloudsearch.us-east-1.amazonaws.com][to
tal kept alive: 0; route allocated: 0 of 50; total allocated: 0 of 50]
Connection leased: [id: 0][route: {s}->https://cloudsearch.us-east-1.amazonaws.c
om][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]
Connecting to cloudsearch.us-east-1.amazonaws.com:443
CookieSpec selected: best-match
Auth cache not set in the context
Proxy auth state: UNCHALLENGED
Sending request: POST / HTTP/1.1
>> "POST / HTTP/1.1[\r][\n]"
>> "Host: cloudsearch.us-east-1.amazonaws.com[\r][\n]"
>> "Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXX/20151130/us-
east-1/cloudsearch/aws4_request, SignedHeaders=host;user-agent;x-amz-date, Signa
ture=e93cd0c22189084ff55c1084[\r][\n]"
>> "X-Amz-Date: 20151130T103603Z[\r][\n]"
>> "User-Agent: aws-internal/3[\r][\n]"
>> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]"
>> "Content-Length: 73[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "[\r][\n]"
>> POST / HTTP/1.1
>> Host: cloudsearch.us-east-1.amazonaws.com
>> Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXX/20151130/us-e
ast-1/cloudsearch/aws4_request, SignedHeaders=host;user-agent;x-amz-date, Signat
ure=e93cd0c22189084ff55c1084
>> X-Amz-Date: 20151130T103603Z
>> User-Agent: aws-internal/3
>> Content-Type: application/x-www-form-urlencoded; charset=utf-8
>> Content-Length: 73
>> Connection: Keep-Alive
>> "Action=DescribeDomains&Version=2013-01-01&DomainNames.member.1=searchdev3"
<< "HTTP/1.1 200 OK[\r][\n]"
<< "x-amzn-RequestId: 2907d340-974e-11e5-a6e5-33953f55803d[\r][\n]"
<< "Content-Type: text/xml[\r][\n]"
<< "Content-Length: 1635[\r][\n]"
<< "Date: Mon, 30 Nov 2015 10:36:05 GMT[\r][\n]"
<< "[\r][\n]"
Receiving response: HTTP/1.1 200 OK
<< HTTP/1.1 200 OK
<< x-amzn-RequestId: 2907d340-974e-11e5-a6e5-33953f55803d
<< Content-Type: text/xml
<< Content-Length: 1635
<< Date: Mon, 30 Nov 2015 10:36:05 GMT
<< "<"
<< "D"
<< "e"
<< "s"
<< "cribeDomainsResponse xmlns=""
<< "http://cloudsearch.amazonaws.com/doc/2013-01-01/">[\n]"
<< " <DescribeDomainsResult>[\n]"
<< " <DomainStatusList>[\n]"
<< " <member>[\n]"
<< " <Created>true</Created>[\n]"
<< " <RequiresIndexDocuments>false</RequiresIndexDocuments>[\n]"
<< " <SearchInstanceType>search.m1.small</SearchInstanceType>[\n]"
<< " <DomainId>613875336225/searchdev3</DomainId>[\n]"
<< " <DocService>[\n]"
<< " <Endpoint>doc-searchdev3-ytmgti5ijnobqprgg65e4yxrhe.us-east-1.clou
dsearch.amazonaws.com</Endpoint>[\n]"
<< " </DocService>[\n]"
<< " <SearchPartitionCount>1</SearchPartitionCount>[\n]"
<< " <Deleted>false</Deleted>[\n]"
<< " <SearchService>[\n]"
<< " <Endpoint>search-searchdev3-ytmgti5ijnobqprgg65e4yxrhe.us-east-1.c
loudsearch.amazonaws.com</Endpoint>[\n]"
<< " </SearchService>[\n]"
<< " <DomainName>searchdev3</DomainName>[\n]"
<< " <Processing>true</Processing>[\n]"
<< " <SearchInstanceCount>1</SearchInstanceCount>[\n]"
<< " <ARN>arn:aws:cloudsearch:us-east-1:613875336225:domain/searchdev3</A
RN>[\n]"
<< " <Limits>[\n]"
<< " <AvailableInstanceTypes>[\n]"
<< " <member>search.m1.small</member>[\n]"
<< " <member>search.m3.medium</member>[\n]"
<< " <member>search.m3.large</member>[\n]"
<< " <member>search.m3.xlarge</member>[\n]"
<< " <member>search.m3.2xlarge</member>[\n]"
<< " </AvailableInstanceTypes>[\n]"
<< " <MaximumReplicationCount>5</MaximumReplicationCount>[\n]"
<< " <MaximumPartitionCount>10</MaximumPartitionCount>[\n]"
<< " </Limits>[\n]"
<< " </member>[\n]"
<< " </DomainStatusList>[\n]"
<< " </DescribeDomainsResult>[\n]"
<< " <ResponseMetadata>[\n]"
<< " <RequestId>2907d340-974e-11e5-a6e5-33953f55803d</RequestId>[\n]"
<< " </ResponseMetadata>[\n]"
<< "</DescribeDomainsResponse>[\n]"
Connection [id: 0][route: {s}->https://cloudsearch.us-east-1.amazonaws.com] can
be kept alive indefinitely
Connection released: [id: 0][route: {s}->https://cloudsearch.us-east-1.amazonaws
.com][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]
Creating S3 source reader
Connection request: [route: {s}->https://mybucket.s3.amazonaws.com][total
kept alive: 0; route allocated: 0 of 50; total allocated: 0 of 50]
Connection leased: [id: 1][route: {s}->https://mybucket.s3.amazonaws.com]
[total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]
Connecting to mybucket.s3.amazonaws.com:443
CookieSpec selected: best-match
Auth cache not set in the context
Proxy auth state: UNCHALLENGED
Sending request: GET /?prefix=html HTTP/1.1
>> "GET /?prefix=html HTTP/1.1[\r][\n]"
>> "Host: mybucket.s3.amazonaws.com[\r][\n]"
>> "Authorization: AWS XXXXXXXX:eA+KR8i3SpRQdTS/pxeBeci7iOw=[\r][\n]
"
>> "User-Agent: aws-internal/3[\r][\n]"
>> "Date: Mon, 30 Nov 2015 10:36:05 +00:00[\r][\n]"
>> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "[\r][\n]"
>> GET /?prefix=html HTTP/1.1
>> Host: mybucket.s3.amazonaws.com
>> Authorization: AWS XXXXXXXX:eA+KR8i3SpRQdTS/pxeBeci7iOw=
>> User-Agent: aws-internal/3
>> Date: Mon, 30 Nov 2015 10:36:05 +00:00
>> Content-Type: application/x-www-form-urlencoded; charset=utf-8
>> Connection: Keep-Alive
<< "HTTP/1.1 403 Forbidden[\r][\n]"
<< "x-amz-bucket-region: us-east-1[\r][\n]"
<< "x-amz-request-id: DA6567BEA36C46F9[\r][\n]"
<< "x-amz-id-2: kCC6NxI43AJi3G+YB0Dxgoc6ZrCGh/V3Wc4nhSnxk+mxYpdH50am3BBgzXfkLHrj
[\r][\n]"
<< "Content-Type: application/xml[\r][\n]"
<< "Transfer-Encoding: chunked[\r][\n]"
<< "Date: Mon, 30 Nov 2015 10:36:06 GMT[\r][\n]"
<< "Server: AmazonS3[\r][\n]"
<< "[\r][\n]"
Receiving response: HTTP/1.1 403 Forbidden
<< HTTP/1.1 403 Forbidden
<< x-amz-bucket-region: us-east-1
<< x-amz-request-id: DA6567BEA36C46F9
<< x-amz-id-2: kCC6NxI43AJi3G+YB0Dxgoc6ZrCGh/V3Wc4nhSnxk+mxYpdH50am3BBgzXfkLHrj
<< Content-Type: application/xml
<< Transfer-Encoding: chunked
<< Date: Mon, 30 Nov 2015 10:36:06 GMT
<< Server: AmazonS3
<< "117[\r][\n]"
<< "<?xml version="1.0" encoding="UTF-8"?>[\n]"
<< "<Error><Code>AccessDenied</Code><Message>AWS authentication requires a valid
Date or x-amz-date header</Message><RequestId>DA6567BEA36C46F9</RequestId><Host
Id>kCC6NxI43AJi3G+YB0Dxgoc6ZrCGh/V3Wc4nhSnxk+mxYpdH50am3BBgzXfkLHrj</HostId></Er
ror>"
<< "[\r][\n]"
<< "0[\r][\n]"
<< "[\r][\n]"
Connection [id: 1][route: {s}->https://mybucket.s3.amazonaws.com] can be
kept alive indefinitely
Connection released: [id: 1][route: {s}->https://mybucket.s3.amazonaws.co
m][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]
AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S
3; Status Code: 403; Error Code: AccessDenied; Request ID: DA6567BEA36C46F9)
Error Type: Client
Error Code: AccessDenied
HTTP Code: 403
Request ID: DA6567BEA36C46F9
[Ljava.lang.StackTraceElement;#12a7737
it seems you are using different version of java.
Please try to degrade java version. hope it will work. i had similar kind of issue before after degrading it to lower version fixed my problem.
I can send notification without any error but I don't get anything on my device. I tried a lot of solutions and I don't know what else I can do.
The certificat and token works in PHP...
Configuration : Java 1.8, Tomcat 7, Javapns 2.2
Thanks
Code
BasicConfigurator.configure();
URL res = getClass().getClassLoader().getResource("KeyNotification/certificat.p12");
File f = new File(res.getFile());
List<PushedNotification> notifications = Push.combined("message", 10, "", f.getAbsolutePath(), "password", false, token);
for (PushedNotification notification : notifications) {
if (notification.isSuccessful()) {
System.out.println("Push notification sent successfully to: " + notification.getDevice().getToken());
} else {
String invalidToken = notification.getDevice().getToken();
Exception theProblem = notification.getException();
theProblem.printStackTrace();
/* If the problem was an error-response packet returned by Apple, get it */
ResponsePacket theErrorResponse = notification.getResponse();
if (theErrorResponse != null) {
System.out.println(theErrorResponse.getMessage());
}
}
}
Log
0 [http-bio-8080-exec-21] DEBUG javapns.communication.ConnectionToAppleServer - Creating SSLSocketFactory
3 [http-bio-8080-exec-21] DEBUG javapns.communication.ConnectionToAppleServer - Creating SSLSocket to feedback.sandbox.push.apple.com:2196
1189 [http-bio-8080-exec-21] DEBUG javapns.feedback.FeedbackServiceManager - Found: [0]
1194 [http-bio-8080-exec-21] DEBUG javapns.notification.Payload - Adding alert [message]
1194 [http-bio-8080-exec-21] DEBUG javapns.notification.Payload - Adding badge [10]
1194 [http-bio-8080-exec-21] DEBUG javapns.notification.Payload - Adding sound []
1199 [http-bio-8080-exec-21] DEBUG javapns.communication.ConnectionToAppleServer - Creating SSLSocketFactory
1201 [http-bio-8080-exec-21] DEBUG javapns.communication.ConnectionToAppleServer - Creating SSLSocket to gateway.sandbox.push.apple.com:2195
1444 [http-bio-8080-exec-21] DEBUG javapns.notification.PushNotificationManager - Initialized Connection to Host: [gateway.sandbox.push.apple.com] Port: [2195]: 1e816c94[SSL_NULL_WITH_NULL_NULL: Socket[addr=gateway.sandbox.push.apple.com/17.110.227.35,port=2195,localport=63887]]
1447 [http-bio-8080-exec-21] DEBUG javapns.notification.PushNotificationManager - Building Raw message from deviceToken and payload
1447 [http-bio-8080-exec-21] DEBUG javapns.notification.PushNotificationManager - Built raw message ID 1 of total length 98
1447 [http-bio-8080-exec-21] DEBUG javapns.notification.PushNotificationManager - Attempting to send notification: {"aps":{"badge":10,"alert":"message","sound":""}}
1447 [http-bio-8080-exec-21] DEBUG javapns.notification.PushNotificationManager - to device: 1d8e0488958b92297923884508bf3714835f3581101ede9d3155ded0d8241c3e
2315 [http-bio-8080-exec-21] DEBUG javapns.notification.PushNotificationManager - Flushing
2315 [http-bio-8080-exec-21] DEBUG javapns.notification.PushNotificationManager - At this point, the entire 98-bytes message has been streamed out successfully through the SSL connection
2315 [http-bio-8080-exec-21] DEBUG javapns.notification.PushNotificationManager - Notification sent on first attempt
2315 [http-bio-8080-exec-21] DEBUG javapns.notification.PushNotificationManager - Reading responses
7321 [http-bio-8080-exec-21] DEBUG javapns.notification.PushNotificationManager - Closing connection
Push notification sent successfully to: 1d8e0488958b92297923884508bf3714835f3581101ede9d3155ded0d8241c3e
I solved my problem , the certificate was not compatible !
The command line to create the certificate (aps_development.pem is downloaded from Apple, myprivatekey.p12 is exported from my app private key in the "Keychain Access") :
openssl x509 -in aps_development.cer -inform DER -out aps_development.pem -outform PEM
openssl pkcs12 -nocerts -in myprivatekey.p12 -out myprivatekey.pem
openssl pkcs12 -export -inkey myprivatekey.pem -in aps_development.pem -out cert_dev.p12
I am unable to setup my javamail session to send an email using my private mail service provider. I am using an auhenticated starttls session and am getting this error : 550 5.7.1 Client does not have permissions to send as this sender. Below is my mail session properties and debug trace :
mail.smtp.auth=true
mail.smtp.host=smtp.myprovider.com
mail.smtp.password=*****
mail.smtp.port=587
mail.smtp.starttls.enable=true
mail.smtp.user=****
javax.portlet.action[0] = sendMail
DEBUG: getProvider() returning javax.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Sun Microsystems, Inc]
DEBUG SMTP: useEhlo true, useAuth true
DEBUG SMTP: trying to connect to host "smtp.myprovider.com", port 587, isSSL false
220 smtp.myprovider.com Microsoft ESMTP MAIL Service ready at Tue, 2 Jun 2015 11:35:22 -0400
DEBUG SMTP: connected to host "smtp.myprovider.com", port: 587
EHLO sgmed001
250-smtp.myprovider.com Hello [xx.xx.xx.xx]
250-SIZE 52428800
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-STARTTLS
250-AUTH NTLM
250-8BITMIME
250-BINARYMIME
250 CHUNKING
DEBUG SMTP: Found extension "SIZE", arg "52428800"
DEBUG SMTP: Found extension "PIPELINING", arg ""
DEBUG SMTP: Found extension "DSN", arg ""
DEBUG SMTP: Found extension "ENHANCEDSTATUSCODES", arg ""
DEBUG SMTP: Found extension "STARTTLS", arg ""
DEBUG SMTP: Found extension "AUTH", arg "NTLM"
DEBUG SMTP: Found extension "8BITMIME", arg ""
DEBUG SMTP: Found extension "BINARYMIME", arg ""
DEBUG SMTP: Found extension "CHUNKING", arg ""
STARTTLS
220 2.0.0 SMTP server ready
EHLO sgmed001
250-smtp.myprovider.com Hello [xx.xx.xx.xx]
250-SIZE 52428800
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-AUTH NTLM LOGIN
250-8BITMIME
250-BINARYMIME
250 CHUNKING
DEBUG SMTP: Found extension "SIZE", arg "52428800"
DEBUG SMTP: Found extension "PIPELINING", arg ""
DEBUG SMTP: Found extension "DSN", arg ""
DEBUG SMTP: Found extension "ENHANCEDSTATUSCODES", arg ""
DEBUG SMTP: Found extension "AUTH", arg "NTLM LOGIN"
DEBUG SMTP: Found extension "8BITMIME", arg ""
DEBUG SMTP: Found extension "BINARYMIME", arg ""
DEBUG SMTP: Found extension "CHUNKING", arg ""
DEBUG SMTP: Attempt to authenticate
AUTH LOGIN
334 VXNlcm5hbWU6
****
334 UGFzc3dvcmQ6
***
235 2.7.0 Authentication successful
DEBUG SMTP: use8bit false
MAIL FROM:<sender#xx.xx>
250 2.1.0 Sender OK
RCPT TO:<recipient#xx.xx>
250 2.1.5 Recipient OK
DEBUG SMTP: Verified Addresses
DEBUG SMTP: recipient#xx.xx
DATA
354 Start mail input; end with <CRLF>.<CRLF>
Date: Tue, 2 Jun 2015 11:35:23 -0400 (EDT)
From: sender#xx.xx
To: recipient#xx.xx
Message-ID: <53119439.11433259323689.JavaMail>
Subject: Request
MIME-Version: 1.0
Content-Type: text/html;charset="UTF-8"
Content-Transfer-Encoding: 7bit
X-Auto-Response-Suppress: AutoReply, DR, NDR, NRN, OOF, RN
qweqwe
.
550 5.7.1 Client does not have permissions to send as this sender
com.sun.mail.smtp.SMTPSendFailedException: 550 5.7.1 Client does not have permissions to send as this sender
at com.sun.mail.smtp.SMTPTransport.issueSendCommand(SMTPTransport.java:1388)
at com.sun.mail.smtp.SMTPTransport.finishData(SMTPTransport.java:1215)
at com.sun.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:586)
at com.liferay.util.mail.MailEngine._send(MailEngine.java:563)
at com.liferay.util.mail.MailEngine.send(MailEngine.java:350)
at com.liferay.util.mail.MailEngine.send(MailEngine.java:425)
at com.liferay.mail.messaging.MailMessageListener.doMailMessage(MailMessageListener.java:93)
at com.liferay.mail.messaging.MailMessageListener.doReceive(MailMessageListener.java:108)
at com.liferay.portal.kernel.messaging.BaseMessageListener.receive(BaseMessageListener.java:26)
at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:72)
at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:69)
at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:682)
at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:593)
at java.lang.Thread.run(Thread.java:745)
11:35:23,696 ERROR [liferay/mail-1][MailEngine:77] null
com.sun.mail.smtp.SMTPSendFailedException: 550 5.7.1 Client does not have permissions to send as this sender_ [Sanitized]
at com.sun.mail.smtp.SMTPTransport.issueSendCommand(SMTPTransport.java:1388)
at com.sun.mail.smtp.SMTPTransport.finishData(SMTPTransport.java:1215)
at com.sun.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:586)
at com.liferay.util.mail.MailEngine._send(MailEngine.java:563)
at com.liferay.util.mail.MailEngine.send(MailEngine.java:350)
at com.liferay.util.mail.MailEngine.send(MailEngine.java:425)
at com.liferay.mail.messaging.MailMessageListener.doMailMessage(MailMessageListener.java:93)
at com.liferay.mail.messaging.MailMessageListener.doReceive(MailMessageListener.java:108)
at com.liferay.portal.kernel.messaging.BaseMessageListener.receive(BaseMessageListener.java:26)
at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:72)
at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:69)
at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:682)
at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:593)
at java.lang.Thread.run(Thread.java:745)
Ok thanks to Steffen I was able to work it out, see comment above. In short the real problem was that I tried to send a mail using a sender outside my mail provider space. Using a mail address inside my provider's space made the email go and be sent away.