One of our WSO2 ESB (4.8.1) log shows this error intermittently, What actually causes this error?
TID: [0] [ESB] [2015-07-08 09:30:09,982] WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-6409: Connection time out while in state: REQUEST_DONE {org.apache.synapse.transport.passthru.TargetHandler}
TID: [0] [ESB] [2015-07-08 09:30:09
TID: [0] [ESB] [2015-07-09 12:08:10,018] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-3931 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [ESB] [2015-07-09 12:08:10,319] WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-7634: Connection time out while in state: REQUEST_DONE {org.apache.synapse.transport.passthru.TargetHandler}
TID: [0] [ESB] [2015-07-09 12:08:10,319] WARN {org.apache.synapse.FaultHandler} - ERROR_CODE : 101507 {org.apache.synapse.FaultHandler}
TID: [0] [ESB] [2015-07-09 12:08:10,319] WARN {org.apache.synapse.FaultHandler} - ERROR_MESSAGE : Error in Sender {org.apache.synapse.FaultHandler}
TID: [0] [ESB] [2015-07-09 12:08:10,320] WARN {org.apache.synapse.FaultHandler} - ERROR_DETAIL : Error in Sender {org.apache.synapse.FaultHandler}
TID: [0] [ESB] [2015-07-09 12:08:10,320] WARN {org.apache.synapse.FaultHandler} - ERROR_EXCEPTION : null {org.apache.synapse.FaultHandler}
TID: [0] [ESB] [2015-07-09 12:08:10,320] WARN {org.apache.synapse.FaultHandler} - FaultHandler : AnonymousEndpoint {org.apache.synapse.FaultHandler}
TID: [0] [ESB] [2015-07-09 12:08:10,320] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : AnonymousEndpoint will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [0] [ESB] [2015-07-09 12:08:10,320] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : AnonymousEndpoint - current suspend duration is : 30000ms - Next retry after : Thu Jul 09 12:08:40 IST 2015 {org.apache.synapse.endpoints.EndpointContext}
WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-3931
The above log says that the connection between the client and the ESB got timeout before ESB sends the response to the client. By default this timeout is 60 seconds (the socket timeout of http listener). So ESB is taking more than 60 seconds to send a response to the client. Reason might be because of your slow backend. You can increase this socket timeout of the passthrough http transport by adding http.socket.timeout=120000 to passthru-http.properties file in $ESB_HOME/repository/conf/ directory. Here socket timeout is set to 120seconds
WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-6409: Connection time out while in state: REQUEST_DONE
The above log says that the connection between the ESB and the backend got timeout before ESB gets the response from the backend. By default this timeout is 60 seconds (the socket timeout of http sender). So your backend is taking more than 60 seconds to respond. You can increase the socket timeout of the passthrough http transport by adding http.socket.timeout=120000 to passthru-http.properties file in $ESB_HOME/repository/conf directory. Here socket timeout is set to 120seconds
Please follow this troubleshoot guide to configure timeout values correctly.
Related
I am using Spring Boot 2.4.0 with Spring Boot Data JPA to connect to PostgreSQL and perform typical read and write operations with JPA based repositories. Since the database is also used by other services, I use the LISTEN/NOTIFY functionality (https://www.postgresql.org/docs/9.1/sql-listen.html) to be notified about changes from PostgeSQL. For this I use the driver com.impossibl.postgres.jdbc.PGDriver instead of the default driver and the following code to make Spring listen for changes to the database:
#Service
class PostgresChangeListener(
val dataSource: HikariDataSource,
#Qualifier("dbToPGReceiverQueue") val postgresQueue: RBlockingQueue<String>
) {
init {
listenToNotifyMessage()
}
final fun listenToNotifyMessage() {
val notificationListener = object:PGNotificationListener {
override fun notification(processId: Int, channelName: String, payload: String) {
log.info("Received change from PostgresQL: $processId, $channelName, $payload")
postgresQueue.add(payload)
}
override fun closed() {
log.debug("Connection to Postgres lost! Try to reconnect...")
listenToNotifyMessage()
}
}
try {
val connection = DataSourceUtils.getConnection(dataSource).unwrap(PGConnection::class.java)
connection.addNotificationListener(notificationListener)
connection.createStatement().use { statement -> statement.execute("LISTEN change_notifier;") }
} catch (e: SQLException) {
throw RuntimeException(e)
}
}
}
This is the Kotlin-like implementation of the listener discribed here: https://impossibl.github.io/pgjdbc-ng/docs/current/user-guide/#extensions-notifications
The listener works, however after one or more days I get the following error:
2021-03-03 06:33:00.185 WARN 1 --- [nio-8080-exec-8] o.s.b.a.jdbc.DataSourceHealthIndicator : DataSource health check failed
org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30001ms.
...
To find the problem, i enabled logging from Hikari as recommended on https://github.com/brettwooldridge/HikariCP/issues/1111#issuecomment-569552070. Here is the output of an excerpt of the logs:
2021-03-02 21:31:59.055 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
...
2021-03-02 21:31:59.055 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
2021-03-02 22:00:53.139 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#201ab69f: (connection has passed maxLifetime)
2021-03-02 22:00:53.162 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#f2ffd1ea
2021-03-02 22:00:54.709 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#3bb847ef: (connection has passed maxLifetime)
2021-03-02 22:00:54.730 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#fd5932d7
2021-03-02 22:00:59.110 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
2021-03-02 22:00:59.111 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Fill pool skipped, pool is at sufficient level.
2021-03-02 22:01:04.782 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#1d081266: (connection has passed maxLifetime)
2021-03-02 22:01:04.803 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#e0b396bc
2021-03-02 22:01:09.295 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#a2b0bd29: (connection has passed maxLifetime)
2021-03-02 22:01:09.313 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#ca9c8226
2021-03-02 22:01:10.075 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#ec8746aa: (connection has passed maxLifetime)
2021-03-02 22:01:10.093 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#aff2bfd8
2021-03-02 22:01:12.820 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#a7e0fc39: (connection has passed maxLifetime)
2021-03-02 22:01:12.840 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#d637554
2021-03-02 22:01:15.099 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#dadcba66: (connection has passed maxLifetime)
2021-03-02 22:01:15.119 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#e29805ef
2021-03-02 22:01:21.558 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#762f0753: (connection has passed maxLifetime)
2021-03-02 22:01:21.576 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#d5b8d008
2021-03-02 22:01:23.351 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#5e4721b0: (connection has passed maxLifetime)
2021-03-02 22:01:23.370 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#a8606b56
2021-03-02 22:01:29.111 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
2021-03-02 22:01:29.111 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Fill pool skipped, pool is at sufficient level.
2021-03-02 22:01:59.112 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
...
For me the log looks correct but after a while the active connections increase more and more...
...
2021-03-03 06:31:29.664 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=9, idle=1, waiting=0)
2021-03-03 06:31:48.687 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#4fa5ec41: (connection is dead)
2021-03-03 06:31:48.707 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#693052fe
2021-03-03 06:31:48.709 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Fill pool skipped, pool is at sufficient level.
2021-03-03 06:31:59.665 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=10, idle=0, waiting=1)
2021-03-03 06:31:59.665 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Fill pool skipped, pool is at sufficient level.
2021-03-03 06:32:20.199 DEBUG 1 --- [io-8080-exec-10] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Timeout failure stats (total=10, active=10, idle=0, waiting=2)
2021-03-03 06:32:20.208 WARN 1 --- [io-8080-exec-10] o.s.b.a.jdbc.DataSourceHealthIndicator : DataSource health check failed
org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
...
... until it comes to the described error message.
I wonder how I need to configure Hikari correctly or change my code to avoid the errors described? I hope you can help.
Not the same issue, but I had a similar problem. When my database was restarted, Hikari couldn't close the active listener connection, and the whole notification stopped working.
I found a possible solution for this. The reason why Hikari can't close the connection when it's dead because you are unwrapping the connection from the proxied Connection here:
DataSourceUtils.getConnection(dataSource).unwrap(PGConnection::class.java)
After this you are attaching a notificationListener to the PGConnection, so it remains alive.
First thing first to avoid hikaripool leaking you should seperate the 2 connection, and after initializing the listener you should close the hikariConnection.
private hikariConnection: Connection;
...
hikariConnection = DataSourceUtils.getConnection(dataSource)
val pgConnection: PGConnection = hikariConnection.unwrap(PGConnection::class.java)
... init the listener
hikariConnection.close()
And in the PGNotificationListener.closed() you have to reinitalize the listener, get a new Connection from the datasource. But beware, getting new connection while the Hikaripool filling it's pool(because the database outage was only a few seconds), can block each other. We solved it by getting the new connection on a dedicated new thread.
override fun closed() {
... get a new PGConnection, and start listening for the notifications
}
Sorry if it's not correctly answering your question, but it may help to some.
I am trying to do channel.basicReject() to requeue message based on some condition by creating an MethodInterceptor ConsumerAdvice and adding it to SMLC factor.setAdviceChain(new ConsumerAdvice()). I also have concurrentConsumer configuration which is set to 10. The moment my reject condition is met I issue basicReject command and it gets redelivered and processed by another consumer. During this redelivery process I get the below error,
2019-11-07 17:34:13.268 ERROR 29385 --- [ 127.0.0.1:5672] o.s.a.r.c.CachingConnectionFactory : Channel shutdown: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - unknown delivery tag 1, class-id=60, method-id=80)
2019-11-07 17:34:13.268 DEBUG 29385 --- [ool-2-thread-13] o.s.a.r.listener.BlockingQueueConsumer : Received shutdown signal for consumer tag=amq.ctag-HUaN71TZUqMfLDR7k6LwGQ
com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - unknown delivery tag 1, class-id=60, method-id=80)
at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:516)
at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:346)
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:178)
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:111)
at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:670)
at com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.java:48)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:597)
at java.lang.Thread.run(Thread.java:748)
My message is not getting lost but I am seeing a bunch of above errors and unable to understand why this is happening. If anyone has any clues please guide me.
Below are the trace logs,
2019-11-08 02:11:31.883 TRACE 8695 --- [askExecutor-138] o.s.a.r.c.CachingConnectionFactory : AMQChannel(amqp://guest#127.0.0.1:5672/,99) channel.getChannelNumber()
2019-11-08 02:11:31.883 INFO 8695 --- [askExecutor-138] c.g.s.w.consumer.advice.ArgumentUtils : Channel number before triggering redelivery : 99
2019-11-08 02:11:31.883 TRACE 8695 --- [askExecutor-138] o.s.a.r.c.CachingConnectionFactory : AMQChannel(amqp://guest#127.0.0.1:5672/,99) channel.basicReject([2, true])
2019-11-08 02:11:31.883 INFO 8695 --- [askExecutor-138] c.g.s.w.consumer.advice.ArgumentUtils : ==============================================================================
2019-11-08 02:11:31.883 INFO 8695 --- [askExecutor-138] c.g.s.w.consumer.advice.ConsumerAdvice : Requeue Message attempted, status : true
2019-11-08 02:11:31.884 TRACE 8695 --- [askExecutor-138] o.s.a.r.l.SimpleMessageListenerContainer : Waiting for message from consumer.
2019-11-08 02:11:31.884 TRACE 8695 --- [askExecutor-138] o.s.a.r.listener.BlockingQueueConsumer : Retrieving delivery for Consumer#7783912f: tags=[[amq.ctag-eY7LN-1pSXPX8FKRBgt-ug]], channel=Cached Rabbit Channel: AMQChannel(amqp://guest#127.0.0.1:5672/,99), conn: Proxy#37ffe4f3 Shared Rabbit Connection: SimpleConnection#708dfe10 [delegate=amqp://guest#127.0.0.1:5672/, localPort= 58638], acknowledgeMode=AUTO local queue size=0
2019-11-08 02:11:31.884 DEBUG 8695 --- [askExecutor-138] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it
com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - unknown delivery tag 1, class-id=60, method-id=80)
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.checkShutdown(BlockingQueueConsumer.java:436)
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.nextMessage(BlockingQueueConsumer.java:501)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:843)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:832)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:78)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1073)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - unknown delivery tag 1, class-id=60, method-id=80)
at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:516)
at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:346)
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:178)
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:111)
at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:670)
at com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.java:48)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:597)
... 1 common frames omitted
2019-11-08 02:11:31.884 ERROR 8695 --- [ 127.0.0.1:5672] o.s.a.r.c.CachingConnectionFactory : Channel shutdown: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - unknown delivery tag 2, class-id=60, method-id=90)
You need to show your code and configuration.
It seems like the SMLC has its default configuration to automatically acknowledge messages and this failure is because you already rejected it; why are you interacting with the channel directly?
You can simply throw an exception and the container will reject the message on your behalf.
I don't know if it will helpful for someone.
I had the same error because of invalid input data. But when I added next properties:
"rabbit.listener.acknowledgeMode": "MANUAL",
"rabbit.listener.defaultRequeueRejected": "true",
"rabbit.listener.prefetchCount": "1",
the problem stop to break my program but only had stopping my listener
This is the scenario:
- Three sequences: s1, s2, s3
- One variable stored in a registry (localEntry) sets to 1
- Get the seqName with a concat
- Try to call the sequence with get-property
What I've done until now:
1) Extract the value
<property name="getSequence" expression="get-property('registry','conf:repository/myVersion2.xml')" scope="default" type="OM"/>
<log level="custom">
<property name="TestVersion::localEntry" expression="$ctx:getSequence//localEntry"/>
</log>
2) Concat
<property name="seqName"
expression="concat('s', $ctx:getSequence//localEntry)"
scope="default"
type="STRING"/>
3) Call the sequence
<sequence key="get-property('seqName')"/>
4) Execute the test
5) Get error:
TID: [0] [ESB] [2015-07-08 10:27:36,325] INFO {org.apache.synapse.mediators.builtin.LogMediator} - TestVersion::info = setting property {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2015-07-08 10:27:36,327] INFO {org.apache.synapse.mediators.builtin.LogMediator} - TestVersion::localEntry = 1 {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2015-07-08 10:27:36,327] INFO {org.apache.synapse.mediators.builtin.LogMediator} - TestVersion::Sequence = s1 {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2015-07-08 10:27:36,327] INFO {org.apache.synapse.mediators.builtin.LogMediator} - TestVersion::info = filter1 {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2015-07-08 10:27:36,327] INFO {org.apache.synapse.mediators.builtin.LogMediator} - TestVersion::info = LOG_S1_TRUE {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2015-07-08 10:27:36,329] ERROR {org.apache.synapse.mediators.base.SequenceMediator} - Sequence named Value {name ='null', keyValue ='get-property('seqName')'} cannot be found {org.apache.synapse.mediators.base.SequenceMediator}
So the problem seems how the information stored in 'seqName' could be read by the sequence as the sequence name and not as a string.
I tried also with
<sequence key="{get-property('seqName')}"/>
But this doesn't work too...
Thank you in advance.
Claudio
Fixed!
<sequence xmlns:local="ws.apache.org/ns/synapse" key="{get-property('seqName')}"/>
Thanks to everybody!
Regards
Claudio
I am trying to use my API through the Swagger UI through the WSO2 API Manager:
Me->Browser->SwaggerUI->WSO2->MyBackend
Now I got the problem, that Swagger UI always gives me:
{ "error": "no response from server" }
I was making some research and figured out, that the Problem here is associated to CORS. While I am attempting a GET request, the Firefox sends an OPTION request first (because of CORS) to the WSO2 gateway.
Now the same scenario is explained and described here as well:
https://wso2.org/jira/browse/APIMANAGER-1819
The solution/workarounds described in the links is, that in the API publisher you have to create an endpoint for the requested resource with method OPTION and to set the auth type of that OPTION endpoint to "none".
However this doesn't work for me, since I get an exception in WSO, when the request is passed in.
The HTTP Request going into WSO2:
TID: [0] [AM] [2015-05-07 09:32:50,707] DEBUG {org.apache.synapse.transport.http.wire} - >> "OPTIONS /bankapi/v1/cashAccounts HTTP/1.1[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [AM] [2015-05-07 09:32:50,707] DEBUG {org.apache.synapse.transport.http.wire} - >> "Host: 10.88.13.104:8243[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [AM] [2015-05-07 09:32:50,707] DEBUG {org.apache.synapse.transport.http.wire} - >> "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:37.0) Gecko/20100101 Firefox/37.0[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [AM] [2015-05-07 09:32:50,707] DEBUG {org.apache.synapse.transport.http.wire} - >> "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [AM] [2015-05-07 09:32:50,708] DEBUG {org.apache.synapse.transport.http.wire} - >> "Accept-Language: en-US,en;q=0.5[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [AM] [2015-05-07 09:32:50,708] DEBUG {org.apache.synapse.transport.http.wire} - >> "Accept-Encoding: gzip, deflate[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [AM] [2015-05-07 09:32:50,708] DEBUG {org.apache.synapse.transport.http.wire} - >> "Origin: http://10.88.13.104:8080[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [AM] [2015-05-07 09:32:50,708] DEBUG {org.apache.synapse.transport.http.wire} - >> "Access-Control-Request-Method: GET[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [AM] [2015-05-07 09:32:50,708] DEBUG {org.apache.synapse.transport.http.wire} - >> "Connection: keep-alive[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [AM] [2015-05-07 09:32:50,708] DEBUG {org.apache.synapse.transport.http.wire} - >> "[\r][\n]" {org.apache.synapse.transport.http.wire}
The exception and stack trace:
TID: [0] [AM] [2015-05-07 09:32:50,878] ERROR {org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler} - API authentication failure {org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler}
org.wso2.carbon.apimgt.gateway.handlers.security.APISecurityException: Access failure for API: /bankapi, version: v1 with key: null
at org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator.authenticate(OAuthAuthenticator.java:212)
at org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler.handleRequest(APIAuthenticationHandler.java:94)
at org.apache.synapse.rest.API.process(API.java:284)
at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:83)
at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:64)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:220)
at org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:83)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:344)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:168)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
The important thing here is 2.... there is an error while authenticating even though the auth type is none and the authentication should always be successful. The request is not even forwarded to the backend (this is why my question is not a duplicate, but there is a very different problem underlying).
I am running WSO2 API Manager in version 1.8.0 on Windows (the binary package you can download on the WSO2 site).
Can you help me out here? Am I missing something? Is this a plain bug (it looks to me that way)?
i followed these guidelines in this WSO2 documentation page: http://docs.wso2.org/pages/viewpage.action?pageId=29918203
I configured the ELB, ESB Manager and 2 ESB worker in cluster. I had not error logged in the log files and, when i create a sample-passthrough proxy in the manager i see the xml files correctly deployed in the two worker instances too (looking at the file system repository folder).
The problem is that if on one hand i can login in the admin console of the manager ESB, every time i try to login in the worker 1 and 2 conoles i get the "Login Failed" message.
Looking in the woker log file i see:
TID: [0] [ESB] [2014-02-25 12:26:13,088] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin#carbon.super [-1234]' logged in at [2014-02-25 12:26:13,088+0100] {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
TID: [0] [ESB] [2014-02-25 12:26:19,108] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin#carbon.super [-1234]' logged in at [2014-02-25 12:26:19,107+0100] {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
TID: [0] [ESB] [2014-02-25 12:26:29,082] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin#carbon.super [-1234]' logged in at [2014-02-25 12:26:29,082+0100] {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
TID: [0] [ESB] [2014-02-25 12:27:10,669] ERROR {org.wso2.carbon.user.core.jdbc.JDBCUserStoreManager} - Using sql : null {org.wso2.carbon.user.core.jdbc.JDBCUserStoreManager}
TID: [0] [ESB] [2014-02-25 12:27:10,670] ERROR {org.wso2.carbon.user.core.common.AbstractUserStoreManager} - org.wso2.carbon.user.core.UserStoreException: Authentication Failure {org.wso2.carbon.user.core.common.AbstractUserStoreManager}
TID: [0] [ESB] [2014-02-25 12:27:10,670] WARN {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - Failed Administrator login attempt 'admin[-1234]' at [2014-02-25 12:27:10,670+0100] {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
TID: [0] [ESB] [2014-02-25 12:27:25,702] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin#carbon.super [-1234]' logged in at [2014-02-25 12:27:25,702+0100] {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
TID: [0] [ESB] [2014-02-25 12:27:28,155] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin#carbon.super [-1234]' logged in at [2014-02-25 12:27:28,155+0100] {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
TID: [0] [ESB] [2014-02-25 12:27:30,862] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin#carbon.super [-1234]' logged in at [2014-02-25 12:27:30,862+0100] {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
TID: [0] [ESB] [2014-02-25 12:27:32,466] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin#carbon.super [-1234]' logged in at [2014-02-25 12:27:32,466+0100] {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
TID: [0] [ESB] [2014-02-25 12:27:34,194] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin#carbon.super [-1234]' logged in at [2014-02-25 12:27:34,194+0100] {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
TID: [0] [ESB] [2014-02-25 12:27:34,826] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin#carbon.super [-1234]' logged in at [2014-02-25 12:27:34,826+0100] {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
Where is the error? What am i missing?
With the 4.7.0 version i was able to login each of the three esb.
thanks