I am trying to setup remote cache by externalising the Infinispan server from Keycloak using the Cross-Datacenter Replication Mode. I am following the Keycloak official documentation here
https://www.keycloak.org/docs/latest/server_installation/#crossdc-mode
As per the documentation, the Keycloak and Infinispan version being used as as follows (I am hoping these versions are compatible to each other since this information is coming from Keycloak’s official. site) -
Keycloak Version - 11.0.2
Infinispan Version - 9.4.18
To my surprise, I have found a bug into Infinispan V9.4.18. Below exception is being constantly reported from infinispan server.
I downloaded Infinispan server (version infinispan-server-9.4.18) from their official site https://downloads.jboss.org/infinispan/9.4.18.Final/infinispan-server-9.4.18.Final.zip
Without making any changes into any of the infinispan config file, when I boot up the Infinispan server, it starts without any issue but when I try to bring up the Hotrod console at http://127.0.0.1:11222
(11222 is the port where hotrod listens on my localhost), it does not load. I get the issue as shown below :
¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 71
¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 69
¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 84
¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 32
¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 47
¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 32
¡�P�`org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or
While on the server’s console, the exception is as follows :
18:41:28,551 ERROR [org.infinispan.server.hotrod.BaseRequestProcessor] (HotRod-ServerIO-4-1) ISPN005003: Exception reported: org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 71
at org.infinispan.server.hotrod.HotRodDecoder.switch0(HotRodDecoder.java:208)
at org.infinispan.server.hotrod.HotRodDecoder.switch1_0(HotRodDecoder.java:153)
at org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.java:143)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
18:41:28,593 ERROR [org.infinispan.server.hotrod.BaseRequestProcessor] (HotRod-ServerIO-4-1) ISPN005003: Exception reported: org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 69
at org.infinispan.server.hotrod.HotRodDecoder.switch0(HotRodDecoder.java:208)
at org.infinispan.server.hotrod.HotRodDecoder.switch1_0(HotRodDecoder.java:153)
I tried with multiple versions of Infinispan servers (e.g. 9.4.18, 9.4.20, 10.0.1, 11.0.3) but the Infinispan versions 9.4.x gives me the issue while 10.x.x and 11.x.x) does not and I can load the Hotrod console . without any problem (with versions 10.x.x and 11.x.x).
Keycloak documentation I am following ( https://www.keycloak.org/docs/latest/server_installation/#crossdc-mode ) for integrating Keycloak with external remote Infinispan , is based on Infinispan version v9.4.18 which differs quite a bit on architectural front from versions v10.0.1 or v11.0.3
I see one solution suggested by Redhat here https://access.redhat.com/solutions/3920921 around this issue which sounds like a hotrod client's protocol version issue (probably not being compatible with hotrod server etc) but in my case I am trying to load the hotrod server console thru browser. I don't have any hotrod client application per se), so the solution isn't of much use.
Has any one ever come across or have any idea/pointers around the below infinispan issue i am facing and how to overcome this??
Appreciate pointers/suggestions here.
Thanks much
Jameel
On Infinispan versions before 10.0 the admin console lives on port 9990.
Related
we had a Mulesoft app that basically picks message from queue (ActiveMQ), then posts to target app via HTTP request to target's API.
Runtime: 4.3.0
HTTP Connector version: v1.3.2
Server: Windows, On-premise standalone
However, sometimes the message doesn't get sent successfully after picking from queue , and below message can be found in the log -
WARN 2021-07-10 01:24:46,080 [[masked-app].http.requester.requestConfig.02 SelectorRunner] [event: ] org.glassfish.grizzly.nio.transport.TCPNIOTransport: GRIZZLY0005: Can not set SO_KEEPALIVE to false
java.net.SocketException: Invalid argument: no further information
at sun.nio.ch.Net.setIntOption0(Native Method) ~[?:1.8.0_281]
The flow completed silently without any error after above message, hence no error handling happens.
I found this mentioning it is a known bug on Windows server and won’t affect the well behavior of the application, but the document is failing to set SO_KEEPALIVE to true rather than false.
Looks the message didn't get posted successfully as the target system team can't find corresponding incoming request in their log.
It is not acceptable as the message is critical and no one knows unless the target system realizes something is wrong... Not sure if the SO_KEEPALIVE is failing to be set to false is the root cause, could you please share some thoughts? Thanks a lot in advance.
The is probably unrelated to the warning you mentioned but there doesn't seem to be enough information to identify the actual root cause.
Having said that the version of the HTTP connector is old and it's missing almost 3 years of fixes. Updating the version to the last one should improve the reliability of the application.
I was making a simple file upload&download service and found out that, as far as I understand, Netty doesn't release direct buffers until request processing is over. As a result, I can't upload bigger files.
I was trying to make sure that the problem is not inside my code, so I created the most simple tiny Ktor application:
routing {
post("upload") {
call.receiveMultipart().forEachPart {}
call.respond(HttpStatusCode.OK)
}
}
The default direct memory size is about 3Gb, to make test simpler I limit it with:
System.setProperty("io.netty.maxDirectMemory", (10 * 1024 * 1024).toString())
before starting the NettyApplicationEngine.
Now if I upload a large file, for example with httpie, I got "Connection reset":
http -v --form POST http://localhost:42195/upload file#/tmp/FileStorageLoadTest-test-data1.tmp
http: error: ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) while doing POST request to URL: http://localhost:42195/upload
On the server side there is no information about the problem except for the "java.io.IOException: Broken delimiter occurred" exception. But if I put the breakpoint in NettyResponsePipeline#processCallFailed, the real exception is:
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 65536 byte(s) of direct memory (used: 10420231, max: 10485760)
It is a pity that this exception is not logged.
Also, I found out that the same code works without problems if I use Jetty engine instead.
Environment:
Ubuntu Linux
Java 8
Ktor=1.2.5
netty-transport-native-epoll=4.1.43.Final
(but if Netty started without native-epoll support, the problem is the same)
My Node Red application in IBM BlueMix is repeatedly crashing - once an hour - with no real error message other than "exited with status: 1."
How can I troubleshoot this issue?
Is there someone from IBM BlueMix support that monitors this that could take a look?
I looked at my logs and there's nothing in there that really says what's going on.
Edit per requests:
The regular log for "OUT/ERR" is scrolling so fast with HTTPD logs that I can't get it to copy/paste. Filtering to "ERR" Channel the only thing I see is below. I believe this is an error which occurs during deploy when the application restarts.
[App/0] ERR js-bson: Failed to load c++ bson extension, using pure JS version
My Node Red application is gathering data from Wink, LIFX, and other IoT services and compiles them together into a Freeboard dashboard.
Caught crash on screenshot here -- not enough cred to post images so it'll only post as a link
The zlib error was fixed in the 0.13.2 Node-RED release (that shipped 19/02/16).
If you re-stage your application is should pick up the new version of Node-RED
You can re-stage the application using the cf command line management application:
cf restage <app name>
I'm receiving the following message on server log in IBM Mobile First 6.3 every time an Adapter is getting called:
Stacktrace
[ERROR ] Error sending bulk request: java.lang.RuntimeException:
failure in bulk execution: [2]: index [worklight], type [devices], id
[b2deefe7-0d15-4ed4-b199-7e42440fc372], message
[VersionConflictEngineException[[worklight][1]
[devices][b2deefe7-0d15-4ed4-b199-7e42440fc372]: version conflict,
current [58], provided [57]]] at
com.ibm.elasticsearch.servlet.DataReceiver.processData(DataReceiver.java:132)
at
com.ibm.elasticsearch.servlet.DataReceiver.processDataLegacy(DataReceiver.java:85)
at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source) ...
The adapter is executed correctly and the response is returned to the app.
Any idea why this error is happening?
Help will be appreciated.
Thanks.
This is an internal error in analytics. The error itself is actually harmless, however the analytics platform should be catching it... A defect will be logged for the message. In the meantime, if you're not using analytics, you can disable it by removing the WAR files from the Liberty server.
If you are using analytics, then I would recommend clearing out the analytics data folder and restarting the IMF platform (this would remove any data you have stored in analytics). This is assuming that you are running in development mode. The analytics data folder can be in the same directory as the server.xml file for your Liberty server.
My code is consistently generating the following error:
org.apache.axis2.AxisFault: Timeout waiting for connection
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:203)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:400)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:225)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:435)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:402)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
After extensive searching the solution appears to be here:
http://amilachinthaka.blogspot.com/2009/05/improving-axis2-client-http-transport.html
Except I cannot access http client directly (that is done in auto generated code).
Instead I have tried:
http://wiki.apache.org/ws/FrontPage/Axis/AxisCommonsHTTP
Which seems to help slightly. However it only delays the problem I always get the timeout. Also the instructions appear to be for an old version of Axis since Axis2 does not have AxisProperties. I am using version 1.5.1. Any suggestions?
I think this was a bug, see here and has been fixed with Axis2 1.5.2 (at least for generated code). With custom code, you need to tackle this problem manually.