Our customer is running four standalone JBoss AS 7.1 nodes that are merged into a cluster by means of session replication and load balancer.
We are getting these errors often without additional details. I am also no expert in JBoss and I have exhausted all my Google-fu.
I don't personally have access to server configuration, nor has my technical bridge person. But we have right to ask questions.
2019-03-29 11:38:08 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-15) ISPN000136: Error executing command CommitCommand, writing keys [0qsZ5PfJNsgD9SYAjeyMLeSrEXJVGbtuwVgwtjd-, 0qsZ5PfJNsgD9SYAjeyMLeSrEXJVGbtuwVgwtjd-]: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from xxxxxx:stc00Server2373, see cause for remote stack trace
at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44) [infinispan-core-8.1.6.Final-redhat-1.jar:8.1.6.Final-redhat-1]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:766) [infinispan-core-8.1.6.Final-redhat-1.jar:8.1.6.Final-redhat-1]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:615) [infinispan-core-8.1.6.Final-redhat-1.jar:8.1.6.Final-redhat-1]
........
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_111]
at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111]
Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000362: Could not find status for remote transaction GlobalTransaction:<xxxxxx:stc00Server2374>:5359:remote, please increase transaction.completedTxTimeout
at org.infinispan.commands.tx.CommitCommand.invalidRemoteTxReturnValue(CommitCommand.java:52) [infinispan-core-8.1.6.Final-redhat-1.jar:8.1.6.Final-redhat-1]
at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.perform(AbstractTransactionBoundaryCommand.java:89) [infinispan-core-8.1.6.Final-redhat-1.jar:8.1.6.Final-redhat-1]
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92) [infinispan-core-8.1.6.Final-redhat-1.jar:8.1.6.Final-redhat-1]
at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34) [infinispan-core-8.1.6.Final-redhat-1.jar:8.1.6.Final-redhat-1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_111]
at org.jboss.as.clustering.infinispan.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:48) [wildfly-clustering-infinispan-extension-7.0.4.GA-redhat-2.jar:7.0.4.GA-redhat-2]
... 1 more
2019-03-29 11:38:08 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-15) ISPN000188: Error while processing a commit in a two-phase transaction: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from xxxxx:stc00Server2373, see cause for remote stack trace
at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44) [infinispan-core-8.1.6.Final-redhat-1.jar:8.1.6.Final-redhat-1]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:766) [infinispan-core-8.1.6.Final-redhat-1.jar:8.1.6.Final-redhat-1]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:615) [infinispan-core-8.1.6.Final-redhat-1.jar:8.1.6.Final-redhat-1]
.....
at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111]
Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000362: Could not find status for remote transaction GlobalTransaction:<xxxxx:stc00Server2374>:5359:remote, please increase transaction.completedTxTimeout
at org.infinispan.commands.tx.CommitCommand.invalidRemoteTxReturnValue(CommitCommand.java:52) [infinispan-core-8.1.6.Final-redhat-1.jar:8.1.6.Final-redhat-1]
at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.perform(AbstractTransactionBoundaryCommand.java:89) [infinispan-core-8.1.6.Final-redhat-1.jar:8.1.6.Final-redhat-1]
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92) [infinispan-core-8.1.6.Final-redhat-1.jar:8.1.6.Final-redhat-1]
at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34) [infinispan-core-8.1.6.Final-redhat-1.jar:8.1.6.Final-redhat-1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_111]
at org.jboss.as.clustering.infinispan.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:48) [wildfly-clustering-infinispan-extension-7.0.4.GA-redhat-2.jar:7.0.4.GA-redhat-2]
... 1 more
I have not just redacted references to customer, but also redundant rows. The full stack trace is available as a paste.
In our environment, where we are unable to replicate the issue, the best I could find that configures Infinispan is the following block:
<subsystem xmlns="urn:jboss:domain:infinispan:4.0">
<cache-container name="server" aliases="singleton cluster" default-cache="default" module="org.wildfly.clustering.server">
<transport lock-timeout="60000"/>
<replicated-cache name="default" mode="SYNC">
<transaction mode="BATCH"/>
</replicated-cache>
</cache-container>
<cache-container name="web" default-cache="repl" module="org.wildfly.clustering.web.infinispan">
<transport lock-timeout="60000"/>
<replicated-cache name="repl" mode="SYNC">
<locking isolation="READ_COMMITTED"/>
<transaction locking="OPTIMISTIC" mode="BATCH"/>
<file-store/>
</replicated-cache>
</cache-container>
<cache-container name="manager" default-cache="repl" module="org.wildfly.clustering.web.infinispan">
<transport lock-timeout="60000"/>
<replicated-cache name="repl" mode="SYNC">
<locking isolation="READ_COMMITTED"/>
<transaction locking="OPTIMISTIC" mode="BATCH"/>
<file-store/>
</replicated-cache>
</cache-container>
<cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
<cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan">
<transport lock-timeout="60000"/>
<local-cache name="local-query">
<eviction strategy="LRU" max-entries="10000"/>
<expiration max-idle="100000"/>
</local-cache>
<invalidation-cache name="entity" mode="SYNC">
<transaction mode="NON_XA"/>
<eviction strategy="LRU" max-entries="10000"/>
<expiration max-idle="100000"/>
</invalidation-cache>
<replicated-cache name="timestamps" mode="ASYNC"/>
</cache-container>
</subsystem>
Question is: where is this parameter (transaction.completedTxTimeout) configured in JBoss standalone.xml? That seems to point at this attribute
Related
We want to set up Orbeon Forms PE replication but we cannot use multicasting as proposed in the docs.
We have two nodes - 172.13.238.241 and 172.13.238.242 and the problem seems to be with the EhCache part. I open the form, load balancer (haproxy) directs me to a node, I turn of the second node and then for several minutes all the requests in the brower fail. Eventually the request start to work again but they are very slow.
This is what I have in node 1 (another node has same config with IP-s replaced with each other and it has a different uniqueId value):
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="asynchronous"
channelStartOptions="3">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Interceptor className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor">
<Member className="org.apache.catalina.tribes.membership.StaticMember"
port="4100"
host="172.13.238.242"
uniqueId="{0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2}" />
</Interceptor>
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="172.13.238.241"
port="4100"
autoBind="0"
maxThreads="6"
selectorTimeout="5000" />
<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.*\.html|.*\.css|.*\.txt"/>
<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
I wonder if anyone could spot some mistakes in my ehcache configuration.
We currently have a 5-node infinispan server cluster running in production and it is able to handle up to 2 thousand requests per second. Our desire is to have the cache scale linearly, up 10x or more. In our stress tests, our app servers start getting SocketTimeoutExceptions from infinspan after 30 minutes under planned future load.
Our most active cache runs in distributed mode with 2 owners and 20 segments. One of the bottlenecks appear to be with our pub/sub system using the remote event listeners. Basically whenever a cache entry is modified, our remote event listeners get notified, who then in turn respond to long-polling requests. The remote event listeners are running in Java servlets on tomcat.
About a month ago, we ran into the issue found HERE
and after patching infinispan server with an increased event queue size (now at 1 million) we were able to scale up quite far, just not as much as we'd like.
In terms of actual errors on the server, what we see are things like:
ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (pool-6-thread-1) ISPN000136: Error executing command RemoveExpiredCommand, writing keys [[B0x033e183537626430..[27]]: org.infinispan.util.concurrent.TimeoutExcepti
on: ISPN000299: Unable to acquire lock after 30 seconds for key [B0x033e183537626430..[27] and requestor CommandUUID{address=XYZ, id=851936}. Lock is held by CommandUUID{address=XYZ, id=851819}
And below is the cache config:
<subsystem xmlns="urn:infinispan:server:core:8.2" default-cache-container="clustered">
<cache-container name="clustered" default-cache="default" statistics="true">
<transport lock-timeout="60000" />
<distributed-cache name="default" mode="SYNC" segments="20" owners="2" remote-timeout="30000" start="EAGER">
<locking acquire-timeout="30000" concurrency-level="10000" striping="false"/>
<transaction mode="NONE"/>
<expiration lifespan="86400000" max-idle="900000" interval="60000"/>
<eviction strategy="LIRS" size="1000000"/>
</distributed-cache>
<distributed-cache name="gameStateCache" mode="SYNC" remote-timeout="30000" start="EAGER">
<locking acquire-timeout="30000" concurrency-level="10000" striping="false"/>
<expiration lifespan="86400000" max-idle="900000" interval="60000"/>
<eviction strategy="LIRS" size="1000000"/>
</distributed-cache>
</cache-container>
</subsystem>
Is anyone aware of high-load issues with the remote event listener system or perhaps suggest alternative configuration?
Cheers,
Brian
I am trying to get my JSF Web Application deployed in a "domain" cluster with two nodes.
I am unable to get the HTTP Session replicated between the nodes.
I have configured my web.xml
<?xml version="1.0" encoding="UTF-8"?>
<web-app version="3.0" xmlns="http://java.sun.com/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd">
<session-config>
<session-timeout>30</session-timeout>
<tracking-mode>URL</tracking-mode>
</session-config>
<distributable></distributable>
</web-app>
and jboss-web.xml
<?xml version="1.0" encoding="UTF-8"?>
<jboss-web>
<context-root>thumbzup-admin</context-root>
<!--replication-config>
<replication-trigger>SET_AND_NON_PRIMITIVE_GET</replication-trigger>
<replication-granularity>SESSION</replication-granularity>
</replication-config-->
<replication-config>
<cache-name>custom-session-cache</cache-name>
<replication-trigger>SET</replication-trigger>
<replication-granularity>ATTRIBUTE</replication-granularity>
<!--replication-field-batch-mode>true</replication-field-batch-mode-->
<use-jk>false</use-jk>
<max-unreplicated-interval>30</max-unreplicated-interval>
<snapshot-mode>INSTANT</snapshot-mode>
<snapshot-interval>1000</snapshot-interval>
<replication-mode>SYNCHRONOUS</replication-mode>
<!--session-notification-policy>com.example.CustomSessionNotificationPolicy</session-notification-policy-->
</replication-config>
</jboss-web>
and Infinispan Cache Container in domain.xml;
<cache-container name="web" aliases="standard-session-cache" default-cache="repl" module="org.jboss.as.clustering.web.infinispan">
<transport lock-timeout="60000"/>
<replicated-cache name="repl" mode="ASYNC" batching="true">
<file-store/>
</replicated-cache>
<replicated-cache name="sso" mode="SYNC" batching="true"/>
<distributed-cache name="dist" l1-lifespan="0" mode="ASYNC" batching="true">
<file-store/>
</distributed-cache>
</cache-container>
But I am unable to retrieve a session created on NODE1 when using the same SESSIONID on NODE2.
Any suggestions?
The name
"custom-session-cache"
you defined in jboss-web.xml does not seem to match any cache inside domain.xml. Remove "cache-name" property completely to use the value of "default-cache" property of the "web" container or add a cache entry with that name
I'm using Windows 8. In drive C 's root directory (C:) there are 27 WCF Trace file; 26 of them are named like <guid>Traces.svclog and one is Traces.svclog. I'm trying to find out which applications are creating these files; because I don't have any application that uses wcf trace described here.
And here is my C:\ directory's view:
I've looked all of the traces: there are only 2 unique trace and the rest is exact copies of them. Here comes the details from one of them;
<E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent">
<System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">
<EventID>589828</EventID>
<Type>3</Type>
<SubType Name="Information">0</SubType>
<Level>8</Level>
<TimeCreated SystemTime="2012-10-15T06:37:57.4222685Z" />
<Source Name="System.ServiceModel" />
<Correlation ActivityID="{00000000-0000-0000-0000-000000000000}" />
<Execution ProcessName="Microsoft.VisualStudio.Web.Host" ProcessID="7104" ThreadID="78" />
<Channel />
<Computer>MYCOMP</Computer>
</System>
<ApplicationData>
<TraceData>
<DataItem>
<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Information">
<TraceIdentifier>http://msdn.microsoft.com/tr-TR/library/System.ServiceModel.Activation.WebHostCompilation.aspx</TraceIdentifier>
<Description>ASP.Net hosted compilation.</Description>
<AppDomain>7f594460-3-129947566686622036</AppDomain>
<Source>System.ServiceModel.Activation.ServiceParser/17864371</Source>
<ExtendedData xmlns="http://schemas.microsoft.com/2006/08/ServiceModel/StringTraceRecord">
<VirtualPath>/BI/DataService.svc</VirtualPath>
</ExtendedData>
</TraceRecord>
</DataItem>
</TraceData>
</ApplicationData>
</E2ETraceEvent>
So the C:\ directory is filling with rubbish traces. I've installed WCF&WF samples, and maybe that caused this. Can you help me on this; to find out which application is doing this?
Search your drive for app.config files that have systems.diagnostics sections that enable trace on the system.servicemodel tracesource.
I followed all the steps in this blogpost http://virgo47.wordpress.com/2010/08/23/tomcat-web-application-with-ssl-client-certificates/ except for the fact that I'm using JBoss7.0.2 and not the 6.x version.
The goal is to ask any clients to provide a client certificate and achieve mutual authentication between the client and the server.
I have created a certification authority (CA) to sign the client and server certificates.
I have imported the server certificate into the keystore and added an HTTPS connector to the standalone.xml configuration file to serve HTTPS requests on the 8443 port.
I have imported the CA root certificate into the Certificate Manager under Authorities in client's Firefox.
Everything works fine and when I request https://localhost:8443 I get a page with a valid server certificate.
The problem is, when I import the client certificate into the Certificate Manager in Firefox and set the server configuration to verify client certificates (verify-client="true" in standalone.xml) I get a browser error:
Secure Connection Failed:
An error occurred during a connection to localhost:8443.
SSL peer cannot verify your certificate.
(Error code: ssl_error_bad_cert_alert)
while the jboss log on the server states:
11:01:31,142 DEBUG [org.apache.tomcat.util.net.JIoEndpoint] (http-localhost-127.0.0.1-8443-1) Handshake failed: java.io.IOException: SSL handshake failed. Ciper suite in SSL Session is SSL_NULL_WITH_NULL_NULL
at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:191) [jbossweb-7.0.1.Final.jar:7.0.2.Final]
at org.apache.tomcat.util.net.JIoEndpoint.setSocketOptions(JIoEndpoint.java:1144) [jbossweb-7.0.1.Final.jar:7.0.2.Final]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:952) [jbossweb-7.0.1.Final.jar:7.0.2.Final]
at java.lang.Thread.run(Thread.java:662) [:1.6.0_30]
Do you have any idea how to resolve this problem?
My setup:
Localhost server:
sovo#sovo-pc:~$ cat /etc/issue
Ubuntu 10.10
JBoss 7.0.2 Final standalone.xml (relevant parts):
<management>
<security-realms>
<security-realm name="PropertiesMgmtSecurityRealm">
<authentication>
<properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
</authentication>
</security-realm>
</security-realms>
<management-interfaces>
<native-interface interface="management" port="9999"/>
<http-interface interface="management" port="9990"/>
</management-interfaces>
</management>
<profile>
<subsystem xmlns="urn:jboss:domain:security:1.0">
<security-domains>
<security-domain name="other" cache-type="default">
<authentication>
<login-module code="Disabled" flag="required"/>
</authentication>
</security-domain>
</security-domains>
</subsystem>
<subsystem xmlns="urn:jboss:domain:web:1.0" default-virtual-server="default-host">
<connector name="https" protocol="HTTP/1.1" socket-binding="https" scheme="https" enable-lookups="false" secure="true">
<ssl name="ssl" key-alias="sercer" password="changeit" certificate-key-file="/usr/share/jboss7.0.2/standalone/configuration/certificates/keystore.jks" protocol="TLSv1" verify-client="true" ca-certificate-file="/usr/share/jboss7.0.2/standalone/configuration/certificates/cacerts.jks"/>
</connector>
<virtual-server name="default-host" enable-welcome-root="true">
<alias name="localhost"/>
<alias name="example.com"/>
</virtual-server>
</subsystem>
<subsystem xmlns="urn:jboss:domain:weld:1.0"/>
</profile>
<interfaces>
<interface name="management">
<inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
</interface>
<interface name="public">
<inet-address value="${jboss.bind.address:127.0.0.1}"/>
<inet-address value="${jboss.bind.address:localhost}"/>
</interface>
</interfaces>
<socket-binding-group name="standard-sockets" default-interface="public">
<socket-binding name="http" port="8080"/>
<socket-binding name="https" port="8443"/>
<socket-binding name="jmx-connector-registry" port="1090" interface="management"/>
<socket-binding name="jmx-connector-server" port="1091" interface="management"/>
<socket-binding name="jndi" port="1099"/>
<socket-binding name="osgi-http" port="8090" interface="management"/>
<socket-binding name="remoting" port="4447"/>
<socket-binding name="txn-recovery-environment" port="4712"/>
<socket-binding name="txn-status-manager" port="4713"/>
</socket-binding-group>
Java version:
sovo#sovo-pc:~$ java -version
java version "1.6.0_30"
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) Server VM (build 20.5-b03, mixed mode)
sovo#sovo-pc:~$ javac -version
javac 1.6.0_30
I'll be happy to provide other relevant information if needed.
You might want to give a try by adding cipher suite you your ssl connector:
<ssl name="ssl" key-alias="sercer" password="changeit" certificate-key-file="/usr/share/jboss7.0.2/standalone/configuration/certificates/keystore.jks" protocol="TLSv1" verify-client="true" ca-certificate-file="/usr/share/jboss7.0.2/standalone/configuration/certificates/cacerts.jks" cipher-suite="AES+RSA"/>