Double Tomcat behind mod_jk load balancer - apache

I am in the process of setting up two Tomcat instances on the same server with an Apache mod_jk load balancer in front of it. I have been using a guide and the Apache Tomcat documentation and stuck to the basic setup suggested. When i try to start up any of the Tomcat instances, i get a BindException from when it tries to start up the SimpleTcpCluster. The error message is "Cannot assign requested address".
I googled for solutions to this issue and came across two suggestions, the first one being to ensure that Java is configured to prefer IPv4 addresses. Tried it - no change. The second suggested to replace the auto value on the address parameter on the Receiver component inside the cluster config (see config below).
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.4"
port="45564" frequency="500"
dropTime="3000"/>
<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="localhost" port="4000" autoBind="100"
selectorTimeout="5000" maxThreads="6"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/>
<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
<ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
I tried changing "auto" to "localhost", which led to a different error message on Tomcat startup, saying "Address already in use :8009".
At this point i really don't know where to look. Is localhost a bad value? Should i be using auto but make a change somewhere else? Is there anyone out there with a little more experience on this that can give me a helping hand?

We got around this issue by changing the address parameter in the Receiver tag inside the Cluster configuration from "auto" to the actual IP address of the server. I was never able to figure out why this was not working and didn't want to spend any more time once we got the calls through.
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="123.123.x.x" port="4000" autoBind="100"
selectorTimeout="5000" maxThreads="6"/>

Related

Orbeon Forms PE (Ehcache) replication without multicasting

We want to set up Orbeon Forms PE replication but we cannot use multicasting as proposed in the docs.
We have two nodes - 172.13.238.241 and 172.13.238.242 and the problem seems to be with the EhCache part. I open the form, load balancer (haproxy) directs me to a node, I turn of the second node and then for several minutes all the requests in the brower fail. Eventually the request start to work again but they are very slow.
This is what I have in node 1 (another node has same config with IP-s replaced with each other and it has a different uniqueId value):
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="asynchronous"
channelStartOptions="3">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Interceptor className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor">
<Member className="org.apache.catalina.tribes.membership.StaticMember"
port="4100"
host="172.13.238.242"
uniqueId="{0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2}" />
</Interceptor>
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="172.13.238.241"
port="4100"
autoBind="0"
maxThreads="6"
selectorTimeout="5000" />
<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.*\.html|.*\.css|.*\.txt"/>
<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
I wonder if anyone could spot some mistakes in my ehcache configuration.

Let webservice use SSL

With WildFly 8.2.1, I am trying to make existing webservice (JAX-WS) use SSL, but I haven't seen any use of SSL in quickstarts and the information I was able to google is limited. So far I've added this to web.xml:
<security-constraint>
<display-name>Foo security</display-name>
<web-resource-collection>
<web-resource-name>FooService</web-resource-name>
<url-pattern>/foo/FooService</url-pattern>
<http-method>POST</http-method>
</web-resource-collection>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>
and this is in my standalone.xml:
<subsystem xmlns="urn:jboss:domain:webservices:1.2">
<wsdl-host>${jboss.bind.address:127.0.0.1}</wsdl-host>
<endpoint-config name="Standard-Endpoint-Config"/>
<endpoint-config name="Recording-Endpoint-Config">
<pre-handler-chain name="recording-handlers" protocol-bindings="##SOAP11_HTTP ##SOAP11_HTTP_MTOM ##SOAP12_HTTP ##SOAP12_HTTP_MTOM">
<handler name="RecordingHandler" class="org.jboss.ws.common.invocation.RecordingServerHandler"/>
</pre-handler-chain>
</endpoint-config>
<client-config name="Standard-Client-Config"/>
</subsystem>
but apparently that's not enough; when I look into standalone/data/wsdl/foo.ear/foo.war/FooService/Bar.wsdl I see:
<service name="FooService">
<port binding="foowsb:FooBinding" name="FooBinding">
<soap:address location="http://localhost:8080/foo/FooService"/>
</port>
</service>
Note that in the EAR/WAR, the soap:address.location is filled just with a placeholder (I suppose that the value is ignored).
I've found some info about setting up security realm, and creating the self-signed certificate using keytool (which I did), but I completely miss how this should be linked together.
I've also tried to setup wsdl-uri-scheme=https, but this is supported only in later versions of CXF.
Seems that the soap:address.location value is not ignored when it's being replaced, since changing that from REPLACE_WITH_ACTUAL_URL to https://REPLACE_WITH_ACTUAL_URL did the trick - now the service got exposed on https://localhost:8443.
There is a couple of more steps I had to do in standalone.xml: in undertow, add https-listener:
<https-listener name="secure" socket-binding="https" security-realm="SslRealm"/>
define the SslRealm:
<security-realm name="SslRealm">
<server-identities>
<ssl>
<keystore path="foo.keystore" relative-to="jboss.server.config.dir" keystore-password="foo1234" alias="foo" key-password="foo1234"/>
</ssl>
</server-identities>
<authentication>
<truststore path="foo.truststore" relative-to="jboss.server.config.dir" keystore-password="foo1234"/>
</authentication>
</security-realm>
Note that I reuse the same keystore for server and clients here. And since my clients are ATM in the same WF node during development, I had to setup the client-side part there, too:
<system-properties>
<property name="javax.net.ssl.trustStore" value="${jboss.server.config.dir}/foo.keystore"/>
<property name="javax.net.ssl.trustStorePassword" value="foo1234"/>
<property name="org.jboss.security.ignoreHttpsHost" value="true"/>
</system-properties>
The last property should be replaced in WF 9+ with cxf.tls-client.disableCNCheck.

Tomcat 8 Session replication - problems

need help in figuring out what`s missing with the configuration that have setup - clustering seems to be work fine,
3 node tomcat cluster on 3 machines
machine A - apache webserver A + Tomcat A
machine B - apache webserver B + Tomcat B
machine C - apache webserver C + Tomcat C
all the tomcat instances are aware of other tomcat nodes and during restart, i see tomcat instances are getting added into cluster and each cluster is being aware of presence of other instances.
but up on accessing a web application (where have enabled ) , session is not being replicated across tomcat instances.
Here is my server.xml configuration
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="8">
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.8"
bind="machineA ip"
port="45564"
frequency="500"
dropTime="3000" />
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="machineA ip"
port="4200"
autoBind="100"
selectorTimeout="5000"
maxThreads="6" />
<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=".*\.gif;.*\.js;.*\.jpg;.*\.htm;.*\.html;.*\.txt;" />
<Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
tempDir="D:/cluster/temp/war-temp/"
deployDir="D:/cluster/temp/war-deploy/"
watchDir="D:/cluster/temp/war-listen/"
watchEnabled="false" />
<ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
and after going through some online blogs and posts i have moved the contents to global context.xml file
'$catalina_base/conf/context.xml'
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true" />
but still the session is not being replicated across the cluster
please provide your inputs/suggestions

Tomcat Clustering Multicast throws error - Unable to perform failure detection check, assuming member down

Hi I am new to clustering concept, so I tried to establish a Tomcat cluster with 2 instances. I followed the examples, mainly at http://www.mulesoft.com/tcat/tomcat-clustering and few other web sources.
My sample set up is as,
in workers.properties
# Define worker names
worker.list=jkstatus, loadbalancer
# Create virtual workers
worker.jkstatus.type=status
worker.loadbalancer.type=lb
# Declare Tomcat server worker 1
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=7009
# Declare Tomcat server worker 2
worker.worker2.type=ajp13
worker.worker2.port=9009
worker.worker2.host=localhost
# Associate real workers with virtual LoadBalancer worker
worker.loadbalancer.balance_workers=worker1,worker2
And Apache httpd.conf as,
# ADDED CLUSTER CONFIG
# Load module
LoadModule jk_module modules/mod_jk.so
# Specify path to worker configuration file
JkWorkersFile C:/tomcat_clustered/workers.properties
# Configure logging and memory
JkShmFile C:/tomcat_clustered/log/location/mod_jk.shm
JkLogFile C:/tomcat_clustered/log/location/mod_jk.log
JkLogLevel info
# Configure monitoring
JkMount /jkmanager/* jkstatus
<Location /jkmanager>
Order deny,allow
deny from all
allow from localhost
</Location>
# Configure applications
JkMount /* loadbalancer
# END CLUSTER CONFIG
With each tomcat(worker) server.xml as,
<Engine name="Catalina" defaultHost="localhost" jvmRoute="worker1">
<!--For clustering, please take a look at documentation at:
/docs/cluster-howto.html (simple how to)
/docs/config/cluster.html (reference documentation) -->
<!--
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
-->
<!-- CLUSTER BEGIN -->
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<!---->
<Membership className="org.apache.catalina.tribes.membership.McastService"
address="239.0.0.1"
port="45564" frequency="500"
dropTime="3000"/>
<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto" port="4000" autoBind="100"
selectorTimeout="5000" maxThreads="6"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/>
<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
<ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
<!-- CLUSTER END -->
<!-- Use the LockOutRealm to prevent attempts to guess user passwords
via a brute-force attack -->
<Realm className="org.apache.catalina.realm.LockOutRealm">
For worker1, worker2. I am working on Windows 7. It looks like some multicast option is needed but I am not sure. On starting up of tomcat, one is starting well, and on starting second(say worker2) both tomcat console's are throwing errors as,
at org.apache.catalina.tribes.group.ChannelInterceptorBase.heartbeat(Cha
nelInterceptorBase.java:103)
at org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.
ava:155)
at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(Gro
pChannel.java:690)
ep 15, 2014 10:11:03 AM org.apache.catalina.tribes.group.interceptors.TcpFailur
Detector memberAlive
EVERE: Unable to perform failure detection check, assuming member down.
ava.net.SocketException: Permission denied: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketI
pl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.ja
a:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocket
mpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java
Not sure if its due to some multicast permissions on my Windows or any config related issues. Please suggest some good pointers in this regard
You need to enable multicast. see this post how to enable multicast is windows, https://serverfault.com/questions/262634/how-do-i-know-if-ip-multicasting-is-enabled-on-my-network-in-windows
In Linux Environment most of the system kernel is capable to process the multicast address.
but we need to add route entry in kernel routing table.
sudo route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0
The error that you're reporting looks like it might be OS / networking related. Hard to say exactly. I can tell you this though.
You've declared a TcpFailureDetector.
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
This is part of Tomcat's cluster configuration and it's responsibility is to make a TCP connection to any node that is suspected of failure. If the listener can't connect to the node, it'll be marked as down. If it can connect, then the node remains active.
In this case, the listener is attempting to make a connection to one of your nodes and it's failing with an error from the JDK. Googling that specific error turns up some suggestions, one of which is to use "-Djava.net.preferIPv4Stack=true".
I get java.net.SocketException: Permission denied: connect when sending an email in Jenkins
On a separate note, if you've note seen the official documentation on Tomcat clustering, I would strongly recommend you check that out. Here's the link.
http://tomcat.apache.org/tomcat-7.0-doc/cluster-howto.html

Cannot get SSL to work on Azure instance

I've been tearing my hear out trying to figure out why SSL works in one of my Azure projects but not in another.
When I navigate to my site, say https://foo.com, I can't even connect to the site. Browsers can't connect at all and curl says "couldn't connect to host". However, if I go to my cloudapp.net URL (e.g. https://foo.cloudapp.net), it can connect but browsers will complain and say my cert is for *.foo.com. Note: I am able to connect to http://foo.com without any trouble.
Here's my code with certain values obfuscated.
ServiceDefinition.csdef:
<?xml version="1.0" encoding="utf-8"?>
<ServiceDefinition name="MyApp" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition">
<WebRole name="www" vmsize="Small">
<Sites>
<Site name="Web">
<VirtualApplication name="r" physicalDirectory="../Foo/Bar" />
<Bindings>
<Binding name="Endpoint1" endpointName="Endpoint1" />
<Binding name="Endpoint2" endpointName="Endpoint2" />
</Bindings>
</Site>
</Sites>
<Endpoints>
<InputEndpoint name="Endpoint1" protocol="http" port="80" />
<InputEndpoint name="Endpoint2" protocol="https" port="443" certificate="STAR.foo.com" />
</Endpoints>
<Imports>
<Import moduleName="Diagnostics" />
</Imports>
<Certificates>
<Certificate name="STAR.foo.com" storeLocation="LocalMachine" storeName="My" />
</Certificates>
</WebRole>
</ServiceDefinition>
my cert is uploaded, the thumbprint matches (in this example it's also "1234567890")
ServiceConfiguration.csfg:
<?xml version="1.0" encoding="utf-8"?>
<ServiceConfiguration serviceName="myApp" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" osFamily="1" osVersion="*">
<Role name="www">
<Instances count="2" />
<ConfigurationSettings>
<Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="UseDevelopmentStorage=true" />
</ConfigurationSettings>
<Certificates>
<Certificate name="STAR.foo.com" thumbprint="1234567890" thumbprintAlgorithm="sha1" />
</Certificates>
</Role>
</ServiceConfiguration>
Azure Console:
I have verified that:
My cert is uploaded
It's SHA1
It's thumbprint matches what I've specified in ServiceConfiguration.cscfg (in this example it's "1234567890")
The certs for the Certificate Authorities are also present (for me it's "PositiveSSL CA" and "AddTrust External CA root")
For the Azure instance, it confirms there are 2 endpoints (port 80 and port 443)
Why would I not be able to connect at all via https://foo.com, but my https://foo.cloudapp.net will load (although triggering a browser warning)? This seem to indicate my configuration is correct but something else is off... ideas?
I think you may be looking in the wrong place for your problem!
How have you mapped foo.com to your site's address?
Note that Azure instances are given dynamic IP addresses - what address your site may be on NOW may not be what its on tomorrow. The recommendation for Azure is to add a "www" CNAME DNS entry in your domain records that points at "foo.cloudapp.net".
This way, when someone browses to www.foo.com, the DNS server will (invisibly) say "hey, actually, that site is as foo.cloudapp.net. The browser will then ask for the IP address of foo.cloudapp.net. This domain is managed by Microsoft who will return the current IP address for your site.
If you want foo.com to still get you to www.foo.com, you'll have to setup DNS redirection so that whenever someone types foo.com into their browser, they're redirected to www.foo.com. This will then cuase the browser to resolve foo.cloudapp.net and then the HTTP request will be sent to your site. Some domain hosters charge for this (typically a nominal fee), some offer it as a free service.
HTH.