Multiple Tomcat instances, starting one kills the other - apache

I have 2 different java applications running in two Tomcat instances (Ubuntu OS, Tomcat 7.0.57). For my Tomcat configuration, I follow instructions provided by this video, but as far as I saw, it is a very standard way to do it.
Running each application separately is working fine, but as soon as I try to run both at same time, the first started one becomes unavailable (HTTP 503 error). Tomcat instance logs do not provide any information about any kind of shutdown, keeping the last "INFO: Server startup in xxx ms". It seems the first tomcat process is simply killed. If I re-start that first application, then the same scenario applies to the second app.
All troubleshooting information I could find talk about port issues. I double checked my port numbers, they are different:
app-1: conf/server.xml:
<Server port="8105" shutdown="SHUTDOWN">
<Connector port="8180" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8143" />
<Connector port="8109" protocol="AJP/1.3" redirectPort="8143" />
...
</Server>
app-2: conf/server.xml:
<Server port="8205" shutdown="SHUTDOWN">
<Connector port="8280" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8243" />
<Connector port="8209" protocol="AJP/1.3" redirectPort="8243" />
...
</Server>
app1.sh
export CATALINA_HOME=/home/tomcat/apache-tomcat-7
export CATALINA_BASE=/home/tomcat/app-1
cd $CATALINA_HOME/bin
./startup.sh
app2.sh
export CATALINA_HOME=/home/tomcat/apache-tomcat-7
export CATALINA_BASE=/home/tomcat/app-2
cd $CATALINA_HOME/bin
./startup.sh
Any idea on what can happen, or how I can get any logs to dig this?

My server was hosted on a EC2 t1.micro instance, with 600MB memory.
I finally decide to update it to an instance with more memory and the problem disappear.

Related

Tomcat 8.5.29 HTTP/2 is not supporting GZIP compression

I am using Tomcat 8.5.29 and using the respective configuration,I have enabled the HTTP2 support for the site. Below is the configuration in server.xml file.
<Connector port="443" protocol="org.apache.coyote.http11.Http11AprProtocol"
maxThreads="150" SSLEnabled="true" compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json" compression="on" compressionMinSize="1024"
>
<UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
<SSLHostConfig>
<Certificate certificateKeyFile="conf/localhost-key.pem"
certificateFile="conf/localhost-cert.pem"
certificateChainFile="conf/cacert.pem"
type="RSA" />
</SSLHostConfig>
</Connector>
When i tried to compare the page load time for the site which is supporting HTTPS 1.1 and HTTP2, it is not consistent. Sometime it is taking more time to load and sometime it is taking less time to load compare to HTTPS 1.1.
To measure the page load time i am using Developer tools from Chrome Browser.
Later, I have found that Load time for HTTP/2 is higher because "content-encoding" in response is not gzip. Using HTTP/1.1, a script file size is 4 MB ( gzip) where as with HTTP/2, same file is of 8 MB ( no content-encoding). Compression is not happening. We have also tried with useSendfile and compression as "force" but it did not work.
We have also tried with Tomcat version 8.5.31 but no luck !! Facing same issue.
We have used Tomcat version 9 and using following configuration, got content in gzip (compressed). Now file size got reduced by 2 MB with a quick page load.
<Connector port="443" protocol="org.apache.coyote.http11.Http11AprProtocol"
maxThreads="150" SSLEnabled="true"
>
<UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json" compression="on" compressionMinSize="1024" />
<SSLHostConfig>
<Certificate certificateKeyFile="conf/localhost-key.pem"
certificateFile="conf/localhost-cert.pem"
certificateChainFile="conf/cacert.pem"
type="RSA" />
</SSLHostConfig>
</Connector>
It looks like Tomcat version 8.5 is not supporting gzip with HTTP2. Any idea on this?
According to the Tomcat documentation:
Note: There is a tradeoff between using compression (saving your bandwidth) and using the sendfile feature (saving your CPU cycles). If the connector supports the sendfile feature, e.g. the NIO2 connector, using sendfile will take precedence over compression. The symptoms will be that static files greater that 48 Kb will be sent uncompressed. You can turn off sendfile by setting useSendfile attribute of the protocol, as documented below, or change the sendfile usage threshold in the configuration of the DefaultServlet in the default conf/web.xml or in the web.xml of your web application.
So try adding useSendfile="false" to your UpgradeProtocol setting
<UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json" compression="on" compressionMinSize="1024" useSendfile="false" />
Note this option does not appear in the 8.5 documentation so not sure if it exists for that (there is a reference to it "You can turn off sendfile by setting useSendfile attribute of the protocol, as documented below" but then nothing below), though you can also turn off at a server level.
Weirdly I wouldn't have expected this to work in Tomcat 9 either without turning off this option, but you say it does? Are you sure you have not turned this off? Or are you testing with a smaller than 48KB file on that?
Tomcat 8.5.33 fixes this issue; compression works with http/2 then.

symmetricDS two way ssl handshake(clientAuth=false)

Problem summary:
I am trying to run symmetric ds on tomcat with ssl. Setup details
1). MasterNode M1: running on tomcat.
2). Cleint node C1: running on tomcat.
I am able to run symmetricds on tomcat without ssl. This setup works fine all sync happens.
Now when I am trying to run tomcat with following setting for ssl, it gives bad_certificate error on client node:
connection setting on Master node:
<Connector SSLEnabled="true" acceptCount="10"
algorithm="PKIX" allowTrace="false"
ciphers=""
clientAuth="true"
enableLookups="true"
keystoreFile="/root/symmetric-server-3.8.28/security/keystore.p12"
keystorePass="changeit#1"
keystoreType="pkcs12"
port="31415"
redirectPort="-1"
scheme="https"
secure="true"
sslEnabledProtocols="TLSv1.2"
sslProtocol="TLSv1.2"
truststoreFile="/root/symmetric-server-3.8.28/security/keystore.p12"
truststorePass="changeit#1"
truststoreType="pkcs12"/>
connector setting on Client node:
<Connector SSLEnabled="true" acceptCount="10"
algorithm="PKIX" allowTrace="false"
ciphers=""
clientAuth="true"
enableLookups="true"
keystoreFile="/root/symmetric-server-3.8.28/security/keystore2.p12"
keystorePass="changeit#1"
keystoreType="pkcs12"
port="31415"
redirectPort="-1"
scheme="https"
secure="true"
sslEnabledProtocols="TLSv1.2"
sslProtocol="TLSv1.2"
truststoreFile="/root/symmetric-server-3.8.28/security/keystore2.p12"
truststorePass="changeit#1"
truststoreType="pkcs12"/>
Note: If I run Master node tomcat with clientAuth=false then everything works fine. On further debugging I found that client is not providing its certificate during TLS handshake. Following is error: Experts please help what config I need to do so that client provide certificate:
Error:
javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
Upgrade to SymmetricDS 3.8.31. There was an issue that was fixed that allows for client TLS authentication now:
http://www.symmetricds.org/issues/view.php?id=2886

How to Change SSL Version for HTTPS Connections in JBoss EAP 6.1

I have a simple HTTPS connector configured on my JBoss EAP 6.1 server for SSL connections to a bunch of RESTful web services I am working on. I am not sure if JBoss EAP 6.1 comes with TLS 1.2 (or SSL 3.2, since I believe TLS is really just later versions of SSL), but I want to use that version of TLS or later.
What is the default SSL version of JBoss EAP 6.1 , if my standalone.xml file tags that handle this connector look like this? :
<connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">
<ssl name="https" key-alias="localhost" password="something" certificate-key-file="${jboss.server.config.dir}/localhost.jks"/>
</connector>
There is no SSL version information given in the standalone.xml entry above, as you can see, so I've no idea.
I have read on the JBoss community web site that you can add something like sslProtocol="TLS" in the tag, and protocol="TLSv2" in the tag, but is that really all there is to it?
i.e.
<connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true" sslProtocol="TLS">
<ssl name="https" key-alias="localhost" password="something" certificate-key-file="${jboss.server.config.dir}/localhost.jks" protocol="TLSv2" />
</connector>
Will the above work, and if so, how can I tell?
According to redhat documentation for EAP 6.1
protocol
The version of the SSL protocol to use. Supported values include SLv2, SSLv3, >TLSv1, SSLv2+SSLv3, and ALL. The default is ALL
Adding sslProtocol="TLS" and protocol="TLSv2" should work fine. Not to sound condescending, but the easiest way to see if it works is by testing it.
If this question did get moved to another SE site could you please provide the link?

Enabling JBoss AS 7 Directory Listings

I have the following directory structure deployed to JBoss AS 7.1.1.Final (under standalone/deployments):
doc.war
-> module1
-> index.html
-> module2
-> index.html
As you can see, there's no index.html under doc.war. When I browse to localhost:8080/doc/module1/, the correct index.html is displayed, but when I browse to localhost:8080/doc/, JBoss shows an error message (404 - The requested resource is not available).
I think this is related to the fact that directory listings are turned off by default in JBoss AS 7. How can I enable directory listings, either globally or more specifically for this one application?
Edit
Based on Mukul Goel's answer, I ran the CLI command to add the static-resources feature, restarted the server and retried the request, but it didn't work.
Here's the relevant snippet from the standalone.xml file. Please note that I have the native connector enabled.
<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="true">
<configuration>
<static-resources listings="true"/>
</configuration>
<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
<connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https"/>
<virtual-server name="default-host" enable-welcome-root="true">
<alias name="localhost"/>
<alias name="example.com"/>
</virtual-server>
</subsystem>
Here's the error message that JBoss is showing:
Update
So the conclusion to this seems to be that there is an issue with the official JBoss 7.1.1.Final (http://www.jboss.org/jbossas/downloads) download. I didn't manage to get Directory Listings working with this version. Trying a later version (from the JBoss CI server at https://ci.jboss.org/jenkins/job/JBoss-AS-7.x-latest/), I was able to see the directory listings after applying the config change that Mukul Goel had suggested below.
A potential source of this issue could be the version of JBossWeb that is used in JBoss. The official 7.1.1.Final bundles JBossWeb 7.0.13. Mukul (see below) was able to get it working running a version of JBoss that bundles JBossWeb 7.0.16.
I'm accepting Mukul Goel's answer as solving this issue, but be aware that it will probably not work with the official 7.1.1.Final download.
Yes you are right, directory listings are by default disabled (a security measure)
To enable directory listing in JBOSS
Try running the following CLI command to enable diectory listing:
In Domain Mode
/profile=full/subsystem=web/configuration=static-resources/:write-attribute(name=listings,value=true)
.
In Standalone Mode
/subsystem=web/configuration=static-resources/:write-attribute(name=listings,value=true)
It will generate following kind of configuration :
<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">
<configuration>
<static-resources listings="true"/>
</configuration>
<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
<virtual-server name="default-host" enable-welcome-root="true">
<alias name="localhost"/>
<alias name="example.com"/>
</virtual-server>
</subsystem>
UPDATE:
Tried it myself as the user is facing problems
RAN CLI command for the standaloneMode
This was generated, note that the native is disabled also there is no HTTPS connector generated for me (Don`t know why it is showing enabled for you? are you using openSSL somewhere? )
I created a sample webprojet(a client side project) with two htmls ,published it to jboss and hit url
http://localhost:8080/sample/
and this is the screenshot of directory listing
The command worked it for me, So that does turns on Directory listing on JBOSS AS7.1.1 Final(I am also using the same version)
So the question comes down to rest of your server configuration, your application structure, technologies you are using, springs etc and also if you are using some ssl library.
UPDATE 2
Suggested a fresh download and reconfigure environment and see
nwinkler was still facing issues even with a fresh distribution (JbossWeb 7.0.13) So suggested to take nightly builds from
ci.jboss.org/jenkins/job/JBoss-AS-7.x-latest
Just go to the link and download whats under last successful artifact
And the problem was resolved. Looks like some bug with jbossWeb 7.0.13
Do go through the comments, the discussion might be helpful

How can I get client certificate authentication working in JBoss 5.1.0.GA when I'm using APR, and not all web deployments use CLIENT-CERT auth?

Note: I will be answering my own question... just wanted to add this tidbit to the collective wisdom of The Internets.
I've successfully configured certificate authentication on my JBoss 5.1.0.GA server, largely with the help of the information on this page: http://docs.jboss.org/jbossas/jboss4guide/r1/html/ch8.chapter.html
I have one context (let's call it /openContext) that doesn't require any authentication, and another context (let's call it /securedContext) that requires client certificate authentication (i.e., it's configured to use CLIENT-CERT in web.xml). When using JBoss's default web connector, this works splendidly. I can hit http://myhost/openContext and I'm not prompted for a certificate, but when I hit http://myhost/securedContext, I'm prompted for a client certificate as I'd expect.
However, when I install JBossWeb Native and use APR as my web connector, I'm no longer prompted for a certificate when I hit http://myhost/securedContext.
My APR connector config in server.xml looks like:
<Connector protocol="HTTP/1.1" SSLEnabled="true"
port="8443" address="${jboss.bind.address}"
scheme="https" secure="true" clientAuth="false"
SSLProtocol="SSLv3+TLSv1"
SSLCipherSuite="ALL:!ADH:!SSLv2:!EXPORT40:!EXP:!LOW"
SSLRandomSeed="/dev/urandom"
SSLCertificateFile="/etc/pki/tls/certs/mycert.crt"
SSLCertificateKeyFile="/etc/pki/tls/private/mycert.key"
SSLPassword="mypasswordwhichiassureyouisbetterthanthisone"
SSLCACertificateFile="/etc/pki/tls/certs/clientCAs.crt"
/>
I've also tried adding the SSLVerifyClient parameter to that configuration and setting it to optional, but that prompts for a certificate in both /openContext and /securedContext, which isn't the behavior I want.
How can I get JBoss with APR to require certificate authentication for one web context, but not another web context?
What worked for me was to just add a whole new web connector, and have clients use that alternate port for the secured web context. My connectors config now looks like:
<Connector protocol="HTTP/1.1" SSLEnabled="true"
port="8443" address="${jboss.bind.address}"
scheme="https" secure="true" clientAuth="false"
SSLProtocol="SSLv3+TLSv1"
SSLCipherSuite="ALL:!ADH:!SSLv2:!EXPORT40:!EXP:!LOW"
SSLRandomSeed="/dev/urandom"
SSLCertificateFile="/etc/pki/tls/certs/mycert.crt"
SSLCertificateKeyFile="/etc/pki/tls/private/mycert.key"
SSLPassword="mypasswordwhichiassureyouisbetterthanthisone"
/>
<Connector protocol="HTTP/1.1" SSLEnabled="true"
port="8543" address="${jboss.bind.address}"
scheme="https" secure="true" clientAuth="true"
SSLProtocol="SSLv3+TLSv1"
SSLCipherSuite="ALL:!ADH:!SSLv2:!EXPORT40:!EXP:!LOW"
SSLRandomSeed="/dev/urandom"
SSLCertificateFile="/etc/pki/tls/certs/mycert.crt"
SSLCertificateKeyFile="/etc/pki/tls/private/mycert.key"
SSLPassword="mypasswordwhichiassureyouisbetterthanthisone"
SSLCACertificateFile="/etc/pki/tls/certs/clientCAs.crt"
SSLVerifyClient="require"
/>
Now, if I hit http://myhost:8443/openContext, I'm not prompted for a certificate, but when I hit http://myhost:8543/securedContext, I am prompted for a certificate. Of course, I can still access either web app with the "wrong" port, but the consequences are negligible for my purposes. If a client hits http://myhost:8443/securedContext, they simply get an HTTP authentication error. If a client hits http://myhost:8543/openContext, they're prompted for a client certificate. If they provide one, great (though I don't care who you are), and if they don't provide one or provide an invalid one, they get an HTTP auth error (they should have used the correct port in the first place).
I'm pretty sure there's an alternative way to get this working without requiring a second connector by putting httpd in front of JBoss and doing some clever configuration there, but this worked well enough for my purposes.