Tomcat 9.x.x Client Authentication using X.509 Certificates - authentication

Question

I’m using Tomcat 9.0.19 and trying to enable X.509 cert.-based client authentication (AKA I&A) for a particular Web application.
In summary, the Tomcat works for an application that has basic I&A enabled over one-way TLS. When accessing the Web application that has certificate-based I&A, Tomcat does not seem to request a client certificate as part of the Server Hello message, prior to sending Server Hello Done and it later fails the authentication check:
02-Jan-2020 13:00:40.371 FINE [https-jsse-nio-443-exec-10] org.apache.catalina.authenticator.SSLAuthenticator.doAuthenticate Looking up certificates
02-Jan-2020 13:00:40.830 FINE [https-jsse-nio-443-exec-10] org.apache.catalina.authenticator.SSLAuthenticator.doAuthenticate No certificates included with this request
Traced the TLS flow in Wireshark and saw the TLS 1.2 handshake. Shortly after encrypted data is exchanged, the Tomcat sends an “Encrypted Alert” message and the socket is closed. Trying to contact the Tomcat from the browser, doing a GET. The browser does not prompt me to select a certificate, which also seems to point to Tomcat not requesting it from the browser.
Any help will be greatly appreciated!
More Details:
We have a set of certificates for the Tomcat and the client, issued by an Intermediate CA, which is signed (issued) by a Root CA. The trust stores have been setup on both sides (client and server) as well as key stores with the right certs/keys in them. The Web application is setup to require certificate I&A (web.xml):
<security-constraint>
<web-resource-collection>
<web-resource-name>All by default</web-resource-name>
<url-pattern>/*</url-pattern>
<http-method>GET</http-method>
<http-method>POST</http-method>
</web-resource-collection>
<auth-constraint>
<role-name>OTService</role-name>
</auth-constraint>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>
<login-config>
<auth-method>CLIENT-CERT</auth-method>
<realm-name>certificate</realm-name>
</login-config>
The OTService role is setup in the Tomcat-Users.xml, along with a single user account:
Now, the Connector in server.xml is configured as follows:
<Connector port="443" protocol="org.apache.coyote.http11.Http11NioProtocol"
maxThreads="100" SSLEnabled="true" scheme="https" secure="true">
<SSLHostConfig>
<Certificate certificateKeystoreFile="/apache-tomcat-9.0.19/conf/km/keyStore.jks"
certificateKeystorePassword="PASSWORD"
certificateKeyAlias="tomcat"
type="RSA" />
truststoreFile="/apache-tomcat-9.0.19/conf/km/trust_store.jks"
truststorePass="PASSWORD"
truststoreType="JKS"
certificateVerification="required"
clientAuth="true"
protocols="TLSv1.2"
</SSLHostConfig>
</Connector>
Any ideas why Tomcat would not request a client certificate?

Answer 1

The first issue that I discovered was that Tomcat ignored the Connector->SSLHostConfig settings for the trust store and used the JRE default trust store anyway. The way I discovered it was to have a browser save the negotiated TLS session key to a file (Google SSLKEYLOGFILE), then configured the Wireshark to use that file, captured the browser-Tomcat session and then was able to see every message in plaintext.
Next, I discovered that Tomcat was actually asking for a client cert., but the list of accepted Root CAs it was sending was from the default JRE cacerts file, not from the file specified by the truststoreFile attribute. Can have Tomcat use a different file across the board by adding a setenv.sh file to the Tomcat bin directory with Java properties to override default trust store location.
Now, I was in business, the browser was able to complete the TLS handshake, but then the authentication and authorization steps were failing. I finally determinate that the proper way to provide the cert. subject field in tomcat_users.xml file was not "CN=OU Client, OU=Control Systems, O=IoTOY, L=Scottsdale, S=AZ, C=US", but "CN=OU Client, OU=Control Systems, O=IoTOY, L=Scottsdale, ST=AZ, C=US". Finally, I had 2-way TLS working.
One thing to keep in mind is if anything running on the Tomcat attempts to connect over TLS to another system that uses commercial CA certs, it will fail because the truststore you're using now does not have commercial Root CAs' certs. One way to remediate it is to make a copy of the default JRE cacerts file and add your system-specific CA cert(s) to it and point to it from the setenv.sh file noted above.

Answer 2

When you have:
<Connector ...>
<SSLHostConfig>
<Certificate A=1 B=2 C=3 />
D=4 E=5 F=6
</SSLHostConfig>
</Connector>
then A,B,C are attributes of the Certificate object but D,E,F are NOT attributes of the SSLHostConfig object -- they are XML content which is different. Attributes need to be put IN THE TAG:
<Connector ... >
<SSLHostConfig certificateVerification="required" truststoreFile=... >
<Certificate ...keystore... />
</SSLHostConfig>
</Connector>
and that does cert-request on the initial handshake as desired (for me, tested on tomcat 9.0.14).