WildFly 10 Jgroups allways binding to localhost interface - infinispan

Hi I'm trying to develop a clustered application that uses Infinispan for caching. First I tried to run in replicated mode by starting two instance of wildfly using the localhost as binding interface (with port offsets). This worked fine. But once I start the server using interface IP, cluster is not forming. Still I can access other services using the interface IP.
I tried to telnet the Jgroups port using interface IP address and it failed. But telnetting to localhost works for Jgorups port.
(Then entered localhsot[port] IP's to initial host configuration element in tcpping. Then cluster formation worked.)
So my question is why does it bind to localhost even after starting wildfly using interface IP.
Here is my configuration. (I cant use UDP, therefore need to use tcpping for cluster formation)
Started the wilfly server using
standalone.bat -Djboss.server.base.dir=../standalone_isuru -c standalone-full-ha.xml -b 192.168.17.33 -Djboss.node.name=isuru -Djboss.socket.binding.port-offset=1
Jgourps configuration
<subsystem xmlns="urn:jboss:domain:jgroups:4.0">
<channels default="ee">
<channel name="ee" stack="tcpping"/>
</channels>
<stacks>
<stack name="udp">
.
.
</stack>
<stack name="tcp">
.
.
</stack>
<stack name="tcpping">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<protocol type="TCPPING">
<property name="initial_hosts">
192.168.17.33[7601], 192.168.14.39[7700], 192.168.17.33[7800]
</property>
<property name="num_initial_members">
2
</property>
<property name="port_range">
5
</property>
<property name="timeout">
1000
</property>
</protocol>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
</stack>
</stacks>
Infinispan cache config
<cache-container name="replicated_cache" default-cache="default" module="org.wildfly.clustering.server" jndi-name="infinispan/replicated_cache">
<transport lock-timeout="60000"/>
<replicated-cache name="customer" jndi-name="infinispan/replicated_cache/customer" mode="SYNC">
<transaction locking="OPTIMISTIC" mode="FULL_XA"/>
<eviction strategy="NONE"/>
</replicated-cache>
</cache-container>

I posted the same question in Jboss developer since I didn't get any answer here.
And this is the answer I got from there.
By default Jgroups bind to private interface. When starting the server this IP can be provided as well.
standalone.bat -b 192.168.17.39 -bprivate=192.168.17.39
You can refer to the interfaces section for interface configuration.
<interfaces>
<interface name="management">
<inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
</interface>
<interface name="public">
<inet-address value="${jboss.bind.address:127.0.0.1}"/>
</interface>
<interface name="private">
<inet-address value="${jboss.bind.address.private:127.0.0.1}"/>
</interface>
<interface name="unsecure">
<inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>
</interface>
</interfaces>
socket bindings, binds jgroups to private interface
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
.
<socket-binding name="jgroups-tcp" interface="private" port="7600"/>
.
</socket-binding-group>
Jgroups subsystem
<stack name="tcpping">
<transport type="TCP" socket-binding="jgroups-tcp"/>
.
</stack>

Related

Infinispan, cross-dc replication: java.lang.IllegalStateException: Site <sitename> not defined in all the cluster members

I'm trying to setup a cross-datacenter replication mode between two infinispans 9.4.x as per Keycloak documentation, but the thing is that I'm trying to do this in sligtly modified environment:
multicast doesn't work between DC for obvious reasons
I have to use the port 7601 because 7600 is already used on this host by Keycloak JGroups transport (yup, by it's internal infinispan, and my future question would be "why do I need to use external extra Infinispan instance instead of setiing up replication between internal Keycloak's Infinispans, but first things first).
These a parts of my config that I added/modified:
[...]
<replicated-cache-configuration name="sessions-cfg" mode="SYNC" start="EAGER" batching="false">
<backups>
<backup site="site2" failure-policy="FAIL" strategy="SYNC" enabled="true">
<take-offline after-failures="3" min-wait="60000"/>
</backup>
</backups>
<locking acquire-timeout="0"/>
</replicated-cache-configuration>
[...]
<subsystem xmlns="urn:infinispan:server:jgroups:9.4">
<channels default="infinicluster">
<channel name="infinicluster" stack="tcp"/>
<channel name="xsite" stack="tcp"/>
</channels>
<stacks default="${jboss.default.jgroups.stack:udp}">
<stack name="udp">
<transport type="UDP" socket-binding="jgroups-udp"/>
<protocol type="PING"/>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="UFC_NB"/>
<protocol type="MFC_NB"/>
<protocol type="FRAG3"/>
<relay site="site1">
<remote-site name="site2" channel="xsite"/>
</relay>
</stack>
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<protocol type="TCPPING">
<property name="initial_hosts">
host1.tld[7601],host2.tld[7601]
</property>
<property name="ergonomics">
false
</property>
</protocol>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2">
<property name="use_mcast_xmit">
false
</property>
Of course, I changed the port numbers in JGroups socket bindings accordingly.
Both instances seem to be starting okay (complaining only about rest https bindings, which seems to be a minor error), I can even see the communications between instances in the logs:
2020-05-06 23:28:54,713 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t20)[Context=___hotRodTopologyCache]ISPN100002: Starting rebalance with members [host1.tld, host2.tld], phase READ_OLD_WRITE_ALL, topology id 2
2020-05-06 23:28:54,779 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t2) [Context=___hotRodTopologyCache]ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 3
2020-05-06 23:28:54,807 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t21)[Context=___hotRodTopologyCache]ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 4
2020-05-06 23:28:54,834 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t4)[Context=___hotRodTopologyCache]ISPN100010: Finished rebalance with members [host1.tld,host2.tld], topology id 5
The main issue is, that as soon as I open the web management page of any of the instances, I get the error on the logs (suppose I open the management page from site1, host1.tld):
2020-05-06 23:30:49,057 ERROR [org.jboss.as.controller.management-operation] (External Management Request Threads -- 1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
("subsystem" => "datagrid-infinispan"),
("cache-container" => "clustered")]): java.lang.IllegalStateException: Site host2.tld not defined in all the cluster members
at org.infinispan.xsite.XSiteAdminOperations.clusterStatus(XSiteAdminOperations.java:78)
at org.infinispan.xsite.GlobalXSiteAdminOperations.globalStatus(GlobalXSiteAdminOperations.java:93)
at org.jboss.as.clustering.infinispan.subsystem.CacheContainerMetricsHandler.filterSitesByStatus(CacheContainerMetricsHandler.java:343)
at org.jboss.as.clustering.infinispan.subsystem.CacheContainerMetricsHandler.executeRuntimeStep(CacheContainerMetricsHandler.java:297)
at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:59)
at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1411)
at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:265)
at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:231)
at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:243)
at org.jboss.as.domain.http.server.DomainApiHandler.handleRequest(DomainApiHandler.java:212)
at io.undertow.server.handlers.encoding.EncodingHandler.handleRequest(EncodingHandler.java:72)
at org.jboss.as.domain.http.server.DomainApiCheckHandler.handleRequest(DomainApiCheckHandler.java:93)
at org.jboss.as.domain.http.server.security.ElytronIdentityHandler.lambda$handleRequest$0(ElytronIdentityHandler.java:62)
at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:246)
at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:254)
at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:225)
at org.jboss.as.domain.http.server.security.ElytronIdentityHandler.handleRequest(ElytronIdentityHandler.java:61)
at io.undertow.server.handlers.BlockingHandler.handleRequest(BlockingHandler.java:56)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:485)
If I open the web management page from another site, the error is mirrored - this time it complains about host1.tld. It's obvious that I did something wrong, but I have no idea what exactly. Will be glad if someone could help me.

IHS not redirecting the request to WAS

So, I have installed WAS, and IHS.
I'm able to access the application directly from WAS. But IHS is trying to reply the request and its not using the plugin to reply the request.
I'm able to access the default IHS page, but not to the test application.
To create the IHS, I have done the standard stuff:
1- Install IHS and Plugin
2- Configure the plugin within the IHS
3- Create an unmanaged node on the DMGR
4- Create the webserver instance on the DMGR
I'm just trying to call a sample application called hello (for helloworld)
I see the request on the webservers logs, but they are not being redirected to the WAS.
This is the mention of the plugin on the httpd.conf:
LoadModule was_ap22_module /opt/ibm/plugin-ihs-85/test_webserver_instance1/bin/64bits/mod_was_ap22_http.so
WebSpherePluginConfig /opt/ibm/plugin-ihs-85/test_webserver_instance1/config/test_webserver_instance1/plugin-cfg.xml
This is the plugin-cfg.xml:
<?xml version="1.0" encoding="ISO-8859-1"?><!--HTTP server plugin config file for the webserver test_Cell.test_webserver_01_unmanagedNode_1.test_webserver_instance1 generated on 20
17.06.25 at 07:37:51 PM CDT-->
<Config ASDisableNagle="false" AcceptAllContent="true" AppServerPortPreference="HostHeader" ChunkedResponse="false" FIPSEnable="false" FailoverToNext="false" HTTPMaxHeaders="
300" IISDisableFlushFlag="false" IISDisableNagle="false" IISPluginPriority="High" IgnoreDNSFailures="false" KillWebServerStartUpOnParseErr="false" MarkBusyDown="false" OS400C
onvertQueryStringToJobCCSID="false" RefreshInterval="60" ResponseChunkSize="64" SSLConsolidate="true" StrictSecurity="false" TrustedProxyEnable="false" VHostMatchingCompat="f
alse">
<Log LogLevel="Error" Name="/opt/ibm/plugin-ihs-85/test_webserver_instance1/logs/test_webserver_instance1/http_plugin.log"/>
<Property Name="ESIEnable" Value="true"/>
<Property Name="ESIMaxCacheSize" Value="1024"/>
<Property Name="ESIInvalidationMonitor" Value="false"/>
<Property Name="ESIEnableToPassCookies" Value="false"/>
<Property Name="ESICacheidFull" Value="false"/>
<Property Name="PostSizeLimit" Value="-1"/>
<Property Name="PostBufferSize" Value="0"/>
<Property Name="PluginInstallRoot" Value="/opt/ibm/plugin-ihs-85/test_webserver_instance1/"/>
<Property Name="Keyfile" Value="/opt/ibm/plugin-ihs-85/test_webserver_instance1/config/test_webserver_instance1/plugin-key.kdb"/>
<Property Name="Stashfile" Value="/opt/ibm/plugin-ihs-85/test_webserver_instance1/config/test_webserver_instance1/plugin-key.sth"/>
<VirtualHostGroup Name="default_host">
<VirtualHost Name="*:9080"/>
<VirtualHost Name="*:80"/>
<VirtualHost Name="*:9443"/>
<VirtualHost Name="*:5060"/>
<VirtualHost Name="*:5061"/>
<VirtualHost Name="*:443"/>
<VirtualHost Name="*:9061"/>
<VirtualHost Name="*:9044"/>
<VirtualHost Name="*:9062"/>
<VirtualHost Name="*:9081"/>
<VirtualHost Name="*:9444"/>
<VirtualHost Name="*:9045"/>
</VirtualHostGroup>
<ServerCluster CloneSeparatorChange="false" GetDWLMTable="false" IgnoreAffinityRequests="false" LoadBalance="Round Robin" Name="test" PostBufferSize="0" PostSizeLimit="-1"
RemoveSpecialHeaders="true" RetryInterval="60" ServerIOTimeoutRetry="-1">
<Server CloneID="1bjgscrk8" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="2" MaxConnections="-1" Name="test_app_01_node_1_test1" ServerIOTimeout="900" W
aitForContinue="false">
<Transport Hostname="test_app_01" Port="9080" Protocol="http"/>
<Transport Hostname="test_app_01" Port="9443" Protocol="https">
<Property Name="keyring" Value="/opt/ibm/plugin-ihs-85/test_webserver_instance1/config/test_webserver_instance1/plugin-key.kdb"/>
<Property Name="stashfile" Value="/opt/ibm/plugin-ihs-85/test_webserver_instance1/config/test_webserver_instance1/plugin-key.sth"/>
</Transport>
</Server>
<Server CloneID="1bjgscsei" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="2" MaxConnections="-1" Name="test_app_01_node_1_test2" ServerIOTimeout="900" W
aitForContinue="false">
<Transport Hostname="test_app_01" Port="9081" Protocol="http"/>
<Transport Hostname="test_app_01" Port="9444" Protocol="https">
<Property Name="keyring" Value="/opt/ibm/plugin-ihs-85/test_webserver_instance1/config/test_webserver_instance1/plugin-key.kdb"/>
<Property Name="stashfile" Value="/opt/ibm/plugin-ihs-85/test_webserver_instance1/config/test_webserver_instance1/plugin-key.sth"/>
</Transport>
</Server>
<Server CloneID="1bjgscstv" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="2" MaxConnections="-1" Name="test_app_02_node_1_test3" ServerIOTimeout="900" W
aitForContinue="false">
<Transport Hostname="test_app_02" Port="9080" Protocol="http"/>
<Transport Hostname="test_app_02" Port="9443" Protocol="https">
<Property Name="keyring" Value="/opt/ibm/plugin-ihs-85/test_webserver_instance1/config/test_webserver_instance1/plugin-key.kdb"/>
<Property Name="stashfile" Value="/opt/ibm/plugin-ihs-85/test_webserver_instance1/config/test_webserver_instance1/plugin-key.sth"/>
</Transport>
</Server>
<Server CloneID="1bjgsctbv" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="2" MaxConnections="-1" Name="test_app_02_node_1_test4" ServerIOTimeout="900" W
aitForContinue="false">
<Transport Hostname="test_app_02" Port="9081" Protocol="http"/>
<Transport Hostname="test_app_02" Port="9444" Protocol="https">
<Property Name="keyring" Value="/opt/ibm/plugin-ihs-85/test_webserver_instance1/config/test_webserver_instance1/plugin-key.kdb"/>
<Property Name="stashfile" Value="/opt/ibm/plugin-ihs-85/test_webserver_instance1/config/test_webserver_instance1/plugin-key.sth"/>
</Transport>
</Server>
<PrimaryServers>
<Server Name="test_app_01_node_1_test1"/>
<Server Name="test_app_01_node_1_test2"/>
<Server Name="test_app_02_node_1_test3"/>
<Server Name="test_app_02_node_1_test4"/>
</PrimaryServers>
</ServerCluster>
<UriGroup Name="default_host_test_URIs">
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/hello/*"/>
</UriGroup>
<Route ServerCluster="test" UriGroup="default_host_test_URIs" VirtualHostGroup="default_host"/>
<RequestMetrics armEnabled="false" loggingEnabled="false" rmEnabled="false" traceLevel="HOPS">
<filters enable="false" type="URI">
<filterValues enable="false" value="/snoop"/>
<filterValues enable="false" value="/hitcount"/>
</filters>
<filters enable="false" type="SOURCE_IP">
<filterValues enable="false" value="255.255.255.255"/>
<filterValues enable="false" value="254.254.254.254"/>
</filters>
<filters enable="false" type="JMS">
<filterValues enable="false" value="destination=aaa"/>
</filters>
<filters enable="false" type="WEB_SERVICES">
<filterValues enable="false" value="wsdlPort=aaa:op=bbb:nameSpace=ccc"/>
</filters>
</RequestMetrics>
</Config>
Normally IHS is going to use port 80 to receive communication and rely to the application server.
The port 80 is in the default virtual host. So I added the listening port I'm using (1080) to the default virtual host.

How to Set Jgroups UDP to unicast instead of default multicast in standalone-ha.xml

Jgroups use "IP multicasting by default to send messages to all members (UDP) and for discovery of the initial members . However, if multicasting cannot be used, the UDP can be configured to send multiple unicast messages instead of one multicast message. To configure UDP to use multiple unicast messages to send a group message instead of using IP multicasting, the ip_mcast property has to be set to false." (as per jboss documentation https://developer.jboss.org/ )
My question is how can I pass "ip_mcast" value to false in wildfly? Below is the sample jgroups subsystem in the standalone-ha.xml. In the xsd I don't see a way to pass this value. Please help!!
<subsystem xmlns="urn:jboss:domain:jgroups:4.0">
<channels default="ee">
<channel name="ee" stack="udpgossip"/>
</channels>
<stacks>
<stack name="udpgossip">
<transport type="UDP" socket-binding="jgroups-tcp"/>
<protocol type="TCPGOSSIP">
<property name="initial_hosts">172.17.0.2[12001]</property>
</protocol>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
</stack>
</stacks>
</subsystem>
In the schema, <transport/> extends <protocol/>, and protocols can have properties, as your config sample already shows. So the correct way to set it should be
<transport type="UDP" socket-binding="jgroups-tcp">
<property name="ip_mcast">false</property>
</transport>

AWS EC2 with S3 PING Wildfly 8.2 session not shared

The two wildfly instances are not in the same session.
I have configured the following parts of the standalone-full-ha.xml :
<subsystem xmlns="urn:jboss:domain:modcluster:1.2">
<mod-cluster-config advertise-socket="modcluster" proxy- list="10.0.0.146:6666" balancer="mycluster" advertise="false" connector="ajp">
</subsystem>
10.0.0.146 is the apache instance private IP.
<subsystem xmlns="urn:jboss:domain:jgroups:2.0" default-stack="tcp">
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<protocol type="S3_PING">
<property name="access_key">
XXXXXXXXXXXXXXXX
</property>
<property name="secret_access_key">
XXXXXXXXXXXXXXXXXXXXXXX
</property>
<property name="location">
wildfly-bucket
</property>
<property name="timeout">
16000
</property>
</protocol>
<protocol type="MERGE2"/>
<protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
<protocol type="RSVP"/>
</stack>
</subsystem>
....
<interfaces>
<interface name="management">
<inet-address value="${jboss.bind.address.management:10.0.0.210}"/>
</interface>
<interface name="public">
<inet-address value="${jboss.bind.address:10.0.0.210}"/>
</interface>
<interface name="unsecure">
<inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>
</interface>
then I have started the instances (server1 and server2):
./standalone.sh -Djboss.node.name=server1 -c standalone-full-ha.xml
In the server logging i get it right:
07:41:27,019 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 63) ISPN000094: Received new cluster view: [server2/web|1] (2) [server2/web, server1/web]
Apache mod_cluster recognizes the two nodes.
In S3 I have granted all the permission to the “wildfly-bucket” to authenticated user, everyone and to myself.
I have also tried adding a bucket policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::XXXXXXXXXXXXXX:user/laura"
},
"Action": [
"s3:GetObjectVersion",
"s3:DeleteObject",
"s3:DeleteObjectVersion",
"s3:GetObject",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::wildfly-bucket/*"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::XXXXXXXXXXX:user/laura"
},
"Action": [
"s3:ListBucketVersions",
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::wildfly-bucket"
}
]
}
The user “laura” is in a Group with AdministratorAccess.
I have set the two instances+apache in the same security group and even tried with the same availability zone.
For the security group I have set the following rules
INBOUND:
Custom TCP Rule TCP 7600 0.0.0.0/0
Custom TCP Rule TCP 8080 0.0.0.0/0
SSH TCP 22 0.0.0.0/0
HTTP TCP 80 0.0.0.0/0
All traffic All All 0.0.0.0/0
Custom ICMP Rule Echo Reply N/A 0.0.0.0/0
All ICMP All N/A 0.0.0.0/0
Custom TCP Rule TCP 9990 0.0.0.0/0
OUTBOUND:
Custom TCP Rule TCP 7600 0.0.0.0/0
All traffic All All 0.0.0.0/0
What am I missing?
Can you please help me?
Thank you.

Jboss application-policy ignored in mutual/client-cert auth with PBESecurityDomain

With the following, mutual client cert, SSL (TLS) handshake works for a rest endpoint (yay!) - validated via testing and debugging: javax.net logging & wireshark. But...
1st observation: HTTPServletRequest and JAX-RS annotated SecurityContext has null Principal info
2nd observation: Tampering with the login-config.xml, containing application-policy elements, has no effect
In short, TLS works but the transfer of the cert DN to the HTTPServletRequest object in the request thread does not preventing the application from picking up on the caller's ID. Does anyone have any advice?
On JBoss 6:
deploy/jbossweb.sar/server.xml:
<Connector protocol="HTTP/1.1" debug="10"
SSLEnabled="true"
...
secure="true"
clientAuth="true"
sslProtocol = "TLS"
securityDomain="java:/jaas/mydomain"
SSLImplementation="org.jboss.net.ssl.JBossImplementation" />
deploy/jbossweb.sar/META-INF/jboss-beans.xml:
<depends>jboss.security:service=PBESecurityDomain</depends>
deploy/security-service.xml:
<?xml version="1.0" encoding="UTF-8"?>
<server>
<mbean code="org.jboss.security.plugins.JaasSecurityDomain"
name="jboss.security:service=PBESecurityDomain">
<constructor> <arg type="java.lang.String" value="mydomain"/>
</constructor>
<attribute name="KeyStoreURL">${jboss.server.home.dir}/mykeystore.jks</attribute>
<attribute name="KeyStorePass">{CLASS}org.jboss.security.plugins.FilePassword:${jboss.server.home.dir}/mykeystorepass.pbe</attribute>
<attribute name="TrustStoreURL">${jboss.server.home.dir}/mytruststore.jks</attribute>
<attribute name="TrustStorePass">password</attribute>
<attribute name="Salt">abunchofrandomchars</attribute>
<attribute name="IterationCount">13</attribute>
<depends>jboss.security:service=JaasSecurityManager</depends>
</mbean>
</server>
deploy/security/security-jboss-beans.xml:
<bean name="XMLLoginConfig" class="org.jboss.security.auth.login.XMLLoginConfig">
<property name="configResource">login-config.xml</property>
</bean>
<bean name="SecurityConfig" class="org.jboss.security.plugins.SecurityConfig">
<property name="mbeanServer"><inject bean="JMXKernel" property="mbeanServer"/></property>
<property name="defaultLoginConfig"><inject bean="XMLLoginConfig"/></property>
</bean>
conf/login-config.xml:
<application-policy name="mydomain">
<authentication>
<login-module code="org.jboss.security.auth.spi.BaseCertLoginModule"
flag = "required">
<module-option name="password-stacking">useFirstPass</module-option>
<module-option name="securityDomain">java:/jaas/mydomain</module-option>
<module-option name="verifier">org.jboss.security.auth.certs.AnyCertVerifier</module-option>
<module-option name="principalClass">org.jboss.security.auth.certs.SubjectDNMapping</module-option>
</login-module>
<login-module code="org.jboss.security.auth.spi.UserRolesLoginModu"
flag = "required">
<module-option name="password-stacking">useFirstPass</module-option>
<module-option name="usersProperties">users.properties</module-option>
<module-option name="rolesProperties">roles.properties</module-option>
</login-module>
</authentication>
</application-policy>
war/WEB-INF/jboss-web.xml:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE jboss-web PUBLIC
"-//JBoss//DTD Web Application 2.4//EN"
"http://www.jboss.org/j2ee/dtd/jboss-web_4_0.dtd">
<jboss-web>
<security-domain>java:/jaas/mydomain</security-domain>
<context-root>/myapp</context-root>
</jboss-web>
Add the special ClientLoginModule to login-context.xml to fix the null principal issue.
<login-module code="org.jboss.security.ClientLoginModule" flag="required"></login-module>