Keycloak Invalid_code when authenticating through load balancer - authentication

Question

I'm trying to setup Keycloak for my company and have run into some issues that I can't solve. I currently have keycloak installed on two servers in standalone clustered mode. I have these servers behind and external load balancer owned by another group. I've created two realms, one for Jenkins and one for Grafana. When I configure each of them to point directly to the server, it works with no problem. The issue comes into play when I try to authenticate through the load balanced url. Below are some logs from Keycloak, grafana and Jenksins. Both servers are configured with standalone-ha.xml and are configured exactly the same.
Keycloak Log:
2020-02-19 06:30:19,599 WARN [org.keycloak.events] (default task-1) type=CODE_TO_TOKEN_ERROR, realmId=CICD, clientId=grafana, userId=null, ipAddress=24.43.182.84, error=invalid_code, grant_type=authorization_code, code_id=08c751b6-d6a1-4b55-9f69-4b4c28c0c9c4, client_auth_method=client-secret
Grafana Log:
t=2020-02-19T14:32:09+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/login/generic_oauth status=302 remote_addr=172.23.0.5 time_ms=0 size=345 referer=https://example.com/grafana/login
t=2020-02-19T14:32:52+0000 lvl=info msg="state check" logger=oauth queryState=0c5576040cb0984602e4e5a8ccc891e425065c740f9c6e4b3331494ad5c69b9b cookieState=0c5576040cb0984602e4e5a8ccc891e425065c740f9c6e4b3331494ad5c69b9b
t=2020-02-19T14:32:52+0000 lvl=eror msg=login.OAuthLogin(NewTransportWithCode) logger=context userId=0 orgId=0 uname= error="oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\",\"error_description\":\"Code not valid\"}"
t=2020-02-19T14:32:52+0000 lvl=eror msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/login/generic_oauth status=500 remote_addr=172.23.0.5 time_ms=21 size=1751 referer="https://load_balanced_example.com/auth/realms/CICD/protocol/openid-connect/auth?access_type=online&client_id=grafana&redirect_uri=https%3A%2F%2Fexample.com%2Fgrafana%2Flogin%2Fgeneric_oauth&response_type=code&scope=openid+email+profile&state=kyersRiz2wsAryohnLlZPPdtQjA6MJO8wanOnDaXgaY%3D"
Response from Jenkins:
com.google.api.client.auth.oauth2.TokenResponseException: 400 Bad Request
{
"error" : "invalid_grant",
"error_description" : "Code not valid"
}
Keycloak service:
[Unit]
Description=Keycloak
After=network.target
[Service]
Type=idle
User=keycloak
Group=keycloak
ExecStart=/opt/keycloak/current/bin/standalone.sh --server-config=standalone-ha.xml -b 0.0.0.0
TimeoutStartSec=600
TimeoutStopSec=600
[Install]
WantedBy=multi-user.target
Screenshots of my client setup within keycloak:
My configuration in Grafana:
GF_AUTH_GENERIC_OAUTH_ENABLED=True
GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=True
GF_AUTH_GENERIC_OAUTH_NAME=KeyCloakOAuth
GF_AUTH_GENERIC_OAUTH_CLIENT_ID=grafana
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=MYSECRETKEY
GF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile
GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://load_balanced_example/auth/realms/CICD/protocol/openid-connect/auth
GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://load_balanced_example.com/auth/realms/CICD/protocol/openid-connect/token
GF_AUTH_GENERIC_OAUTH_API_URL=https://load_balanced_example.com/auth/realms/CICD/protocol/openid-connect/userinfo
GF_AUTH_GENERIC_OAUTH_TLS_SKIP_VERIFY_INSECURE=True
GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH=contains(roles[*], 'Admin') && 'Admin' || contains(roles[*], 'Editor') && 'Editor' || 'Viewer'
Some of my standalone-ha.xml config:
<subsystem xmlns="urn:jboss:domain:datasources:5.0">
<datasources>
<datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true" statistics-enabled="${wildfly.datasources.statistics-enabled:${wildfly.statistics-enabled:false}}">
<connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>
<driver>h2</driver>
<security>
<user-name>sa</user-name>
<password>sa</password>
</security>
</datasource>
<datasource jndi-name="java:jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true" use-java-context="true" statistics-enabled="${wildfly.datasources.statistics-enabled:${wildfly.statistics-enabled:false}}">
<connection-url>jdbc:mariadb://example_db.com:3306/keycloak</connection-url>
<driver>mariadb</driver>
<security>
<user-name>keycloak</user-name>
<password>mydbpassword</password>
</security>
</datasource>
<drivers>
<driver name="h2" module="com.h2database.h2">
<xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
</driver>
<driver name="mariadb" module="org.mariadb">
<xa-datasource-class>org.mariadb.jdbc.MariaDbDataSource</xa-datasource-class>
</driver>
</drivers>
</datasources>
<server name="default-server">
<ajp-listener name="ajp" socket-binding="ajp"/>
<http-listener name="default" socket-binding="http" proxy-address-forwarding="true" redirect-socket="proxy-https"/>
<https-listener name="https" socket-binding="https" security-realm="ApplicationRealm" enable-http2="true"/>
<host name="default-host" alias="localhost">
<location name="/" handler="welcome-content"/>
<http-invoker security-realm="ApplicationRealm"/>
</host>
</server>
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
<socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
<socket-binding name="http" port="${jboss.http.port:8080}"/>
<socket-binding name="https" port="${jboss.https.port:8443}"/>
<socket-binding name="jgroups-mping" interface="private" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
<socket-binding name="jgroups-tcp" interface="private" port="7600"/>
<socket-binding name="jgroups-tcp-fd" interface="private" port="57600"/>
<socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
<socket-binding name="jgroups-udp-fd" interface="private" port="54200"/>
<socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
<socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
<socket-binding name="modcluster" multicast-address="${jboss.modcluster.multicast.address:224.0.1.105}" multicast-port="23364"/>
<socket-binding name="txn-recovery-environment" port="4712"/>
<socket-binding name="txn-status-manager" port="4713"/>
<socket-binding name="proxy-https" port="443"/>
<outbound-socket-binding name="mail-smtp">
<remote-destination host="localhost" port="25"/>
</outbound-socket-binding>
Another thing that might be worth pointing out is I've setup the keycloak servers behind a apache reverse proxy and pointed the load balancer to that, and I've taken apache out of the equation and had the load balancer point directly to the keycloak service on port 8443.
I've also verified from the team doing the load balancing that X-Forwarded-For and X-Forwarded-Proto are enabled and they are forwarding the client IP. If there are other settings to look for on the LB, let me know.
If anyone could help point me in the right direction that would be great! If there is a log that wasn't included or a config that could help troubleshoot, please let me know.

Answer 1

I got it working. The issue was with the clustering of the two nodes. I had to configure the jgroups to use TCP instead of UDP and added TCPPING.
<subsystem xmlns="urn:jboss:domain:jgroups:7.0">
<channels default="ee">
<channel name="ee" stack="tcp" cluster="ejb"/>
</channels>
<stacks>
<stack name="udp">
<transport type="UDP" socket-binding="jgroups-udp"/>
<protocol type="PING"/>
<protocol type="MERGE3"/>
<socket-protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG3"/>
</stack>
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<protocol type="TCPPING">
<property name="initial_hosts">server1_IP[7600],server2_IP[7600]</property>
<property name="ergonomics">false</property>
</protocol>
<protocol type="MERGE3"/>
<protocol type="MERGE3"/>
<socket-protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="MFC"/>
<protocol type="FRAG3"/>
</stack>
</stacks>
</subsystem>
I then had to change the service to bind my private IP.
[Unit]
Description=Keycloak
After=network.target
[Service]
Type=idle
User=keycloak
Group=keycloak
ExecStart=/opt/keycloak/current/bin/standalone.sh --server-config=standalone-ha.xml -b 0.0.0.0 -bprivate PRIVATEIP
TimeoutStartSec=600
TimeoutStopSec=600
[Install]
WantedBy=multi-user.target