Connecting a containerised LDAP-backed Nifi to a containerised Nifi Registry via third party SSL certificates - ssl

Question

Note: This is not a question, I'm providing information that may help others.
Hi all,
I recently spent way too much time beating my head against the keyboard trying to work out how to connect Nifi to a Nifi registry in a corporate environment. After eventually working it out I thought I'd post my findings here to save the next poor soul that comes along seeking help with Nifi and Nifi registry.
Apologies in advance for the long post, but I thought the details would be useful.
I had a requirement to setup containerised instances of Nifi and Nifi-registry, both backed by LDAP, leveraging corporate SSL certificates and using an internal Container registry (no direct Internet access). As of this morning, this is now working, here's an overview of how I got it to work on RHEL 8 servers:
In a corporate environment the hosts need SSL certs setup for HTTPS, and to ensure they can communicate securely.
SSL Cert setup
Generate SSL private keys for each host in a Java keystore on the respective machines
Generate CSRs from the keystores, with appropriate SANs as required
Get CSRs Signed - Ensure that the "Client Auth" and "Server Auth" Extended Key Usage attributes are set for the Nifi cert (This is required for Nifi to successfully connect to a Nifi Registry). The registry cert just needs the Server Auth attribute.
Import corporate CA chain into the keystores, to ensure full trust chain of the signed cert is resolvable
Create a Java keystore (truststore) containing the CA cert chain
I can provide further details of the above steps if needed
Now that we have some SSL certs, the steps to setup the containers were as follows:
Container setup
Install podman (or docker if you prefer)
For Podman - Update the /etc/containers/registries.conf to turn off the default container registries
For Podman - Update /usr/share/containers/libpod.conf to replace the path to the pause container with the path the container in our internal registry
Setup folders for the containers, ensuring they have an SELinux file context of "container_file_t", and have permissions of 1000:1000 (UID & GID of nifi user in the containers).
Setup an ENV file to define all of the environment variables to pass to the containers (there's a lot for Nifi and the Registry, they each share this info). This saves a lot of CLI parameters, and stops passwords appearing in the process list (note password encryption for nifi is possible, but not covered in this post).
KEYSTORE_PATH=/path/to/keystore.jks
TRUSTSTORE_PATH=/path/to/truststore.jks
KEYSTORE_TYPE=JKS
TRUSTSTORE_TYPE=JKS
KEYSTORE_PASSWORD=InsertPasswordHere
TRUSTSTORE_PASSWORD=InsertPasswordHere
LDAP_AUTHENTICATION_STRATEGY=LDAPS
LDAP_MANAGER_DN=CN=service account,OU=folder its in,DC=domain,DC=com
LDAP_MANAGER_PASSWORD=InsertPasswordHere
LDAP_TLS_KEYSTORE=/path/to/keystore.jks
LDAP_TLS_TRUSTSTORE=/path/to/truststore.jks
LDAP_TLS_KEYSTORE_TYPE=JKS
LDAP_TLS_TRUSTSTORE_TYPE=JKS
LDAP_TLS_KEYSTORE_PASSWORD=InsertPasswordHere
LDAP_TLS_TRUSTSTORE_PASSWORD=InsertPasswordHere
LDAP_TLS_PROTOCOL=TLSv1.2
INITIAL_ADMIN_IDENTITY=YourUsername
AUTH=ldap
LDAP_URL=ldaps://dc.domain.com:636
LDAP_USER_SEARCH_BASE=OU=user folder,DC=domain,DC=com
LDAP_USER_SEARCH_FILTER=cn={0}
LDAP_IDENTITY_STRATEGY=USE_USERNAME
Start both the Nifi & Nifi-Registry containers, and copy out the contents of their respective conf folders to the host (/opt/nifi-registry/nifi-registry-current/conf and /opt/nifi/nifi-current/conf). This allows us to customise and persist the configuration.
Modify the conf/authorizers.xml file for both Nifi and the Nifi-registry
to setup LDAP authentication, and add a composite auth provider (allowing both local & ldap users). We need both in order to add user locals accounts for any Nifi nodes connecting to the registry (can be done via LDAP, but is easier this way).
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizers>
<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<!--<property name="Initial User Identity 1"></property>-->
</userGroupProvider>
<userGroupProvider>
<identifier>ldap-user-group-provider</identifier>
<class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
<property name="Authentication Strategy">LDAPS</property>
<property name="Manager DN">CN=service account,OU=folder its in,DC=domain,DC=com</property>
<property name="Manager Password">InsertPasswordHere</property>
<property name="TLS - Keystore">/path/to/keystore.jks</property>
<property name="TLS - Keystore Password">InsertPasswordHere</property>
<property name="TLS - Keystore Type">JKS</property>
<property name="TLS - Truststore">/path/to/truststore.jks</property>
<property name="TLS - Truststore Password">InsertPasswordHere</property>
<property name="TLS - Truststore Type">jks</property>
<property name="TLS - Client Auth">WANT</property>
<property name="TLS - Protocol">TLS</property>
<property name="TLS - Shutdown Gracefully">true</property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldaps://dc.domain.com:636</property>
<property name="Page Size"/>
<property name="Sync Interval">30 mins</property>
<property name="User Search Base">OU=user folder,DC=domain,DC=com</property>
<property name="User Object Class">user</property>
<property name="User Search Scope">ONE_LEVEL</property>
<property name="User Search Filter"/>
<property name="User Identity Attribute">cn</property>
<property name="Group Search Base">OU=group folder,DC=domain,DC=com</property>
<property name="Group Object Class">group</property>
<property name="Group Search Scope">ONE_LEVEL</property>
<property name="Group Search Filter"/>
<property name="Group Name Attribute">cn</property>
<property name="Group Member Attribute">member</property>
<property name="Group Member Attribute - Referenced User Attribute"/>
</userGroupProvider>
<userGroupProvider>
<identifier>composite-user-group-provider</identifier>
<class>org.apache.nifi.authorization.CompositeConfigurableUserGroupProvider</class>
<property name="Configurable User Group Provider">file-user-group-provider</property>
<property name="User Group Provider 1">ldap-user-group-provider</property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">composite-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">YourUsername</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1">DN of Nifi Instance (OPTIONAL - more details on this later)</property>
<property name="Node Group"></property>
</accessPolicyProvider>
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers>
Performance Mod - Optional - Modify conf/bootstrap.conf to increase the Java Heap Size (if required). Also update Security limits (files & process limits).
Extract the OS Java keystore from the containers, and add the corporate cert chain to it. Note: Nifi and nifi-registry java keystores are in slightly different locations in the containers. I needed to inject CA certs into these keystores to ensure Nifi processors can resolve SSL trust chains (I needed this primarily for a number of custom nifi processors we wrote which interrogated LDAP).
Run the containers, mounting volumes for persistent data and include your certs folder and the OS Java keystores:
podman run --name nifi-registry \
--hostname=$(hostname) \
-p 18443:18443 \
--restart=always \
-v /path/to/certs:/path/to/certs \
-v /path/to/OS/Java/Keystore:/usr/local/openjdk-8/jre/lib/security/cacerts:ro \
-v /path/to/nifi-registry/conf:/opt/nifi-registry/nifi-registry-current/conf \
-v /path/to/nifi-registry/database:/opt/nifi-registry/nifi-registry-current/database \
-v /path/to/nifi-registry/extension_bundles:/opt/nifi-registry/nifi-registry-current/extension_bundles \
-v /path/to/nifi-registry/flow_storage:/opt/nifi-registry/nifi-registry-current/flow_storage \
-v /path/to/nifi-registry/logs:/opt/nifi-registry/nifi-registry-current/logs \
--env-file /path/to/.env/file \
-d \
corporate.container.registry/apache/nifi-registry:0.7.0
podman run --name nifi \
--hostname=$(hostname) \
-p 443:8443 \
--restart=always \
-v /path/to/certs:/path/to/certs \
-v /path/to/certs/cacerts:/usr/local/openjdk-8/lib/security/cacerts:ro \
-v /path/to/nifi/logs:/opt/nifi/nifi-current/logs \
-v /path/to/nifi/conf:/opt/nifi/nifi-current/conf \
-v /path/to/nifi/database_repository:/opt/nifi/nifi-current/database_repository \
-v /path/to/nifi/flowfile_repository:/opt/nifi/nifi-current/flowfile_repository \
-v /path/to/nifi/content_repository:/opt/nifi/nifi-current/content_repository \
-v /path/to/nifi/provenance_repository:/opt/nifi/nifi-current/provenance_repository \
-v /path/to/nifi/state:/opt/nifi/nifi-current/state \
-v /path/to/nifi/extensions:/opt/nifi/nifi-current/extensions \
--env-file /path/to/.env/file \
-d \
corporate.container.registry/apache/nifi:1.11.4
Note: Please ensure that the SELinux contexts (if applicable to your OS), and permissions (1000:1000) are correct for the mounted volumes prior to starting the containers.
Configuring the Containers
Browse to https://hostname.domain.com/nifi (we redirected 8443 to 443) and https://hostname2.domain.com:18443/nifi-registry
Login to both as the initial admin identity you provided in the config files
Add a new user account using the full DN of the SSL certificate, e.g. CN=machinename, OU=InfoTech, O=Big Company, C=US. This account is needed on both ends for Nifi & the registry to connect and getting the name correct is important. There's probably an easier way to determine the DN, but I reverse engineered after inspecting the cert in a browser. I took everything listed under the "Subject Name" heading and wrote it out from the bottom entry up.
Set permissions for the account in nifi, adding "Proxy User Request", "Access the controller (view)" and "Access the controller (modify)".
Set permissions for account in nifi registry, adding "Can proxy user request", "Read buckets".
Set other user/group permissions as needed
Setup and Connect to the Registry
Create a bucket in Nifi Registry
In Nifi (Controller Settings -> Registry Clients), add the url of the registry: https://hostname.domain.com:18443.
Select a Processor or Process group, right-click, Version -> Start Version Control
That should be it!
I found that Nifi is terrible at communicating errors when connecting to the registry. I got a range of errors whilst attempting to connect. The only way to get useful errors is to add a new entry to conf/bootstrap.conf on the nifi registry:
java.arg.XX=--Djavax.net.debug=ssl,handshake
After restarting the Nifi Registry container you should start seeing SSL debug information in logs/nifi-registry-bootstrap.log.
e.g. When Nifi was reporting "Unknown Certificate", the Nifi Registry debug logs contained:
INFO [NiFi logging handler] org.apache.nifi.registry.StdOut sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication
I hope this is helpful.