I'm trying to connect to my company's LDAP via SSL using GlassFish. Everything works as expected, except that sometimes connecting takes a very long time.
After enabling the ssl debugging, I found that there is sometimes, but not always, a 30 second delay in the Change Cipher Spec. An example of the delay is shown below.
[2020-06-18T09:11:51.806+0100] [glassfish 4.1] [INFO] [] [] [tid: _ThreadID=32 _ThreadName=Thread-8] [timeMillis: 1592467911806] [levelValue: 800] [[
http-listener-1(1), WRITE: TLSv1.2 Handshake, length = 40]]
[2020-06-18T09:12:22.030+0100] [glassfish 4.1] [INFO] [] [] [tid: _ThreadID=32 _ThreadName=Thread-8] [timeMillis: 1592467942030] [levelValue: 800] [[
http-listener-1(1), READ: TLSv1.2 Change Cipher Spec, length = 1]]
[2020-06-18T09:12:22.030+0100] [glassfish 4.1] [INFO] [] [] [tid: _ThreadID=32 _ThreadName=Thread-8] [timeMillis: 1592467942030] [levelValue: 800] [[
http-listener-1(1), READ: TLSv1.2 Handshake, length = 40]]
Here is the code used to connect to the LDAP
final Hashtable<String, String> env = new Hashtable<String, String> ();
final String url = "ldaps://" + ldapHostAddress + ":" + ldapPort;
env.put (Context.SECURITY_PROTOCOL, "ssl");
env.put (Context.PROVIDER_URL, url);
env.put (Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.ldap.LdapCtxFactory");
env.put (Context.SECURITY_AUTHENTICATION, "simple");
env.put (Context.SECURITY_PRINCIPAL, principalDN);
env.put (Context.SECURITY_CREDENTIALS, principalPassword);
// Ensure the objectGUID is handled as a binary object, rather than a string.
env.put ("java.naming.ldap.attributes.binary", "objectGUID");
LdapContext connection = new InitialLdapContext (env, null);
Running this code from the command line does not appear to suffer from the 30 delay, so I can only assume it's an issue with GlassFish. Any suggestions would be appreciated.
Turns out that GlassFish was occasionally having problems resolving the DNS name of the LDAP server. Replacing the DNS name with the IP address removed the delay when performing the handshake.
Related
Been almost two months and I cannot figure out how to make the following logs to parse. Challenges faced:
There are double quotes around the logs and the format of the logs is not very consistent
Many tab and odd spaces between the logs
Appreciate any guide on how to start
"[5/10/22 16:07:39:393 GTS] 00000330 SystemErr R at com.ibm.mdr.DrStateMgr.eventFromUser(DrStateMgr.java:2952)"
"[5/10/22 16:07:39:393 GTS] 00000330 SystemErr R at com.ibm.mdr.DrStateMgr.dequeueAndFireEvents(DrStateMgr.java:5010)"
[5/10/22 16:03:49:982 GTS] 000000a4 WebContainer E com.ibm.ws.webcontainer.internal.WebContainer handleRequest TEST_SERVER: A WebGroup/Virtual Host to handle / has not been defined.
[5/8/22 6:43:42:236 GTS] 00000001 SSLConfigMana W AAPKI0003A: The runtime has at least one SSL configuration that supports only weak TLSv1 or TLSv1.1 handshake protocols. For increased security, modify the configuration to use only stronger protocols such as TLSv1.2 or later. Find instructions to update your configuration at https://www.ibm.com/support/pages/node/1077951. SSL configurations that use the weaker SSL protocols include: [XDADefaultSSLSettings((cell):AFDJP01PCell01)].
[5/8/22 6:43:42:220 GTS] 00000001 WSKeyStore W SSPKI0002A: One or more key stores are using the default password.
[5/8/22 6:43:42:204 GTS] 00000001 SSLConfigMana I DDPKI0004A: The process has the java security property jdk.tls.disabledAlgorithms set to [SSLv3, TLSv1, TLSv1.1, RC4, DES, MD5withRSA, DH keySize < 1024, DESede, EC keySize < 224, 3DES_EDE_CBC, anon, NULL, DES_CBC]. The WebSphere Application server is setting the java security property jdk.tls.disabledAlgorithms to [SSLv3, RC4, DH keySize < 768, MD5withRSA].
[5/8/22 6:43:42:204 GTS] 00000001 SSLConfigMana I DDPKI0004A: The process has the java security property jdk.certpath.disabledAlgorithms set to [MD2, MD5, SHA1 jdkCA & usage TLSServer, RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224]. The WebSphere Application server is setting the java security property jdk.certpath.disabledAlgorithms to [MD2, RSA keySize < 1024, MD5].
[5/8/22 6:43:42:204 GTS] 00000001 FIPSManager I EEPKI0005A: FIPS security mode is : No FIPS property found.
[5/8/22 6:43:42:204 GTS] 00000001 SSLConfigMana I GGPKI0007A: The SSL configuration is initializing.
[5/8/22 6:43:42:189 GTS] 00000001 SSLComponentI I HHPKI0008A: SSL service is initializing the configuration
[5/8/22 6:43:42:095 GTS] 00000001 PluginConfigS I PLGC0044B: The plug-in configuration service started successfully.
[5/8/22 6:43:41:345 GTS] 00000001 AdminInitiali A ADMN0054E: The administration service is initialized.
[5/8/22 6:43:41:048 GTS] 00000001 ProviderTrack I com.ibm.ffdc.osgi.ProviderTracker AddingService FFDC1007I: FFDC Provider Installed: com.ibm.ws.ffdc.impl.FfdcProvider#ed46329b
[5/8/22 6:43:40:908 GTS] 00000001 ComponentMeta I ASVR0150U: The runtime provisioning feature is disabled. All components will be started.
[5/8/22 6:43:39:923 GTS] 00000001 ModelMgr I ASVR0180U: Initializing core configuration models
[5/8/22 6:43:39:783 GTS] 00000001 ManagerAdmin I TRAS0555T: The message IDs that are in use are deprecated
[5/8/22 6:43:39:783 GTS] 00000001 ManagerAdmin I TRAS0787K: The startup trace state is *=info.
"[5/8/22 7:37:18:809 GTS] FFDC Exception:java.io.FileNotFoundException SourceId:com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters -IOE ProbeId:1044
java.io.FileNotFoundException: DEAV0180D: File not found: /favicon.ico
at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor._processEDR(DefaultExtensionProcessor.java:977)
at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.processEDR(DefaultExtensionProcessor.java:958)
at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.handleRequest(DefaultExtensionProcessor.java:486)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1114)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:4075)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1019)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:213)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:287)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1187)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:694)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)
Reporter:null"
Expected output
{
"month": [
[
"5"
]
],
"day": [
[
"10"
]
],
"year": [
[
"22"
]
],
"time": [
[
"16:03:49:982"
]
],
"instance": [
[
"000000a4"
]
]
"process": [
[
"WebContainer E com.ibm.ws.webcontainer.internal.WebContainer handleRequest TEST_SERVER: A WebGroup/Virtual Host to handle / has not been"
]
]
"server": [
[
"TEST_SERVER"
]
]
"error": [
[
"A WebGroup/Virtual Host to handle / has not been"
]
]
}
Grok pattern in use
\[%{MONTHNUM:month}\/%{MONTHDAY:day}\/%{YEAR:year} %{TIME:time} GTS\] %{GREEDYDATA:host}
For this multiple pattern in a single log you can try like the below. if any of grok parsing failed it might be due to extra space in between hence constructed it and add those patterns in this.
filter
{
grok
{
match => {"message" => ["\[%{MONTHNUM:month}\/%{MONTHDAY:day}\/%{YEAR:year} %{TIME:time} GTS\] %{DATA:host} %{WORD:source} %{WORD:logtype} %{DATA:lib} %{WORD:request_type} %{DATA:server}: %{GREEDYDATA:detailed_message}","\[%{MONTHNUM:month}\/%{MONTHDAY:day}\/%{YEAR:year} %{TIME:time} GTS\] %{DATA:host} %{WORD:source} %{WORD:logtype} %{DATA:code}: %{GREEDYDATA:detailed_message}"]}
}
}
However, the above pattern would only work single line for W,R,I,A log type but it won't work in multi line
i.e.,
"[5/8/22 7:37:18:809 GTS] FFDC Exception:java.io.FileNotFoundException SourceId:com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters -IOE ProbeId:1044
java.io.FileNotFoundException: DEAV0180D: File not found: /favicon.ico
at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor._processEDR(DefaultExtensionProcessor.java:977)
at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.processEDR(DefaultExtensionProcessor.java:958)
at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.handleRequest(DefaultExtensionProcessor.java:486)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1114)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:4075)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1019)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:213)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:287)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1187)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:694)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)
Reporter:null"
Keep posted on how it goes !!! Thanks !!!
I am new to KFServing and Kubeflow.
I was following https://github.com/kubeflow/kfserving/tree/master/docs/samples/v1alpha2/tensorflow to deploy a simple inference service.
However, when looking at the logs, I am unable to find the container storage-initializer. The only containers my predict service pod has are kfserving and queue-proxy.
I am currently on Kubeflow 1.2 and Kubernetes 1.17 on IBM Cloud.
Error Message Image
storage-initializer is an init container, so if you describe the pod you won't find it in the containers section of pod spec but in the initContainers section.
$ kubectl get pod flowers-sample-predictor-default-00002-deployment-58bb9557sf7g2 -o json | jq .status.initContainerStatuses
[
{
"containerID": "docker://e40e5f86401b3715118b873fec4ae6c3ef57765ffbb5c9ab48757234c4f53b6f",
"image": "gcr.io/kfserving/storage-initializer:v0.5.0",
"imageID": "docker-pullable://gcr.io/kfserving/storage-initializer#sha256:1d396c0c50892f5562a1c24d925691ec786e5d48e08200f3f9bb17bb48da40ae",
"lastState": {},
"name": "storage-initializer",
"ready": true,
"restartCount": 0,
"state": {
"terminated": {
"containerID": "docker://e40e5f86401b3715118b873fec4ae6c3ef57765ffbb5c9ab48757234c4f53b6f",
"exitCode": 0,
"finishedAt": "2021-02-27T20:13:25Z",
"reason": "Completed",
"startedAt": "2021-02-27T20:13:11Z"
}
}
}
]
I'm not familiar with the model label you are using, can you retry by using the app label or the pod name directly?
$ kubectl logs -l app=flowers-sample-predictor-default-00002 -c storage-initializer
[I 210227 20:13:12 initializer-entrypoint:13] Initializing, args: src_uri [gs://kfserving-samples/models/tensorflow/flowers] dest_path[ [/mnt/models]
[I 210227 20:13:12 storage:43] Copying contents of gs://kfserving-samples/models/tensorflow/flowers to local
[W 210227 20:13:15 _metadata:104] Compute Engine Metadata server unavailable onattempt 1 of 3. Reason: timed out
[W 210227 20:13:15 _metadata:104] Compute Engine Metadata server unavailable onattempt 2 of 3. Reason: [Errno 113] No route to host
[W 210227 20:13:18 _metadata:104] Compute Engine Metadata server unavailable onattempt 3 of 3. Reason: timed out
[W 210227 20:13:18 _default:250] Authentication failed using Compute Engine authentication due to unavailable metadata server.
[I 210227 20:13:19 storage:127] Downloading: /mnt/models/0001/saved_model.pb
[I 210227 20:13:19 storage:127] Downloading: /mnt/models/0001/variables/variables.data-00000-of-00001
[I 210227 20:13:25 storage:127] Downloading: /mnt/models/0001/variables/variables.index
[I 210227 20:13:25 storage:76] Successfully copied gs://kfserving-samples/models/tensorflow/flowers to /mnt/models
when I execute this command to generate kubernetes certificate:
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
kubernetes-csr.json | cfssljson -bare kubernetes
Why the cfssl took shows:
[root#iZuf63refzweg1d9dh94t8Z ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem \
> -config=ca-config.json \
> -profile=kubernetes \
> kubernetes-csr.json | cfssljson -bare kubernetes
2019/08/25 20:02:12 [INFO] generate received request
2019/08/25 20:02:12 [INFO] received CSR
2019/08/25 20:02:12 [INFO] generating key: rsa-2048
2019/08/25 20:02:13 [INFO] encoded CSR
2019/08/25 20:02:13 [INFO] signed certificate with serial number 540759253485135214776496461610290604881680785507
2019/08/25 20:02:13 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
this is my kubernetes(kubernetes-csr.json) config:
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"172.19.104.230",
"172.19.150.82",
"172.19.104.231"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
obviously it contains hosts field. I am using cfssl version 1.2 .Is this a bug?
update cfssl version from v1.2 to v1.3.4(latest version):
go get -u github.com/cloudflare/cfssl/cmd/cfssl
I am trying pass HTTP from Oracle HTTP (OHS) to External, works via non-ssl, However, SSL is a bit challenging "OHS---NON-SSL--->External System(ssl)", and via SSL I am seeing " " error need some light on this.
Errors seen in ohs logs:
[] [OHS] [ERROR:32] [] [core.c] [host_id: xxxxxxxxx.xxxx.xxxx] [host_addr: aa.aa.a.aaa] [tid: ] [user: oracle] [ecid:] [rid: 0] [VirtualHost: zzzzzzzzz:bbbb] NZ Library Error: SSL fatal alert
[] [OHS] [ERROR:32] [] [core.c] [host_id: xxxxxxxxx.xxxx.xxxx] [host_addr: aa.aa.a.aaa] [tid: ] [user: oracle] [ecid: ] [rid: 0] [VirtualHost: zzzzzzzzz:bbbb] (20014)Internal error: : pass body failed to xx.xx.x.xxx:cccc (dddddddddddd.ddd.dddd)
[] [OHS] [ERROR:32] [] [core.c] [host_id: xxxxxxxxx.xxxx.xxxx] [host_addr: aa.aa.a.aaa] [tid: ] [user: oracle] [ecid: ] [rid: 0] [VirtualHost: zzzzzzzzz:bbbb] : pass body failed to xx.xx.x.xxx:cccc (dddddddddddd.ddd.dddd) from yy.yy.yy.yyy ()
Pass used in
<IfModule mod_proxy.c>
SSLProxyEngine On
SSLPROXYWALLET "/zzz/zzzzzz/zzzzzzzzWallet"
ProxyPass / https://###.###.#####.######:XXXX/sss/aaa.sddd.sss.dddd/XXXX
ProxyPassReverse / https://###.###.#####.######:XXXX/sss/aaa.sddd.sss.dddd/XXXX
ProxyPreserveHost On
ProxyRequests off
</IfModule>
I think port 443 has been blocked in firewall level..just check whether port is opened or not by using netstat command.
First of all I'd like to mention, that my setup works like a charm when there's no TLS enabled. It works even in Docker Swarm on AWS.
The problem starts when I enable TLS. When I deploy my .bna file via Composer, my newly created chaincode container produces the following logs:
2017-08-23 13:14:16.389 UTC [Composer] Info -> INFO 001 Setting the Composer pool size to 8
2017-08-23 13:14:16.402 UTC [shim] userChaincodeStreamGetter -> ERRO 002 Error trying to connect to local peer: x509: certificate signed by unknown authority
Error starting chaincode: Error trying to connect to local peer: x509: certificate signed by unknown authority
Funny thing is, that this works when deploying .bna via the composer playground (when the TLS is still enabled in my fabric)...
Below is my connection profile:
{
"name": "test",
"description": "test",
"type": "hlfv1",
"orderers": [
{
"url": "grpcs://orderer.company.com:7050",
"cert": "-----BEGIN CERTIFICATE-----blabla1\n-----END CERTIFICATE-----\n"
}
],
"channel": "channelname",
"mspID": "CompanyMSP",
"ca": {
"url": "https://ca.company.com:7054",
"name": "ca-company",
"trustedRoots": [
"-----BEGIN CERTIFICATE-----\nblabla2\n-----END CERTIFICATE-----\n"
],
"verify": true
},
"peers": [
{
"requestURL": "grpcs://peer0.company.com:7051",
"eventURL": "grpcs://peer0.company.com:7053",
"cert": "-----BEGIN CERTIFICATE-----\nbalbla3\n-----END CERTIFICATE-----\n"
}
],
"keyValStore": "/home/composer/.composer-credentials",
"timeout": 300
}
My certs have been generated by cryptogen tool, hence:
orderers.0.cert contains value of crypto-config/ordererOrganizations/company.com/orderers/orderer.company.com/msp/tlscacerts/tlsca.company.com-cert.pem
peers.0.cert contains value of crypto-config/peerOrganizations/company.com/peers/peer0.company.com/msp/tlscacerts/tlsca.company.com-cert.pem
ca.trustedRoots.0 contains crypto-config/peerOrganizations/company.com/peers/peer0.company.com/tls/ca.crt
I've got the feeling, that my trustedRoots certificate is wrong...
UPDATE
When I do docker inspect chaincode_container I can see that it misses ENV variable: CORE_PEER_TLS_ROOTCERT_FILE=/etc/hyperledger/fabric/peer.crt, while the chaincode container deployed via playground does have it...
When the chaincode image is built, the TLS certificate that it uses to build the trusted roots is the rootcert from:
# TLS Settings
# Note that peer-chaincode connections through chaincodeListenAddress is
# not mutual TLS auth. See comments on chaincodeListenAddress for more info
tls:
enabled: false
cert:
file: tls/server.crt
key:
file: tls/server.key
rootcert:
file: tls/ca.crt
The TLS certificate that the peer uses to run the gRPC service is the cert one.
By the way - You're using the release branch code, not the one in master - is that correct?