Hyperledger Fabric - Peer unable to connect to (raft) Orderer with Mutual TLS - ssl

Question

I am running a HLF on kubernetes - (3 raft orderers & 2 peers)
Now as raft requires Mutual TLS I had to setup some certificates.
The 3 raft orderers are able to communicate with eachother, as they are electing a leader, and re-electing another leader when I bring that leader down.
When I setup the peer, I used the same CA to generate the certificates. I am able to create the channel & join it from the peer. However I have to run CORE_PEER_MSPCONFIGPATH=$ADMIN_MSP_PATH prior to those commands, otherwise I get Access Denied error.
I am also forced to append the following flags to every peer channel x command I run.
--tls --cafile $ORD_TLS_PATH/cacert.pem --certfile $CORE_PEER_TLS_CLIENTCERT_FILE --keyfile $CORE_PEER_TLS_CLIENTKEY_FILE --clientauth
I am able to create, fetch, join the channel using the admin msp.
Now once the channel is joined, the peer is unable to connect with the orderer, somehow a bad certificate is given.
Orderer Logs
A bad certificate is used ?
2019-08-15 16:07:55.699 UTC [core.comm] ServerHandshake -> ERRO 221 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=10.130.2.148:53922
2019-08-15 16:07:55.699 UTC [grpc] handleRawConn -> DEBU 222 grpc: Server.Serve failed to complete security handshake from "10.130.2.148:53922": remote error: tls: bad certificate
Peer Logs
These suggest that it could not validate it with the ca.crt ?
2019-08-15 16:10:17.990 UTC [grpc] DialContext -> DEBU 03a parsed scheme: ""
2019-08-15 16:10:17.990 UTC [grpc] DialContext -> DEBU 03b scheme "" not registered, fallback to default scheme
2019-08-15 16:10:17.991 UTC [grpc] watcher -> DEBU 03c ccResolverWrapper: sending new addresses to cc: [{orderer-2.hlf-orderers.svc.cluster.local:7050 0 <nil>}]
2019-08-15 16:10:17.991 UTC [grpc] switchBalancer -> DEBU 03d ClientConn switching balancer to "pick_first"
2019-08-15 16:10:17.991 UTC [grpc] HandleSubConnStateChange -> DEBU 03e pickfirstBalancer: HandleSubConnStateChange: 0xc00260b710, CONNECTING
2019-08-15 16:10:18.009 UTC [grpc] createTransport -> DEBU 03f grpc: addrConn.createTransport failed to connect to {orderer-2.hlf-orderers.svc.cluster.local:7050 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority". Reconnecting...
2019-08-15 16:10:18.012 UTC [grpc] HandleSubConnStateChange -> DEBU 040 pickfirstBalancer: HandleSubConnStateChange: 0xc00260b710, TRANSIENT_FAILURE
2019-08-15 16:10:18.991 UTC [grpc] HandleSubConnStateChange -> DEBU 041 pickfirstBalancer: HandleSubConnStateChange: 0xc00260b710, CONNECTING
2019-08-15 16:10:19.003 UTC [grpc] createTransport -> DEBU 042 grpc: addrConn.createTransport failed to connect to {orderer-2.hlf-orderers.svc.cluster.local:7050 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority". Reconnecting...
2019-08-15 16:10:19.003 UTC [grpc] HandleSubConnStateChange -> DEBU 043 pickfirstBalancer: HandleSubConnStateChange: 0xc00260b710, TRANSIENT_FAILURE
2019-08-15 16:10:20.719 UTC [grpc] HandleSubConnStateChange -> DEBU 044 pickfirstBalancer: HandleSubConnStateChange: 0xc00260b710, CONNECTING
2019-08-15 16:10:20.731 UTC [grpc] createTransport -> DEBU 045 grpc: addrConn.createTransport failed to connect to {orderer-2.hlf-orderers.svc.cluster.local:7050 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority". Reconnecting...
2019-08-15 16:10:20.733 UTC [grpc] HandleSubConnStateChange -> DEBU 046 pickfirstBalancer: HandleSubConnStateChange: 0xc00260b710, TRANSIENT_FAILURE
2019-08-15 16:10:20.990 UTC [ConnProducer] NewConnection -> ERRO 047 Failed connecting to {orderer-2.hlf-orderers.svc.cluster.local:7050 [OrdererMSP]} , error: context deadline exceeded
I generated the used certificates as follows:
Orderer Admin
fabric-ca-client enroll -u https://u:p#ca.example.com -M ./OrdererMSP
Orderer Node X
As I use the same certificates for TLS I added the used hosts here for TLS purposes
orderer-x.hlf-orderers.svc.cluster.local #kubernetes
orderer-x.hlf-orderers #kubernetes
orderer-x #kubernetes
localhost #local debug
fabric-ca-client enroll -m orderer-x \
-u https://ox:px#ca.example.com \
--csr.hosts orderer-x.hlf-orderers.svc.cluster.local,orderer-x.hlf-orderers,orderer-x,localhost \
-M orderer-x-MSP
Peer Admin
fabric-ca-client enroll -u https://u:p#ca.example.com -M ./PeerMSP
Peer Node X
fabric-ca-client enroll -m peer-x \
-u https://ox:px#ca.example.com \
--csr.hosts peer-x.hlf-peers.svc.cluster.local,peer-x.hlf-peers,peer-x,localhost \
-M peer-x-MSP
Now all of these, have the same ca.crt (/cacerts/ca.example.com.pem)
configtx.yaml
Orderer:
<<: *OrdererDefaults
OrdererType: etcdraft
EtcdRaft:
Consenters:
- Host: orderer-1.hlf-orderers.svc.cluster.local
Port: 7050
ClientTLSCert: orderer-1-MSP/signcerts/cert.pem
ServerTLSCert: orderer-1-MSP/signcerts/cert.pem
- Host: orderer-2.hlf-orderers.svc.cluster.local
Port: 7050
ClientTLSCert: orderer-2-MSP/signcerts/cert.pem
ServerTLSCert: orderer-2-MSP/signcerts/cert.pem
- Host: orderer-3.hlf-orderers.svc.cluster.local
Port: 7050
ClientTLSCert: orderer-3-MSP/signcerts/cert.pem
ServerTLSCert: orderer-3-MSP/signcerts/cert.pem
Addresses:
- orderer-1.hlf-orderers.svc.cluster.local:7050
- orderer-2.hlf-orderers.svc.cluster.local:7050
- orderer-3.hlf-orderers.svc.cluster.local:7050
I have checked multiple times if the correct certificates are mounted on the correct places and configured.
On the peer side I made sure that:
CORE_PEER_TLS_CLIENTROOTCAS_FILES is set correctly and that the (correct) file gets mounted (CORE_PEER_TLS_CLIENTROOTCAS_FILES: "/var/hyperledger/tls/client/cert/ca.crt")
Idem for CORE_PEER_TLS_CLIENTKEY_FILE & CORE_PEER_TLS_CLIENTCERT_FILE
CORE_PEER_TLS_CLIENTAUTHREQUIRED is set to true
On the orderer side I made sure that:
ORDERER_GENERAL_TLS_CLIENTAUTHREQUIRED is set to true
ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE is set correctly
ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY is set correctly
ORDERER_GENERAL_TLS_CLIENTROOTCAS is set correctly
It seems strange to me that the orderers are able to talk to eachother (as they are electing leaders), but that the peer is not able to do so

Answer 1

So it appears to be, that the tlscacerts should be in the msp(s) directory(ies) PRIOR to creating genesis / channel block. Simply mounting them in the pod at runtime is not enough
My msp directories (used in configtx.yaml) look like:
admincerts
tlscacerts
cacerts
...
After this it all started to work

Answer 2

seems like you have got below error
E0923 16:30:14.963567129 31166 ssl_transport_security.cc:989] Handshake failed with fatal error SSL_ERROR_SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate.
E0923 16:30:15.964456710 31166 ssl_transport_security.cc:188] ssl_info_callback: error occured.
According to your details, All seems to be correct
However check below
certificate signed by unknown authority -> This makes me bit doubt on your certificate mapping
MAKE SURE
PEER:
CORE_PEER_TLS_ENABLED=true
CORE_PEER_TLS_CERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/tls/server.crt
CORE_PEER_TLS_KEY_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/tls/server.key
CORE_PEER_TLS_ROOTCERT_FILE=/data/maersksea-rca-maersksea-chain.pem
CORE_PEER_TLS_CLIENTCERT_FILE=/data/tls/maersksea-peer-maersksea-client.crt
CORE_PEER_TLS_CLIENTKEY_FILE=/data/tls/maersksea-peer-maersksea-client.key
CORE_PEER_TLS_CLIENTAUTHREQUIRED=true
CORE_PEER_TLS_CLIENTROOTCAS_FILES=/data/maersksea-rca-maersksea-chain.pem
Orderer:
ORDERER_GENERAL_TLS_CLIENTAUTHREQUIRED=true
ORDERER_GENERAL_TLS_CLIENTROOTCAS=[/data/maersksea-rca-maersksea-chain.pem]