Kubernetes apiserver TLS handshake error with webhook for authentication - authentication

Question

Iam trying to perform kubernetes webhook authentication using pamhook. But I get a TLS handshake error at the webhook when apiserver tries to contact it.
Following are the steps I followed:
apiserver IP: 192.168.20.30
pam webhook server IP: 192.168.20.50
Created a server certificate for my pam webhook server using the ca.crt present in /etc/kubernetes/pki/ (in master node)
> openssl genrsa -out ubuntuserver.key 2048 openssl req -new -key
> ubuntuserver.key -out ubuntuserver.csr -config myconf.conf openssl
> x509 -req -in ubuntuserver.csr -CA ca.crt -CAkey ca.key
> -CAcreateserial -out ubuntuserver.crt -days 10000
Started my pam webhook server using this certificate.
./pam_hook-master -cert-file /root/newca/ubuntuserver.crt -key-file /root/newca/ubuntuserver.key -signing-key rootroot -bind-port 6000
I1109 07:21:41.388836 3882 main.go:327] Starting pam_hook on :6000
Created a kubeconfig file as follows:
$cat webhook-config.yaml
apiVersion: v1
clusters:
- cluster:
certificate-authority: /etc/kubernetes/pki/ca.crt
server: https://192.168.20.50:6000/authenticate
name: 192.168.20.50
users:
- name: root
user:
client-certificate: /etc/kubernetes/pki/client.crt
client-key: /etc/kubernetes/pki/client.key
current-context: 192.168.20.50
contexts:
- context:
cluster: 192.168.20.50
user: root
name: 192.168.20.50
Configured api server manifest file in
/etc/kubernetes/manifest/kube-apiserver.yaml
> ...
> - --authentication-token-webhook-config-file=/etc/kubernetes/pki/webhook-config.yaml
> - --runtime-config=authorization.k8s.io/v1beta1=true ...
apiserver gets restarted.
Obtain token by requesting pamhook server. (I did this from my master node)
$ curl https://192.168.20.50:6000/token --cacert /etc/kubernetes/pki/ca.crt -u root
Enter host password for user 'root':
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiIiLCJleHAiOjE1MTAyMjY3ODksImlhdCI6MTUxMDIyNjE4OSwiaXNzIjoiIiwidXNlcm5hbWUiOiJyb290In0.3LmHBy_anjR62WNqKICCx_b8YWFpF4HSKMWLmyORU0M
Made a request to apiserver using this token, which in turn should communicate with webhook to provide authentication. But Iam getting a 401 Error.
$ curl -vvv --insecure -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiIiLCJleHAiOjE1MTAyMjY3ODksImlhdCI6MTUxMDIyNjE4OSwiaXNzIjoiIiwidXNlcm5hbWUiOiJyb290In0.3LmHBy_anjR62WNqKICCx_b8YWFpF4HSKMWLmyORU0M" https://192.168.20.38:6443/api/
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "Unauthorized",
"reason": "Unauthorized",
"code": 401
}
Message at the webhook server is:
> 2017/11/09 07:50:53 http: TLS handshake error from 192.168.20.38:49712: remote error: tls: bad certificate
Document says that api server sends a http request with the token in its payload. If I try to recreate the same call in curl using the token and ca.crt, it gets authenticated.
> $ curl -X POST https://192.168.20.50:6000/authenticate --cacert /etc/kubernetes/pki/ca.crt -d '{"ApiVersion":"authentication.k8s.io/v1beta1", "Kind": "TokenReview", "Spec":{"Token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiIiLCJleHAiOjE1MTAyMjY3ODksImlhdCI6MTUxMDIyNjE4OSwiaXNzIjoiIiwidXNlcm5hbWUiOiJyb290In0.3LmHBy_anjR62WNqKICCx_b8YWFpF4HSKMWLmyORU0M"}}'
{"apiVersion":"authentication.k8s.io/v1beta1","kind":"TokenReview","status":{"authenticated":true,"user":{"username":"root","uid":"0","groups":["root"]}}}
But when it is requested by apiserver, TLS handshake gets failed.
My understanding is that the TLS verification is done by checking against the certificate-authority file mentioned in webhook-config.yaml right? If so, the TLS verification should have been successful with ca.crt. But it is failing.
Does that mean api server is performing validation using some other CA? Which CA does it use? How do I go past this TLS verification successfully?

Answer 1

Finally I've fixed this. The mistake I've made is I used IP of the webhook server throughout (192.168.20.50 in this case). I replaced it with the FQDN of the webhook server machine and things worked out.
I changed IP to FQDN in the following places:
CN while generating the server certificate
-server field in the kubeconfig file
$cat webhook-config.yaml
apiVersion: v1
clusters:
- cluster:
certificate-authority: /etc/kubernetes/pki/ca.crt
server: https://ubuntuserver:6000/authenticate
name: 192.168.20.50
users:
- name: root
user:
client-certificate: /etc/kubernetes/pki/client.crt
client-key: /etc/kubernetes/pki/client.key
current-context: 192.168.20.50
contexts:
- context:
cluster: 192.168.20.50
user: root
name: 192.168.20.50