How to access Kubernetes UI via browser? - ssl

I have installed Kubernetes using contrib/ansible scripts.
When I run cluster-info:
[osboxes#kube-master-def ~]$ kubectl cluster-info
Kubernetes master is running at http://localhost:8080
Elasticsearch is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging
Heapster is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/heapster
Kibana is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kibana-logging
KubeDNS is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubedash is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kubedash
Grafana is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana
InfluxDB is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb
The cluster is exposed on localhost with insecure port, and exposed on secure port 443 via ssl
kube 18103 1 0 12:20 ? 00:02:57 /usr/bin/kube-controller-manager --logtostderr=true --v=0 --master=https://10.57.50.161:443 -- kubeconfig=/etc/kubernetes/controller-manager.kubeconfig --service-account-private-key-file=/etc/kubernetes/certs/server.key --root-ca-file=/etc/kubernetes/certs/ca.crt
kube 18217 1 0 12:20 ? 00:00:15 /usr/bin/kube-scheduler --logtostderr=true --v=0 --master=https://10.57.50.161:443 --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
root 27094 1 0 12:21 ? 00:00:00 /bin/bash /usr/libexec/kubernetes/kube-addons.sh
kube 27300 1 1 12:21 ? 00:05:36 /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd-servers=http://10.57.50.161:2379 --insecure-bind-address=127.0.0.1 --secure-port=443 --allow-privileged=true --service-cluster-ip-range=10.254.0.0/16 --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota --tls-cert-file=/etc/kubernetes/certs/server.crt --tls-private-key-file=/etc/kubernetes/certs/server.key --client-ca-file=/etc/kubernetes/certs/ca.crt --token-auth-file=/etc/kubernetes/tokens/known_tokens.csv --service-account-key-file=/etc/kubernetes/certs/server.crt
I have copied the certificates from kube-master machine to my local machine, I have installed the ca root certificate. The chrome/safari browsers are accepting the ca root certificate.
When I'm trying to access the https://10.57.50.161/ui
I'm getting the 'Unauthorized'
How can I access the kubernetes ui?

You can use kubectl proxy.
Depending if you are using a config file, via command-line run
kubectl proxy
or
kubectl --kubeconfig=kubeconfig proxy
You should get a similar response
Starting to serve on 127.0.0.1:8001
Now open your browser and navigate to
http://127.0.0.1:8001/ui/ (deprecated, see kubernetes/dashboard)
http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
You need to make sure the ports match up.

This works for me that you can access from network
kubectl proxy --address 0.0.0.0 --accept-hosts '.*'

Quick-n-dirty (and unsecure) way to access the Dashboard:
$ kubectl edit svc/kubernetes-dashboard --namespace=kube-system
This will load the Dashboard config (yaml) into an editor where you can edit it.
Change line type: ClusterIP to type: NodePort.
Get the tcp port:
$ kubectl get svc kubernetes-dashboard -o json --namespace=kube-system
The line with the tcp port will look like:
"nodePort": 31567
In newer releases of kubernetes you can get the nodeport from get svc:
# This is kubernetes 1.7:
donn#host37:~$ sudo kubectl get svc --namespace=kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard 10.3.0.234 <nodes> 80:31567/TCP 2h
Do kubectl describe nodes to get a node IP address.
Browse to:
http://NODE_IP:31567
Good for testing. Not good for production due to lack of security.

Looking at your apiserver configuration, you will need to either present a bearer token (valid tokens will be listed in /etc/kubernetes/tokens/known_tokens.csv) or client certificate (signed by the CA cert in /etc/kubernetes/certs/ca.crt) to prove to the apiserver that you should be allowed to access the cluster.
https://github.com/kubernetes/kubernetes/issues/7307#issuecomment-96130676 describes how I was able to configure client certificates for a GKE cluster on my Mac.
To pass bearer tokens, you need to pass an HTTP header Authorization with a value Bearer ${KUBE_BEARER_TOKEN}. You can see an example of how this is done with curl here; in a browser, you will need to install an add-on/plugin to pass custom headers.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
kubectl proxy &
Run the following command in your local laptop(or where you want to access the GUI)
ssh -L 8877:127.0.0.1:8001 -N -f -l root master_IPADDRESS
Get the secret key
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | awk '/^deployment-controller-token-/{print $1}') | awk '$1=="token:"{print $2}'
Open the dashboard
http://localhost:8877/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
Perform role-binding if you get any errors.

You can use kubectl proxy --address=clusterIP --port 8001 --accept-hosts '.*'

api server is already accessible on 6443 port on the node, but not authorize accesss to https://:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
i've generated client certificats signed by kubernetes ca cert, and converted to pkcs12 and integreted to my browser... when try to access to the this url they says that user are not authorized to access to the uri...

Related

Certbot unable to find AWS credentials when issuing certificate via dns for route53

I need to get an certificate for my domain hosted on AWS Route 53 from LetsEncrypt. I do not have any port 80 or 443 exposed since the server is used for VPN and does not have a public access.
So the only way to do this is via DNS validation of route 53.
So far I have installed certbot and dns-route53 plugin
sudo snap install --beta --classic certbot
sudo snap set certbot trust-plugin-with-root=ok
sudo snap install --beta certbot-dns-route53
sudo snap connect certbot:plugin certbot-dns-route53
I have created a special user in my AWS account who has access to Route53 and I have added the access key id and secret access key in the ~/.aws/config and also ~/.aws/credentials which looks something like this
[default]
aws_access_key_id=foo
aws_secret_access_key=bar
Basically followed every step given here: https://certbot-dns-route53.readthedocs.io/en/stable/
Now when I run the following command:
sudo certbot certonly -d mydomain.com --dns-route53
It gives the following output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-route53, Installer None
Requesting a certificate for mydomain.com
Performing the following challenges:
dns-01 challenge for mydomain.com
Cleaning up challenges
Unable to locate credentials
To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
I went to the documentation given in the error message: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#best-practices-for-configuring-credentials
but I do not think there is anything wrong I am doing
I even went to the root level by doing sudo su and exported the AWS keys as env vars there and even exported the AWS keys in the home as well but it still throws me the same error.
so I also ran into this same issue, and it's likely because of you running certbot with sudo, when do you do that, whatever user you've used as ~/, is ignored, as instead, it's looking in /root/.
I fixed it by (centos) is my user where I have the .aws/ directory with config and credential files.
sudo -s
ln -s /home/centos/.aws/ ~/.aws
ls -lsa ~/.aws
... /root/.aws -> /home/centos/.aws/

How to remove certificate from Traefik acme storage when saved to consul KV

I have Traefik running with a Consul KV store. How do I remove a record from the acme certificate storage in Consul, or force a renewal for just one domain/frontend?
Problem:
Somehow one of the frontend domains has saved with the wrong certificate. It's referencing a certificate from a different domain (which is also a frontend in Traefik).
I was able to inspect the acme json by getting the consul value for the traefik/acme/account/object key, decode and unzip it and this is the record from the Certs array:
{
"Domains":{
"Main":"my.domain1.com",
"SANs":null
},
"Certificate":{
"Domain":"my.domain2.com",
"CertURL":"https://acme-v02.api.letsencrypt.org/acme/cert/idfordomain2",
"CertStableURL":"https://acme-v02.api.letsencrypt.org/acme/cert/idfordomain2",
"PrivateKey":"...",
"Certificate":"..."
}
}
As you can see, somehow the cert for my.domain2.com has been saved against the record for my.domain1.com so this results in an invalid certificate warning in the browser. I want to clear out the whole record so Traefik will get a fresh cert. I'm using Consul and its saved in binary so I can't just edit the json.
Here is how I solved this issue:
Your traefik network should be marked as attachable: true
Run on host:
docker run -it --rm --name consul-client --network traefik_traefik consul sh
Then run in created container:
export CONSUL_HTTP_ADDR=consul:8500
# get value from consul and store it to acme.json
consul kv get traefik/acme/account/object | gzip -dc > acme.json
# remove invalid domain and store it to acme-fixed.json
cat acme.json | jq -r 'del (.DomainsCertificate.Certs[] | select(.Domains.Main=="'yourdomain.com'"))' > acme-fixed.json
# gzip it
cat acme-fixed.json | gzip -c > acme-fixed.json.gz
# upload fixed and gzipped json back to consul
consul kv put traefik/acme/account/object #acme-fixed.json.gz
Simplest way is to use consul CLI utility. The utility is also used to run server and ideally you should use same version as the one used for your servers. Make sure you export environment variables: CONSUL_HTTP_ADDR - points to consul server, default is http://127.0.0.1:8500 and CONSUL_HTTP_TOKEN - is ACL token, if you have ACLs on your server enabled, as it should be on production environments.
Then you just run following command
consul kv put traefik/acme/account/object #traefik.json
Where trafik.json is json file that has updated values, that you wish to change in Consul KV store.
Or you can use HTTP API: Consul Create/Update Key
curl -X PUT --data #traefik.json http://<your-server-url>:<port>/v1/kv/traefik/acme/account/object
If your server is ACL enabled, you need to add following header to curl request, with <your-acl-token> that was issued to you: -H "X-Consul-Token: <your-acl-token>"

Docker container not connecting to https endpoints

From inside a docker container, I'm running
# openssl s_client -connect rubygems.org:443 -state -nbio 2>&1 | grep "^SSL"
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:error in SSLv2/v3 read server hello A
That's all I get
I can't connect to any https site from within the docker container. The container is running on an openstack vm. The vm can connect via https.
Any advice?
UPDATE
root#ce239554761d:/# curl -vv https://google.com
* Rebuilt URL to: https://google.com/
* Hostname was NOT found in DNS cache
* Trying 216.58.217.46...
* Connected to google.com (216.58.217.46) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
and then it hangs.
Also, I'm getting intermittent successes now.
Sanity Checks:
changing the docker ips doesn't fix the problem
The docker containers work on my local machine
The docker containers work on other clouds
Docker 1.10.0 doesn't work in the vms
Docker 1.9.1 works in the vms
I was given a solution by the Docker community
OpenStack network seems to use lower MTU values and Docker does not infer the MTU settings from the host's network card since 1.10.
To run docker daemon with custom MTU settings, you can follow this blog post, that says:
$ cp /lib/systemd/system/docker.service /etc/systemd/system/docker.service
Edit a line in the new file to look like this:
ExecStart=/usr/bin/docker daemon -H fd:// --mtu=1454
Or (as suggested below by Dionysius), create and edit the file
/etc/systemd/system/docker.service.d/fixmtu.conf as follow:
[Service]
# Reset ExecStart & update mtu (see original command in /lib/systemd/system/docker.service)
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --mtu=1454.
MTU of 1454 is the value that seems to be common with OpenStack. You can look it up in your host using ifconfig.
Finally restart Docker:
$ sudo systemctl daemon-reload
$ sudo service docker restart

Setting up a Docker registry with Letsencrypt certificate

I'm setting up a domain registry as described here:
https://docs.docker.com/registry/deploying/
I generated a certificate for docker.mydomain.com and started the docker using their command on my server:
docker run -d -p 5000:5000 --restart=always --name registry \
-v `pwd`/certs:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
registry:2
I've started the docker and pointed to certificates I obtained using letsencrypt (https://letsencrypt.org/).
Now, when I browse to https://docker.mydomain.com:5000/v2/ I will get a page with just '{}', with a green lock (succesful secure page request).
But when I try to do a docker login docker.mydomain.com:5000 from a different server I see a error in the registry docker:
TLS handshake error from xxx.xxx.xxx.xxx:51773: remote error: bad certificate
I've tried some different variations in setting up the certificates, and gotten errors like:
remote error: unknown certificate authority
and
tls: first record does not look like a TLS handshake
What am I missing?
Docker seams to not support SNI : https://github.com/docker/docker/issues/9969
Update : Docker now should support SNI.
It's mean, when connecting to your server during the tls transaction, the docker client do not specify the domain name, so your server show the default certificate.
The solution could be to change to default certificate of your server to be to one valid for the docker domain.
This site works only in browsers with SNI support.
To check if your (sub-)domain works with clients not SNI-aware, you can use ssllabs.com/ssltest : If you DONT see the message, "This site works only in browsers with SNI support. " then it will works.

Cannot validate certificate for ip because it doesn't contain any IP SANs

I have installed OpenShift3 with Docker and Kubernetes with the ansible installer.
After the installation I want to create my docker registration on my master but I get the following error (I read it was something with SSL but I can't find a solution):
commands (from the sample):
[root#ip-10-0-0-x centos]# export CURL_CA_BUNDLE=`pwd`/openshift.local.config/master/ca.crt
[root#ip-10-0-0-x centos]# sudo chmod a+rwX openshift.local.config/master/admin.kubeconfig
[root#ip-10-0-0-x centos]# sudo chmod +r openshift.local.config/master/openshift-registry.kubeconfig
[root#ip-10-0-0-x centos]# oadm registry --create --credentials=openshift.local.config/master/openshift-registry.kubeconfig --config=openshift.local.config/master/admin.kubeconfig
error:
error: error getting client: couldn't read version from server: Get https://10.0.0.x:8443/api: x509: cannot validate certificate for 10.0.0.x because it doesn't contain any IP SANs
additional info
[root#ip-10-0-0-x centos]# kubectl version
Client Version: version.Info{Major:"", Minor:"", GitVersion:"v1.1.0-alpha.0-1605-g44c91b1", GitCommit:"44c91b1", GitTreeState:"not a git tree"}
Server Version: version.Info{Major:"", Minor:"", GitVersion:"v1.1.0-alpha.0-1605-g44c91b1", GitCommit:"44c91b1", GitTreeState:"not a git tree"}
[root#ip-10-0-0-191 centos]# oc get services
NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE
kubernetes 172.30.0.1 <none> 443/TCP <none> 1d
[root#ip-10-0-0-x centos]# kubernetes apiserver
F0924 12:15:13.674745 75545 server.go:223] No --service-cluster-ip-range specified
The Ansible installer should generate certs for you that have the right IPs in the certs. Your local kubeconfig file (that oadm is using to connect to the server) should have been generated by the Ansible installer - can you verify that is the case? The file is in ~/.kube/config - does it point to the system that the Ansible installer used? Are you using an IaaS for OpenShift, deploying to local machines, or Vagrant?