I'm trying to do a simple build of a nodejs app I wrote in OpenShift Origin using the following yaml:
kind: "BuildConfig"
apiVersion: "v1"
metadata:
name: "dyn-kickstart"
spec:
triggers:
- type: "GitHub"
github:
secret: "secret101"
source:
git:
uri: git#bitbucket.org:serverninja02/dynamic-kickstart.git
sourceSecret:
name: "github"
strategy:
type: Docker
dockerStrategy:
dockerfilePath: .
forcePull: true
noCache: true
output:
to:
kind: "DockerImage"
name: "docker-registry-default.apps.reedfamily.local/serverninja/dynamic-kickstart:v0.0.1
The command I'm running to create the build:
$ cat dynamic-kickstart.yml | oc create -f -
What I'm running into is that the build service account doesn't seem to be able to access the github url to clone:
Cloning "git#bitbucket.org:serverninja02/dynamic-kickstart.git" ...
error: build error: Warning: Permanently added 'bitbucket.org,192.168.1.81' (ECDSA) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
I did follow the instructions on creating the ssh-privatekey secret, placing in the secret store, and linking to the build sa. I also double-checked that key and tested through ssh forwarding that I can log into the OpenShift node and ssh git#bitbucket.org.
I'm not sure what I'm doing wrong but even with using the http git url and making it a public repo, it still doesn't work as it complains about the peer certificate not being trusted:
Cloning "https://serverninja02#bitbucket.org/serverninja02/dynamic-kickstart.git" ...
error: build error: fatal: unable to access 'https://serverninja02#bitbucket.org/serverninja02/dynamic-kickstart.git/': Peer's certificate issuer has been marked as not trusted by the user.
At this point, I'm unsure where to go with this as OpenShift Origin doesn't seem to want to build anything from git as a source.
Any help or suggestions would be greatly appreciated!
OpenShift Version: 1.3.0
OpenShift Kubernetes Version: v1.3.0+52492b4
This is a flat network behind a router. DNS is on Active Directory with a wildcard entry for *.apps.reedfamily.local.
This is a test bed environment in a .local domain. However I'm using this build to potentially build this out as a POC for my company to host OpenShift.
I figured out the answer to my problem!!! So I'll share:
The /etc/resolv.conf was configured automatically during the build of my OpenShift nodes when I ran openshift-ansible. Unfortunately, there was a search domain placed in /etc/resolv.conf that must have been causing issues.
# Generated by NetworkManager
search apps.reedfamily.local
nameserver 192.168.1.40
Once I removed "search apps.reedfamily.local", that fixed the problem immediately on the next build!
Related
I am using Jenkins with Kubernetes agents, trying to build a Gitea-hosted git repository using an organizational folder configuration. When my build triggers, I get
stderr: fatal: unable to access 'https://<servername.com>/homelab/java-spring-microservice.git/': server certificate verification failed. CAfile: none
The repository (also hosted within the kubernetes cluster) has a valid LetsEncrypt certificate on its ingress (managed by cert-manager.) I'm able to clone this repo fine from git command-line (without having TLS disabled.)
My Jenkinsfile looks like this:
podTemplate(containers: [
containerTemplate(
name: 'maven',
image: 'maven:3.8.4-openjdk-11',
command: 'sleep',
args: '30d'
)
]) {
node(POD_LABEL) {
stage('Checkout') {
checkout scm
container('maven') {
stage('Build') {
sh '''
mvn clean package
'''
}
}
}
}
}
I've looked around and seen ways to get around this by disabling TLS for the git operation, but that seems wrong-headed to me, since TLS appears to be working. I'll admit to being a bit uncertain of how exactly all this works when things are under kubernetes (where should I be looking to see if the CA trust chain is correct, etc.)
After some digging, I determined that both the version of jenkins/inbound-agent and the version of jenkins itself were built against base images that did not have up-to-date CA trust chains. I was able to resolve the problem by updating to the latest Jenkins, updating the pod template for kubernetes, and updating the kubernetes plugin for jenkins.
Everything was working fine from months I was able to clone/fetch/push/pull from my Gitlab repo on my mac:
Suddenly started to get this error on mac:
When I run the command:
git fetch "repo_link" I get the following error
unable to access : SSL peer handshake failed, the server most likely requires a client certificate to connect.
git config http.sslVerify false is already set.
Even if try to fetch/push/pull from Android studio getting the same error.
Tried clone/fetch on another mac it's working fine.
Please help.
It worked for me by using SSL instead of HTTPS.
I simply added another remote repo using the following command
git remote add org git#gitlab.com:***/***.git
Replace *** with your SSH path details
Then add an SSH to your GitLab account from here
To add an SSH key you need to generate one or use an existing key.
Once the key has been added to your GitLab account you can try
git push -u org master
Try to enable the sslVerify
locally at repo git config http.sslVerify true or globally git config --global http.sslVerify true
I've rolled out the Traefik Ingress Controller on my AKS Cluster (Kubernetes) on Azure. I've got the TOML file talking to the POD, but I receive the following on the logs:
ime="2018-12-21T00:09:36Z" level=error msg="Failed to read new
account, ACME data conversion is not available : permissions 755 for
certs are too open, please use 600" time="2018-12-21T00:09:36Z"
level=error msg="Unable to add ACME provider to the providers list:
unable to get ACME account : permissions 755 for certs are too open,
please use 600"
I have resolved this before in a docker environment where I simply chmod 600 the directory in question. However, I can't do that as I don't have direct access to the underlying storage.
If I open a shell to the container, the system and try to chmod that way the system tells me its a read only filesystem.
Any help is really appreciated.
Rutnet solved the issue by deploing a custom Traefik container which includes the required permissions.
I'm using Kubernetes and I recently updated my admin certs used in the kubeconfig. However, after I did that, all the helm commands fail thus:
Error: Get https://cluster.mysite.com/api/v1/namespaces/kube-system/pods?labelSelector=app%3Dhelm%2Cname%3Dtiller: x509: certificate signed by unknown authority
kubectl works as expected:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-1-0-34.eu-central-1.compute.internal Ready master 42d v1.7.10+coreos.0
ip-10-1-1-51.eu-central-1.compute.internal Ready master 42d v1.7.10+coreos.0
ip-10-1-10-120.eu-central-1.compute.internal Ready <none> 42d v1.7.10+coreos.0
ip-10-1-10-135.eu-central-1.compute.internal Ready <none> 27d v1.7.10+coreos.0
ip-10-1-11-71.eu-central-1.compute.internal Ready <none> 42d v1.7.10+coreos.0
ip-10-1-12-199.eu-central-1.compute.internal Ready <none> 8d v1.7.10+coreos.0
ip-10-1-2-110.eu-central-1.compute.internal Ready master 42d v1.7.10+coreos.0
As far as I've been able to read, helm is supposed to use the same certificates as kubectl, which makes me curious as how how kubectl works, but helm doesn't?
This is a production cluster with internal releases handled through helm charts, so it being solved is imperative.
Any hints would be greatly appreciated.
As a workaround you can try to disable certificate verification. Helm uses the kube config file (by default ~/.kube/config). You can add insecure-skip-tls-verify: true for the cluster section:
clusters:
- cluster:
server: https://cluster.mysite.com
insecure-skip-tls-verify: true
name: default
Did you already try to reinstall helm/tiller?
kubectl delete deployment tiller-deploy --namespace kube-system
helm init
Also check if you have configured an invalid certificate in the cluster configuration.
In my case, I was running for a single self-manage and the config file was also container ca-file, so the following the above answer was throwing below error
Error: Kubernetes cluster unreachable: Get "https://XX.XX.85.154:6443/version?timeout=32s": x509: certificate is valid for 10.96.0.1, 172.31.25.161, not XX.XX.85.154
And my config was
- cluster:
certificate-authority-data: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
server: https://54.176.85.154:6443
insecure-skip-tls-verify: true
So I had to remove the certificate-authority-data.
- cluster:
server: https://54.176.85.154:6443
insecure-skip-tls-verify: true
Use --insecure-skip-tls-verify to skip tls verification via command line
helm repo add stable --insecure-skip-tls-verify https://charts.helm.sh/stable
In my case the error was caused by an untrusted certificate from the Helm repository.
Downloading the certificate and specifying it using the --ca-file option solved the issue (at least in Helm version 3).
helm repo add --ca-file /path/to/certificate.crt repoName https://example/repository
--ca-file string, verify certificates of HTTPS-enabled servers using this CA bundle
Adding the line below the -cluster to /home/centos/.kube/config file fixed my issue
insecure-skip-tls-verify: true
fixed my issue.
my config file now looks like this.
apiVersion: v1
clusters:
- cluster:
certificate-authority: /home/centos/.minikube/ca.crt
extensions:
- extension:
last-update: Tue, 02 Nov 2021 20:51:44 EDT
provider: minikube.sigs.k8s.io
version: v1.23.2
name: cluster_info
server: https://192.168.49.2:8443
insecure-skip-tls-verify: true
name: minikube
contexts:
I encountered an edge case for this. You can also get this error if you have multiple kubeconfig files referenced in the KUBECONFIG variable, and more than one file has clusters with the same name.
For my case, it was an old version of helm (v. 3.6.3 in my case) after I upgraded to helm v.3.9.0 brew upgrade helm everything worked again.
Although adding repo with --ca-file did the thing, when it tried to download from that repo with the command posted under, I still got the x509: certificate signed by unknown authority
helm dependency update helm/myStuff
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "myRepo" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 18 charts
Downloading myService from repo https://myCharts.me/
Save error occurred: could not download https://myCharts.me/stuff.tgz ...
x509: certificate signed by unknown authority
Deleting newly downloaded charts, restoring pre-update state
What I needed to do, apart from adding repo with --ca-file was to download the repository certificate and install it as Current User:
Place all certificates in the following store: Trusted Root Certification Authorities:
After installing the certificate I also needed to restart the computer. After restart, when you open the browser and paste the repo URL it should connect without giving a warning and trusting the site (this way you know you installed the certificate successfully).
You can go ahead and run the command, it should pick the certificate this time.
helm dependency update helm/myStuff
....
Saving 18 charts
Downloading service1 from repo https://myCharts.me/
Downloading service2 from repo https://myCharts.me/
....
i'm getting an error when running kubectl one one machine (windows)
the k8s cluster is running on CentOs 7 kubernetes cluster 1.7
master, worker
Here's my .kube\config
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://10.10.12.7:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: system:node:localhost.localdomain
name: system:node:localhost.localdomain#kubernetes
current-context: system:node:localhost.localdomain#kubernetes
kind: Config
preferences: {}
users:
- name: system:node:localhost.localdomain
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
the cluster is built using kubeadm with the default certificates on the pki directory
kubectl unable to connect to server: x509: certificate signed by unknown authority
One more solution in case it helps anyone:
My scenario:
using Windows 10
Kubernetes installed via Docker Desktop ui 2.1.0.1
the installer created config file at ~/.kube/config
the value in ~/.kube/config for server is https://kubernetes.docker.internal:6443
using proxy
Issue: kubectl commands to this endpoint were going through the proxy, I figured it out after running kubectl --insecure-skip-tls-verify cluster-info dump which displayed the proxy html error page.
Fix: just making sure that this URL doesn't go through the proxy, in my case in bash I used export no_proxy=$no_proxy,*.docker.internal
So kubectl doesn't trust the cluster, because for whatever reason the configuration has been messed up (mine included). To fix this, you can use openssl to extract the certificate from the cluster
openssl.exe s_client -showcerts -connect IP:PORT
IP:PORT should be what in your config is written after server:
Copy paste stuff starting from -----BEGIN CERTIFICATE----- to -----END CERTIFICATE----- (these lines included) into a new text file, say... myCert.crt If there are multiple entries, copy all of them.
Now go to .kube\config and instead of
certificate-authority-data: <wrongEncodedPublicKey>`
put
certificate-authority: myCert.crt
(it assumes you put myCert.crt in the same folder as the config file)
If you made the cert correctly it will trust the cluster (tried renaming the file and it no longer trusted afterwards).
I wish I knew what encoding certificate-authority-data uses, but after a few hours of googling I resorted to this solution, and looking back I think it's more elegant anyway.
Run:
gcloud container clusters get-credentials standard-cluster-1 --zone us-central1-a --project devops1-218400
here devops1-218400 is my project name. Replace it with your project name.
I got the same error while running $ kubectl get nodes as a root user. I fixed it by exporting kubelet.conf to environment variable.
$ export KUBECONFIG=/etc/kubernetes/kubelet.conf
$ kubectl get nodes
For my case, its simple worked by adding --insecure-skip-tls-verify at end of kubectl commands, for single time.
Sorry I wasn't able to provide this earlier, I just realized the cause:
So on the master node we're running a kubectl proxy
kubectl proxy --address 0.0.0.0 --accept-hosts '.*'
I stopped this and voila the error was gone.
I'm now able to do
kubectl get nodes
NAME STATUS AGE VERSION
centos-k8s2 Ready 3d v1.7.5
localhost.localdomain Ready 3d v1.7.5
I hope this helps those who stumbled upon this scenario.
I my case I resolved this issue copying the kubelet configuration to my home kube config
cat /etc/kubernetes/kubelet.conf > ~/.kube/config
This was happening because my company's network does not allow self signing certificates through their network. Try switching to a different network
For those of you that were late to the thread like I was and none of these answers worked for you I may have the solution:
When I copied over my .kube/config file to my windows 10 machine (with kubectl installed) I didn't change the IP address from 127.0.0.1:6443 to the master's IP address which was 192.168.x.x. (running windows 10 machine connecting to raspberry pi cluster on the same network). Make sure that you do this and it may fix your problem like it did mine.
On GCP
check: gcloud version
-- localMacOS# gcloud version
Run:
--- localMacOS# gcloud container clusters get-credentials 'clusterName' \ --zone=us-'zoneName'
Get clusterName and zoneName from your console -- here: https://console.cloud.google.com/kubernetes/list?
ref: .x509 #market place deployments on GCP #Kubernetes
I got this because I was not connected to the office's VPN
In case of the error you should export all the kubecfg which contains the certs. kops export kubecfg "your cluster-name and export KOPS_STATE_STORE=s3://"paste your S3 store" .
Now you should be able to access and see the resources of your cluster.
This is an old question but in case that also helps someone else here is another possible reason.
Let's assume that you have deployed Kubernetes with user x. If the .kube dir is under the /home/x user and you connect to the node with root or y user it will give you this error.
You need to switch to the user profile so kubernetes can load the configuration from the .kube dir.
Update: When copying the ~/.kube/config file content on a local pc from a master node make sure to replace the hostname of the loadbalancer with a valid IP. In my case the problem was related to the dns lookup.
Hope this helps.