We are running a small test cluster of Custom Kubernetes pods on Google cloud, that internally are running Apache and PHP-FPM.
The Cluster has the following key config:
Master version: 1.10.6-gke.2
Kubernetes alpha features: Disabled
Total size: 3
StackDriver Logging: Enabled
StackDriver Monitoring:
Enabled
Once the cluster comes up a kubectl get pods --all-namespaces is showing the fluentd and heapster services running along side our services as I would expect.
kube-system event-exporter-v0.2.1-5f5b89fcc8-r89d5 2/2 Running 0 13d
kube-system fluentd-gcp-scaler-7c5db745fc-gbrqx 1/1 Running 0 21d
kube-system fluentd-gcp-v3.1.0-76mr4 2/2 Running 0 13d
kube-system fluentd-gcp-v3.1.0-kl4xp 2/2 Running 0 13d
kube-system fluentd-gcp-v3.1.0-vxsq5 2/2 Running 0 13d
kube-system heapster-v1.5.3-95c7549b8-fdlmm 3/3 Running 0 13d
kube-system kube-dns-788979dc8f-c9v2d 4/4 Running 0 99d
kube-system kube-dns-788979dc8f-rqp7d 4/4 Running 0 99d
kube-system kube-dns-autoscaler-79b4b844b9-zjtwk 1/1 Running 0 99d
We can get the logging from our application code (that runs inside our pods) to show up in Stackdriver Logging, but we want to aggregate the logging for Apache (/var/log/httpd/access_log and error_log) and PHP-FPM in Stackdriver as well.
This page from Google's Docs implies that this should be enabled by default.
https://cloud.google.com/kubernetes-engine/docs/how-to/logging
Note: Stackdriver Logging is enabled by default when you create a new cluster using the gcloud command-line tool or Google Cloud Platform Console.
However that is obviously not the case for us. We have tried a few different approaches to get this to work (listed below), but without success.
Including:
redirecting the log output from Apache to stdout and/or stderr, as described in this post.
https://serverfault.com/questions/711168/writing-apache2-logs-to-stdout-stderr
Installing the stackdriver agent inside each pod as described in https://cloud.google.com/monitoring/agent/plugins/apache#configuring
It didn't appear that this step should be required as the documentation implies that you only need to do this on a VM instance, but we tried it anyway on our k8s pods. As part of this step we made sure that Apache has mod_status enabled (/server-status) and PHP-FPM has /fpm-status enabled, and then installed the module Apache plugin following the docs.
Piping the Apache logging to STDOUT
How to Redirect Apache Logs to both STDOUT and Apache Log File
This seems like it should be a simple thing to do, but we have obviously missed something. Any help would be most appreciated.
Cheers, Julian Cone
Related
I deploy the infinispan cluster using these following code:
https://github.com/infinispan/infinispan-helm-charts
On Minikube: Its working fine
When i run above command infinispan server got created and goes into cluster
on EKS : clustering is not happening
When i run above command infinispan server got created but NOT goes into cluster
helm lint ./infinispan-helm-charts
helm install -n qa infinispan-server ./infinispan-helm-charts
And then port forward to access
kubectl port-forward service/infinispan-server 11222:11222 -n qa
First time trying EKS Anywhere docker provider deployment as given in below link
https://anywhere.eks.amazonaws.com/docs/getting-started/local-environment/
It gets stuck at 'waiting for cert-manager' . Working on CentOS 7 .System is behind proxy.
Installing cert-manager Version="v1.5.3+66e1acc"
Using Override="cert-manager.yaml" Provider="cert-manager" Version="v1.5.3+66e1acc"
Waiting for cert-manager to be available...
Error: timed out waiting for the condition
Only cert-manager pods are not able to pull images
NAMESPACE NAME READY STATUS RESTARTS AGE
cert-manager cert-manager-7988d4fb6c-bjhsv 0/1 ImagePullBackOff 0 5m54s
cert-manager cert-manager-cainjector-6bc8dcdb64-hvdx5 0/1 ImagePullBackOff 0 5m55s
cert-manager cert-manager-webhook-68979bfb95-q8ttt 0/1 ImagePullBackOff 0 5m54s
kube-system coredns-745c7986c7-2wrx5 1/1 Running 0 5m57s
kube-system coredns-745c7986c7-kx594 1/1 Running 0 5m57s
kube-system etcd-dev-cluster-eks-a-cluster-control-plane 1/1 Running 0 5m52s
kube-system kindnet-4jcvt 1/1 Running 0 5m57s
kube-system kube-apiserver-dev-cluster-eks-a-cluster-control-plane 1/1 Running 0 5m52s
kube-system kube-controller-manager-dev-cluster-eks-a-cluster-control-plane 1/1 Running 0 5m52s
kube-system kube-proxy-4dk2r 1/1 Running 0 5m57s
kube-system kube-scheduler-dev-cluster-eks-a-cluster-control-plane 1/1 Running 0 5m52s
local-path-storage local-path-provisioner-666bfc797f-nkhqf 1/1 Running 0 5m57s
same images are getting pulled using docker pull
public.ecr.aws/eks-anywhere/jetstack/cert-manager-webhook v1.5.3-eks-a-6 194bcfda671e 3 months ago 46MB
public.ecr.aws/eks-anywhere/jetstack/cert-manager-controller v1.5.3-eks-a-6 1e6749016508 3 months ago 61.3MB
public.ecr.aws/eks-anywhere/jetstack/cert-manager-cainjector v1.5.3-eks-a-6 45723d794a88 3 months ago 42.4MB
kubectl describe gives below (i/o timeout) error as well as 'server misbehaving' error
Failed to pull image "public.ecr.aws/eks-anywhere/jetstack/cert-manager-controller:v1.5.3-eks-a-6": rpc error: code = Unknown desc = failed to pull and unpack image "public.ecr.aws/eks-anywhere/jetstack/cert-manager-controller:v1.5.3-eks-a-6": failed to resolve reference "public.ecr.aws/eks-anywhere/jetstack/cert-manager-controller:v1.5.3-eks-a-6": failed to do request: Head "https://public.ecr.aws/v2/eks-anywhere/jetstack/cert-manager-controller/manifests/v1.5.3-eks-a-6": dial tcp: lookup public.ecr.aws on 172.19.0.1:53: read udp 172.19.0.2:38941->172.19.0.1:53: i/o timeout
It was a proxy related issue. Resolved by adding proxy config in containerd service of docker container of node and restarting containerd service.
docker exec -it <container name> bash
Inside container
cd /etc/systemd/system/
mkdir containerd.service.d
touch http-proxy.conf
cat <<EOF >/etc/systemd/system/containerd.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://proxy ip:proxy port"
Environment="HTTPS_PROXY=http://proxy ip:proxy port"
Environment="NO_PROXY=${NO_PROXY:-localhost},${LOCAL_NETWORK}"
EOF
systemctl daemon-reload
systemctl restart containerd
Our Spinnaker is deployed on Ubuntu 18, Spinnaker version is 1.20.3. The only way we can view the logs is to run journalctl -u $microservice there are no logs on /var/log/spinnaker.
Is this normal?
Yes. The preferred way of installation for Spinnaker is in Kubernetes. A quick and easy way for you to get started and easily migrate is to backup all you config with halyard, export the pipelines as json and run Minnaker in any Ubuntu 18 Compute box
Then import your old spinnaker data and pipelines.
The Ubuntu18 debian deploy flavor that you are running could be useful to debug cloud driver issues or for development purposes.
I suggest that you perform the migration to a Kubernetes cluster.
The reason why none of the Spinnaker microservices output any logs to their log file directories in /var/log/spinnaker is because the preferred method of installation for Spinnaker is to use Kubernetes.
If the microservices were to create log files in /var/log/spinnaker, there is a good chance that the Kubernetes pods would die due to running out of storage, hence they all output their logs to STDOUT, and can be retrieved from Kubernetes by running:
kubectl -n spinnaker logs POD_NAME > my_logfile_name.log
If you prefer to run Spinnaker on a VM rather than in Kubernetes and want to enable the log files so that you can debug a specific issue instead of using journalctl, you can edit the systemd service file for the particular microservice, for example Clouddriver, and add the following line in the [Service] section:
StandardOutput=append:/var/log/spinnaker/clouddriver/clouddriver.log
Then you reload the systemctl daemon and restart the service and it will then output its logs to the specified log file instead of STDOUT, for example:
sudo systemctl daemon-reload
sudo systemctl restart clouddriver.service
Created a 2 node Kubernetes cluster as:
KUBERNETES_PROVIDER=aws NUM_NODES=2 kube-up.sh
This shows the output as:
Found 2 node(s).
NAME STATUS AGE
ip-172-20-0-226.us-west-2.compute.internal Ready 57s
ip-172-20-0-227.us-west-2.compute.internal Ready 55s
Validate output:
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health": "true"}
etcd-1 Healthy {"health": "true"}
Cluster validation succeeded
Done, listing cluster services:
Kubernetes master is running at https://52.33.9.1
Elasticsearch is running at https://52.33.9.1/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging
Heapster is running at https://52.33.9.1/api/v1/proxy/namespaces/kube-system/services/heapster
Kibana is running at https://52.33.9.1/api/v1/proxy/namespaces/kube-system/services/kibana-logging
KubeDNS is running at https://52.33.9.1/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://52.33.9.1/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
Grafana is running at https://52.33.9.1/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana
InfluxDB is running at https://52.33.9.1/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb
I can see the instances in EC2 console. How do I ssh into the master node?
Here is the exact command that worked for me:
ssh -i ~/.ssh/kube_aws_rsa admin#<masterip>
kube_aws_rsa is the default key generated, otherwise controlled with AWS_SSH_KEY environment variable. For AWS, it is specified in the file cluster/aws/config-default.sh.
More details about the cluster can be found using kubectl.sh config view.
"Creates an AWS SSH key named kubernetes-. Fingerprint here is the OpenSSH key fingerprint, so that multiple users can run the script with different keys and their keys will not collide (with near-certainty). It will use an existing key if one is found at AWS_SSH_KEY, otherwise it will create one there. (With the default Ubuntu images, if you have to SSH in: the user is ubuntu and that user can sudo"
https://github.com/kubernetes/kubernetes/blob/master/docs/design/aws_under_the_hood.md
You should see the ssh key-fingerprint locally in ssh config or set the ENV and recreate.
If you are throwing up your cluster on AWS with kops, and use CoreOS as your image, then the login name would be "core".
I'm trying to follow beginner tutorial on Docker's website and I suffer with an error on login.
OS is Ubuntu 14.04, I'm not using VirtualBox and I'm not behind any proxy and want to push to the "regular" docker repository (not private one).
All threads I've found mention proxies and private repositories but that isn't my case, I'm just trying to do simple beginner tutorial.
Here is my attempt:
$ sudo docker login
[sudo] password for myuname:
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: myDHuname
Password:
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
My docker info:
Containers: 5
Running: 0
Paused: 0
Stopped: 5
Images: 5
Server Version: 1.11.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 28
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 3.19.0-58-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.686 GiB
Name: myuname-ThinkPad-T420
ID: 6RW3:X3FC:T62N:CWKI:JQW5:YIPY:RAHO:ZHF4:DFZ6:ZL7X:JPOD:V7EC
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Epilogue
Now docker login is passing. I have not touched anything since yesterday when it was broken...
I can't reproduce the behavior anymore.
I encounter this issue when my first use docker. I've shadowsocks proxy on, and configed as pac mode. When I try to run docker run hello-world, I get this timeout error. When I set the proxy mode to global, the error is aslo there.
But when I disable the proxy, docker runs well. It pull remote image successfully.
docker for windows
Note: Some users reported problems connecting to Docker Hub on Docker
for Windows stable version. This would manifest as an error when
trying to run docker commands that pull images from Docker Hub that
are not already downloaded, such as a first time run of docker run
hello-world. If you encounter this, reset the DNS server to use the
Google DNS fixed address: 8.8.8.8. For more information, see
Networking issues in Troubleshooting.
The error Client.Timeout exceeded while awaiting headers indicates:
GET request to the registry https://registry-1.docker.io/v2/ timedout
The library responsible (most likely libcurl) timed out before a response was heard
The connection never formed (proxy/firewall gobbled it up)
If you see the below result you can rule out timed out and network connectivity
$ curl https://registry-1.docker.io/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
If you get the above response next would be to check if your user environment has some proxy configuration.
env | grep "proxy"
Note: The docker runs as root. Maybe you have http_proxy in your env. Most likely I am wrong. Anywho see what happens with the curl GET request
change the proxy settings in the firefox. May be you are in access restricted mode. Just add the server address in the firefox settings -> preferences -> advanced -> network -> configuration (settings). Add the server ip in the no proxy for the issue can be resolved