propertySources empty why?
application.yaml
server:
port: 8888
spring:
cloud:
config:
server:
git:
uri: file:///C:/Users/hp/IdeaProjects/cloudconfig
default-label: master
application:
name: spring-cloud-config-server
limit-service.yaml
limit-service:
min: 10
max: 1000
Related
I deployed minio and the console in K8S, used ClusterIP to expose ports 9000 & 5000
Listening for port 80 and 5000 forwarding requests to minio.service(ClusterIP)
Request console all right through port 5000
By requesting the console on port 80, you can see the console, but the request is 404 in the browser
enter image description here
enter image description here
apiVersion: v1
kind: Service
metadata:
namespace: {{ .Release.Namespace }}
name: minio-headless
labels:
app: minio-headless
spec:
type: ClusterIP
clusterIP: None
ports:
- name: server
port: 9000
targetPort: 9000
- name: console
port: 5000
targetPort: 5000
selector:
app: minio
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: ingress-route-minio
namespace: {{ .Release.Namespace }}
spec:
entryPoints:
- minio
- web
routes:
- kind: Rule
match: Host(`minio-console.{{ .Release.Namespace }}.k8s.zszc`)
priority: 10
services:
- kind: Service
name: minio-headless
namespace: {{ .Release.Namespace }}
port: 5000
responseForwarding:
flushInterval: 1ms
scheme: http
strategy: RoundRobin
weight: 10
traefik access log
{
"ClientAddr": "192.168.4.250:55485",
"ClientHost": "192.168.4.250",
"ClientPort": "55485",
"ClientUsername": "-",
"DownstreamContentSize": 19,
"DownstreamStatus": 404,
"Duration": 688075,
"OriginContentSize": 19,
"OriginDuration": 169976,
"OriginStatus": 404,
"Overhead": 518099,
"RequestAddr": "minio-console.etb-0-0-1.k8s.zszc",
"RequestContentSize": 0,
"RequestCount": 1018,
"RequestHost": "minio-console.etb-0-0-1.k8s.zszc",
"RequestMethod": "GET",
"RequestPath": "/api/v1/login",
"RequestPort": "-",
"RequestProtocol": "HTTP/1.1",
"RequestScheme": "http",
"RetryAttempts": 0,
"RouterName": "traefik-traefik-dashboard-6e26dcbaf28841493448#kubernetescrd",
"StartLocal": "2023-01-27T13:20:06.337540015Z",
"StartUTC": "2023-01-27T13:20:06.337540015Z",
"entryPointName": "web",
"level": "info",
"msg": "",
"time": "2023-01-27T13:20:06Z"
}
It looks to me like the request for /api is conflicting with rules for the Traefik dashboard. If you look at the access log in your question, we see:
"RouterName": "traefik-traefik-dashboard-6e26dcbaf28841493448#kubernetescrd",
If you have installed Traefik from the Helm chart, it installs an IngressRoute with the following rules:
- kind: Rule
match: PathPrefix(`/dashboard`) || PathPrefix(`/api`)
services:
- kind: TraefikService
name: api#internal
In theory those are bound only to the traefik entrypoint, but it looks like you may have customized your entrypoint configuration.
Take a look at the IngressRoute resource for your Traefik dashboard and ensure that it's not sharing an entrypoint with minio.
I have a problem attaching a volume to the docker image running inside tekton pipelines. I have used the below task
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: distributor-base
namespace: cicd
labels:
app.kubernetes.io/version: "0.1"
annotations:
tekton.dev/pipelines.minVersion: "0.12.1"
tekton.dev/platforms: "linux/amd64"
spec:
params:
- name: builder_image
description: The location of the docker builder image.
default: docker:stable
- name: dind_image
description: The location of the docker-in-docker image.
default: docker:dind
- name: context
description: Path to the directory to use as context.
default: .
workspaces:
- name: source
steps:
- name: docker-build
image: docker
env:
# Connect to the sidecar over TCP, with TLS.
- name: DOCKER_HOST
value: tcp://localhost:2376
# Verify TLS.
- name: DOCKER_TLS_VERIFY
value: '1'
# Use the certs generated by the sidecar daemon.
- name: DOCKER_CERT_PATH
value: /certs/client
- name: DOCKER_USER
valueFrom:
secretKeyRef:
key: username
name: docker-auth
- name: DOCKER_TOKEN
valueFrom:
secretKeyRef:
key: password
name: docker-auth
- name: DIND_CONFIG
valueFrom:
configMapKeyRef:
key: file
name: dind-env
workingDir: $(workspaces.source.path)
args:
- --storage-driver=vfs
- --debug
securityContext:
privileged: true
script: |
#!/usr/bin/env sh
set -e
pwd
ls -ltr /workspace/source
docker run --privileged -v "/workspace/source:/workspace" busybox ls -ltr /workspace
volumeMounts:
- mountPath: /certs/client
name: dind-certs
sidecars:
- image: $(params.dind_image)
name: server
args:
- --storage-driver=vfs
- --debug
- --userland-proxy=false
resources:
requests:
memory: "512Mi"
securityContext:
privileged: true
env:
# Write generated certs to the path shared with the client.
- name: DOCKER_TLS_CERTDIR
value: /certs
volumeMounts:
- mountPath: /certs/client
name: dind-certs
# Wait for the dind daemon to generate the certs it will share with the
# client.
readinessProbe:
periodSeconds: 1
exec:
command: ['ls', '/certs/client/ca.pem']
volumes:
- name: dind-certs
emptyDir: {}
in the above task workspace comes from another git-clone task
workspaces:
- name: source
in this task, I am trying to run a docker image that has access to the workspace folder , because I have to modify some files in the workspace folder.
when we look into the script
pwd
ls -ltr /workspace/source
docker run --privileged -v "/workspace/source:/workspace"
below is the console log of above 3 commands
workspace/source
total 84
-rwxr-xr-x 1 50381 50381 3206 Jun 1 10:13 README.md
-rwxr-xr-x 1 50381 50381 10751 Jun 1 10:13 Jenkinsfile.next
-rwxr-xr-x 1 50381 50381 5302 Jun 1 10:13 wait-for-it.sh
drwxr-xr-x 4 50381 50381 6144 Jun 1 10:13 overlays
-rwxr-xr-x 1 50381 50381 2750 Jun 1 10:13 example-distributor.yaml
drwxr-xr-x 5 50381 50381 6144 Jun 1 10:13 bases
-rw-r--r-- 1 50381 50381 0 Jun 1 10:13 semantic.out
-rw-r--r-- 1 50381 50381 44672 Jun 1 10:13 final.yaml
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
462eb288b104: Pulling fs layer
462eb288b104: Verifying Checksum
462eb288b104: Download complete
462eb288b104: Pull complete
Digest: sha256:ebadf81a7f2146e95f8c850ad7af8cf9755d31cdba380a8ffd5930fba5996095
Status: Downloaded newer image for busybox:latest
total 0
basically pwd command is giving me results
and ls -ltr command also gives me the results
but when I try to attach /workspace/source folder as a volume to busybox docker, I am not able to see the content.
i mean since I have attached a volume into the directory /workspace I would expect the contents from local folder /workspace/source but I see 0 results from the above log.
basically volume is not getting attached properly.
can anyone please help me to fix this issue.
below is my pipeline run triggered by tekton-triggers
apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
name: github-gitops-template
namespace: cicd
spec:
params:
- name: gitRevision
description: The git revision (SHA)
default: master
- name: gitRepoUrl
description: The git repository url ("https://github.com/foo/bar.git")
- name: gitRepoName
description: The git repository name
- name: branchUrl
description: The git repository branch url
- name: repoFullName
description: The git repository full name
- name: commitSha
description: The git commit sha
resourcetemplates:
- apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: $(tt.params.gitRepoName)-
spec:
timeout: 0h10m
pipelineRef:
name: gitops-pipeline
serviceAccountName: github-service-account
params:
- name: url
value: $(tt.params.gitRepoUrl)
- name: branch
value: $(tt.params.gitRevision)
- name: repoName
value: $(tt.params.gitRepoName)
- name: branchUrl
value: $(tt.params.branchUrl)
- name: repoFullName
value: $(tt.params.repoFullName)
- name: commitSha
value: $(tt.params.commitSha)
workspaces:
- name: ws
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
below is my task run:
completionTime: '2022-06-01T10:13:47Z'
conditions:
- lastTransitionTime: '2022-06-01T10:13:47Z'
message: All Steps have completed executing
reason: Succeeded
status: 'True'
type: Succeeded
podName: gitops-core-business-tzb7f-distributor-base-pod
sidecars:
- container: sidecar-server
imageID: 'docker-pullable://gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/nop#sha256:1d65a20cd5fbc79dc10e48ce9d2f7251736dac13b302b49a1c9a8717c5f2b5c5'
name: server
terminated:
containerID: 'docker://d5e96143812bb4912c6297f7706f141b9036c6ee77efbffe2bcb7edb656755a5'
exitCode: 0
finishedAt: '2022-06-01T10:13:49Z'
message: Sidecar container successfully stopped by nop image
reason: Completed
startedAt: '2022-06-01T10:13:37Z'
startTime: '2022-06-01T10:13:30Z'
steps:
- container: step-docker-build
imageID: 'docker-pullable://docker#sha256:5bc07a93c9b28e57a58d57fbcf437d1551ff80ae33b4274fb60a1ade2d6c9da4'
name: docker-build
terminated:
containerID: 'docker://18aa9111f180f9cfc6b9d86d5ef1da9f8dbe83375bb282bba2776b5bbbcaabfb'
exitCode: 0
finishedAt: '2022-06-01T10:13:46Z'
reason: Completed
startedAt: '2022-06-01T10:13:42Z'
taskSpec:
params:
- default: 'docker:stable'
description: The location of the docker builder image.
name: builder_image
type: string
- default: 'docker:dind'
description: The location of the docker-in-docker image.
name: dind_image
type: string
- default: .
description: Path to the directory to use as context.
name: context
type: string
sidecars:
- args:
- '--storage-driver=vfs'
- '--debug'
- '--userland-proxy=false'
env:
- name: DOCKER_TLS_CERTDIR
value: /certs
image: $(params.dind_image)
name: server
readinessProbe:
exec:
command:
- ls
- /certs/client/ca.pem
periodSeconds: 1
resources:
requests:
memory: 512Mi
securityContext:
privileged: true
volumeMounts:
- mountPath: /certs/client
name: dind-certs
steps:
- args:
- '--storage-driver=vfs'
- '--debug'
env:
- name: DOCKER_HOST
value: 'tcp://localhost:2376'
- name: DOCKER_TLS_VERIFY
value: '1'
- name: DOCKER_CERT_PATH
value: /certs/client
- name: DOCKER_USER
valueFrom:
secretKeyRef:
key: username
name: docker-auth
- name: DOCKER_TOKEN
valueFrom:
secretKeyRef:
key: password
name: docker-auth
- name: DIND_CONFIG
valueFrom:
configMapKeyRef:
key: file
name: dind-env
image: docker
name: docker-build
resources: {}
script: |
#!/usr/bin/env sh
set -e
pwd
ls -ltr /workspace/source
docker run --privileged -v "/workspace/source:/workspace" busybox ls -ltr /workspace
securityContext:
privileged: true
volumeMounts:
- mountPath: /certs/client
name: dind-certs
workingDir: $(workspaces.source.path)
volumes:
- emptyDir: {}
name: dind-certs
workspaces:
- name: source
basically we have to attach volume to sidecar, since docker run happens in side card
volumeMounts:
- mountPath: /certs/client
name: dind-certs
- name: $(workspaces.source.volume)
mountPath: $(workspaces.source.path)
I am trying to setup a 3 node vault cluster with raft storage enabled. I am currently at a loss to why the readiness probe (also the liveness probe) is returning
Readiness probe failed: Get "https://10.42.4.82:8200/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204": http: server gave HTTP response to HTTPS client
I am using helm 3 for 'helm install vault hashicorp/vault --namespace vault -f override-values.yaml'
global:
enabled: true
tlsDisable: false
injector:
enabled: false
server:
image:
repository: "hashicorp/vault"
tag: "1.5.5"
resources:
requests:
memory: 1Gi
cpu: 2000m
limits:
memory: 2Gi
cpu: 2000m
readinessProbe:
enabled: true
path: "/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204"
livenessProbe:
enabled: true
path: "/v1/sys/health?standbyok=true"
initialDelaySeconds: 60
VAULT_CACERT: /vault/userconfig/tls-ca/ca.crt
# extraVolumes is a list of extra volumes to mount. These will be exposed
# to Vault in the path `/vault/userconfig/<name>/`.
extraVolumes:
# holds the cert file and the key file
- type: secret
name: tls-server
# holds the ca certificate
- type: secret
name: tls-ca
auditStorage:
enabled: true
standalone:
enabled: false
# Run Vault in "HA" mode.
ha:
enabled: true
replicas: 3
raft:
enabled: true
setNodeId: true
config: |
ui = true
listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/userconfig/tls-server/tls.crt"
tls_key_file = "/vault/userconfig/tls-server/tls.key"
tls_ca_cert_file = "/vault/userconfig/tls-ca/ca.crt"
}
storage "raft" {
path = "/vault/data"
retry_join {
leader_api_addr = "https://vault-0.vault-internal:8200"
leader_ca_cert_file = "/vault/userconfig/tls-ca/ca.crt"
leader_client_cert_file = "/vault/userconfig/tls-server/tls.crt"
leader_client_key_file = "/vault/userconfig/tls-server/tls.key"
}
retry_join {
leader_api_addr = "https://vault-1.vault-internal:8200"
leader_ca_cert_file = "/vault/userconfig/tls-ca/ca.crt"
leader_client_cert_file = "/vault/userconfig/tls-server/tls.crt"
leader_client_key_file = "/vault/userconfig/tls-server/tls.key"
}
retry_join {
leader_api_addr = "https://vault-2.vault-internal:8200"
leader_ca_cert_file = "/vault/userconfig/tls-ca/ca.crt"
leader_client_cert_file = "/vault/userconfig/tls-server/tls.crt"
leader_client_key_file = "/vault/userconfig/tls-server/tls.key"
}
}
service_registration "kubernetes" {}
# Vault UI
ui:
enabled: true
serviceType: "ClusterIP"
serviceNodePort: null
externalPort: 8200
Return from describe pod vault-0
Name: vault-0
Namespace: vault
Priority: 0
Node: node4/10.211.55.7
Start Time: Wed, 11 Nov 2020 15:06:47 +0700
Labels: app.kubernetes.io/instance=vault
app.kubernetes.io/name=vault
component=server
controller-revision-hash=vault-5c4b47bdc4
helm.sh/chart=vault-0.8.0
statefulset.kubernetes.io/pod-name=vault-0
vault-active=false
vault-initialized=false
vault-perf-standby=false
vault-sealed=true
vault-version=1.5.5
Annotations: <none>
Status: Running
IP: 10.42.4.82
IPs:
IP: 10.42.4.82
Controlled By: StatefulSet/vault
Containers:
vault:
Container ID: containerd://6dfde76051f44c22003cc02a880593792d304e74c56d717eef982e0e799672f2
Image: hashicorp/vault:1.5.5
Image ID: docker.io/hashicorp/vault#sha256:90cfeead29ef89fdf04383df9991754f4a54c43b2fb49ba9ff3feb713e5ef1be
Ports: 8200/TCP, 8201/TCP, 8202/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/bin/sh
-ec
Args:
cp /vault/config/extraconfig-from-values.hcl /tmp/storageconfig.hcl;
[ -n "${HOST_IP}" ] && sed -Ei "s|HOST_IP|${HOST_IP?}|g" /tmp/storageconfig.hcl;
[ -n "${POD_IP}" ] && sed -Ei "s|POD_IP|${POD_IP?}|g" /tmp/storageconfig.hcl;
[ -n "${HOSTNAME}" ] && sed -Ei "s|HOSTNAME|${HOSTNAME?}|g" /tmp/storageconfig.hcl;
[ -n "${API_ADDR}" ] && sed -Ei "s|API_ADDR|${API_ADDR?}|g" /tmp/storageconfig.hcl;
[ -n "${TRANSIT_ADDR}" ] && sed -Ei "s|TRANSIT_ADDR|${TRANSIT_ADDR?}|g" /tmp/storageconfig.hcl;
[ -n "${RAFT_ADDR}" ] && sed -Ei "s|RAFT_ADDR|${RAFT_ADDR?}|g" /tmp/storageconfig.hcl;
/usr/local/bin/docker-entrypoint.sh vault server -config=/tmp/storageconfig.hcl
State: Running
Started: Wed, 11 Nov 2020 15:25:21 +0700
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 11 Nov 2020 15:19:10 +0700
Finished: Wed, 11 Nov 2020 15:20:20 +0700
Ready: False
Restart Count: 8
Limits:
cpu: 2
memory: 2Gi
Requests:
cpu: 2
memory: 1Gi
Liveness: http-get https://:8200/v1/sys/health%3Fstandbyok=true delay=60s timeout=3s period=5s #success=1 #failure=2
Readiness: http-get https://:8200/v1/sys/health%3Fstandbyok=true&sealedcode=204&uninitcode=204 delay=5s timeout=3s period=5s #success=1 #failure=2
Environment:
HOST_IP: (v1:status.hostIP)
POD_IP: (v1:status.podIP)
VAULT_K8S_POD_NAME: vault-0 (v1:metadata.name)
VAULT_K8S_NAMESPACE: vault (v1:metadata.namespace)
VAULT_ADDR: https://127.0.0.1:8200
VAULT_API_ADDR: https://$(POD_IP):8200
SKIP_CHOWN: true
SKIP_SETCAP: true
HOSTNAME: vault-0 (v1:metadata.name)
VAULT_CLUSTER_ADDR: https://$(HOSTNAME).vault-internal:8201
VAULT_RAFT_NODE_ID: vault-0 (v1:metadata.name)
HOME: /home/vault
VAULT_CACERT: /vault/userconfig/tls-ca/ca.crt
Mounts:
/home/vault from home (rw)
/var/run/secrets/kubernetes.io/serviceaccount from vault-token-lfgnj (ro)
/vault/audit from audit (rw)
/vault/config from config (rw)
/vault/data from data (rw)
/vault/userconfig/tls-ca from userconfig-tls-ca (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-vault-0
ReadOnly: false
audit:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: audit-vault-0
ReadOnly: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: vault-config
Optional: false
userconfig-tls-ca:
Type: Secret (a volume populated by a Secret)
SecretName: tls-ca
Optional: false
home:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
vault-token-lfgnj:
Type: Secret (a volume populated by a Secret)
SecretName: vault-token-lfgnj
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 18m default-scheduler Successfully assigned vault/vault-0 to node4
Warning Unhealthy 17m (x2 over 17m) kubelet Liveness probe failed: Get "https://10.42.4.82:8200/v1/sys/health?standbyok=true": http: server gave HTTP response to HTTPS client
Normal Killing 17m kubelet Container vault failed liveness probe, will be restarted
Normal Pulled 17m (x2 over 18m) kubelet Container image "hashicorp/vault:1.5.5" already present on machine
Normal Created 17m (x2 over 18m) kubelet Created container vault
Normal Started 17m (x2 over 18m) kubelet Started container vault
Warning Unhealthy 13m (x56 over 18m) kubelet Readiness probe failed: Get "https://10.42.4.82:8200/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204": http: server gave HTTP response to HTTPS client
Warning BackOff 3m41s (x31 over 11m) kubelet Back-off restarting failed container
Logs from vault-0
2020-11-12T05:50:43.554426582Z ==> Vault server configuration:
2020-11-12T05:50:43.554524646Z
2020-11-12T05:50:43.554574639Z Api Address: https://10.42.4.85:8200
2020-11-12T05:50:43.554586234Z Cgo: disabled
2020-11-12T05:50:43.554596948Z Cluster Address: https://vault-0.vault-internal:8201
2020-11-12T05:50:43.554608637Z Go Version: go1.14.7
2020-11-12T05:50:43.554678454Z Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
2020-11-12T05:50:43.554693734Z Log Level: info
2020-11-12T05:50:43.554703897Z Mlock: supported: true, enabled: false
2020-11-12T05:50:43.554713272Z Recovery Mode: false
2020-11-12T05:50:43.554722579Z Storage: raft (HA available)
2020-11-12T05:50:43.554732788Z Version: Vault v1.5.5
2020-11-12T05:50:43.554769315Z Version Sha: f5d1ddb3750e7c28e25036e1ef26a4c02379fc01
2020-11-12T05:50:43.554780425Z
2020-11-12T05:50:43.672225223Z ==> Vault server started! Log data will stream in below:
2020-11-12T05:50:43.672519986Z
2020-11-12T05:50:43.673078706Z 2020-11-12T05:50:43.543Z [INFO] proxy environment: http_proxy= https_proxy= no_proxy=
2020-11-12T05:51:57.838970945Z ==> Vault shutdown triggered
I am running a 6 node rancher k3s cluster v1.19.3ks2 on my mac.
Any help would be appreciated
I followed the steps given in this post to deploy my tensorflow model for prediction using GPUs on Google Kubernetes Engine and Kubeflow. I have exposed the service as a load balancer by modifying the YAML file in this way where I changed the type from ClusterIP to LoadBalancer.
spec:
clusterIP: A.B.C.D
externalTrafficPolicy: Cluster
ports:
- name: grpc-tf-serving
nodePort: 30098
port: 9000
protocol: TCP
targetPort: 9000
- name: http-tf-serving-proxy
nodePort: 31399
port: 8000
protocol: TCP
targetPort: 8000
selector:
app: my-model
sessionAffinity: None
type: LoadBalancer
The status changed to:
status:
loadBalancer:
ingress:
- ip: W.X.Y.Z
Service specs (kubectl describe services my-model):
Name: my-model
Namespace: default
Labels: app=my-model
app.kubernetes.io/deploy-manager=ksonnet
ksonnet.io/component=model2
Annotations: getambassador.io/config:
---
apiVersion: ambassador/v0
kind: Mapping
name: tfserving-mapping-my-model-get
prefix: /models/my-model/
rewrite: /
method: GET
service: my-model.default:8000
---
apiVersion: ambassador/v0
kind: Mapping
name: tfserving-mapping-my-model-post
prefix: /models/my-model/
rewrite: /model/my-model:predict
method: POST
service: my-model.default:8000
ksonnet.io/managed:
{"pristine":"H4sIAAAAAAAA/7SRMY/UQAyFe35F5DpzCVweRcHW4QQBWKlQzQMhS/jZEckHmvGt9xplf+OZvfYjXRCgoIyz+/L8xsfgTR+5VxiEkA4vIYWfkQJgHDH+RAHhhYWNgpkB...
Selector: app=my-model
Type: LoadBalancer
IP: A.B.C.D
LoadBalancer Ingress: W.X.Y.Z
Port: grpc-tf-serving 9000/TCP
TargetPort: 9000/TCP
NodePort: grpc-tf-serving 30098/TCP
Endpoints: P.Q.R.S:9000
Port: http-tf-serving-proxy 8000/TCP
TargetPort: 8000/TCP
NodePort: http-tf-serving-proxy 31399/TCP
Endpoints: R.Q.R.S:8000
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
Pods Specs (kubectl describe pods):
Name: my-model-v1-bd6ccb757-qrwdv
Namespace: default
Node: gke-kuberflow-xyz-gpu-pool-5d4ebf17-56mf/SOME_IP
Start Time: Mon, 18 Feb 2019 18:11:24 +0530
Labels: app=my-model
pod-template-hash=682776313
version=v1
Annotations: <none>
Status: Running
IP: P.Q.R.S
Controlled By: ReplicaSet/my-model-v1-bd6ccb757
Containers:
my-model:
Container ID: docker://d14e8261ddfe606393da2ee45badac0136cee98rwa5611c47ad85733ce5d2c925
Image: tensorflow/serving:1.11.1-gpu
Image ID: docker-pullable://tensorflow/serving#sha256:907d7db828b28ewer234d0b3ca10e2d66bcd8ef82c5cccea761fcd4f1190191d2f
Port: 9000/TCP
Host Port: 0/TCP
Command:
/usr/bin/tensorflow_model_server
Args:
--port=9000
--model_name=my-model
--model_base_path=gs://xyz_kuber_app-xyz-identification/export/
State: Running
Started: Mon, 18 Feb 2019 18:11:25 +0530
Ready: True
Restart Count: 0
Limits:
cpu: 4
memory: 4Gi
nvidia.com/gpu: 1
Requests:
cpu: 1
memory: 1Gi
nvidia.com/gpu: 1
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-b6dpn (ro)
my-model-http-proxy:
Container ID: docker://c98e06ad75f3456c353395e9ad2e2e3bcbf0b38cd2634074704439cd5ebf335d
Image: gcr.io/kubeflow-images-public/tf-model-server-http-proxy:v20180606-asdasda
Image ID: docker-pullable://gcr.io/kubeflow-images-public/tf-model-server-http-proxy#sha256:SHA
Port: 8000/TCP
Host Port: 0/TCP
Command:
python
/usr/src/app/server.py
--port=8000
--rpc_port=9000
--rpc_timeout=10.0
State: Running
Started: Mon, 18 Feb 2019 18:11:25 +0530
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 1Gi
Requests:
cpu: 500m
memory: 500Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-b6dpn (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
default-token-b6dpn:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-fsdf3
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
nvidia.com/gpu:NoSchedule
Events: <none>
I used the command python predict.py --url=http://W.X.Y.Z:8000/model/my-model:predict to perform the prediction from the serving_script folder but I am getting the a 500 Internal server error as the response. What is going wrong here?
The code for prediction can be found here: https://github.com/kubeflow/examples/tree/master/object_detection/serving_script
It was a mistake from my end. I was using a different input image array format for the model. I was sending an image tensor instead of encoded image string tensor.
I am trying to set up a cluster of Apache Ignite with persistence enabled. I am trying to start the cluster on Azure Kubernetes with 10 nodes. The problem is that the cluster activation seems to get stuck, but I am able to activate a cluster with 3 nodes in less than 5 minutes.
Here is the configuration I am using to start the cluster:
apiVersion: v1
kind: Service
metadata:
name: ignite-main
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
labels:
main: ignite-main
spec:
type: LoadBalancer
externalTrafficPolicy: Cluster
ports:
- port: 10800 # JDBC port
targetPort: 10800
name: jdbc
- port: 11211 # Activating the baseline (port)
targetPort: 11211
name: control
- port: 8080 # REST port
targetPort: 8080
name: rest
selector:
main: ignite-main
---
#########################################
# Ignite service configuration
#########################################
# Service for discovery of ignite nodes
apiVersion: v1
kind: Service
metadata:
name: ignite
labels:
app: ignite
spec:
clusterIP: None
# externalTrafficPolicy: Cluster
ports:
# - port: 9042 # custom value.
# name: discovery
- port: 47500
name: discovery
- port: 47100
name: communication
- port: 11211
name: control
selector:
app: ignite
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ignite-cluster
labels:
app: ignite
main: ignite-main
spec:
selector:
matchLabels:
app: ignite
main: ignite-main
replicas: 5
template:
metadata:
labels:
app: ignite
main: ignite-main
spec:
volumes:
- name: ignite-storage
persistentVolumeClaim:
claimName: ignite-volume-claim # Must be equal to the PersistentVolumeClaim created before.
containers:
- name: ignite-node
image: ignite.azurecr.io/apacheignite/ignite:2.7.0-SNAPSHOT
env:
- name: OPTION_LIBS
value: ignite-kubernetes
- name: CONFIG_URI
value: https://file-location
- name: IGNITE_H2_DEBUG_CONSOLE
value: 'true'
- name: IGNITE_QUIET
value: 'false'
- name: java.net.preferIPv4Stack
value: 'true'
- name: JVM_OPTS
value: -server -Xms10g -Xmx10g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
ports:
- containerPort: 47100 # communication SPI port number.
- containerPort: 47500 # discovery SPI port number.
- containerPort: 49112 # JMX port number.
- containerPort: 8080 # REST port number.
- containerPort: 10800 # SQL port number.
- containerPort: 11211 # Activating the baseline (port)
imagePullSecrets:
- name: docker-cred
I was trying to activate the cluster remotely by providing --host parameter, like:
./control.sh --host x.x.x.x --activate
Instead, I tried activating the cluster by logging into one of the kubernetes nodes and activating from there. The detailed steps are mentioned here