Cannot use spring cloud config and istio 1.1.1 together-cannot recover when HTTP 404 error to get remote config - spring-cloud-config

Question

when I'm tring to mix the spring cloud config with istio 1.1.1, When my app container(with istio envoy auto-injected) starts, the spring cloud config client will try to get config(applicationContext.yaml) from remote cloud config server(started in advance with good status), unfornately it fails with HTTP 404 error. Even if I've configged my app to have retry for cloud config client, it keeps retring alway with HTTP 404 error(I've confirmed the config server URL is correct from another container) and there's no chance to recover. It happens sometimes. I knew that Istio envoy and my app are in the same kubernetes POD, the app may start before istio envoy, in which case there might be network error but as soon as the envoy is up, everything should be OK. I really don't understand why my app cannot recover automatically. Here're my diagnostic steps:
1. Add retry mechanism in my app(with retry libs included in POM and modified yaml. - retry works but each retry failed with HTTP 404 error
spring-config/
fail-fast: true
retry:
initial-interval: 10000
max-attempts: 100
2. Add 'sleep xx' before my java app starts in my app k8s deployment file - less chance to have HTTP 404 error, but problem is not eliminated
command: ["/bin/sh","-c","sleep 20; java -jar -Xms512m -Xmx1024m app.jar"]
3. get the istio envoy's access log and compare the victim app's and good app's - it sounds like the good log has values for upstream_cluster and upstream_cluster key; the fields for the bad log are empty
the good access log
{
"response_code": "200",
"user_agent": "Java/1.8.0_121",
"response_flags": "-",
"start_time": "2019-06-25T01:17:29.661Z",
"method": "2019-06-25T01:17:29.661Z",
"request_id": "d3d27512-161b-4303-bb48-05a6e19e05b7",
"upstream_host": "172.20.3.104:9083",
"x_forwarded_for": "-",
"requested_server_name": "-",
"bytes_received": "0",
"istio_policy_status": "-",
"bytes_sent": "1144",
"upstream_cluster": "outbound|9083||fota-spring-config.ns-fota.svc.cluster.local",
"downstream_remote_address": "172.20.2.115:45816",
"path": "/fota-spring-config/fota-task/dev/master",
"authority": "fota-spring-config.ns-fota.svc.cluster.local:9083",
"protocol": "HTTP/1.1",
"upstream_service_time": "289",
"upstream_local_address": "-",
"duration": "290",
"downstream_local_address": "172.21.1.152:9083"
}
the bad access log:
{
"upstream_cluster": "-",
"downstream_remote_address": "172.20.2.118:41980",
"path": "/fota-spring-config/fota-dmserver/dev/master",
"authority": "fota-spring-config.ns-fota.svc.cluster.local:9083",
"protocol": "HTTP/1.1",
"upstream_service_time": "-",
"upstream_local_address": "-",
"duration": "0",
"downstream_local_address": "172.21.1.152:9083",
"response_code": "404",
"user_agent": "Java/1.8.0_121",
"response_flags": "NR",
"start_time": "2019-06-25T01:21:24.197Z",
"method": "2019-06-25T01:21:24.197Z",
"request_id": "346716e4-1def-465f-b370-cb1e71e30d25",
"upstream_host": "-",
"x_forwarded_for": "-",
"requested_server_name": "-",
"bytes_received": "0",
"istio_policy_status": "-",
"bytes_sent": "0"
}

Answer 1

the K8S deployment file is attached.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: fota-car
spec:
template:
metadata:
labels:
app: fota-car
version: v1
spec:
serviceAccountName: fota-serviceaccount
imagePullSecrets:
- name: uaes-docker2
containers:
- name: fota-car
image: 192.168.119.22:18080/uaes-fota/fota-car:dev-release-1.0.0
imagePullPolicy: Always
ports:
- containerPort: 8085
env:
- name: SPRING_DATASOURCE_URL
value: jdbc:mysql://mysql-ali-dev.ns-fota-ext-svc/fota-car?useUnicode=true&characterEncoding=utf-8&useSSL=false
- name: SPRING_DATASOURCE_USERNAME
valueFrom:
secretKeyRef:
name: mysql-ali-dev-secret
key: username
- name: SPRING_DATASOURCE_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-ali-dev-secret
key: password
command: ["/bin/sh","-c","java -jar -Xms512m -Xmx1024m app.jar"]
readinessProbe:
httpGet:
path: /actuator/health
port: 18085
initialDelaySeconds: 60
timeoutSeconds: 1
kind: Service
apiVersion: v1
metadata:
labels:
app: fota-car
name: fota-car
spec:
ports:
- name: http
port: 8085
selector:
app: fota-car