Only 1 of 2 pods deploy, other pod is not authorised to pull container - openshift-origin

Question

I set up a production project using the same structure as my staging project minus the BuildConfigurations and then tagged my containers from the staging image stream to the prod image stream.
oc tag my-staging/nginx:latest my-prod/nginx:prod
oc tag my-staging/gunicorn:latest my-prod/gunicorn:prod
oc tag my-staging/celery-worker:latest my-prod/celery-worker:prod
Each of these as a DeploymentConfig for 2 replicas. The first two have come up with both pods, but the celery-worker container is only coming up with a single pod. The other pod generates an error:
Failed to pull image
"172.x.x.x:5000/my-staging/celery-worker#sha256:xxx":
unauthorized: authentication required
I don't get how one kubelet can have access but not another. Especially since all of the other pods are up.
Here's the logs from the registry:
10.1.3.1 - - [22/Feb/2016:02:52:58 +0000] "GET /v2/cwl-staging/cwl-leadershift-20-celery-worker/manifests/sha256:7a2608ce648b767d65209410fd9f0e8d2fe3f559367c77ba45ba9a713940f83a HTTP/1.1" 401 176 "" "docker/1.8.2-el7.centos go/go1.4.2 kernel/3.10.0-327.4.5.el7.x86_64 os/linux arch/amd64"
time="2016-02-22T02:52:58.297372303Z" level=error msg="OpenShift access denied: User \"system:serviceaccount:cwl-production:default\" cannot get imagestreams/layers in project \"cwl-staging\"" go.version=go1.4.2 http.request.host="172.30.140.184:5000" http.request.id=71a32c41-9e91-40be-9774-166bfa7264f8 http.request.method=GET http.request.remoteaddr="10.1.3.1:48777" http.request.uri="/v2/cwl-staging/cwl-leadershift-20-celery-worker/manifests/sha256:7a2608ce648b767d65209410fd9f0e8d2fe3f559367c77ba45ba9a713940f83a" http.request.useragent="docker/1.8.2-el7.centos go/go1.4.2 kernel/3.10.0-327.4.5.el7.x86_64 os/linux arch/amd64" instance.id=180a3a82-b568-40ab-aaa0-538588e8e765 vars.name="cwl-staging/cwl-leadershift-20-celery-worker" vars.reference="sha256:7a2608ce648b767d65209410fd9f0e8d2fe3f559367c77ba45ba9a713940f83a"
time="2016-02-22T02:52:58.297449598Z" level=error msg="error authorizing context: access denied" go.version=go1.4.2 http.request.host="172.30.140.184:5000" http.request.id=71a32c41-9e91-40be-9774-166bfa7264f8 http.request.method=GET http.request.remoteaddr="10.1.3.1:48777" http.request.uri="/v2/cwl-staging/cwl-leadershift-20-celery-worker/manifests/sha256:7a2608ce648b767d65209410fd9f0e8d2fe3f559367c77ba45ba9a713940f83a" http.request.useragent="docker/1.8.2-el7.centos go/go1.4.2 kernel/3.10.0-327.4.5.el7.x86_64 os/linux arch/amd64" instance.id=180a3a82-b568-40ab-aaa0-538588e8e765 vars.name="cwl-staging/cwl-leadershift-20-celery-worker" vars.reference="sha256:7a2608ce648b767d65209410fd9f0e8d2fe3f559367c77ba45ba9a713940f83a"

Answer 1

The problem is that the system:image-puller role wasn't granted to my-prod.
Grant the role on the my-staging project:
oc policy add-role-to-user system:image-puller system:serviceaccount:my-prod:default -n my-staging
Delete the stuck pods so they gain the new credentials to pull the image.
Appropriate section of the Openshift documentation.