What's the right setup to replicate a remote docker registry on artifactory - reverse-proxy

Question

I've got two artifactory instances with one serving as primary docker registry behind an apache2 proxy. Now, I'd like to have the second one acting also as a docker registry but with a remote registry which points to the primary instance.
When trying that I got this message when testing the active replication:
Error testing pull replication config: Unknown host 'api: Name or service not known
Here is the full stacktrace in the logs:
2017-10-26 15:30:58,004 [art-exec-3] [ERROR] (o.a.a.c.BasicStatusHolder:212) - Error occurred while performing folder replication for 'private-docker-registry:': api
java.net.UnknownHostException: api
at java.net.InetAddress.getAllByName0(InetAddress.java:1280) ~[na:1.8.0_121]
at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[na:1.8.0_121]
at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[na:1.8.0_121]
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:111) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71) ~[httpclient-4.5.1.jar:4.5.1]
at org.jfrog.client.http.CloseableHttpClientDecorator.doExecute(CloseableHttpClientDecorator.java:90) ~[jfrog-http-client-1.2.4.jar:na]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.1.jar:4.5.1]
at org.artifactory.repo.HttpRepo.doExecuteMethod(HttpRepo.java:493) ~[artifactory-core-5.2.0.jar:na]
at org.artifactory.repo.HttpRepo.executeMethod(HttpRepo.java:510) ~[artifactory-core-5.2.0.jar:na]
at org.artifactory.repo.HttpRepo.executeMethod(HttpRepo.java:461) ~[artifactory-core-5.2.0.jar:na]
at org.artifactory.addon.replication.core.context.RemoteReplicationRequestExecutor.execute(RemoteReplicationRequestExecutor.java:28) ~[artifactory-addon-replication-5.2.0.jar:na]
at org.artifactory.addon.replication.core.context.server.TargetServerInfoResolver.executeRequestAndSetDetails(TargetServerInfoResolver.java:92) ~[artifactory-addon-replication-5.2.0.jar:na]
at org.artifactory.addon.replication.core.context.server.TargetServerInfoResolver.resolveTargetInfo(TargetServerInfoResolver.java:49) ~[artifactory-addon-replication-5.2.0.jar:na]
at org.artifactory.addon.replication.core.BaseReplicationProducer.resolveTargetInfo(BaseReplicationProducer.java:92) ~[artifactory-addon-replication-5.2.0.jar:na]
at org.artifactory.addon.replication.core.BaseReplicationProducer.run(BaseReplicationProducer.java:78) ~[artifactory-addon-replication-5.2.0.jar:na]
at org.artifactory.addon.replication.core.remote.RemoteReplicator.replicate(RemoteReplicator.java:56) [artifactory-addon-replication-5.2.0.jar:na]
at org.artifactory.addon.replication.core.remote.RemoteReplicator.replicate(RemoteReplicator.java:29) [artifactory-addon-replication-5.2.0.jar:na]
at org.artifactory.addon.replication.core.ReplicationAddonImpl.performRemoteReplication(ReplicationAddonImpl.java:91) [artifactory-addon-replication-5.2.0.jar:na]
at org.artifactory.repo.replication.RemoteReplicationJob.onExecute(RemoteReplicationJob.java:101) [artifactory-core-5.2.0.jar:na]
at org.artifactory.schedule.quartz.QuartzCommand.execute(QuartzCommand.java:52) [artifactory-storage-common-5.2.0.jar:na]
at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.1.jar:na]
at org.artifactory.schedule.ArtifactoryConcurrentExecutor$RunnableWrapper.run(ArtifactoryConcurrentExecutor.java:104) [artifactory-storage-common-5.2.0.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
What am I doing wrong ?
Same thing occurs when trying to replicate my private account from dockerhub
Thank you for your help
EDIT:
Ok now, I managed to partially get this working. I'm saying "partially" because it's actually not anymore displaying a stacktrace and the different image references are synced but for some reason the layers themselves are not copied over to the remote. Looking closer to the logs, it looks like something wrong with permissions:
2017-11-26 23:30:08,611 [replication-consumer-1511713803259-0] [WARN ] (o.a.r.s.RepositoryServiceImpl:901) - Cannot set properties on 'remote-docker-cache:my/irc-base/0.1.0-b25-09e84a42/sha256__096495a59c0e938508a5c9d4cb003d5e4556e0fb8f1befd9469903a6d446e797': Item not found.
2017-11-26 23:30:08,611 [replication-consumer-1511713803259-0] [ERROR] (o.a.a.c.BasicStatusHolder:214) - Unable to set properties for remote-docker-cache:my/image/0.1.0-b25-09e84a42/sha256__096495a59c0e938508a5c9d4cb003d5e4556e0fb8f1befd9469903a6d446e797
2017-11-26 23:30:10,947 [replication-consumer-1511713803259-0] [WARN ] (o.a.r.s.RepositoryServiceImpl:901) - Cannot set properties on 'remote-docker-cache:my/image/0.1.0-b25-09e84a42/sha256__5523a881c6c86f188888bba730867591402f40db1be718c64726b1723c5abbf5': Item not found.
2017-11-26 23:30:10,947 [replication-consumer-1511713803259-0] [ERROR] (o.a.a.c.BasicStatusHolder:214) - Unable to set properties for remote-docker-cache:my/image/0.1.0-b25-09e84a42/sha256__5523a881c6c86f188888bba730867591402f40db1be718c64726b1723c5abbf5
2017-11-26 23:30:12,145 [replication-consumer-1511713803259-0] [WARN ] (o.a.r.s.RepositoryServiceImpl:901) - Cannot set properties on 'remote-docker-cache:my/image/0.1.0-b25-09e84a42/sha256__6b888ef3098531f0c7000584ce049b24e4559cfab8c4141fcf62bcfd60f6b177': Item not found.
2017-11-26 23:30:12,145 [replication-consumer-1511713803259-0] [ERROR] (o.a.a.c.BasicStatusHolder:214) - Unable to set properties for remote-docker-cache:my/image/0.1.0-b25-09e84a42/sha256__6b888ef3098531f0c7000584ce049b24e4559cfab8c4141fcf62bcfd60f6b177
In the same time on the registry hosting the layers:
20171126173008|1|REQUEST|192.168.210.102|admin|GET|/api/storage/docker/remote-docker/my/image/0.1.0-b25-09e84a42/sha256__096495a59c0e938508a5c9d4cb003d5e4556e0fb8f1befd9469903a6d446e797|HTTP/1.0|200|0
20171126173008|1|REQUEST|192.168.210.102|anonymous|HEAD|/api/docker/remote-docker/my/image/0.1.0-b25-09e84a42/sha256__5523a881c6c86f188888bba730867591402f40db1be718c64726b1723c5abbf5|HTTP/1.0|403|0
20171126173010|0|REQUEST|192.168.210.102|non_authenticated_user|GET|/api/storage/docker/remote-docker/my/image/0.1.0-b25-09e84a42/sha256__5523a881c6c86f188888bba730867591402f40db1be718c64726b1723c5abbf5|HTTP/1.0|401|0
20171126173010|0|REQUEST|192.168.210.102|admin|GET|/api/storage/docker/remote-docker/my/image/0.1.0-b25-09e84a42/sha256__5523a881c6c86f188888bba730867591402f40db1be718c64726b1723c5abbf5|HTTP/1.0|200|0
20171126173011|0|REQUEST|192.168.210.102|anonymous|HEAD|/api/docker/docker/remote-docker/my/image/0.1.0-b25-09e84a42/sha256__6b888ef3098531f0c7000584ce049b24e4559cfab8c4141fcf62bcfd60f6b177|HTTP/1.0|403|0
20171126173011|0|REQUEST|192.168.210.102|non_authenticated_user|GET|/api/storage/docker/remote-docker/my/image/0.1.0-b25-09e84a42/sha256__6b888ef3098531f0c7000584ce049b24e4559cfab8c4141fcf62bcfd60f6b177|HTTP/1.0|401|0
I'm trying to do the replication with the admin user which should have full access to the different registries.
Now the funny thing (maybe not so funny) is that if I allow the user Anonymous to have access to the registry, the replication works just fine. However, from a security point of view I cannot just let Anonymous access on these private registries.
Thanks again for your help

Answer 1

All credit goes to Yonatan from JFrog support
I finally managed to solve the issue thanks to JFrog support and here is what I was doing wrong and how it should be solved:
In the remote repository settings, I put the following as target url:
https://myregistry.example.com.
I was then kindly suggested by jFrog support to prefix the url with /api/docker/myregistry having the the following as target url:
https://myregistry.example.com/api/docker/myregistry
More inforation can be found here.
Edit:
Here is the exact reply from JFrog support (which might be more accurate than my trial to bulky translate what I understood):
"The issue you are experiencing is due to a misconfiguration in the target URL.
For some packaging formats, when using the corresponding client to access a repository through Artifactory, the repository key in the URL needs to be prefixed with api/ in the path. For example, in the case of Docker repositories, the repository key should be prefixed with api/docker.
Nevertheless, there are exceptions to this rule. For example, when replicating Maven repositories, you do not need to add a prefix the remote repository path. (This is the reason why you did not encounter issues with replicating Maven repositories)
You can find the full list here.
With regards to your scenario, please try to configure the following URL:
https://myregistry.example.com/api/docker/myregistry
Please note you have to add the target repository name."
Thank you Yonatan