RavenDB Replication Issue - database cannot be found - ravendb

Question

Have followed the documentation, but am unable to make replication work for RavenDB over the WAN.
Scenario:
Using Raven build #2261
Master DB: has a local name of "it23"
Slave DB: has a remote name of "http://184.169.xxx.xxx" (xxx's are for
privacy)
On both servers I have created a database called "TonyTest".
On the Master db, I have set up replication using the following document:
{
"Destinations": [
{
"Url": "http://184.169.xxx.xxx:8080",
"Username": null,
"Password": null,
"Domain": null,
"ApiKey": null,
"Database": "TonyTest",
"TransitiveReplicationBehavior": "None",
"IgnoredClient": false,
"Disabled": false,
"ClientVisibleUrl": null
}
]
}
When browsing to the remote server using the same URL of: http://184.169.xxx.xxx:8080, the RavenDB studio launches correctly, and I can see the TestTony database. This seems to confirm that the URL is formatted correctly.
However, the master database immediately generates a document showing failures:
{
"Destination": "http://184.169.xxx.xxx:8080/databases/TonyTest",
"FailureCount": 142
}
When we look at the logs for the REMOTE db, we see that there IS communication with the master, but the replication doesn't complete.
Debug 3/9/2013 12:19:44 AM Document with key 'Raven/Replication/Sources/http://it23:8080/databases/TonyTest' was not found Raven.Storage.Esent.StorageActions.DocumentStorageActions
It looks like the remote server is saying that the db "TonyTest' can't be found, but it IS created.
Can anyone spot my mistake?
Per Ayende's request, here are some log samples from LOCAL server after attempting to setup replication (again I replaced IPs with xxx for privacy). We do not see any errors in the LOCAL db's log. And we do see errors popup in the REMOTE db log. This seems to imply that the LOCAL db is connecting to the REMOTE db, but the replication does not happen. Here are the LOCAL logs:
Debug 3/11/2013 3:17:00 PM No work was found, workerWorkCounter: 17626, for: ReducingExecuter, will wait for additional work Raven.Database.Indexing.WorkContext
Debug 3/11/2013 3:17:00 PM Going to index 1 documents in IndexName: Raven/DocumentsByEntityName, LastIndexedEtag: 00000001-0000-0100-0000-000000002265: (Raven/Replication/Destinations/184.169.xxx.xxx8080databasesTonyTest) Raven.Database.Indexing.AbstractIndexingExecuter
Debug 3/11/2013 3:17:00 PM Document with key 'Raven/Studio/PriorityColumns' was not found Raven.Storage.Esent.StorageActions.DocumentStorageActions
Debug 3/11/2013 3:16:56 PM Going to index 1 documents in IndexName: Raven/DocumentsByEntityName, LastIndexedEtag: 00000001-0000-0100-0000-000000002256: (Raven/Replication/Destinations/184.169.xxx.xxx8080databasesTonyTest) Raven.Database.Indexing.AbstractIndexingExecuter
Update 3/11 8:24p Pacific time
I am now seeing the following errors in the MASTER/Local raven logs:
Failed to close response
System.AggregateException: One or more errors occurred. ---> System.Net.HttpListenerException: An operation was attempted on a nonexistent network connection
at System.Net.HttpResponseStream.Dispose(Boolean disposing)
at System.IO.Stream.Close()
at Raven.Database.Util.Streams.BufferPoolStream.Dispose(Boolean disposing) in c:\Builds\RavenDB-Stable\Raven.Database\Util\Streams\BufferPoolStream.cs:line 144
at System.IO.Stream.Close()
at Raven.Database.Impl.ExceptionAggregator.Execute(Action action) in c:\Builds\RavenDB-Stable\Raven.Database\Impl\ExceptionAggregator.cs:line 23
--- End of inner exception stack trace ---
at Raven.Database.Impl.ExceptionAggregator.ThrowIfNeeded() in c:\Builds\RavenDB-Stable\Raven.Database\Impl\ExceptionAggregator.cs:line 38
at Raven.Database.Server.Abstractions.HttpListenerResponseAdapter.Close() in c:\Builds\RavenDB-Stable\Raven.Database\Server\Abstractions\HttpListenerResponseAdapter.cs:line 94
at Raven.Database.Server.Abstractions.HttpListenerContextAdpater.FinalizeResponse() in c:\Builds\RavenDB-Stable\Raven.Database\Server\Abstractions\HttpListenerContextAdpater.cs:line 92
---> (Inner Exception #0) System.Net.HttpListenerException (0x80004005): An operation was attempted on a nonexistent network connection
at System.Net.HttpResponseStream.Dispose(Boolean disposing)
at System.IO.Stream.Close()
at Raven.Database.Util.Streams.BufferPoolStream.Dispose(Boolean disposing) in c:\Builds\RavenDB-Stable\Raven.Database\Util\Streams\BufferPoolStream.cs:line 144
at System.IO.Stream.Close()
at Raven.Database.Impl.ExceptionAggregator.Execute(Action action) in c:\Builds\RavenDB-Stable\Raven.Database\Impl\ExceptionAggregator.cs:line 23<---

Answer 1

Solved this. Although I don't understand the full reason why.
On the SLAVE server, you must set the raven.server.exe config file to have the following key:
<add key="Raven/AnonymousAccess" value="All"/>
The default was
<add key="Raven/AnonymousAccess" value="Get"/>.
The default worked fine when the master and slave were on the same machine. But when the master and slave were on separate machines (either on the LAN, or across the WAN) replication failed.
I could never find a log entry on the master that pointed toward his problem. The only log entry I could see was on the slave which said that Raven/Replication/Sources/ was not found. I realized that the master was connecting to the slave, but the slave was unable to create the "Raven/Replication/Sources/" document remotely.