We have seen a noticeable uptick in problems related to RavenDB connections. We're using the IIS server connection method.
I've checked the index errors and I don't see anything listed.
This server is replicated with a MySQL server and there is one error related to replication in the log from today. Could the client stack trace errors actually be coming from the replicated server and not Raven?
EDIT
The client in this case is a single job that runs a few hundred times a day...succeeding many of those times but increasing in failures due to these errors.
Sql Replication failure to replication
Here are the partial stack trace errors from the client log:
[WebException: Unable to connect to the remote server]
System.Net.HttpWebRequest.GetRequestStream(TransportContext& context):367
System.Net.HttpWebRequest.GetRequestStream():3
Raven.Abstractions.Connection.HttpRequestHelper.WriteDataToRequest(HttpWebRequest req, String data, Boolean disableCompression):7
Raven.Client.Connection.ServerClient.DirectBatch(IEnumerable`1 commandDatas, String operationUrl):171
Raven.Client.Connection.ReplicationInformer.TryOperation[T](Func`2 operation, String operationUrl, Boolean avoidThrowing, T& result, Boolean& wasTimeout):35
Raven.Client.Connection.ReplicationInformer.ExecuteWithReplication[T](String method, String primaryUrl, Int32 currentRequest, Int32 currentReadStripingBase, Func`2 operation):169
Raven.Client.Connection.ServerClient.ExecuteWithReplication[T](String method, Func`2 operation):33
Raven.Client.Document.DocumentSession.SaveChanges():65
and
[WebException: Unable to connect to the remote server]
System.Net.HttpWebRequest.GetResponse():570
Raven.Client.Connection.HttpJsonRequest.ReadJsonInternal(Func`1 getResponse):45
Raven.Client.Connection.HttpJsonRequest.ReadResponseJson():206
Raven.Client.Connection.ServerClient.DirectGet(String[] ids, String operationUrl, String[] includes, String transformer, Dictionary`2 queryInputs, Boolean metadataOnly):631
Raven.Client.Connection.ServerClient+<>c__DisplayClass77.<Get>b__76(String u):51
Raven.Client.Connection.ReplicationInformer.TryOperation[T](Func`2 operation, String operationUrl, Boolean avoidThrowing, T& result, Boolean& wasTimeout):35
Raven.Client.Connection.ReplicationInformer.ExecuteWithReplication[T](String method, String primaryUrl, Int32 currentRequest, Int32 currentReadStripingBase, Func`2 operation):169
Raven.Client.Connection.ServerClient.ExecuteWithReplication[T](String method, Func`2 operation):33
Raven.Client.Document.HiLoKeyGenerator.GetDocument(IDatabaseCommands databaseCommands):41
Raven.Client.Document.HiLoKeyGenerator.GetNextRange(IDatabaseCommands databaseCommands):109
Raven.Client.Document.HiLoKeyGenerator.NextId(IDatabaseCommands commands):58
Raven.Client.Document.HiLoKeyGenerator.GenerateDocumentKey(IDatabaseCommands databaseCommands, DocumentConvention convention, Object entity):9
Raven.Client.Document.MultiTypeHiLoKeyGenerator.GenerateDocumentKey(IDatabaseCommands databaseCommands, DocumentConvention conventions, Object entity):174
Raven.Client.Document.DocumentStore+<>c__DisplayClass4.<Initialize>b__2(String dbName, IDatabaseCommands databaseCommands, Object entity):20
Raven.Client.Document.DocumentConvention.GenerateDocumentKey(String dbName, IDatabaseCommands databaseCommands, Object entity):164
Raven.Client.Document.GenerateEntityIdOnTheClient.GenerateDocumentKeyForStorage(Object entity):46
Raven.Client.Document.InMemoryDocumentSessionOperations.StoreInternal(Object entity, Etag etag, String id, Boolean forceConcurrencyCheck):79
Raven.Client.Document.InMemoryDocumentSessionOperations.Store(Object entity):23
The error says the client cannot connect to the the server. Maybe the client is offline. Or the server is. Or a firewall is in the way. Or maybe the server is there but blows up before returning a response to the client. If the issue is intermittent, and users arent complaining, it's probably just a connectivity thing that you can ignore. If users are complaining, you should look in server-side logs.
Related
I have a dacpac file that is used from SQL 2005 to SQL 2016 databases. Our developers have been using SQL 2016 for the past few months, so we are now moving to production with it.
When attempting to upgrade a database (no previous dacpackage) using our .dacpac file, it will give this error below
===================================
Method not found: 'Void Microsoft.SqlServer.TransactSql.ScriptDom.AlterTableAlterColumnStatement.set_Encryption(Microsoft.SqlServer.TransactSql.ScriptDom.ColumnEncryptionDefinition)'. (Microsoft.Data.Tools.Schema.Sql)
------------------------------
Program Location:
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeploymentPlanGenerator.AlterStatementGenerator.GenerateForSimpleColumn(SqlSimpleColumn oldColumn, SqlSimpleColumn newColumn)
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeploymentPlanGenerator.AlterStatementGenerator.GenerateAlterTableForColumn(IModelElement oldElement, IModelElement newElement)
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeploymentPlanGenerator.AlterStatementGenerator.GenerateStatement(IModelElement oldElement, IModelElement newElement)
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeploymentPlanGenerator.Sql90AlterStatementGenerator.GenerateScriptDom(IModelElement oldElement, IModelElement newElement)
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeploymentPlanGenerator.DeploymentScriptDomGenerator.GenerateTableAlter(SqlTable sourceTable, SqlTable targetTable)
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeploymentPlanGenerator.DeploymentScriptDomGenerator.GenerateFragment(Int32 operation, IModelElement element)
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeploymentPlanGenerator.DeploymentScriptDomGenerator.GenerateSteps(Int32 operation, IModelElement element)
at Microsoft.Data.Tools.Schema.Sql.Deployment.OrderedStepGenerator.Add(DeploymentScriptDomGenerator scriptDom, IModelElement element, Int32 operation)
at Microsoft.Data.Tools.Schema.Sql.Deployment.Analyzers.PlanMediator.BuildDependencyOrderedSteps(Int32 operation, List`1 classOrder, List`1 operationOrder, Dictionary`2 changes, Boolean preserveGraphs, Dictionary`2& relating, Dictionary`2& related)
at Microsoft.Data.Tools.Schema.Sql.Deployment.Analyzers.PlanMediator.BuildOperations()
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeploymentPlanGenerator.OnGeneratePlan()
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeploymentPlanGenerator.GeneratePlan(List`1 drops)
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeployment.BuildPlan()
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeployment.Controller.CreatePlan()
at Microsoft.SqlServer.Dac.DacServices.<>c__DisplayClass25.<CreateDeploymentArtifactGenerationOperation>b__23(Object operation, CancellationToken token)
at Microsoft.SqlServer.Dac.Operation.Microsoft.SqlServer.Dac.IOperation.Run(OperationContext context)
at Microsoft.SqlServer.Dac.ReportMessageOperation.Microsoft.SqlServer.Dac.IOperation.Run(OperationContext context)
at Microsoft.SqlServer.Dac.OperationExtension.Execute(IOperation operation, DacLoggingContext loggingContext, CancellationToken cancellationToken)
at Microsoft.SqlServer.Dac.DacServices.GenerateDeployReport(DacPackage package, String targetDatabaseName, DacDeployOptions options, Nullable`1 cancellationToken)
at Microsoft.SqlServer.Management.Dac.DacWizard.ReviewPlanPage.ActionStepCalculator_DoWork(Object sender, DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)
As far as I can tell, we don't have Encryption enabled at all on any of the tables. This is a fresh database. We've even tried recreating it from scratch, but we get the same error once we try to apply the dacpac file.
Does anyone know what I can do to get the server to recognize the set_Encryption method that it seems to be missing? Any suggestions are appreciated.
Edit: I've further determined that it is not a SQL 2016 issue, but rather a SQL Server Management Studio 2016 issue.
I was able to get around this error, by first running the dacpac through SqlPackage.exe. After that- I was able to publish through management studio.
I'm trying to install TFS2013 (en_visual_studio_team_foundation_server_2013_with_update_4_x86_x64) on my machine (windows 8.1 x64)
I had a existing SQL instance installed and tried to use that one during my TFS installation but I kept on getting error messages about TFS not being able to connect to my SQL instance.
I started googling te error message and found guides about how you needed to config the SQL to be put on port 1433, add Firewall Rules,.. I tried literally everything, nothing worked.
So I decided to completely remove my SQL instance and install the SQL SERVER 2012 Express edition that comes with the TFS installer. The SQL installation completes successfully, but when configuring TFS again, the same error popped up.
So I went another step further and removed the SQL Server instance again and downloaded the SQL SERVER 2014 Express edition. Yet, same problem.
I've been looking around the web for the error but can't seem to find anything useful. Note that using a database on the SQL SERVER instance from code or connecting using the SQL Management Studio works like charm!
I started inspecting the full blown error log TFS provided me with and found this:
TF400129: TF400129: Error from readiness check: Verifying SQL
connection [Info #18:58:47.025] +-+-+-+-+-| Verifying SQL connection
|+-+-+-+-+- [Info #18:58:47.025] Starting Node: VSQLCONNECT [Info
#18:58:47.025] NodePath :
VINPUTS/Progress/Conditional/VSQLNOTLOCALDB/VSQLISRUNNING/VSQLCONNECT
> [Error #18:58:47.025] System.OverflowException: Arithmetic operation
resulted in an overflow. at
SNIOpenSyncExWrapper(SNI_CLIENT_CONSUMER_INFO* , SNI_ConnWrapper** )
at SNINativeMethodWrapper.SNIOpenSyncEx(ConsumerInfo consumerInfo,
String constring, IntPtr& pConn, Byte[] spnBuffer, Byte[]
instanceName, Boolean fOverrideCache, Boolean fSync, Int32 timeout,
Boolean fParallel) at
System.Data.SqlClient.SNIHandle..ctor(ConsumerInfo myInfo, String
serverName, Byte[] spnBuffer, Boolean ignoreSniOpenTimeout, Int32
timeout, Byte[]& instanceName, Boolean flushCache, Boolean fSync,
Boolean fParallel) at
System.Data.SqlClient.TdsParserStateObject.CreatePhysicalSNIHandle(String
serverName, Boolean ignoreSniOpenTimeout, Int64 timerExpire, Byte[]&
instanceName, Byte[] spnBuffer, Boolean flushCache, Boolean async,
Boolean fParallel) at
System.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo,
SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout,
Int64 timerExpire, Boolean encrypt, Boolean trustServerCert, Boolean
integratedSecurity, Boolean withFailover)
...
For completeness, this is the error screen I can't pass:
I'm really at a dead end here. Ive been trying to get this fixed for days now and I come here as my last resort. Thanks in advance.
I am getting some random disconnects from Booksleeve when deployed in the production environment. I have run tests on my machine and I can not seem to reproduce the errors. I am not sure what to check so I decided to post here.
My set-up:
2 x Redis 2.6.16 (Debian Linux)
2 x Redis sentinel (running on the same machines as redis)
4 x Windows Server 2008 R2
Application connection handling
My app has one static connection that is opened using ConnectionUtils. I also have a timer (System.Timers.Timer) that runs every connection.Ping every 5'' to check whether the connection is still open. If I get an exception for the Ping I log it, close and dispose the connection and replace it with a new one using ConnectionUtils again.
I am also logging Task.UnobservedExceptions
Here is what I am getting in my logs:
A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. ---> System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
at BookSleeve.RedisMessage.WriteUnified(Stream stream, Byte[] value) in d:\Dev\BookSleeve\BookSleeve\RedisCommand.cs:line 271
at BookSleeve.RedisMessage.RedisMessageMulti.Write(Stream stream) in d:\Dev\BookSleeve\BookSleeve\RedisCommand.cs:line 564
at BookSleeve.RedisConnectionBase.WriteMessage(Int32& db, RedisMessage next, IList`1 queued) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1597
at BookSleeve.RedisConnectionBase.EnqueueMessage(RedisMessage message, Boolean queueJump) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1965
at BookSleeve.RedisConnectionBase.ExecuteInt64(RedisMessage message, Boolean queueJump, Object state) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1688
at System.Threading.Tasks.Task.Execute()
--- End of inner exception stack trace ---
---> (Inner Exception #0) System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
at BookSleeve.RedisMessage.WriteUnified(Stream stream, Byte[] value) in d:\Dev\BookSleeve\BookSleeve\RedisCommand.cs:line 271
at BookSleeve.RedisMessage.RedisMessageMulti.Write(Stream stream) in d:\Dev\BookSleeve\BookSleeve\RedisCommand.cs:line 564
at BookSleeve.RedisConnectionBase.WriteMessage(Int32& db, RedisMessage next, IList`1 queued) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1597
at BookSleeve.RedisConnectionBase.EnqueueMessage(RedisMessage message, Boolean queueJump) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1965
at BookSleeve.RedisConnectionBase.ExecuteInt64(RedisMessage message, Boolean queueJump, Object state) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1688
at System.Threading.Tasks.Task.Execute()
And this is my connection monitor reporting on the error in the ping:
Redis connection is broken: System.InvalidOperationException: The connection has been closed (Error); no new messages can be delivered; the last command was sent 1453ms ago
at BookSleeve.RedisConnectionBase.EnqueueMessage(RedisMessage message, Boolean queueJump) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1965
at BookSleeve.RedisConnectionBase.ExecuteInt64(RedisMessage message, Boolean queueJump, Object state) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1688
at BookSleeve.RedisConnectionBase.PingImpl(Boolean queueJump, Boolean duringInit, Object state) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 129
at Atcom.Data.Caching.RedisConnectionContainer.CheckConnections(Object sender, EventArgs e) in C:\Projects\Atcom.Data.Caching.Redis\Atcom.Data.Caching.Redis\RedisConnectionContainer.cs:line 66
I have tried connecting through both the Sentinels and directly to Redis but I am getting the same thing.
I have asked my network administrator to check but he says there is no problem (and I cant really argue with what I dont know oO)
I have reviewed Redis settings and documentation and as far as I can tell clients that do not use monitor etc. should not get disconnected by redis. Idle time disconnect is also disabled on redis.
Could sentinel (since its beta) be the culprit?
Any help troubleshooting the problem would be GREATLY appreciated.
I have the following scenario.
First write something to a db
Send message on bus
I want to execute those like an atomic action, that is, running them in a transaction but can't get it to work when it is really distributed. Everything was working fint on local machine, but as soon as the service was on another machine it stopped working.
My code looks something like this:
using(var ts = new TransactionScope())
{
// Write something to db
_bus.Send(SomeMessage);
ts.Complete();
}
I have done everything I could think of and find information about, like enabling DTC on client, server and db with inbound and outbound enabled, set the port DCOM protocols to 5000-6000. The error I get is:
FailedToSendMessageException: "Failed to send message to address: myserver"
and the stacktrace is:
at NServiceBus.Transports.Msmq.MsmqMessageSender.ThrowFailedToSendException(Address address, Exception ex) in c:\BuildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Transports\Msmq\MsmqMessageSender.cs:line 89
at NServiceBus.Transports.Msmq.MsmqMessageSender.Send(TransportMessage message, Address address) in c:\BuildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Transports\Msmq\MsmqMessageSender.cs:line 80
at NServiceBus.Unicast.UnicastBus.SendMessage(List`1 addresses, String correlationId, MessageIntentEnum messageIntent, Object[] messages) in c:\BuildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Unicast\UnicastBus.cs:line 658
at NServiceBus.Unicast.UnicastBus.SendMessage(Address address, String correlationId, MessageIntentEnum messageIntent, Object[] messages) in c:\BuildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Unicast\UnicastBus.cs:line 583
at NServiceBus.Unicast.UnicastBus.NServiceBus.IBus.Send(Object[] messages) in c:\BuildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Unicast\UnicastBus.cs:line 480
at Rapporteringsregisteret.Web.Controllers.RapporteringController.Post(OpprettRapportering opprettRapportering) in c:\dev\git\Rapporteringsregisteret\src\Rapporteringsregisteret.Web\Controllers\RapporteringController.cs:line 37
at lambda_method(Closure , Object , Object[] )
at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass10.<GetExecutor>b__9(Object instance, Object[] methodParameters)
at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.Execute(Object instance, Object[] arguments)
at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)
Why isn't this working?
When running in a distributed scenario your endpoint mapping needs to explicitely define the machine name where the queue of the receiver is located. When running on the same box this is not necessary. Therefore your mapping should not contain "myserver" but "myserver#machinename". This error is not related to DTC. For future questions always include Nservicebus version number and transport layer
I have a webservice that works perfectly from localhost but when I hosted it on the testing server I get an error on one of the methods. I am using wsHttp Binding. everything else works fine on the client side except for this method. This is the error that I am getting
client error:
System.ServiceModel.Security.SecurityAccessDeniedException: Access is denied.
Server stack trace:
at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter)
at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Server error:
namespace.Service Error: 10001 : Error occurred in methodname().
System.Security.SecurityException: The source was not found, but some or all event logs could not be searched. Inaccessible logs: Security.
at System.Diagnostics.EventLog.FindSourceRegistration(String source, String machineName, Boolean readOnly)
at System.Diagnostics.EventLog.SourceExists(String source, String machineName)
at System.Diagnostics.EventLog.VerifyAndCreateSource(String sourceName, String currentMachineName)
at System.Diagnostics.EventLog.WriteEvent(EventInstance instance, Byte[] data, Object[] values)
at System.Diagnostics.EventLog.WriteEvent(EventInstance instance, Object[] values)
at System.Diagnostics.EventLogTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType severity, Int32 id, String format, Object[] args)
at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String format, Object[] args)
at AutoWatch.Entity.WcfService.TrackingService.UpdateIncidentStatusHistory(Int64 incidentId, String status, String username, String comment, Boolean SuspectFaultyUnit) in C:\..servicename.cs:line 566
at AutoWatch.Entity.WcfService.TrackingService.GetNewIncidentMessage(String username) in C:\..servicename.cs:line 444
The Zone of the assembly that failed was:
MyComputer
I added in the error I am getting on the server.
Is it possible I am getting this error because the service cannot write to the event log?
Please help.
Ensure that account running your process where the service is hosted has access rights to the database. For example in case of IIS the account running the application pool where the service is hosted must have login to database server and it must have permissions to do all necessary operations in your database.
Edit:
The server stack trace looks quite straightforward. You have a problem with writing to Windows Event Log! It cannot find source you requested and it doesn't have permission to create it.
I got the same error when I tried to use the following statement in a Web Method of a WCF Service:
string myTypeName = typeof(ErrorHandlerBehavior).AssemblyQualifiedName;
where ErrorHandlerBehavior derives from BehaviorExtensionElement.
The line works well when I test the service on my localhost. On the web, the call of the containing method raises the Exception "Access is denied".
In this case, I guess that the cause was that my IP does not allow the creation of a BehaviourExtensionElement in a Partially Trusted environment (my service is in a shared hosting environment).
Finally, I succeded following the first mechanism described at https://learn.microsoft.com/en-us/dotnet/framework/wcf/extending/configuring-and-extending-the-runtime-with-behaviors about Service Behaviors:
"Using an attribute on the service class. When a ServiceHost is constructed, the ServiceHost implementation uses reflection to discover the set of attributes on the type of the service. If any of those attributes are implementations of IServiceBehavior, they are added to the behaviors collection on ServiceDescription. This allows those behaviors to participate in the construction of the service run time."
I just modified the derivation (no other modification required):
public class ErrorServiceBehavior : Attribute, IServiceBehavior
{ ... }
and use the class as Attribute of my service:
[ErrorServiceBehavior()]
public partial class MyService : IMyService
{...}
No other modification required. Check the original sample at How do I create a global exception handler for a WCF Services?.