I made a mistake. After realizing that SB was causing crashes (and had been for a while), I had it patched. Now, I had millions of messages backed up in a queue, it was trying to catch up on messages from other machines, and disk space was running out. Even with 4 readers it was falling behind.
So I did the only thing I could think of at the time. Which was stupid of me.
ALTER DATABASE ... SET NEW_BROKER WITH ROLLBACK IMMEDIATE;
Now, I'm trying to clean up from that mistake. The first thing I tried to do is alter the ROUTEs on the sending servers so they matched. That doesn't seem to be working - now the sys.transmission_queue on the senders says "Target service name could not be found". And I'm stumped on that - I see the service on the receiver, and I don't believe I changed anything with it. I'm scripting out the CREATE ROUTE from the box via SSMS, then changing the broker instance with the results of service_broker_guid from sys.databases for the receiving database.
Looking at a profiler trace with broker, I'm seeing (on the receiving server) these messages:
Could not forward the message because forwarding is disabled in this SQL Server instance.
The message could not be delivered because it could not be classified.
Enable broker message classification trace to see the reason for the failure.
Next up is doing endpoint cleanup on the senders, pulling the conversation_handle from sys.transmission_queue and using that to end it.
Update: okay, so I've cleaned out msdb.sys.transmission_queue, but I still have a 15gb MSDB, and it's got to be service broker (no tables using more than a few MB that I can see). Considering doing the NEW_BROKER there as well, since I've turned everything off. But that still seems like A Bad Idea.
The receiver is a R2 box, just patched to SP3.
At this point, I'm at a loss. Any help appreciated. Thanks in advance.
We ran into this issue where service broker suddenly and inexplicably stops delivering messages, they just accumulate in sys.transmission_queue and a profiler trace shows the error message Could not forward the message because forwarding is disabled in this SQL Server instance.
Executing this command fixed it:
alter endpoint ServiceBrokerEndpoint for service_broker (message_forwarding=enabled)
..which is odd since we never disabled message forwarding and never had to explicitly enable it before.
Related
I am dealing with some legacy code. There is a program written in vb6. That connects to an sql server 2000.
When a transaction is begun I am getting No transaction is active error.
The problem is this error occurs only on one of the clients. There are 3 pcs one server 2 clients work just fine. One doesn't.
The TCP protocol is enabled on the server. I have uninstalled and installed msdtc on server.
A connection is made. Other queries execute just fine. I am unsure what might be the problem.
If I had sufficient reputation, I would have added this as comment instead of answer...
Because it is not entirely clear to me what you mean with "When a transaction is begun" and what the actual error is for "No transaction is active error"
There are a few settings that can be done on any connection. The settings can be part of the connection configuration or set explicitly afterwards. One of them is "implicit transactions". On my machines, it is always set to "off". Maybe this setting differs between your machines.
To test that theory, you could add the line
set implicit_transactions off
or
set implicit_transactions on
as first line in the batch / stored procedure that throws the error.
If that solves the problem, you should fix the connection configuration of the troublesome machine and change the batch / stored procedure back to the original.
If these servers communicate with each other, dns information must be defined in the host file. You can check it out.
Im trying to get SQL Notifications to work with BizTalk, but im struggling a one point.
The Binding of the Receivelocation is the following:
The SQL Server is supporting Notifications, and the connection string is correct.
When i start the Receivelocation it is working exactly one time in a correct way, but when i disable it and start it again, i get the following error in the eventlog.
The Messaging Engine failed to add a receive location
"RL.MDM.SQL" with URL
"mssql://.//Database?InboundId=GetNewMDMChanges" to the adapter
"WCF-SQL". Reason:
"Microsoft.ServiceModel.Channels.Common.TargetSystemException: The
notification callback returned an error. Info=Invalid.
Source=Statement. Type=Subscribe.
I cant start the Receivelocation again till i Execute the following command on the Database to enable the Broker.
alter database MDMDEV set enable_broker with rollback immediate;
The strange thing here is when i check if the broker is still enabled before i execute the command above, i see that the broker is indeed still enabled.
So the command to enable the broker fixes my problem for exactly one other notification and than i have to do this again.
Has anybody ever had this problem or can tell me what im doing wrong?
Thanks in advance.
Regarding the Notifications feature in general, my recommendation is to not use it.
With both SQL Server and Oracle, the Notifications feature is quite fragile and will stop receiving event with no warning or error. When this happens, the only way to recover is Disable/Enable the Receive Location.
Basically, I have found it not reliable enough to use in production apps.
If you or your organization own the database, Polling [+ Triggers if needed] are 100% reliable.
This article describes some different Polling scenarios: BizTalk Server: SQL Patterns for Polling and Batch Retrieve
In the last release of my app, I added a command that tells it to wait when something arrives in the Service Broker queue
WAITFOR (RECEIVE CONVERT(int, message_body) AS Message FROM MyQueue)
The DBAs tell me that since the addition, the log sizes have gone through the roof. Could this be correct? Or should I be looking elsewhere?
I haven't tested this in service broker but I assume the same ACID compliance mechanisms would be in play. It would depend on if it's leaving a transaction open or not in your code. If it is leaving a transaction open and not committing it, the log will continue to grow until something closes it and only at that point will it finally mark the old areas for re-use.
I haven't rolled service broker in prod yet but the testing/reading I did did not include any WAITFOR.
Instead, the Server Broker MVPs like Denny Cherry would typically keep querying the queue instead of doing a WAITFOR.
Can you post some of the other code and also tell us why you're using WAITFOR? Maybe there's something I'm not getting that would be a good use case scenario. Thanks!
I am concerned with my NServiceBus solution.
I have a "MessageHub" that publishes some very important messages. But sometimes it loses track of its subscriptions and just discards the message because it thinks no one is listening.
I have tried turning on "NServiceBus.Integration" to store the subscriptions. But despite that, I still have issues with bad start up order where it thinks nothing is listening.
Is there a way to debug this process? Try to figure out why it is getting confused?
I don't even know a way to look at what subscriptions it "thinks" it has...
I went with NServiceBus because it is not supposed to lose data ever. Now I am losing large chucks. I know it is a config issue, but it is causing much grief.
What is probably happening in your case is that you are using MSMQ for subscription storage. Even though it's possible for subscriptions to endure for a while, using MSMQ to store things long term is always going to be volatile.
For durable subscriptions storage (which survive "forever") you should be using SQL server as your subscription storage.
Note: You can always view your current subscriptions whether you are using sql or msmq to store them. In SQL just look in the subscriptions table and for msmq look in the publisher's subscription queue.
UPDATE
Since version 3 I have been using RavenDb which is the default.
In my experiance, to get the subscriptions assigned correctly, one should first start the EventHandler projects and then when they are all idle, start the CommandHandlers (Publishers).
You can see what messages are being Subscribed to using Service Bus MQ Manager, it has a dialog listing all "messages" and their subscribers/publishers. A side project of mine, its free and open sourced.
http://blog.halan.se/page/Service-Bus-MQ-Manager.aspx
Background WCF Stack, Data Access Implemented in Entity Framework, Simple ASP.NET Front End
This is a two part question.
Recently we ran into an issue with periodic crashes with an exception that read:
A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The specified network name is no longer available
We had been running our application without issues for over a week, and then all the sudden we were hit with this random crash/ If I had to guess I would say it was network related, but we were unable to determine the exact source. Has anyone periodically gotten this message? If so what was the root cause?
Second question is someone suggested to set "async=true" in our Entity Framework connection string. I was under the impression this just enables the async api. Does this do anything when you are using EF? Does switching this flag do anything with the queries that get generated by EF?
To be that guy I will answer this one on my own.
First I posted the question about the "async=true"s effect on entity framrwork to MS and no one answered ... as usual(if they answer i will update this post).
Our issue:
A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The specified network name is no longer available
Was environment related. Something was causing the DB to run a little bit slower, but it was hinting to a larger issue. Apparently EF has horrible issues when you share context between threads (not an easy problem to solve), so we were seeing a race condition with opening connections.
We basically had a "read only context" that only did gets. Our issue was two threads attempt to open the connection at the same time, one wins, the other loses resulting in some variation of the exception below:
The connection was not closed. The connection's current state is connecting.
Our solution was to convert our singleton to be thread specific. Not exactly what we wanted, but it worked, and when we pushed this fix our other issue magically went away.
The second half to this question was what does async=true do. When it comes to EF, it made our system crash. We had a block of code that did a join, and if async=true and MARS=false we got a:
There is already an open DataReader associated with this Command which must be closed first
Once we cut back on MARS, and disabled async things were good again.