I am currently using sql dependency notification to detect changes in a table and process them. I am having a problem where the notification gets called while its still in the middle of completing the first request which causes duplicate processing
private void ProcessData()
{
try
{
m_Guids = new List<Guid>();
using (SqlCommand command = new SqlCommand("SP_XXX_SELECT", m_sqlConn))
{
command.CommandType = CommandType.StoredProcedure;
command.Notification = null;
SqlDependency dependency = new SqlDependency(command);
dependency.OnChange += new OnChangeEventHandler(OnDependencyChange);
SqlDependency.Start(m_ConnectionString, m_QueueName);
if (m_sqlConn.State == ConnectionState.Closed)
{
m_sqlConn.Open();
}
using (SqlDataReader reader = command.ExecuteReader())
{
if (reader.HasRows)
{
while (reader.Read())
{
m_Guids.Add(reader.GetGuid(0));
}
}
}
Console.WriteLine(m_Guids.Count.ToString());
ProcessGuids();
}
}
}
catch (Exception ex)
{
//SendFailureEmail
}
}
private void OnDependencyChange(object sender, SqlNotificationEventArgs e)
{
SqlDependency dependency = sender as SqlDependency;
dependency.OnChange -= OnDependencyChange;
ProcessData();
}
public void OnStart()
{
SqlDependency.Stop(m_ConnectionString, m_QueueName);
SqlDependency.Start(m_ConnectionString, m_QueueName);
m_sqlConn = new SqlConnection(m_ConnectionString);
}
ProcessData method gets called again while its still in the middle of processing (processGuids) Should I subscribe to the event after processing all the data?
If I don't subscribe until processing is complete, what happens to the data that was changed during the process, which I believe doesn't get notified until next change happens?. What is the correct way of doing this or am I doing something wrong.
Thanks
SqlDependency.OnChange is called not only on data change.
In the OnDependencyChange you must check e.Type/e.Source/e.Info.
F.e., combination of {Type = Subscribe, Source = Statement, Info = Invalid} means "Statement not ready for notification, no notification started".
See Creating a Query for Notification for SQL statement requirements for notification. You must follow these requirements in SELECT statements in your SP.
Additional requirements for stored procedures are not well documented. Known restrictions for SP:
Use of SET NOCOUNT (ON and OFF) is prohibited.
Use of RETURN is prohibited.
Related
I'm writing a Windows service. It periodically connects to a SQL database and looks for new records. I can read the SQL database without an issue and it gets the data using a SqlDataReader. I then have a while loop that just does a while Read. Part way through the while loop I create a web request to send that data to a server side component. For that I am using RestSharp v107. When I call ExecuteAsync with the request, I can see the data being sent and it ends up on my server side without an issue. However, when the response comes back, it has wiped out my SqlDataReader so on the next loop I get "Exception: Invalid attempt to call Read when reader is closed." as the reader is null. Here is the code, truncated where necessary for brevity.
private async Task AddUpdateCustomers(SqlConnection poConn)
{
string sQuery = "select * from customers";
try
{
SqlCommand oCmd = new SqlCommand(sQuery, poConn);
poConn.Open();
using (SqlDataReader dataReader = oCmd.ExecuteReader())
{
if (dataReader.HasRows)
{
while (dataReader.Read())
{
int iRecordId = dataReader.GetInt32(dataReader.GetOrdinal("ID"));
Customer customer = new Customer()
{
CustomerId = dataReader.GetString(dataReader.GetOrdinal("CustomerCodeFinancialReference")),
Name = dataReader.GetString(dataReader.GetOrdinal("Name"))
// more fields here removed for brevity
};
RestRequest request = new RestRequest();
request.AddHeader("Profile", aCLIntacctClientServiceConfiguration.ProfileName);
request.AddHeader("Entity", aCLIntacctClientServiceConfiguration.EntityName);
request.AddHeader("CreateUpdate", dataReader.GetBoolean(dataReader.GetOrdinal("IsNew")) ? "create" : "update");
request.AddHeader("Type", "Customer");
request.AddHeader("CorrelationId", Guid.NewGuid().ToString());
string body = ObjectToString(customer);
request.AddStringBody(body, DataFormat.Json);
var options = new RestClientOptions(ClientURI)
{
ThrowOnAnyError = false,
Timeout = 30000
};
RestClient client = new RestClient(options);
RestResponse response = await client.ExecuteAsync(request);
if (!string.IsNullOrEmpty(response.Content))
{
try
{
CustomResponse customerResponseObject = JsonConvert.DeserializeObject<CustomResponse>(response.Content);
if (customerResponseObject.ExitCode == 0)
{
WriteBackToStagingTable(iRecordId, true, "");
}
else
{
string sMessages = "";
foreach (CustomResponseMessage message in customerResponseObject.Messages)
{
sMessages += $"Record ID '{message.RecordId}' failed with the following error: '{message.MessageValue}'" + Environment.NewLine;
}
WriteBackToStagingTable(iRecordId, false, sMessages);
}
}
catch (Exception ex)
{
WriteBackToStagingTable(iRecordId, false, ex.Message + Environment.NewLine + response.Content);
}
}
}
if (!dataReader.IsClosed)
{
dataReader.Close();
}
}
else
{
// No records read from db
}
}
}
catch (Exception ex)
{
HandleLogRequest("Exception: " + ex.Message);
}
}
I've tried using PostAsync but that didn't work and reading the documentation, it isn't what I want. I cannot see why after calling client.ExecuteAsync the dataReader is suddenly null. I'm using RestSharp v107 and so many examples on the web but they are earlier versions and there are a lot of differences.
Rather than pass the connection round as it wasn't particularly necessary, I moved the connection into each function and created it as necessary. With the SQL data reader, I didn't bother with the using block. I also changed it so that it was executing asynchronously.
SqlDataReader dataReader = await oCmd.ExecuteReaderAsync();
Once I'd done that, calling the rest client execute method, it returned and the data reader was still in scope.
I am trying SqlDepenedency for the first time. I am not getting any notifications on Database Update.
I am placing breakpoints inside :
OnChange(object sender, SqlNotificationEventArgs e),
but it never gets hit.
Here is my code :
protected void Page_Load(object sender, EventArgs e)
{
Label1.Text = "Cache Refresh: " + DateTime.Now.ToLongTimeString();
DateTime.Now.ToLongTimeString();
// Create a dependency connection to the database.
SqlDependency.Start(GetConnectionString());
using (SqlConnection connection = new SqlConnection(GetConnectionString()))
{
using (SqlCommand command = new SqlCommand(GetSQL(), connection))
{
SqlDependency dependency =
new SqlDependency(command);
// Refresh the cache after the number of minutes
// listed below if a change does not occur.
// This value could be stored in a configuration file.
connection.Open();
dgHomeRequests.DataSource = command.ExecuteReader();
dgHomeRequests.DataBind();
}
}
}
private string GetConnectionString()
{
// To avoid storing the connection string in your code,
// you can retrieve it from a configuration file.
//return "Data Source=(local);Integrated Security=true;" +"Initial Catalog=AdventureWorks;";
return ConfigurationManager.ConnectionStrings["TestData"].ConnectionString;
}
private string GetSQL()
{
return "Select [Address] From [UserAccount1]";
}
void OnChange(object sender, SqlNotificationEventArgs e)
{
// have breakpoint here:
SqlDependency dependency = sender as SqlDependency;
// Notices are only a one shot deal
// so remove the existing one so a new
// one can be added
dependency.OnChange -= OnChange;
// Fire the event
/* if (OnNewMessage != null)
{
OnNewMessage();
}*/
}
I have also placed some code in Global.asax file :
public class Global : HttpApplication
{
void Application_Start(object sender, EventArgs e)
{
// Code that runs on application startup
RouteConfig.RegisterRoutes(RouteTable.Routes);
BundleConfig.RegisterBundles(BundleTable.Bundles);
SqlDependency.Start(ConfigurationManager.ConnectionStrings["TestData"].ConnectionString);
}
protected void Application_End()
{
// Shut down SignalR Dependencies
SqlDependency.Stop(ConfigurationManager.ConnectionStrings["TestData"].ConnectionString);
}
}
The SQL server is on local machine. I am running the code through Visual Studio(IIS Express).
To enable service broker on database:
ALTER DATABASE SET ENABLE_BROKER
GO
To subscribe query notification, we need to give permission to IIS service account
GRANT SUBSCRIBE QUERY NOTIFICATIONS TO “<serviceAccount>”
I guessed the 2nd point is not needed as it is local. But I tried giving it some permissions. Don't know if they are right as I don't think it is using app pool.And don't need the permission on local env. if I am the user myself and created the schema myself.
One of the questions that I saw was granting :
alter authorization on database::<dbName> to [sa];
I gave that permission too.
I was missing :
dependency.OnChange += new OnChangeEventHandler(OnChange);
The new code would look like :
SqlDependency dependency = new SqlDependency(command);
dependency.OnChange += new OnChangeEventHandler(OnChange);
now I can fire
void OnChange(object sender, SqlNotificationEventArgs e)
I'm wondering if anyone has encountered this before:
I handle a command, and in the handler, I save an event to the eventstore (joliver).
Right after dispatching, the handler for the same command is handled again.
I know its the same command because the guid on the command is the same.
After five tries, nservicebus says the command failed due to the maximum retries.
So obviously the command failed, but I don't get any indication of what failed.
I've put the contents of the dispatcher in a try catch, but there is no error caught. After the code exits the dispatcher, the event handler will always fire as if something errored out.
Tracing through the code, the events are saved to the database (I see the row), the dispatcher runs, and the Dispatched column is set to true, and then the handler handles the command again, the process repeats, and another row gets inserted into the commits table.
Just what could be failing? Am I not setting a success flag somewhere in the event store?
If I decouple the eventstore from nServicebus, both will run as expected with no retries and failures.
The dispatcher:
public void Dispatch(Commit commit)
{
for (var i = 0; i < commit.Events.Count; i++)
{
try
{
var eventMessage = commit.Events[i];
var busMessage = (T)eventMessage.Body;
//bus.Publish(busMessage);
}
catch (Exception ex)
{
throw ex;
}
}
}
The Wireup.Init()
private static IStoreEvents WireupEventStore()
{
return Wireup.Init()
.LogToOutputWindow()
.UsingSqlPersistence("EventStore")
.InitializeStorageEngine()
.UsingBinarySerialization()
//.UsingJsonSerialization()
// .Compress()
//.UsingAsynchronousDispatchScheduler()
// .DispatchTo(new NServiceBusCommitDispatcher<T>())
.UsingSynchronousDispatchScheduler()
.DispatchTo(new DelegateMessageDispatcher(DispatchCommit))
.Build();
}
I had a transaction scope opened on the save that I never closed.
public static void Save(AggregateRoot root)
{
// we can call CreateStream(StreamId) if we know there isn't going to be any data.
// or we can call OpenStream(StreamId, 0, int.MaxValue) to read all commits,
// if no commits exist then it creates a new stream for us.
using (var scope = new TransactionScope())
using (var eventStore = WireupEventStore())
using (var stream = eventStore.OpenStream(root.Id, 0, int.MaxValue))
{
var events = root.GetUncommittedChanges();
foreach (var e in events)
{
stream.Add(new EventMessage { Body = e });
}
var guid = Guid.NewGuid();
stream.CommitChanges(guid);
root.MarkChangesAsCommitted();
scope.Complete(); // <-- missing this
}
}
I am running multiple inserts using transactions. I am using the SqlDependency class to let the client machine know when the server has been updated.
The problem I am having is that whenever I insert using a transaction, no matter what isolation level I set for the transaction, the SqlNotificationEventArgs returns e.Info as Isolation which indicates that I have the wrong isolation level set for that transactions (I think). When I insert without using a transaction, everything runs smoothly.
My questions is, what are the supported Isolation levels, if any, for transactions when using Sql Notification?
Below is some of the code I am using for the notification:
void DataChanged(object sender, SqlNotificationEventArgs e) {
var i = (ISynchronizeInvoke)_form;
if (i.InvokeRequired) {
var tempDelegate = new OnChangeEventHandler(DataChanged);
object[] args = { sender, e };
i.BeginInvoke(tempDelegate, args);
} else {
var dependency = (SqlDependency)sender;
if (e.Type == SqlNotificationType.Change) {
dependency.OnChange -= DataChanged;
GetData(dependency);
}
}
}
And for the transaction:
public void ExecuteNonQueryData(List<string> commandTexts) {
SqlConnection connection = null;
var command = new SqlCommand();
SqlTransaction transaction = null;
try {
connection = new SqlConnection(GetConnectionString());
connection.Open();
transaction = connection.BeginTransaction(IsolationLevel.ReadCommitted);
foreach (var commandText in commandTexts) {
try {
command.Connection = connection;
command.CommandText = commandText;
command.Transaction = transaction;
command.ExecuteNonQuery();
} catch (Exception ex) {
Console.WriteLine(ex.Message);
}
}
transaction.Commit();
} catch (Exception ex) {
Console.WriteLine(ex.Message);
} finally {
command.Dispose();
if (transaction != null) transaction.Dispose();
if (connection != null) {
connection.Close();
connection.Dispose();
}
}
commandTexts.Clear();
}
Edit: I was committing the transaction in the wrong place.
Apparently Query Notification does not support transactions. Removing the transaction code fixed this problem.
According to Microsoft:
Transact-SQL does not provide a way to subscribe to notifications. The CLR data access classes hosted within SQL Server do not support query notifications.
This quote was found at http://msdn.microsoft.com/en-us/library/ms188669.aspx, which describes how Query Notifications work and their requirements.
I am trying to use TransactionScope with NHibernate in order to call several methods in one transactions. Data repository methods are like this:
public virtual void Save(T dataObject)
{
try
{
using (TransactionScope scope = new TransactionScope(TransactionScopeOption.Required, new TransactionOptions { IsolationLevel = IsolationLevel.RepeatableRead }))
{
this.session.SaveOrUpdate(dataObject);
scope.Complete();
}
}
catch (Exception ex)
{
bool rethrow = ExceptionPolicy.HandleException(ex, "Data Layer Policy");
if (rethrow)
{
throw;
}
}
}
public T GetByNumber(string documentNumber)
{
T document = null;
try
{
using (TransactionScope scope = new TransactionScope(TransactionScopeOption.Required, new TransactionOptions { IsolationLevel = IsolationLevel.RepeatableRead }))
{
document = this.Session.CreateCriteria(typeof(T))
.Add(Restrictions.Eq("Number", documentNumber))
.UniqueResult();
scope.Complete();
}
}
catch (Exception ex)
{
bool rethrow = ExceptionPolicy.HandleException(ex, "Data Layer Policy");
if (rethrow)
{
throw;
}
}
return document;
}
I wanted to test row/table locking in transactions so I made several unit tests and some console applications. Here is code from these console applications:
Application which does update:
const string DocumentNumber = "386774321";
Random randomGenerator = new Random();
using (TransactionScope scope = new TransactionScope(TransactionScopeOption.Required, new TransactionOptions { IsolationLevel = IsolationLevel.RepeatableRead }))
{
using (BillingDocumentRepository billingDocumentRepository = new BillingDocumentRepository())
{
BillingOrderData orderData = billingDocumentRepository.GetByNumber(DocumentNumber);
orderData.Notes = randomGenerator.Next().ToString();
Console.WriteLine(string.Format("SECOND: {0}: Updated notes to {1}.", DateTime.Now.ToString("HH:mm:ss.fffff"), orderData.Notes));
Console.WriteLine(string.Format("SECOND: {0}: Updating order.", DateTime.Now.ToString("HH:mm:ss.fffff")));
Console.WriteLine(string.Format("SECOND: {0}: Going to sleep for 10000ms.", DateTime.Now.ToString("HH:mm:ss.fffff")));
Sleep(10000); // My custom sleep method because I didn't want to use Thread.Sleep for simulating long transaction
billingDocumentRepository.Save(orderData);
}
Console.WriteLine(string.Format("SECOND: {0}: Going to sleep for 10000ms.", DateTime.Now.ToString("HH:mm:ss.fffff")));
Sleep(10000);
Console.WriteLine(string.Format("SECOND: {0}: Completing transaction.", DateTime.Now.ToString("HH:mm:ss.fffff")));
scope.Complete();
}
Application which reads the same row in database:
while (true)
{
using (BillingDocumentRepository repository = new BillingDocumentRepository())
{
Console.WriteLine(string.Format("MAIN: {0}: Getting document.", DateTime.Now.ToString("HH:mm:ss.fffff")));
BillingOrderData billingOrderData = repository.GetByNumber("386774321");
Console.WriteLine(string.Format("MAIN: {0}: Got order with notes {1}.", DateTime.Now.ToString("HH:mm:ss.fffff"), billingOrderData.Notes));
Sleep(1000);
}
}
Problem is that first transaction (which updates row) doesn't lock row for reading at any moment. Second application is reading that row all the time with old value before scope.Complete() and than new value after that. How can I achieve locking with this model?
You should lock when reading. Locking later is "too late":
document = this.Session.CreateCriteria(typeof(T))
.Add(Restrictions.Eq("Number", documentNumber))
.SetLockMode(LockMode.Upgrade)
.SetTimeout(5)
.UniqueResult();
Or:
var doc = session.QueryOver<BillingDocument>()
.Where(c => c.Number== "2233445")
.Lock()
.Upgrade
.UnderlyingCriteria.
SetTimeout(5).
List().
FirstOrNull() as BillingDocument;
There is a session.Lock(object) method.
When you call session.Save(object), NHibernate isn't doing anything in the database until it gets flushed.
Flushing is done (depending on the flush mode, which is usually AutoFlush)
before queries (except Get and Load)
when calling flush explicitly
when committing the transaction (if the connection is created by NH I think)
When the session is flushed, the actual update, insert and delete operations are done on the database and locks are set.
In SQL Server, when the lock is set, the reading transaction is waiting until commit of the updating transaction. When it commits, it reads the committed values (when you are in "Read Committed" isolation).