So I have a very weird problem.
I am writing some code in VB.Net under .NET 2.0 which interfaces with MS Exchange 2003. Because of the Exchange 2003 "requirement" I am forced to write this code using WEBDAV.
The code itself is replicating, to some degree, a schedule management process. It's creating Appointments on the Exchange Server in response to inputs from the user and managing it's data internally in a SQL Server database.
The problem situation is this: A new person is assigned to be in charge of a meeting. The requirement says the program should generate a meeting cancellation request to the person removed from the meeting (if such a person existed) and a meeting request sent to the new person.
In the case of there being an existing person, what appears to happen is this:
The meeting cancellation request
gets sent
Exchange barfs and returns status code 500 (internal server error) during the set of
requests which send the meeting request to the new person.
However! While debugging this particular scenario, it works just fine for me, if I step through the code in the Visual Studio debugger. Left to it's own devices, it fails every time.
Just for yuk's sake, I added a Thread.Sleep(500) to the part after sending the cancellation request, and Exchange doesn't barf anymore...
So, my question!
If adding a Thread.Sleep to the code causes this error to go away, a race condition is implied, no? But, my code is running under a web application and is a totally single threaded process, from start to finish. The web requests I am sending are all in synchronous mode so this shouldn't be a problem.
What would I do next to try and track down the issue?
Try and divine if the race condition itself is in the .Net 2.0 BCL networking code?
Try and do some debugging on the Exchange server itself?
Ignore it, be glad the Thread.Sleep masks the problem and keep on going?
Any further suggestions would be wonderful.
In response to comment, I can post the failing function:
Private Shared Sub UpdateMeeting(ByVal folder As String, ByVal meetingId As String, ByVal oldAssignedId As String, ByVal newAssignedTo As String, ByVal transaction As DbTransaction)
If String.IsNullOrEmpty(meetingId) Then
Throw New Exception("Outlook ID for that date and time is empty.")
End If
Dim x As New Collections.Generic.List(Of String)
If oldAssignedId <> newAssignedTo AndAlso Not String.IsNullOrEmpty(oldAssignedId) Then
'send cancellation to old person
Dim lGetCounselorEmail1 As String = GetCounselorEmail(oldAssignedId, transaction)
Common.Exchange.SendCancellation(meetingId, New String() {lGetCounselorEmail1})
' Something very weird here. Running this code through the debugger works fine. Running without causes exchange to return 500 - Internal Server Error.
Threading.Thread.Sleep(500)
End If
x.Add(folder)
If Not String.IsNullOrEmpty(newAssignedTo) Then x.Add(GetCounselorEmail(newAssignedTo, transaction))
x.RemoveAll(AddressOf String.IsNullOrEmpty)
If x.Count > 0 Then
If Not Common.Exchange.UpdateMeetingAttendees(meetingId, x.ToArray()) Then
Throw New Exception("Failure during update of calendar")
End If
End If
End Sub
...but a lot of the implementation details are hidden here, as I wrote a set of classes to interface with Exchange WebDAV.
Ended up sticking with the Sleep and calling it a day.
My 'belief' is that I was erroneous in thinking that a WebRequest/WebResponse combo sent to Exchange through WebDav was an atomic operation.
Related
I'm maintaining an antedeluvian Notes application which connects to a SAP back-end via a manually done 'Webservice'
The server is running Domino Release 7.0.4FP2 HF97.
The Webservice is not the more recently Webservice Consumer, but a large Java agent which is using Apache soap.jar (org.apache.soap). Below an example of the calling code.
private Call setupSOAPCall() {
Call call = new Call();
SOAPHTTPConnection conn = new SOAPHTTPConnection();
call.setSOAPTransport(conn);
call.setEncodingStyleURI(Constants.NS_URI_SOAP_ENC);
There has been a change in the SAP system which is now taking 8 minutes to complete (verified by SAP Team).
I'm getting an error message as follows:
[SOAPException: faultCode=SOAP-ENV:Client; msg=For input string: "906 "; targetException=java.lang.NumberFormatException: For input string: "906 "]
I found a blog article describing the error message quite closely:
https://thejavablog.wordpress.com/category/jmeter/
and I've come to the hypothesis that it is a timeout message that is returning to my Call object and that this timeout message is being incorrectly parsed, hence the NumberFormat Exception.
Looking at my logs I can see that there is a time difference of 62 seconds between my call and the response.
I recommended that the server setting in the server document, tab Internet Protocols/HTTP/Timeouts/Request timeouts be changed from 60 seconds to 600 seconds, and the http task restarted with
tell http restart
I've re-run the tests and I am getting the same error, and the time difference is still slightly more than 60 seconds, which is not what I was expecting.
I read Michael Rulnau's blog entry
http://www.mruhnau.net/2014/06/how-to-overcome-domino-webservice.html
which points to this APR
http://www-01.ibm.com/support/docview.wss?uid=swg1LO48272
but I'm not convinced that this would apply in this case, since there is no way that IBM would know that my Java agent is in fact making a Soap call.
My current hypothesis is that I have to use either the setTimeout() method on
org.apache.axis.client.Call
https://axis.apache.org/axis/java/apiDocs/org/apache/axis/client/Call.html
or on the org.apache.soap.transport.http.SOAPHTTPConnection
https://docs.oracle.com/cd/B13789_01/appdev.101/b12024/org/apache/soap/transport/http/SOAPHTTPConnection.html
and that the timeout value is an apache default, not something that is controlled by the Domino server.
I'd be grateful for any help.
I understand your approach, and I hope this is the correct one to solve your problem.
Add a debug (console write would be fine) that display the default Timeout then try to increase it to 10 min.
SOAPHTTPConnection conn = new SOAPHTTPConnection();
System.out.println("time out is :" + conn.getTimeout());
conn.setTimeout(600000);//10 min in ms
System.out.println("after setting it, time out is :" + conn.getTimeout());
call.setSOAPTransport(conn);
Now keep in mind that Dommino has also a Max LotusScript/Java execution time, check this value and (at least for a try) change it: http://www.ibm.com/support/knowledgecenter/SSKTMJ_9.0.1/admin/othr_servertasksagentmanagertab_r.html (it's version 9 help but this part should be identical)
I've since discovered that it wasn't my code generating the error; the default timeout for the apache axis SOAPHTTPConnetion is 0, i.e. no timeout.
I have a SQLite database that I want to lock for synchronization purposes. I don't want a process that runs async on a different box processing data that has been added from a different box until it has finished with updates. DataAccess is a class that connects to sPackageFileName and reuses the same connection as long as sPackageFileName is the same or unless .Close method is called. So basically DataAccess.ExecCommand executes a command.
In Google I found this ....
DataAccess.ExecCommand("PRAGMA locking_mode = EXCLUSIVE", sPackageFileName)
DataAccess.ExecCommand("BEGIN EXCLUSIVE", sPackageFileName)
DataAccess.ExecCommand("COMMIT", sPackageFileName)
This works as advertise. If I run this on box A and then on box B I get a "database locked" exception. The problem is how long it takes. I found a PRAGMA busy_timeout. This PRAGMA is timeout controls access locks, not database locks. I am stratring to think there is not PRAGMA for database lock timeout. Right now it seems about 3-4 minutes. One other note, the sPackageFileName is not on either box, they (box A and B) connect to it over a share drive.
Also I am using the VB.NET wrapper for the SQLite dll.
CL got me on the right trail. It was the timeout of the .NET command. Here the code setting it up from my class.
Dim con As DbConnection = OpenDb(DatabaseName, StoreNumber, ShareExclusive, ExtType)
Dim cmd As DbCommand = con.CreateCommand()
If _QueryTimeOut > -1 Then cmd.CommandTimeout = _QueryTimeOut
Don't get hang up on the variables, the purpose of posting the code is show I could show the property I was talking about. The default _QueryTimeOut was set the 300 (seconds). I set cmd.ComandTimeout to 1 (second) and it returned as expected.
As CL finally got through to me, the timeout was happening someplace else. Sometimes it takes a kick to get you out of the box. :-)
I am calling 5 external servers to retrieve XML-based data for each request for a particular webpage on my IIS 6 server. Present volume is between 3-5 incoming requests per second, meaning 15-20 outgoing requests per second.
99% of the outgoing requests from my server (the client) to the external servers (the server) work OK but about 100-200 per day end up with a "The operation has timed out" exception.
This suggests I have a resource problem on my server - some shortage of sockets, ports etc or a thread lock but the problem with this theory is that the failures are entirely random - there are not a number of requests in a row that all fail - and two of the external servers account for the majority of the failures.
My question is how can I further diagnose these exceptions to determine if the problem is on my end (the client) or on the other end (the servers)?
The volume of requests precludes putting an analyzer on the wire - it would be very difficult to capture these few exceptions. I have reset CONNECTIONS and THREADS in my machine.config and the basic code looks like:
Dim hRequest As HttpWebRequest
Dim responseTime As String
Dim objWatch As New Stopwatch
Try
' calculate time it takes to process transaction
objWatch.Start()
hRequest = System.Net.WebRequest.Create(url)
' set some defaults
hRequest.Timeout = 5000
hRequest.ReadWriteTimeout = 10000
hRequest.KeepAlive = False ' to prevent open HTTP connection leak
hRequest.SendChunked = False
hRequest.AllowAutoRedirect = True
hRequest.MaximumAutomaticRedirections = 3
hRequest.Accept = "text/xml"
hRequest.Proxy = Nothing 'do not waste time searching for a proxy
hRequest.ServicePoint.Expect100Continue = False
Dim feed As New XDocument()
' use *Using* to auto close connections
Using hResponse As HttpWebResponse = DirectCast(hRequest.GetResponse(), HttpWebResponse)
Using reader As XmlReader = XmlReader.Create(hResponse.GetResponseStream())
feed = XDocument.Load(reader)
reader.Close()
End Using
hResponse.Close()
End Using
objWatch.Stop()
' Work here with returned contents in "feed" document
Return XXX' some results here
Catch ex As Exception
objWatch.Stop()
hRequest.Abort()
Return Nothing
End Try
Any suggestions?
By default, HttpWebRequest limits you to 2 connections per HTTP/1.1 server. So, if your requests take time to complete, and you have incoming requests queuing up on the server, you will run out of connection and thus get timeouts.
You should change the max outgoing connections on ServicePointManager.
ServicePointManager.DefaultConnectionLimit = 20 // or some big value.
You said that you are doing 5 outgoing request for each incoming request to the ASP page. Is that 5 different servers, or the same server?
DO you wait for the previous request to complete, before issuing the next one? Is the timeout happening while it is waiting for a connection, or during the request/response?
If the timeout is happening during the request/response then it means that the target server is under stress. The only way to find out if this is the case, is to run wireshark/netmon on one of the machines, and look at the network trace to see if the request from the app is even making it through to the server, and if it is, whether the target server is responding within the given timeout.
If this is a thread starvation issue, then one of the ways to diagnose it is to attach windbg.exe debugger to w3wp.exe process, when you start getting timeout. Then load the sos.dll debugging extension. And run the !threads command, followed by !threadpool command. It will show you how many Worker threads and completion port threads are utilized/remaining. If the #completionport threads or worker threads are low, then that will contribute to the timeout.
Alternatively, you can monitor ASP.NET and System.net perf counters. See if the ASP.NET request queue is increasing monotonically - this might indicate that your outgoing requests are not completing fast enough.
Sorry, there are no easy answers here. THere is a lot of avenues you will need to explore. If I were you, I would start off by attaching windbg.exe to w3wp when you start getting timeouts and do what I described earlier.
I'm encountering a hang when the program tries to access the fruit database. I've already enabled network access MSDTC on both my development computer and the SQL Server server.
Code:
(pardon the code coloring...SO's misinterpreting my VB .NET)
Using ts As New TransactionScope
Dim fruit As New FruitDataContext
Dim thingies As New ThingiesDataContext
If (From f In fruit.tblApples Where f.Rotten = "Yes" AndAlso f.batch = 1).Count >= 1 Then
'Record today's date as the day that the rotten apples were dumped.
End If
'Other complicated code that uses ThingiesDataContext and FruitDataContext
du.SubmitChanges()
ts.Complete()
End Using
Edit:
I've dug around a bit more and it turns out that the problem lies in the line of LINQ. When I tried to view it with the LINQ to SQL Visualizer, I get the following error:
System.InvalidCastException: Specified cast is not valid.
at LinqToSqlQueryVisualizer.SqlQueryInfo.deserialize(Stream stream)
at LinqToSqlQueryVisualizer.Visualizer.Display(IDialogVisualizerService windowService, Stream rawStream)
at LinqToSqlQueryVisualizer.DialogChooser.Show(IDialogVisualizerService windowService, IVisualizerObjectProvider objectProvider)
at Microsoft.VisualStudio.DebuggerVisualizers.DebugViewerShim.ManagedShim.DelegatedHost.CreateViewer(IntPtr hwnd, HostServicesHelper hsh, SafeProxyWrapper proxy)
I've also edited the LINQ statement to be closer to my real code.
Final edit:
I tried using a normal SqlConnection instead of a "thingies as New ThingiesDataContext" and the problem still occurs.
It appears that TransactionScope cannot handle multiple SQL connections inside the same transaction.
Official Microsoft Note
parallel transactions are not supported by SQL Server.
From MSDN: http://msdn.microsoft.com/en-us/library/bb896149.aspx
This is not an MSDTC issue. If it were, you would get an error saying DTC is not enabled and needs to be. It's also not a deadlock issue, because you would get a specific error about that as well.
If I had to guess, I would say that the 'Other complicated code...' is attempting to perform a database operation and is being blocked by one or the other database context objects.
One way you can determine this is to run SQL Profiler to see what SQL statements are actually being executed on the server, and check for blocks.
I'm using the following code to work out the next unique Order Number in an access database. ServerDB is a "System.Data.OleDb.OleDbConnection"
Dim command As New OleDb.OleDbCommand("", serverDB)
command.CommandText = "SELECT max (ORDERNO) FROM WORKORDR"
iOrder = command.ExecuteScalar()
NewOrderNo = (iOrder + 1)
If I subsequently create a WORKORDR (using a different DB connection), the code will not pick up the new "next order number."
e.g.
iFoo = NewOrderNo
CreateNewWorkOrderWithNumber(iFoo)
iFoo2 = NewOrderNo
will return the same value to both iFoo and iFoo2.
If I Close and then reopen serverDB, as part of the "NewOrderNo" function, then it works. iFoo and iFoo2 will be correct.
Is there any way to force a "System.Data.OleDb.OleDbConnection" to refresh the database in this situation without closing and reopening the connection.
e.g. Is there anything equivalent to serverdb.refresh or serverdb.FlushCache
How I create the order.
I wondered if this could be caused by not updating my transactions after creating the order. I'm using an XSD for the order creation, and the code I use to create the record is ...
Sub CreateNewWorkOrderWithNumber(ByVal iNewOrder As Integer)
Dim OrderDS As New CNC
Dim OrderAdapter As New CNCTableAdapters.WORKORDRTableAdapter
Dim NewWorkOrder As CNC.WORKORDRRow = OrderDS.WORKORDR.NewWORKORDRRow
NewWorkOrder.ORDERNO = iNewOrder
NewWorkOrder.name = "lots of fields filled in here."
OrderDS.WORKORDR.AddWORKORDRRow(NewWorkOrder)
OrderAdapter.Update(NewWorkOrder)
OrderDS.AcceptChanges()
End Sub
From MSDN
Microsoft Jet has a read-cache that is
updated every PageTimeout milliseconds
(default is 5000ms = 5 seconds). It
also has a lazy-write mechanism that
operates on a separate thread to main
processing and thus writes changes to
disk asynchronously. These two
mechanisms help boost performance, but
in certain situations that require
high concurrency, they may create
problems.
If you possibly can, just use one connection.
Back in VB6 you could force the connection to refresh itself using ADO. I don't know whether it's possible with VB.NET. My Google-fu seems to be weak today.
You can change the PageTimeout value in the registry but that will affect all programs on the computer that use the Jet engine (i.e. programmatic use of Access databases)
I always throw away a Connection Object after I used it. Due to Connection Pooling getting a new Connection is cheap.