I'm trying to send a WorkItemChanged notification from TFS to my WCF Service. On my development machine everything works perfectly fine, the notification is delivered (and processed by my programm) almost immediatly.
But now as I wanted to test my software on another system with a "freshly" installed TFS 2012, the notifications are not delivered. I looked into it and found the following error:
elapsed time: 00:00:00.2376893, sql calls: 19, sql connect time: 00:00:00, sql execute time: 00:00:00.1560000, non-sql time: 00:00:00.0816893 (34%), cpu time: 00:00:00.0156001 ( 6.6%), avg connect time: 0.00 ms, avg execute time: 8.2 ms. All methods quick. 0 slowest sql calls: prc_ReadGroupMembership(47 ms) set nocount on
declare #NowUtc as datetime; set #NowUtc=getutcdate()
declare #PersonId as int
declare #rebuildOK as int
declare #PersonN(30 ms) CollectionError: CreateEvent != TransformEvents. CollectionError: CreateEvent != ExpandEvents. CollectionError: AfterReadSubscription != SendNotifications.
There were errors or warnings during notification delivery.
0/0 emails delivered.
0/1 soap notifications delivered.
1 errors.
0 warnings.
-------------------------------
Notification not delivered.
Notification: WorkItemChangedEvent (DeliveryType: Soap; Address: http://localhost:8733/NAMEOFMYSERVICEENDPOINT)
Exception: System.NotSupportedException: Collection is read-only.
at System.Collections.ObjectModel.Collection`1.Clear()
at Microsoft.TeamFoundation.Client.Channels.TfsHttpClientBase.HandleReply(TfsClientOperation operation, TfsMessage message, Object[]& outputs)
at Microsoft.TeamFoundation.Client.Channels.TfsHttpClientBase.Invoke(TfsClientOperation operation, Object[] parameters, Object[]& outputs)
at Microsoft.TeamFoundation.Client.Channels.TfsHttpClientBase.Invoke(TfsClientOperation operation, Object[] parameters)
at Microsoft.TeamFoundation.JobService.Extensions.Core.NotificationJobExtension.SendSoapNotification(TeamFoundationRequestContext requestContext, TeamFoundationNotification notification)
I don't really understand what exactly the problem is or where it comes from. What "Collection" is this error referring to? Is it something on my services side or is it related to a wrong configuration or some missing administrative rights on the TFS's side?
Edit
Almost forgot this question ;)
Well apparently it was caused by an error subscribing to the notification via BisSubscribe.exe. After deleting the notification and re-subscribing, the error was gone.
Related
I'm having an issue with a particular team project collection on my TFS2015.4. This collection caused issues when I wanted to upgrade TFS sometime ago as well. I was able to detach it in TFS2013.3 and then upgraded. Now I want to upgrade to TFS2017 and I don't know how to resolve the issue with this collection.
TF400868: Job definition not found for JobId d891ac97-ddf1-42df-8242-3cd4bd607790
Here is the current status:
Won't detach
Won't start
Status -> ApplyPatch -> won't execute
Stays Offline
One project inside and stays at 'Deleting' state
If I try to start, I get this error:
TF400783: The host 'MyDAS' cannot be started. The host is in the process of being serviced. The servicing may have failed and needs to be restarted and completed before the host can be started.
I did a pre-production upgrade to TFS2017 and there was a validation error with this collection's state that prevented me from finishing the upgrade.
The detailed log for ApplyPatch Rerun has just one failure point:
[12:29:58.457] +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[12:29:58.457] Executing step: Populate commit changes
[12:29:58.457] Executing step: 'Populate commit changes' Git.M83PopulateCommitChanges (1017 of 1201)
[12:29:58.477] [Error] TF400868: Job definition not found for JobId d891ac97-ddf1-42df-8242-3cd4bd607790.
[12:29:58.480] Microsoft.TeamFoundation.Framework.Server.JobDefinitionNotFoundException: TF400868: Job definition not found for JobId d891ac97-ddf1-42df-8242-3cd4bd607790.
[12:29:58.480] at Microsoft.TeamFoundation.Framework.Server.TeamFoundationJobService.ResolveJobPriorityClasses(IVssRequestContext requestContext, IEnumerable`1 jobReferences, ITFLogger logger)
[12:29:58.480] at Microsoft.TeamFoundation.Framework.Server.TeamFoundationJobService.QueueJobsRaw(IVssRequestContext requestContext, IEnumerable`1 jobReferences, JobPriorityLevel priorityLevel, Int32 maxDelaySeconds, ITFLogger logger, Boolean queueAsDormant)
[12:29:58.480] at Microsoft.TeamFoundation.Server.Deploy.TFCollection.GitStepPerformer.M83PopulateCommitChanges(IVssRequestContext requestContext, ServicingContext servicingContext)
[12:29:58.480] at Microsoft.TeamFoundation.Framework.Server.TeamFoundationStepPerformerBase.PerformHostStep(String servicingOperation, ServicingOperationTarget target, IServicingStep servicingStep, String stepData, ServicingContext servicingContext)
[12:29:58.480] at Microsoft.TeamFoundation.Framework.Server.TeamFoundationStepPerformerBase.PerformStep(String servicingOperation, ServicingOperationTarget target, String stepType, String stepData, ServicingContext servicingContext)
[12:29:58.480] at Microsoft.TeamFoundation.Framework.Server.ServicingStepDriver.PerformServicingStep(ServicingStep step, ServicingContext servicingContext, ServicingStepGroup group, ServicingOperation servicingOperation, Int32 stepNumber, Int32 totalSteps)
[12:29:58.480] Step failed: Populate commit changes. Execution time: 23 milliseconds.
[12:29:58.480] [StepDuration] 0.0236576
[12:29:58.480] [GroupDuration] 0.2517195
[12:29:58.480] [OperationDuration] 0.2517302
[12:29:58.587] Clearing dictionary, removing all items.
======================================================================================================
Step execution times in descending order
======================================================================================================
Updates all rows in tbl_GitCommit and sets the Status to ... (GitToDev14M83Collection, ToDev14M83Collection) - 227 milliseconds
Populate commit changes (GitToDev14M83Collection, ToDev14M83Collection) - 23 milliseconds
Write service level to stamp (StartInstallUpdates, StartInstallUpdates) - 20 milliseconds
Configure framework servicing tokens (VsspToDev14M71Collection, VsspToDev14M71Collection) - 20 milliseconds
Setup integration environment (TestManagementToDev12M65FinalConfiguration, ToDev12M65FinalConfiguration) - 3 milliseconds
Setup Git environment (GitToDev14M74Collection, ToDev14M74Collection) - 1 millisecond
Setup Git environment (GitToDev14M83Collection, ToDev14M83Collection) - 1 millisecond
Set the collection partition id tokens in servicing context (GitToDev14M83Collection, ToDev14M83Collection) - 1 millisecond
======================================================================================================
Execution times by group in descending order
======================================================================================================
GitToDev14M83Collection (ToDev14M83Collection) - 250 milliseconds
StartInstallUpdates (StartInstallUpdates) - 20 milliseconds
VsspToDev14M71Collection (VsspToDev14M71Collection) - 20 milliseconds
TestManagementToDev12M65FinalConfiguration (ToDev12M65FinalConfiguration) - 3 milliseconds
GitToDev14M74Collection (ToDev14M74Collection) - 1 millisecond
I was pondering the fact that this is a rouge job that needs to be deleted from the databse manually but I mmight be wrong. Any pointer will be greatfully +1-ed.
The main reason for TFS being unable to upgrade team project collections usually is that their data was already either corrupted, incomplete, or stuck between schema versions.
You will need to contact customer support for assistance troubleshooting these data issues and getting your data back to a workable state.
Since there is only one project inside. If you don't care the source control history and have the back up of project. A crude (not recommend) way should be deleting the special project collection, delete the database from SQL Server and create a new collection, restore the project inside.
Also take a look at this similar question with the same error as you: Starting a team project collection after detaching and attaching again
Ultimately, there was no way to recover the collection. So I used the admin command to permanently remove the collection. In my case, n one came back to me asking where the collection go so it's solved for me. I'm running the latest TFS 2017.2 and the rest of the dev are happy.
I have a job that periodically does some work involving ServerXmlHttpRquest to perform an HTTP POST. The job runs every 60 seconds.
And normally it runs without issue. But there's about a 1 in 50,000 chance (every two or three months) that it will hang:
IXMLHttpRequest http = new ServerXmlHttpRequest();
http.open("POST", deleteUrl, false, "", "");
http.send(stuffToDelete); <---hang
When it hangs, not even the Task Scheduler (with the option enabled to kill the job if it takes longer than 3 minutes to run) can end the task. I have to connect to the remote customer's network, get on the server, and use Task Manager to kill the process.
And then its good for another month or three.
Eventually i started using Task Manager to create a process dump,
so i could analyze where the hang is. After five crash dumps (over the last 11 months or so) i get a consistent picture:
ntdll.dll!_NtWaitForMultipleObjects#20()
KERNELBASE.dll!_WaitForMultipleObjectsEx#20()
user32.dll!MsgWaitForMultipleObjectsEx()
user32.dll!_MsgWaitForMultipleObjects#20()
urlmon.dll!CTransaction::CompleteOperation(int fNested) Line 2496
urlmon.dll!CTransaction::StartEx(IUri * pIUri, IInternetProtocolSink * pOInetProtSink, IInternetBindInfo * pOInetBindInfo, unsigned long grfOptions, unsigned long dwReserved) Line 4453 C++
urlmon.dll!CTransaction::Start(const wchar_t * pwzURL, IInternetProtocolSink * pOInetProtSink, IInternetBindInfo * pOInetBindInfo, unsigned long grfOptions, unsigned long dwReserved) Line 4515 C++
msxml3.dll!URLMONRequest::send()
msxml3.dll!XMLHttp::send()
Contoso.exe!FrobImporter.TFrobImporter.DeleteFrobs Line 971
Contoso.exe!FrobImporter.TFrobImporter.ImportCore Line 1583
Contoso.exe!FrobImporter.TFrobImporter.RunImport Line 1070
Contoso.exe!CommandLineProcessor.TCommandLineProcessor.HandleFrobImport Line 433
Contoso.exe!CommandLineProcessor.TCommandLineProcessor.CoreExecute Line 71
Contoso.exe!CommandLineProcessor.TCommandLineProcessor.Execute Line 84
Contoso.exe!Contoso.Contoso Line 167
kernel32.dll!#BaseThreadInitThunk#12()
ntdll.dll!__RtlUserThreadStart()
ntdll.dll!__RtlUserThreadStart#8()
So i do a ServerXmlHttpRequest.send, and it never returns. It will sit there for days (causing the system to miss financial transactions, until come Sunday night i get a call that it's broken).
It is of no help unless someone knows how to debug code, but the registers in the stalled thread at the time of the dump are:
EAX 00000030
EBX 00000000
ECX 00000000
EDX 00000000
ESI 002CAC08
EDI 00000001
EIP 732A08A7
ESP 0018F684
EBP 0018F6C8
EFL 00000000
Windows Server 2012 R2
Microsoft IIS/8.5
Default timeouts of ServerXmlHttpRequest
You can use serverXmlHttpRequest.setTimeouts(...) to configure the four classes of timeouts:
resolveTimeout: The value is applied to mapping host names (such as "www.microsoft.com") to IP addresses; the default value is infinite, meaning no timeout.
connectTimeout: A long integer. The value is applied to establishing a communication socket with the target server, with a default timeout value of 60 seconds.
sendTimeout: The value applies to sending an individual packet of request data (if any) on the communication socket to the target server. A large request sent to a server will normally be broken up into multiple packets; the send timeout applies to sending each packet individually. The default value is 30 seconds.
receiveTimeout: The value applies to receiving a packet of response data from the target server. Large responses will be broken up into multiple packets; the receive timeout applies to fetching each packet of data off the socket. The default value is 30 seconds.
The KB305053 (a server that decides to keep the connection open will cause serverXmlHttpRequest to wait for the connection to close) seems like it plausibly could be the issue. But the 30 second default timeout would have taken care of that.
Possible workaround - Add myself to a Job
The Windows Task Scheduler is unable to terminate the task; even though the option is enabled to do do.
I will look into using the Windows Job API to add my self process to a job, and use SetInformationJobObject to set a time limit on my process:
CreateJobObject
AssignProcessToJobObject
SetInformationJobObject
to limit my process to three minutes of execution time:
PerProcessUserTimeLimit
If LimitFlags specifies
JOB_OBJECT_LIMIT_PROCESS_TIME, this member is the per-process
user-mode execution time limit, in 100-nanosecond ticks. Otherwise,
this member is ignored.
The system periodically checks to determine
whether each process associated with the job has accumulated more
user-mode time than the set limit. If it has, the process is
terminated.
If the job is nested, the effective limit is the most
restrictive limit in the job chain.
Although since Task Scheduler uses Job objects to also limit a task's time, i'm not hopeful that the Job Object can limit a job either.
Edit: Job objects cannot limit a process by process time - only user time. And with a process idle waiting for an object, it will not accumulate any user time - certainly not three minutes worth.
Bonus Reading
How can a ServerXMLHTTP GET request hang? (GET, not POST)
KB305053: ServerXMLHTTP Stops Responding When You Send a POST Request (which says the timeout should expire; where mine does not)
MS Forums: oHttp.Send - Hangs (HEAD, not POST)
MS Forums: ASP to test SOAP WebService using MSXML2.ServerXMLHTTP Send hangs
CC to MS Support Forums
Consider switching to a newer, supported API.
msxml6.dll using MSXML2.ServerXMLHTTP.6.0
winhttpcom.dll using WinHttp.WinHttpRequest.5.1.
The msxml3.dll library is no longer supported and is only kept around for compatibility reasons. Plus, there were a number of security and stability improvements included with msxml4.dll (and newer) that you are missing out on.
I have an ASP.NET MVC application. It's run on a domain and we determine the user by calling UserPrincipal.Current. This works beautifully, most of the time. Every once in a while (maybe 1 out of 5 times), and only right after I publish my app to IIS, it will throw an AppDomainUnloadedException.
The specific line of my code causing the exception is:
if (UserPrincipal.Current.SamAccountName != null &&
_currentUserId != UserPrincipal.Current.SamAccountName) ...
The remainder of the call stack is:
mscorlib.dll!System.StubHelpers.StubHelpers.GetCOMHRExceptionObject(int hr, System.IntPtr pCPCMD, object pThis) + 0xe bytes
System.DirectoryServices.AccountManagement.dll!System.DirectoryServices.AccountManagement.ADStoreCtx.LoadDomainInfo() + 0x358 bytes
System.DirectoryServices.AccountManagement.dll!System.DirectoryServices.AccountManagement.ADStoreCtx.DnsDomainName.get() + 0x5e bytes
System.DirectoryServices.AccountManagement.dll!System.DirectoryServices.AccountManagement.ADStoreCtx.GetAsPrincipal(object storeObject, object discriminant = {Name = "UserPrincipal" FullName = "System.DirectoryServices.AccountManagement.UserPrincipal"}) + 0x17a bytes
System.DirectoryServices.AccountManagement.dll!System.DirectoryServices.AccountManagement.ADStoreCtx.FindPrincipalByIdentRefHelper(System.Type principalType, string urnScheme, string urnValue, System.DateTime referenceDate, bool useSidHistory) + 0x576 bytes
System.DirectoryServices.AccountManagement.dll!System.DirectoryServices.AccountManagement.ADStoreCtx.FindPrincipalByIdentRef(System.Type principalType, string urnScheme, string urnValue, System.DateTime referenceDate) + 0x35 bytes
System.DirectoryServices.AccountManagement.dll!System.DirectoryServices.AccountManagement.Principal.FindByIdentityWithTypeHelper(System.DirectoryServices.AccountManagement.PrincipalContext context, System.Type principalType, System.DirectoryServices.AccountManagement.IdentityType? identityType, string identityValue, System.DateTime refDate) + 0x9e bytes
System.DirectoryServices.AccountManagement.dll!System.DirectoryServices.AccountManagement.Principal.FindByIdentityWithType(System.DirectoryServices.AccountManagement.PrincipalContext context, System.Type principalType, System.DirectoryServices.AccountManagement.IdentityType identityType, string identityValue) + 0x5b bytes
System.DirectoryServices.AccountManagement.dll!System.DirectoryServices.AccountManagement.UserPrincipal.FindByIdentity(System.DirectoryServices.AccountManagement.PrincipalContext context, System.DirectoryServices.AccountManagement.IdentityType identityType, string identityValue) + 0x1e bytes
System.DirectoryServices.AccountManagement.dll!System.DirectoryServices.AccountManagement.UserPrincipal.Current.get() + 0xc1 bytes
After a great deal of digging around, and due to the random nature of the exception, I concluded the problem was probably related to this:
http://support.microsoft.com/kb/2683913
So I installed the hotfix and as expected, the problem went away. That was Tuesday. This morning, the exception started up again. I verified that the hotifx is in fact installed. I turned on module load messages and got:
'w3wp.exe': Loaded 'C:\Windows\SysWOW64\activeds.dll', Cannot find or open the PDB file.
So I verified that that .dll is in fact the one from the hotfix and it is (size and version #s match).
At the same time I get this exception, I get this event in the event log:
A process serving application pool 'WebSitePool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '7404'. The data field contains the error number.
The only data in the error is: 0x8007006D which I believe simply means there was a fatal communication error which the error already said...
Humorously, when the exception dialog pops up it states, "If there is a handler for this exception, the program may be safely continued." Which is true if your app has no need for UserPrincipal.Current, but calling it again after this exception causes the exception to rethrow, so I'd question their "safely continued" assertion there.
Now, if I rerun the app, I won't get a problem. I have yet to have this happen any time, other than right after publishing. Because of our setup and relations to other projects, this app must be debugged under IIS, which means I have to publish it every time I make a code change which means I normally see this throughout the day (Tuesday afternoon and yesterday being the exceptions).
It seems curious to me that, from digging around, it appears this bug is frequently associated with ASP.NET MVC, but I don't believe I've seen it associated with WinForms or ASP.NET... I can't see how that would matter, but maybe there's a relationship there.
I suspect this is an MS bug. I can't imagine that there's any legitimate condition under which UserPrincipal.Current should cause an AppDomainUnloadedException, but I thought I'd take a shot here on Stackoverflow...
I have not found this answered anywhere. I have a CLR function that exectues a webmethod call of my .NET application (.asmx). The web service successfully executes when called directly but when called via the CLR it times out after 100 seconds with the following error:
Msg 6522, Level 16, State 1, Line 1
A .NET Framework error occurred during execution of user-defined routine or aggregate "fn_ExecuteReport":
System.Net.WebException: The operation has timed out
System.Net.WebException:
at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)
at System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest request)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
at DD.WebServices.WebExec.ExecuteReport(String ddBotID, String serverKey, Int32 ddUserID, String reportReportTypeList, String deliverToUserList)
at ExecuteReport.GetResult(Int32 userID, SqlString reportList, SqlString deliverToUserList)
I have increased the web service proxy timeout in fn_ExecuteReport without effect:
WebExec svc = new WebExec();<br/>
svc.Timeout = 3600000; // set timeout to 1 hour<br/>
result = svc.ExecuteReport(userID, reportTypeList.ToString(),
deliverToUserList.ToString());
I want to capture the returned result so executing the webservice asynchronously is not a solution. Where else might I override timeout settings for the SQL CLR call? Thanks for any help you can provide.
Here's the code for the function. I'm able to execute the webservice, the timeout only occurs when executing via the CLR.
ALTER FUNCTION [dbo].[fn_ExecuteReport]
(#UserID int, #ReportTypeList nvarchar(max), #DeliverToUserList nvarchar(max))
RETURNS [nvarchar](255)
WITH EXECUTE AS CALLER AS EXTERNAL NAME [MyCLRLib].[ExecuteReport].[GetResult]
I've tried both synchronous and asynchronous calls to the web service in the CLR function and both end up with the 100 second timeout. Here's both calls that I've tried:
Synchronous:
WebExec svc = new WebExec();
svc.Timeout = 3600000; // set timeout to 1 hour
result = svc.ExecuteReport(userID, reportTypeList.ToString(), deliverToUserList.ToString());
Asynchronous:
WebExec svc = new WebExec();
IAsyncResult result = svc.BeginExecuteReport(userID, reportTypeList.ToString(), deliverToUserList.ToString(), null, null);
result.AsyncWaitHandle.WaitOne();
retStr = svc.EndExecuteReport(result);
Your error message suggests that the timeout is originating at SQL Server.
Have you tried updating your SQL Server statistics (or rebuilding indexes)?
Can you post the code of your CLR function?
I am using Sping.Net 1.3.1 and Nhibernate 3.0.
I use Spring's Transaction Interceptor in order to create my Transactions.
I mark my Transactional methods with the Transaction Attribute.
My server gets something like 20 - 25 requests per second, each request is
handled on a new thread, using parallel's Task.
I run a stress test in order to verify my server capability of handling the calls.
when i run only two or three calls ion a time, every thing works great, but
when I run 5 -10 calls simultanly I got an exception from Spring.
The exception is:
Spring.Transaction.TransactionSystemException was unhandled by user code
Message=Could not commit Hibernate transaction
Source=Spring.Data.NHibernate30
StackTrace:
at Spring.Data.NHibernate.HibernateTransactionManager.DoCommit(DefaultTransactionStatus status) in c:\_svn\spring-net\tags\spring-net-1.3.1\src\Spring\Spring.Data.NHibernate\Data\NHibernate\HibernateTransactionManager.cs:line 568
at Spring.Transaction.Support.AbstractPlatformTransactionManager.ProcessCommit(DefaultTransactionStatus status)
InnerException: NHibernate.TransactionException
Message=Transaction not connected, or was disconnected
Source=NHibernate
StackTrace:
at NHibernate.Transaction.AdoTransaction.CheckNotZombied() in d:\CSharp\NH\nhibernate\src\NHibernate\Transaction\AdoTransaction.cs:line 408
at NHibernate.Transaction.AdoTransaction.Commit() in d:\CSharp\NH\nhibernate\src\NHibernate\Transaction\AdoTransaction.cs:line 181
at Spring.Data.NHibernate.HibernateTransactionManager.DoCommit(DefaultTransactionStatus status) in c:\_svn\spring-net\tags\spring-net-1.3.1\src\Spring\Spring.Data.NHibernate\Data\NHibernate\HibernateTransactionManager.cs:line 556
InnerException:
Thank you very much,
Or Chubook.
I am sure you have already discovered this answer by now. When you share a NHibernate session among multiple threads, you will run into this concurrency problem. Each thread must have their own session in scope in order to avoid a transaction disconnect state.