Attempting to consume CRM SOAP endpoint occasionally throws "User id is invalid" - wcf

Question

We have an on-premise CRM with roll-up 13. Some of the forms need to be populated with external data that comes from multiple sources and needs to be pre-processed. To do this, we developed a custom WCF service that runs on the same server as CRM and is hosted in IIS as a virtual app in CRM's website (to avoid cross-domain scripting issues) and exposes a REST endpoint to the javascript that runs in CRM forms. The javascript makes requests, the service collects and processes data and then replies back to javascript with some nice simple JSON.
Since some of the data we need comes from CRM itself, we figured we'll let the service collect that also (using CRM's SOAP endpoint), rather than making the requests from javascript to CRM web services directly. We have pass-through authentication set up in IIS, so our custom app makes requests to CRM while impersonating the user that's logged into CRM.
Most of the time, this all works great. However, once we got more testers we started noticing that occasionally CRM will stop replying to our service for some user, throwing an exception. If that user closes the browser and re-opens it and then tries the exact same operation, it all starts working again. I turned on tracing on the CRM server and found that when this happens, CRM is logging "Crm Exception: Message: The user Id is invalid., ErrorCode: -2147214049". Full trace below:
[2013-09-10 08:54:14.492] Process: w3wp |Organization:00000000-0000-0000-0000-000000000000 |Thread: 69 |Category: Exception |User: 00000000-0000-0000-0000-000000000000 |Level: Error |ReqId: 4b901457-a545-4844-9ce8-b017d3833087 | CrmException..ctor ilOffset = 0x36
at CrmException..ctor(Int32 errorCode, Object[] arguments) ilOffset = 0x36
at SecurityLibrary.GetPrivilegedUserCallerAndBusinessGuidsFromThread(WindowsIdentity identity, IOrganizationContext context) ilOffset = 0x6F
at SecurityLibrary.GetCallerAndBusinessGuidsFromThread(WindowsIdentity identity, Guid organizationId, LocatorServiceContext locatorServiceContext) ilOffset = 0x56
at UserManagementFactory.ValidateSpecialUser(WindowsIdentity identity, Guid organizationId) ilOffset = 0x1E
at WindowsIdentityAuthorizationManager.Authenticate(OperationContext operationContext) ilOffset = 0x185
at WindowsIdentityAuthorizationManager.CheckAccessCore(OperationContext operationContext) ilOffset = 0x22
at AuthorizationBehavior.Authorize(MessageRpc& rpc) ilOffset = 0x28
at ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc) ilOffset = 0x293
at MessageRpc.Process(Boolean isOperationContextSet) ilOffset = 0x62
at ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext) ilOffset = 0x1D7
at ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext) ilOffset = 0xF1
at ChannelHandler.AsyncMessagePump(IAsyncResult result) ilOffset = 0x21
at AsyncThunk.UnhandledExceptionFrame(IAsyncResult result) ilOffset = 0x0
at AsyncResult.Complete(Boolean completedSynchronously) ilOffset = 0xC2
at ReceiveItemAndVerifySecurityAsyncResult`2.InnerTryReceiveCompletedCallback(IAsyncResult result) ilOffset = 0x55
at AsyncThunk.UnhandledExceptionFrame(IAsyncResult result) ilOffset = 0x0
at AsyncResult.Complete(Boolean completedSynchronously) ilOffset = 0xC2
at AsyncQueueReader.Set(Item item) ilOffset = 0x21
at InputQueue`1.Dispatch() ilOffset = 0x121
at ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped) ilOffset = 0x22
at IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped) ilOffset = 0x5
at _IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP) ilOffset = 0x3C
>Crm Exception: Message: The user Id is invalid., ErrorCode: -2147214049
[2013-09-10 08:54:14.508] Process: w3wp |Organization:00000000-0000-0000-0000-000000000000 |Thread: 69 |Category: Platform.Sdk |User: 00000000-0000-0000-0000-000000000000 |Level: Error |ReqId: 4b901457-a545-4844-9ce8-b017d3833087 | ServiceModelTraceRedirector.TraceData ilOffset = 0x45
><TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Error"><TraceIdentifier>http://msdn.microsoft.com/en-CA/library/System.ServiceModel.Diagnostics.TraceHandledException.aspx</TraceIdentifier><Description>Handling an exception.</Description><AppDomain>/LM/W3SVC/1/ROOT-6-130232384646037254</AppDomain><Exception><ExceptionType>Microsoft.Crm.CrmException, Microsoft.Crm.Core, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35</ExceptionType><Message>The user Id is invalid.</Message><StackTrace> at Microsoft.Crm.BusinessEntities.SecurityLibrary.GetPrivilegedUserCallerAndBusinessGuidsFromThread(WindowsIdentity identity, IOrganizationContext context)
> at Microsoft.Crm.BusinessEntities.SecurityLibrary.GetCallerAndBusinessGuidsFromThread(WindowsIdentity identity, Guid organizationId, LocatorServiceContext locatorServiceContext)
> at Microsoft.Crm.Authentication.UserManagementFactory.ValidateSpecialUser(WindowsIdentity identity, Guid organizationId)
> at Microsoft.Crm.Authentication.WindowsIdentityAuthorizationManager.Authenticate(OperationContext operationContext)
> at Microsoft.Crm.Authentication.WindowsIdentityAuthorizationManager.CheckAccessCore(OperationContext operationContext)
> at System.ServiceModel.Dispatcher.AuthorizationBehavior.Authorize(MessageRpc&amp; rpc)
> at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc&amp; rpc)
> at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)</StackTrace><ExceptionString>Microsoft.Crm.CrmException: The user Id is invalid.
> at Microsoft.Crm.BusinessEntities.SecurityLibrary.GetPrivilegedUserCallerAndBusinessGuidsFromThread(WindowsIdentity identity, IOrganizationContext context)
> at Microsoft.Crm.BusinessEntities.SecurityLibrary.GetCallerAndBusinessGuidsFromThread(WindowsIdentity identity, Guid organizationId, LocatorServiceContext locatorServiceContext)
> at Microsoft.Crm.Authentication.UserManagementFactory.ValidateSpecialUser(WindowsIdentity identity, Guid organizationId)
> at Microsoft.Crm.Authentication.WindowsIdentityAuthorizationManager.Authenticate(OperationContext operationContext)
> at Microsoft.Crm.Authentication.WindowsIdentityAuthorizationManager.CheckAccessCore(OperationContext operationContext)
> at System.ServiceModel.Dispatcher.AuthorizationBehavior.Authorize(MessageRpc&amp; rpc)
> at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc&amp; rpc)
> at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)</ExceptionString></Exception></TraceRecord>
It seems to me like a security token or cached credentials of some sort expire or become invalid somehow, and our app can no longer authenticate to CRM. Re-opening IE fixes this. We haven't found a way to reliably reproduce this, but it happens often enough to be a concern to the users. Anyone have any ideas?

Answer 1

One workaround is to scrap pass through authentication. Instead change your web services to always authenticate using the same CRM user id and impersonate users once successfully authenticated.
Set up a new user (e.g. "CRM Web Service User") or pick an existing user account. Make sure this user has a security role with the right to impersonate other users, I believe the correct setting for this is under business management on the security role and is called "Act on behalf of other users".
Once you've set this user up you will need to tweak your web services so that they can accept a User ID from the caller. The web service will use its configured "CRM Web Service User" user to authenticate against CRM, but will then impersonate the user id passed in by the caller (which will in turn apply the correct rights for this user).
This might require a little extra work, but it gives you a bit more control over the authentication and how to handle failures.