Topshelf Windows Service times out Error 7000 7009 - vb.net

I have a windows service programmed in vb.NET, using Topshelf as Service Host.
Once in a while the service doesn't start. On the event log, the SCM writes errors 7000 and 7009 (service did not respond in a timely fashion). I know this is a common issue, but I (think) I have tried everything with no result.
The service only relies in WMI, and has no time-consuming operations.
I read this question (Error 1053: the service did not respond to the start or control request in a timely fashion), but none of the answers worked for me.
I Tried:
Set topshelf's start timeout.
Request additional time in the first line of "OnStart" method.
Set a periodic timer wich request additional time to the SCM.
Remove TopShelf and make the service with the Visual Studio Service Template.
Move the initialization code and "OnStart" code to a new thread to return inmediately.
Build in RELEASE mode.
Set GeneratePublisherEvidence = false in the app.config file (per application).
Unchecked "Check for publisher’s certificate revocation" in the internet settings (per machine).
Deleted all Alternate Streams (in case some dll was marked as web and blocked).
Removed any "Debug code"
Increased Window's general service timeout to 120000ms.
Also:
The service doesn't try to communicate with the user's desktop in any way.
The UAC is disabled.
The Service Runs on LOCAL SYSTEM ACCOUNT.
I believe that the code of the service itself is not the problem because:
It has been on production for over two years.
Usually the service starts fine.
There is no exception logged in the Event Log.
The "On Error" options for the service dosn't get called (since the service doesn't actually fails, just doesn't respond to the SCM)
I've commented out almost everything on it, pursuing this error! ;-)
Any help is welcome since i'm completely out of ideas, and i've been strugling with this for over 15 days...

For me the 7009 error was produced by my NET core app because I was using this construct:
var builder = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json");
and appsettings.json file obviously couldn't be found in C:\WINDOWS\system32.. anyway, changing it to Path.Combine(AppContext.BaseDirectory, "appsettings.json") solved the issue.
More general help - for Topshelf you can add custom exception handling where I finally found some meaningfull error info, unlike event viewer:
HostFactory.Run(x => {
...
x.OnException(e =>
{
using (var fs = new StreamWriter(#"C:\log.txt"))
{
fs.WriteLine(e.ToString());
}
});
});

I've hit the 7000 and 7009 issue, which fails straight away (even though the error message says A timeout was reached (30000 milliseconds)) because of misconfiguration between TopShelf and what the service gets installed as.
The bottom line - what you pass in HostConfigurator.SetServiceName(name) needs to match exactly the SERVICE_NAME of the Windows service which gets installed.
If they don't match it'll fail straight away and you get the two event log messages.

I had this start happening to a service after Windows Creator's Edition update installed. Basically it made the whole computer slower, which is what I think triggered the problem. Even one of the Windows services had a timeout issue.
What I learned online is that the constructor for the service needs to be fast, but OnStart has more leeway with the SCM. My service had a C# wrapper and it included an InitializeComponent() that was called in the constructor. I moved that call to OnStart and the problem went away.

Related

WCF InstancePersistenceCommand Exception

I have a WCF application which consists in some async communications with ecternal services. When we start a new expedient, a new instance is created; it process data and send an xml to a external service and waits for the response. This response requires that a person review the xml and send the response so it usually it is delayed for a long time. For this reason, the workflow go to idle and we use persistence with AppFabric.
The fact is that sometime, when we receive the response, the next exception is raised:
The execution of the InstancePersistenceCommand named {urn:schemas-microsoft-com:System.Activities.Persistence/command}LoadWorkflowByInstanceKey was interrupted by an error.
Normally this error does not occur, it can occur very sporadically. However, we are trying to update the app to include a new functionality (it does not modify the workflow) but when the application is deployed to the server, the instances that were created with the old deployment and were waiting for the response, throw this exception when they receive the response from the external service. However, the instances initiated with the new deployment process the response without problem.
I have been looking for information about this problem but I haven't found much. Anybody can help me?
SOLUTION:
Thanks a lot for your answer, it may be helpful for me in the future. In this case, the problem was that I was updating an assembly version of one of the implicated project (to upload a nuget package) and for a reason that I don’t understand, the instances created with an old version raised this exception when the service with the new version had to manipulate the mentioned instances.
If I change the assembly version to upload the nuget and then set the original version and deploy with this version, everything works ok. Anybody knows what is the reason?
Thanks a lot.
This may be because there is a program running in the background and trying to extend the lock on the instance store every 30 seconds, and it seems that whenever the connection to the SQL service fails, it marks the instance store as invalid.
You can try <workflowIdle timeToUnload="0"/>, if it doesn't work you can look at the methods provided by other links.
Windows workflow 4.0 InstancePersistenceCommand Error
Why do I get exception "The execution of the InstancePersistenceCommand named LoadWorkflowByInstanceKey was interrupted by an error"
WF4 InstancePersistenceCommand interrupted

MassTransit Consumer never invoked when using Windsor Integration

I can't seem to get the Castle Windsor Integration working for Mass Transit over RabbitMQ. Everything was working fine until I introduced Windsor into the picture. I referenced Castle.Windsor 3.2 and MassTransit.WindsorIntegration 2.9 and configured the container for use within my application. I'm registering the MassTransit Consumers via:
Container.Register(..., Types.FromThisAssembly().BasedOn<IConsumer>());
When I debug and inspect the container after this line is ran, I can see that it successfully registered all of the consumers along with all of my other components. I then have the following code to initialize and register the service bus:
var serviceBus = ServiceBusFactory.New(sbc =>
{
sbc.UseRabbitMq();
sbc.ReceiveFrom(Config.ServiceBusEndpoint);
sbc.Subscribe(sc => sc.LoadFrom(Container));
});
Container.Register(Component.For<IServiceBus>().Instance(serviceBus));
I am using the LoadFrom(IWindsorContainer container) extension method provided by MassTransit.WindsorIntegration.
All of the examples I've found so far stop here and indicate that this is all you should have to do. Unfortunately for me my Consumers are never being called and messages are just piling up in the queue (eventually timing out and going to error queue). The fact that messages are showing up in the Consumer queue at all (+ I see a single consumer bound to the queue via the RabbitMQ Admin Tool) indicates to me that the consumers are probably being subscribed properly - so I'm not sure where the problem lies.
I added NLog logging for Windsor and MassTransit but no errors are showing up in the logs. I'm not sure how I should proceed troubleshooting at this point. Any ideas?
Also, this application is currently just a console application using Topshelf for development. Ultimately it will be installed as a Windows Service. Not sure if that is relevant or not but I thought I'd mention it just in case.
UPDATE
As a test I created a very simple Consumer with a parameter-less constructor for processing a very simple test message. This Consumer is successfully called! The "real" Consumers however have dependencies that need to be injected into them via the constructor. I was hoping the Container would resolve these but apparently it's having some sort of trouble. Weird that nothing is showing up in the logs about it. Stay tuned...
Okay I figured it out. Somewhere along the way when I was adding/removing NuGet packages I somehow managed to delete a reference to a DLL (ServiceStack.Text.dll) that one of my components needed (RedisClientsManager).
I started the debugger, let all my components get registered then popped open the Immediate Window and attempted to resolve each component one by one (by calling container.Resolve<RegisteredType>()) until I found the one that threw the exception when I attempted to resolve it.
The Exception message from Windsor at that point told me exactly what the problem was. I'm a little lost as to why this wasn't being logged or why the Exception was not raised when the container itself attempted to resolve it. Anyhow, moral of the story is make sure your components resolve.

Task Persistence C#

I'm having a hard time trying to get my task to stay persistent and run indefinitely from a WCF service. I may be doing this the wrong way and am willing to take suggestions.
I have a task that starts to process any incoming requests that are dropped into a BlockingCollection. From what I understand, the GetConsumingEnumerable() method is supposed to allow me to persistently pull data as it arrives. It works with no problem by itself. I was able to process dozens of requests without a single error or flaw using a windows form to fill out the request and submit them. Once I was confident in this process I wired it up to my site via an asmx web service and used jQuery ajax calls to submit request.
The site submits request based on a url that is submitted, the Web Service downloads the html content from the url and looks for other urls within the content. It then proceeds to create a request for each url it finds and submits it to the BlockingCollection. Within the WCF service, if the application is Online (i.e. Task has started) - it pulls the request using the GetConsumingEnumerable via a Parallel.ForEach and Processes the request.
This works for the first few submissions, but then the task just stops unexpectedly. Of course, this is doing 10x more request than I could simulate in testing - but I expected it to just throttle. I believe the issue is in my method that starts the task:
public void Start()
{
Online = true;
Task.Factory.StartNew(() =>
{
tokenSource = new CancellationTokenSource();
CancellationToken token = tokenSource.Token;
ParallelOptions options = new ParallelOptions();
options.MaxDegreeOfParallelism = 20;
options.CancellationToken = token;
try
{
Parallel.ForEach(FixedWidthQueue.GetConsumingEnumerable(token), options, (request) =>
{
Process(request);
options.CancellationToken.ThrowIfCancellationRequested();
});
}
catch (OperationCanceledException e)
{
Console.WriteLine(e.Message);
return;
}
}, TaskCreationOptions.LongRunning);
}
I've thought about moving this into a WF4 Service and just wire it up in a Workflow and use Workflow Persistence, but am not willing to learn WF4 unless necessary. Please let me know if more information is needed.
The code you have shown is correct by itself.
However there are a few things that can go wrong:
If an exception occurs, your task stops (of course). Try adding a try-catch and log the exception.
If you start worker threads in a hosted environment (ASP.NET, WCF, SQL Server) the host can decide arbitrarily (without reason) to shut down any worker process. For example, if your ASP.NET site is inactive for some time the app is shut down. The hosts that I just mentioned are not made to have custom threads running. Probably, you will have more success using a dedicated application (.exe) or even a Windows Service.
It turns out the cause of this issue was with the WCF Binding Configuration. The task suddenly stopped becasue the WCF killed the connection due to a open timeout. The open timeout setting is the time that a request will wait for the service to open a connection before timing out. In certain situations, it reached the limit of 10 max connection and caused the incomming connections to get backed up waiting for a connection. I made sure that I closed all connections to the host after the transactions were complete - so I gave in to upping the max connections and the open timeout period. After this - it ran flawlessly.

Silverlight wcf connection error

I'm about a month away developing my silverlight application (this is my first). Everything went rather smoothly until today, when out of the blue I started getting this message:
An error occurred while trying to make a request to URI 'http://localhost:2682/Services/Authentication/LoginService.svc'. This could be due to attempting to access a service in a cross-domain way without a proper cross-domain policy in place, or a policy that is unsuitable for SOAP services. You may need to contact the owner of the service to publish a cross-domain policy file and to ensure it allows SOAP-related HTTP headers to be sent. This error may also be caused by using internal types in the web service proxy without using the InternalsVisibleToAttribute attribute. Please see the inner exception for more details.
I'm using WCF Services and this issue never appeared until now.
I've added a clientdomain.xml and clientaccesspolicy.xml file to my [projectname].web folder, and re-wrote them about a 1000 different ways.
I've also used Fiddler and it shows me that the error is on both those files, the error is
[Fiddler] The socket connection to localhost failed. ErrorCode: 10061. No connection could be made because the target machine actively refused it 127.0.0.1:2682
I've searched the error "10061" and it has to do with socket definition. But I couldn't find any solution to that.
Don't know if it has anything to do with it, but my "ASP.net Development Server" port is 6939.
Keep in mind that the app has NOT been deployed, so this is only happening locally. I'm using MS VS 2010 and MS SQL Server 2008.
Am I doing anything wrong or is this a silverlight issue??
On a last note, I haven't changed anything on Port, socket or service configuration. Last thing I was doing was editing a XAML file on client side and and the app started throwing me this error.
Need help, can't do anything until this is solved!!!!
Thanks.
i think you are using you app on localhost and a dynamic port is getting assigned and this port is not fixed and every run and that causes the refuse problem. if you want to fix this, create a solid url for example,
http://localhost/apps/Services/Authentication/LoginService.svc
Well, last night, just before I went to bed, I noticed something odd. In my "ServiceReferences.ClientConfig" file, the endpoint ports for each one of my services where diferent from the ones the silverlight machine used, so going on a hunch (and because I was reaching my sanity breakpoint) I decided to eliminate all my Service References and re-add them again.
I worked... go figure. Still don't know why this happened and if anyone could shed some light on the subject, I would appreciate it. It's kinda of annoying having to re-add all my services references. Right now I have only 6 of them, but in the near future they may go over 20, and if this happens again... well, it's going to be a real pain...
Thanks

WCF wsdualhttpbinding strange behaviour at client side that has no clear (Exception details)

to make long story as short as possible, I made a duplex WCF service, using wsdualhttpbinding, tested it on my local machine using Visual Stuido and using IIS7, thigs are working fine (i had to use windows authentication on IIS7 to make it work thu). Any way, I published the duplex service on my company network (LAN network, windows domain, IIS 6) which i used to for many WCF services (basic and ws http bindings and worked fine) and the problem is I can not get the service to work, I tried all possible configurations to the app.config and to the web.config files, still i can not make it work.. The problem is when the client start calling the duplex service It hangs and no error message or whatsoever, and after few minutes it gives me a timeout error.. there is no error at server side ( i have checked the even viewer). There is not error about port 80 or access privileges. Is there anything i am missing? anything i should take care of and i forgot? i have been awake for 20 hours trying to find a solution because i have a time line to follow or my boss will Dispose() Me..
i have tried the following:
1- I set the IsOneWay = true..
2- I run the program as admin.
3- I i set the base address to another port.. same result..
4- I used windows auth, none... same result..
5- i have tried another machine to test the client.. same result.. it hangs with no reponse.
6- I also tried running the client on windows vista, 7 and xp... same result..
7- I have played with all kind of configurations in config files and i made sure its the same on both files (app.config and web.config)..
8- I have added a static constructor in the service side to see if the service was ever reached by the client, i added few lines to add a value in the db.. and nothing was written to db.. so it was never reached..
9- Yes, when i access the service SVC page i see the page and it worked fine.. I could access the metadata as well..
10- restarting the server, restarting IIS server, Recycling AppPool...etc..
I tried all of that and many other solutions and tricks.. same result when the app is on IIS.. the strange thing is, there is no error at server side.. and i only get TimeOut error (if i get it).. While If i tried the service on the IIS installed locally it works fine.. change the config file to point to the company IIS server,,, no luck!
Is there anything i am missing??? could be something simple but i just missed it..