Wix - ServiceControl start takes four minutes to fail, should be 30 sec - wix

Question

My service automatically starts during install...
<ServiceControl Id="StartService" Start="install" Stop="both" Remove="uninstall" Name="HeskaGateway" Wait="yes" />
And it works fine if I provide the service with a valid connection string. If I provide a bad connection string the service starts and stops very quickly... I see this when I go to Services and do a manual start. According to the documentation on MSI ServiceControl Table, a Wait value of "yes" turns into a 1 which means it should wait for 30 seconds and then fail. It takes 4 minutes and 7 seconds. Why so long?
MSI (s) (6C:78) [16:36:41:932]: Executing op: ServiceControl(,Name=HeskaGateway,Action=1,Wait=1,)
StartServices: Service: Heska Gateway
MSI (s) (6C:78) [16:40:48:862]: Note: 1: 2205 2: 3: Error
MSI (s) (6C:78) [16:40:48:862]: Note: 1: 2228 2: 3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1920
Error 1920. Service 'Heska Gateway' (HeskaGateway) failed to start. Verify that you have sufficient privileges to start system services.
EDIT: I never got to find out what my real problem was. I also had an installation sequencing error because my CustomAction (deferred) which would edit the connection string in the JSON file was triggering AFTER the ServiceStart. Trying to move the ServiceStart after that deferred custom action was awful. So I killed off the start from the ServiceControl entry and then added another custom action which silently ran "SC.EXE start HeskaGateway". I'll document that below as a solution.

Answer 1

The installer has a custom UI which asks the user to copy-paste their connection string given to them from support department. The installer edits a JSON file in the app folder using a deferred CustomAction. It is deferred because it needed to be after the files are written to the disk and it also needed to have elevated permissions. This all worked great until I decided to have the service start itself "at the end" of the installation. My first attempt was to use a
<ServiceControl Id="StartService" Start="install" ...>
But that was taking 4 minutes to fail. Troubleshooting showed that the service was being started BEFORE the custom action which writes the connection string into the JSON file. I needed the service start to be delayed till after the custom action. I looked at adding a second ServiceControl entry into its own component that could be scheduled much later but that gave me an uncomfortable feeling that I was going to break uninstall and repair installs. So, I just added another deferred custom action sequenced right after the JSON file edit.
That new action executes "SC.EXE start MyServiceName". SC.EXE is a non-blocking way to start services so succeed or fail, it will finish quickly.
My final solution:
<Component Id="MyCloudSync.exe" Guid="{generate-your-own-guid}">
<File Id="MyCloudSync.exe.file" KeyPath="yes" Source="$(var.RELEASEBINARIES)\MyCloudSync.exe" />
<ServiceInstall Id="MyCloudSync.exe"
Type="ownProcess"
Name="MyGateway"
DisplayName="My Gateway"
Description="Synchronizes laboratory data with Cloud"
Start="auto"
ErrorControl="normal" />
<!--Start is performed by a customer action that calls SC.EXE so it can be delayed after the custom action that writes the JSON file -->
<ServiceControl Id="StartService" Stop="both" Remove="uninstall" Name="MyGateway" Wait="yes" />
</Component>
You should note that the ServiceControl entry above does not have a "start="
The DLL that GetConnectionString and SetConnectionString calls is one of my own making. Wix has its own custom action for running command lines quietly... WixQuietExec
<CustomAction Id= "GetConnectionString"
BinaryKey="MyCustomActions"
DllEntry="GetConnectionString"
Execute="immediate"/>
<CustomAction Id= "SetConnectionString"
BinaryKey="MyCustomActions"
Impersonate="no"
DllEntry="SetConnectionString"
Execute="deferred"/>
<CustomAction Id="SetConnectionStringDeferredParams"
Property="SetConnectionString"
Value=""[INSTALLFOLDER]""[CONNECTIONSTRING]"" />
<Property Id="QtExecStartService" Value=""SC.EXE" start MyGateway"/>
<CustomAction Id="QtExecStartService"
BinaryKey="WixCA"
DllEntry="WixQuietExec"
Impersonate="no"
Execute="deferred"
Return="ignore"/>
Starting the service is just a convenience so for the installer to prevent going to Services.msc to perform the start or requiring a reboot. So I used Return="ignore". Also SC.EXE just puts the service into "Start Pending" so it probably can't return much of an error unless your service doesn't exist.
NOTE: WixQuietExec is documented here. Make sure to quote the EXE and give your property the same Id as your CustomAction that uses WixQuietExec. That info is under "Deferred Execution" but I still got it wrong on my first try.

Answer 2

The service itself might be doing something. The service control protocol includes a service status that's returned from the service itself, and this tells Windows what's going on. One of the items in there is a wait hint. Knowing nothing about the service, it's possible that the service is aware that it might have a slow startup and tells Windows (with a wait hint) that it should wait longer. 30 seconds is really a default, not a fixed value. This post refers to the wait hint for a managed code service:
How to choose value for serviceStatus.dwWaitHint?
You didn't show the ServiceControl used to install the service, but if it's shared with another service in the same process things can get complicated because the process itself can't terminate while it's also hosting another service.

Answer 3

Error 1920: this error would seem to indicate a missing privilege (logon as a service) or access right, or perhaps some sort of interference from an external cause or perhaps an MSI package that is not running elevated (unlikely I think - then you can only write to per-user paths).
Are you running this service with the LocalSystem account or NetworkService or LocalService or with a regular user account? (about the above service accounts).
Or the newer concepts of managed service accounts, group managed service accounts or virtual accounts step-by-step info (concepts that I do not know enough about).
If you use a regular user account, does it have the "logon as a service" privilege set?
If you create the user in your WiX MSI (or define it), you can set LogonAsService="yes" for the User(element) in question. I believe this adds the privilege for the account (SeServiceLogonRight).
A privilege is different from access rights (ACLs) - for the record: a privilege is a pervasive system-wide access to some sort of function / feature - for example change system time, start / stop services, logon as a service, etc... (see section 13 here for more).
Is there a security software / antivirus on your test box? It could be interfering with your setup's API calls. If so, try to disable it (if possible) during the installation process. I'll mention firewalls too.
Is your MSI set to run elevated? (Package Element => InstallPrivileges).
UPDATE: Just adding the issue it turned out to be: the custom action update of config data needed to run with elevated rights or faulty service configuration data resulted which in turn caused the generic 1920 error message. In this case the configuration was in JSON format, it can obviously be in several formats: XML, registry, INI, etc... See OP comment above for more details.
Timeout: As to the long timeout. I have seen this sometimes with security software (locks the whole setup so the service timeout runs only after some scanning delay), or with setups that trigger the creation of a system restore point prior to the installation kicking off in the first place (this is one of the possible reasons why some MSI installations suddenly take a long time when they ordinarily complete quickly - it is possible to prevent this restore point creation - MSIFASTINSTALL). Also, maybe check this answer from serverfault and the registry value described for whether there is a policy on your network to change the default service start timeout: How do I increase windows service startup timeout (ServicesPipeTimeout). Frankly I am not sure whether MSI uses its own timeout or the system default one - maybe someone can illuminate? One could also speculate that the database connection you initiate has its own timeout? (doesn't match your interactive test experience?) Maybe you can check your code and your call and let us know? Does your service depend on another service? (see symantec link below). Any dependencies to files installed to the GAC or to WinSxS as mentioned by Chris in the first link below?
A lot of speculation. Let's hope some of it helps or that it inspires other ideas that solve the problem. Below some links for safekeeping (trying to write a generic answer that may also help others with the same or similar problems - which makes the answer too long, sorry about that).
Links:
Error 1920 service failed to start. Verify that you have sufficient privileges to start system services
http://blog.iswix.com/2008/09/different-year-same-problem.html
https://support.symantec.com/en_US/article.TECH103676.html (adds issues such as: dependency on other services)
How to debug Windows services (service timeout and more)
windows service startup timeout (I would check this)
Service failed to start error 1920 (generatePublisherEvidence)