WiX CustomAction fires in UI but not Execute Sequence - wix

Question

We need our MSI to store the password in the registry, so users can use our installer to change connection strings, etc.
We found MsiExt for its cryptography DLL and are trying to configure the custom actions.
Our users will need to be able to install both through the UI dialogs and silently -- so we need to able to execute in both sequences.
Here is a sample of our Product.wxs
<Property Id="DB_PASSWORD" Secure="yes"/>
<Property Id="P.DB_PASSWORD">
<RegistrySearch Id="S.DB_PASSWORD" Root="HKLM" Key="SOFTWARE\$(var.Manufacturer)\$(var.ProductName)" Name="DB_PASSWORD" Type="raw" Win64="$(var.Win64)"/>
</Property>
<Component Id="c.RegistryEntries" Guid="XXXXX-XXXXX-XXXXX-XXXXX" Directory="INSTALLDIR">
<RegistryKey Root="HKLM" Key="SOFTWARE\$(var.Manufacturer)\$(var.ProductName)" Action="createAndRemoveOnUninstall">
<RegistryValue Id="R.DB_PASSWORD" Name="DB_PASSWORD" Value="[ENCRYPTED_DBPASSWORD]" Type="string" />
</RegistryKey>
</Component>
<!--For encrypting the database password on the way to the registry-->
<Binary Id="Cryptography" SourceFile="..\..\lib\msiext-1.4\CustomActions\Cryptography.dll"/>
<!--This property will receive the encrypted DB_PASSWORD that the user enters and will be encrypted-->
<Property Id="CRYPTPROTECT_DATA" Hidden="yes" />
<Property Id="CRYPTPROTECT_FLAGS" Value="CRYPTPROTECT_LOCAL_MACHINE|CRYPTPROTECT_UI_FORBIDDEN" />
<CustomAction Id="EncryptPassword" BinaryKey="Cryptography" DllEntry="CryptProtectDataHex" Execute="immediate" />
<CustomAction Id="SetDBUSERsPASSWORDForEncryption" Property="CRYPTPROTECT_DATA" Value="[DB_PASSWORD]" />
<Property Id="ENCRYPTED_DBPASSWORD" Hidden="yes" />
<SetProperty Id="ENCRYPTED_DBPASSWORD" Value="[CRYPTPROTECT_RESULT]" Sequence="execute" After="SetDBUSERsPASSWORDForEncryption" />
<!--This is for decrypting the registry value-->
<Property Id="CRYPTUNPROTECT_DATA" Hidden="yes" />
<Property Id="CRYPTUNPROTECT_FLAGS" Value="CRYPTPROTECT_LOCAL_MACHINE|CRYPTPROTECT_UI_FORBIDDEN" />
<SetProperty Id="CRYPTUNPROTECT_DATA" Value="[P.DB_PASSWORD]" Before="DecryptPassword" >NOT(DB_PASSWORD)</SetProperty>
<CustomAction Id="DecryptPassword" BinaryKey="Cryptography" DllEntry="CryptUnprotectDataHex" Execute="firstSequence" />
<CustomAction Id="SetDBUSERsDecryptedPASSWORD" Property="DB_PASSWORD" Value="[CRYPTUNPROTECT_RESULT]"/>
<CustomAction Id="LaunchApplication" BinaryKey="WixCA" DllEntry="WixShellExec" Impersonate="yes" />
<InstallUISequence>
<Custom Action="DecryptPassword" After="CostFinalize"><![CDATA[NOT(DB_PASSWORD)]]></Custom>
<Custom Action="SetDBUSERsDecryptedPASSWORD" After="DecryptPassword" ><![CDATA[NOT(DB_PASSWORD)]]></Custom>
</InstallUISequence>
<InstallExecuteSequence>
<Custom Action="SchedXmlConfig" After="InstallFiles"><![CDATA[(NOT REMOVE~="All")]]></Custom>
<Custom Action="SetDBUSERsPASSWORDForEncryption" Before="InstallInitialize" />
<Custom Action="EncryptPassword" After="SetDBUSERsPASSWORDForEncryption" />
<Custom Action="DecryptPassword" Before="InstallFiles"><![CDATA[CRYPTUNPROTECT_RESULT]]></Custom>
<Custom Action="SetDBUSERsDecryptedPASSWORD" After="DecryptPassword" ><![CDATA[CRYPTUNPROTECT_RESULT]]></Custom>
</InstallExecuteSequence>
The install log shows that the DecryptPassword is firing during the UISequence. The decryption custom action fires, then the DecryptPassword works.
Action ended 17:52:43: CostFinalize. Return value 1.
MSI (c) (CC:A8) [17:52:43:936]: Doing action: SetCRYPTUNPROTECT_DATA
Action 17:52:43: SetCRYPTUNPROTECT_DATA.
Action start 17:52:43: SetCRYPTUNPROTECT_DATA.
MSI (c) (CC:A8) [17:52:43:937]: PROPERTY CHANGE: Adding CRYPTUNPROTECT_DATA property. Its value is '**********'.
Action ended 17:52:43: SetCRYPTUNPROTECT_DATA. Return value 1.
MSI (c) (CC:A8) [17:52:43:937]: Doing action: DecryptPassword
Action 17:52:43: DecryptPassword.
Action start 17:52:43: DecryptPassword.
MSI (c) (CC:7C) [17:52:49:129]: Invoking remote custom action. DLL: C:\Users\kujotx\AppData\Local\Temp\MSI9904.tmp, Entrypoint: CryptUnprotectDataHex
MSI (c) (CC:54) [17:52:49:130]: Cloaking enabled.
MSI (c) (CC:54) [17:52:49:130]: Attempting to enable all disabled privileges before calling Install on Server
MSI (c) (CC:54) [17:52:49:130]: Connected to service for CA interface.
CryptUnprotectDataHex: MSI Extensions 1.4.1114.0
MSI (c) (CC!94) [17:52:49:236]: PROPERTY CHANGE: Adding CRYPTUNPROTECT_RESULT property. Its value is 'password'.
Action ended 17:52:49: DecryptPassword. Return value 1.
MSI (c) (CC:A8) [17:52:49:238]: Doing action: SetDBUSERsDecryptedPASSWORD
Action 17:52:49: SetDBUSERsDecryptedPASSWORD.
Action start 17:52:49: SetDBUSERsDecryptedPASSWORD.
MSI (c) (CC:A8) [17:52:49:239]: PROPERTY CHANGE: Adding DB_PASSWORD property. Its value is '**********'.
Action ended 17:52:49: SetDBUSERsDecryptedPASSWORD. Return value 1.
My problem is that SetCRYPTUNPROTECT_DATA is not executing during ExecuteSequence, so DecryptPassword fails:
Action ended 17:53:47: AppSearch. Return value 1.
MSI (s) (28:1C) [17:53:47:206]: Doing action: DecryptPassword
Action 17:53:47: DecryptPassword.
Action start 17:53:47: DecryptPassword.
MSI (s) (28:98) [17:53:47:217]: Invoking remote custom action. DLL: C:\Windows\Installer\MSI9020.tmp, Entrypoint: CryptUnprotectDataHex
CryptUnprotectDataHex: MSI Extensions 1.4.1114.0
CryptUnprotectDataHex: [CryptUnprotectDataHex] std::exception: 0x80070057 - Error in CryptUnprotectData: The parameter is incorrect.
MSI (s) (28!DC) [17:53:47:237]: PROPERTY CHANGE: Adding CA_ERROR property. Its value is '0x80070057 - Error in CryptUnprotectData: The parameter is incorrect.'.
CustomAction DecryptPassword returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)
Action ended 17:53:47: DecryptPassword. Return value 3.
Action ended 17:53:47: INSTALL. Return value 3.
Can you point out how to schedule our custom actions to get this to decrypt properly?

Answer 1

Your problem is likely due to one of the following:
The <SetProperty> element that sets the CRYPTUNPROTECT_DATA property is being conditionally run based on the value of DB_PASSWORD not being set, but we can't see from the logs that this property is indeed not set. The property action won't run if this evaluates to false.
The default setting for the Sequence attribute on the <SetProperty> element is not working correctly. Try setting the Sequence attribute to first, which will schedule it to be run in the first of either InstallUISequence or InstallExecuteSequence (will be first in silent mode). If that doesn't work, try setting it to both explicitly.
Try changing the Before attribute on the custom action for DecryptPassword in the InstallExecuteSequence to After="InstallInitialize", which is much earlier on in the install sequence than InstallFiles. This will get you behavior that is similar to running with the UI Sequence, which computes and modifies your properties and runs your actions prior to anything actually being installed.
See this page for suggested sequencing (and see relative ordering of events)