How to call DLL functions from InstallScript called by MSI custom action - dll

Question

I have a DLL named FSSetup.dll that I want to call from InstallScript code. And I want to call the InstallScript code from an MSI custom action. But there seems to be a problem in that I don't know where to find FSSetup.dll. The InstallScript function is starting and the whole install aborts when it gets to the line where it calls the DLL function. The MSI log reports:
MSI (c) (1C:3C) [09:39:52:261]: Invoking remote custom action. DLL: C:\Users\bmarty\AppData\Local\Temp\MSIBEAE.tmp, Entrypoint: f1
Action ended 9:39:55: BrowseFSKeyFile. Return value 3.
Info 2896. Executing action BrowseFSKeyFile failed.
Action ended 9:39:55: InstallWelcome. Return value 3.
The InstallScript code looks like this:
export prototype BrowseFSKeyFile(HWND);
prototype FSSetup.FSBrowseFile( BYVAL STRING, BYVAL STRING, BYVAL STRING, BYREF STRING );
function BrowseFSKeyFile(hMSI)
NUMBER nKeyFile;
STRING svKeyFile, szKPath, szKFile, szTemp, szTemp1;
begin
MessageBox("Hi", INFORMATION);
svKeyFile = "";
FSBrowseFile( "*.key", "key", "Please select a KEY File...", svKeyFile );
MessageBox("Bye", INFORMATION);
...
end;
I see the "Hi" message, but the install aborts before "Bye". I suspect it's because FSSetup.dll doesn't exist in my %temp% directory where MSIBEAE.tmp resides, which presumably represents the extracted name of the ISSetup.dll that was generated by the InstallScript compile.
Edit:
The MSI debugger is showing a different value for SUPPORTDIR than the InstallScript is seeing. FSSetup.dll exists in the SUPPORTDIR provided by MSI, but not in the one seen from InstallScript.

Answer 1

You are missing a call to UseDll (and UnUseDll). In order to call UseDll, you will need to know the location of the DLL in question. If you have added it to the support files of the MSI project, you will need to retrieve the Windows Installer Property SUPPORTDIR to find this location - that's where Christopher Painter's answer comes into play - and then call e.g. UseDll(szSupportDir ^ "FSSetup.dll") before calling FSBrowseFile.

Answer 2

The SUPPORTDIR behavior was changed about 9 years ago when fundamental design flaws of InstallScript were fixed in InstallShield 12.0. See:
InstallShield 12 Beta2
and
InstallScript, meet CustomActionData
and
Upgrading Projects to InstallShield 12 (search for SUPPORTDIR)
Basically you need to do call MsiGetProperty() to get SUPPORTDIR. Also you need to pass this to your deferred custom actions as CustomActionData.