Leveraging heat.exe and harvest already localized file names and including them to msi using wix - wix

Question

I have a question whether what i am trying to do is doable, and if the answer is yes how to do it.
I am new to the wix and have been doing some reading on how dynamically to include a folder to an installer and eventually i were able to do a task in nant that uses heat.exe to generate wxs file and latter adding newly generated wxs file to light and candle tasks. This allowed me to add the content of a folder to the msi and subsequently have that folder and its content to be installed.
My problem starts at the point where the folder that i am adding to the msi contains files that has their names already localized (this is a requirement).
When i am adding a file to the directory structure that has its name in Russian for example which is not 1252 codepage i am getting the error:
[exec] ......Templates.wxs(65) : error LGHT0311 : A string was
provided with characters that are not available in the specified
database code page '1252'. Either change these characters to ones that
exist in the database's code page, or update the database's code page
by modifying one of the following attributes: Product/#Codepage,
Module/#Codepage, Patch/#Codepage, PatchCreation/#Codepage, or
WixLocalization/#Codepage.
I tried to set Product/#Codepage to 65001 (UTF-8) however that did not solve the problem.
Eventually what i want to do is to have an ability to add a folder and its content to installer and someone else latter add any number of files that has their names localized into that folder. This way whenever the build runs and subsequent creation of msi happens, msi would contain that folder and its content.
Thank you very much in advance.

Answer 1

This is what WiX.chm says about setting the code page of the MSI database:
You can set this to a valid Windows code page by integer like 1252, or
by web name like Windows-1252. UTF-7 and UTF-8 are not officially
supported because of user interface issues. Unicode is not supported.
As long as you are going to have files named in different languages, that is, File table won't fit into a single Windows code page, you have very little choice. UTF-8 is said to be not officially supported, and this leaves a place for a hope.
If you set the CodePage attribute of the Product element to UTF-8, it will build successfully. And you can install/uninstall the resulting MSI with no problem. I have played a bit with it, and didn't face with any "interface issues" mentioned in that warning above.
Furthermore, I've googled the topic a bit, and found out that InstallShield allows setting the MSI database code page to UTF-8, which is reflected in their docs (search for 'utf-8' on that page). They have more to say about the potential interface issues:
However, some scenarios result in user interface issues. For example,
if an end user specifies the /qb command-line option or uninstalls the
product from Add or Remove Programs, Windows Installer uses very small
fonts to display the user interface text in a UTF-8 database.
They also want to stay on the safe side, hence this setting is false by default (no UTF-8, just ASCII).
So, finally, what would I do in your situation?
if that's a strict requirement to the installation package, use UTF-8 as code page
test all possible scenarios (install / uninstall / repair / upgrade / etc.) on all possible combinations related to localization (English OS, non-English OS, various combinations of current culture and culture UI)
if you face with those ghost "interface issues", show those to your stakeholders, decide whether this is what you can live with and publish known issues if you do
otherwise, recycle this idea and just thank your life for an opportunity to level-up your skills in this area :)
Hope this helps.