npm install using node-gyp works from CLI, fails from TeamCity - npm

Question

I'm stuck with an unreliable build in TeamCity.
When executing an npm install for my project that indirectly uses node-gyp to compile a solution, crucially it only fails with when executed from the TeamCity agent.
[EDIT AFTER ANSWER] The context of execution is from Build Step that executes a custom build.ps1 which automates the build process - covering c#, npm projects, executes unit tests etc. This is to say it isn't a bunch of discrete steps in TeamCity but a single PowerShell command that can execute from any environment.
If I execute from the command line on the same server, same user that the TeamCity Agent runs under, it works fine.
These are the errors from the build agent...
MSBuild: C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\MSBuild.exe
Executing Build
configuration: debug
Restoring npm for src\my-corp-app
> scrypt#6.0.3 preinstall D:\IR\my-corp-app\TcBuild\src\my-corp-app\node_modules\scrypt
> node node-scrypt-preinstall.js
> keccak#1.4.0 install D:\IR\my-corp-app\TcBuild\src\my-corp-app\node_modules\keccak
> npm run rebuild || echo "Keccak bindings compilation fail. Pure JS implementation will be used."
> keccak#1.4.0 rebuild D:\IR\my-corp-app\TcBuild\src\my-corp-app\node_modules\keccak
> node-gyp rebuild
D:\IR\my-corp-app\TcBuild\src\my-corp-app\node_modules\keccak>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild ) else (node "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" rebuild )
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
D:\IR\my-corp-app\TcBuild\src\my-corp-app\node_modules\keccak\build\keccak.vcxproj(21,3): error MSB4019: The imported project "D:\Microsoft.Cpp.Default.props" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.
gyp ERR! build error
gyp ERR! stack Error: `msbuild` failed with exit code: 1
gyp ERR! stack at ChildProcess.onExit (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\build.js:262:23)
gyp ERR! stack at emitTwo (events.js:126:13)
gyp ERR! stack at ChildProcess.emit (events.js:214:7)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
gyp ERR! System Windows_NT 10.0.14393
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd D:\IR\my-corp-app\TcBuild\src\my-corp-app\node_modules\keccak
gyp ERR! node -v v8.12.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! keccak#1.4.0 rebuild: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the keccak#1.4.0 rebuild script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?
npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\teamcity_user\AppData\Roaming\npm-cache\_logs\2019-02-13T11_38_48_872Z-debug.log
"Keccak bindings compilation fail. Pure JS implementation will be used."
The work around is to execute the command line, it succeeds, then execute in TeamCity again and the recent artifacts from the command line execution make it skip this node-gyp process and the build succeeds. This workaround is getting tiresome.
I've unsucessfully followed the various googled suggestions for fixing "C:\Microsoft.Cpp.Default.props" was not found errors:
npm install --global --production windows-build-tools
Set / Clear VCTargetsPath
Have also compared the environment variables between my CLI prompt and TeamCity Agent when executing the build but the only differences are the expected additions for TeamCity JRE and build numbers.
Am currently stuck trying to determine what magic makes the install work from an interactive CLI as opposed to TeamCity agent running as a Windows Service
The complete log referred to in the above log has details like
0 info it worked if it ends with ok
1 verbose cli [ 'C:\\Program Files\\nodejs\\node.exe',
1 verbose cli 'C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js',
1 verbose cli 'run',
1 verbose cli 'rebuild' ]
2 info using npm#6.4.1
3 info using node#v8.12.0
4 verbose run-script [ 'prerebuild', 'rebuild', 'postrebuild' ]
5 info lifecycle keccak#1.4.0~prerebuild: keccak#1.4.0
6 info lifecycle keccak#1.4.0~rebuild: keccak#1.4.0
7 verbose lifecycle keccak#1.4.0~rebuild: unsafe-perm in lifecycle true
8 verbose lifecycle keccak#1.4.0~rebuild: PATH: C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin;D:\IR\my-corp-app\TcBuild\src\my-corp-app-node\node_modules\keccak\node_modules\.bin;D:\IR\my-corp-app\TcBuild\src\my-corp-app-node\node_modules\.bin;C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin;D:\IR\my-corp-app\TcBuild\src\my-corp-app-node\node_modules\keccak\node_modules\.bin;D:\IR\my-corp-app\TcBuild\src\my-corp-app-node\node_modules\.bin;C:\Windows\Microsoft.NET\Framework64\v4.0.30319\;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Amazon\cfn-bootstrap\;C:\ProgramData\chocolatey\bin;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\130\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\140\Tools\Binn\;C:\Program Files\Microsoft SQL Server\140\Tools\Binn\;C:\Program Files\Microsoft SQL Server\140\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\Client SDK\ODBC\130\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\140\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\140\Tools\Binn\ManagementStudio\;C:\Program Files\dotnet\;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\110\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\120\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\130\DTS\Binn\;D:\apps\jdbc;C:\Program Files\Git\cmd;D:\apps\OpenSSL-Win64\bin;C:\Program Files\nodejs\;C:\Users\teamcity_user\AppData\Local\Microsoft\WindowsApps;C:\Users\teamcity_user\AppData\Roaming\npm;C:\Users\teamcity_user\.dotnet\tools
9 verbose lifecycle keccak#1.4.0~rebuild: CWD: D:\IR\my-corp-app\TcBuild\src\my-corp-app-node\node_modules\keccak
10 silly lifecycle keccak#1.4.0~rebuild: Args: [ '/d /s /c', 'node-gyp rebuild' ]
11 silly lifecycle keccak#1.4.0~rebuild: Returned: code: 1 signal: null
12 info lifecycle keccak#1.4.0~rebuild: Failed to exec rebuild script
13 verbose stack Error: keccak#1.4.0 rebuild: `node-gyp rebuild`
13 verbose stack Exit status 1
13 verbose stack at EventEmitter.<anonymous> (C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\index.js:301:16)
13 verbose stack at emitTwo (events.js:126:13)
13 verbose stack at EventEmitter.emit (events.js:214:7)
13 verbose stack at ChildProcess.<anonymous> (C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\lib\spawn.js:55:14)
13 verbose stack at emitTwo (events.js:126:13)
13 verbose stack at ChildProcess.emit (events.js:214:7)
13 verbose stack at maybeClose (internal/child_process.js:915:16)
13 verbose stack at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)
14 verbose pkgid keccak#1.4.0
15 verbose cwd D:\IR\my-corp-app\TcBuild\src\my-corp-app-node\node_modules\keccak
16 verbose Windows_NT 10.0.14393
17 verbose argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "run" "rebuild"
18 verbose node v8.12.0
19 verbose npm v6.4.1
20 error code ELIFECYCLE
21 error errno 1
22 error keccak#1.4.0 rebuild: `node-gyp rebuild`
22 error Exit status 1
23 error Failed at the keccak#1.4.0 rebuild script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]

Answer 1

Finally solved this by adding an inserting extra (and redundant) PowerShell build step:
Runner Type: PowerShell
Script: SourceCode
Working directory: <the obvious folder>
Script Source: npm install
and it magically worked.
Now, of course I already was executing npm install from deep down inside my build automation build.ps1.
For some reason, the above works and the buried one doesn't. I can't explain why. Confirmed that the execution folder was correct and also used get-command npm from both contexts and the result was the same for both:
CommandType Name Version Source
----------- ---- ------- ------
Application npm.cmd 0.0.0.0 C:\Program Files\nodejs\npm.cmd