Running npm inside AWS Lambda - npm

I am trying to run npm install from inside AWS Lambda.
But I'm getting the below error.
Setting --prefix to "/tmp" doesn't work either.
{ Error: Command failed: npm install async npm ERR! code EROFS npm
ERR! syscall mkdir npm ERR! path /home/sbx_user1051 npm ERR! errno -30
npm ERR! rofs EROFS: read-only file system, mkdir '/home/sbx_userXXXX'
npm ERR! rofs Often virtualized file systems, or other file systems
npm ERR! rofs that don't support symlinks, give this error.

You cannot run npm install inside lambda, you need to upload your modules using zip file
A deployment package is a ZIP archive that contains your function code and dependencies. You need to create a deployment package if you use the Lambda API to manage functions, or if you need to include libraries and dependencies other than the AWS SDK. You can upload the package directly to Lambda, or you can use an Amazon S3 bucket, and then upload it to Lambda. If the deployment package is larger than 50 MB, you must use Amazon S3.
https://docs.aws.amazon.com/lambda/latest/dg/nodejs-create-deployment-pkg.html

I was able to work around this same npm issue by creating a .zip, then an AWS Layer, then finally configuring the Lambda function to use that Layer; specific steps below:
make a new empty directory:
cd newdir && cd newdir
install whatever npm things: npm install --save xyz
make a directory skeleton that matches the expected Lambda structure for Node14 (there's a different structure for Node12, or various other languages; see https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html?icmpid=docs_lambda_help): mkdir -p nodejs/node14
copy the "node_modules" directory into that newly made directory skeleton:
cp -R node_modules nodejs/node14
zip the whole thing up (name it whatever you want):
zip -r custom-drivers-node14.zip nodejs
from there, go to AWS console, Lambda, then "Layers" and create a new layer. In the dialog, upload your .zip file ("custom-drivers-node14.zip").
finally, edit your Lambda function in AWS console, and add a new Layer – the interface might change, but as of now, this is under the main screen for a single function, then scroll way down to the bottom. Follow the "Add a layer" flow, choose the Layer you made, and then try your code.
One final note, this code structure worked:
const xyz = require('xyz');
exports.handler = async (event) => {
xyz.doSomething();
}

Related

Replicating `npm pack` behavior

I am trying to replicate the behavior of npm pack because it has a limitation where it does not write to stdout, it can only write to a local file (see this issue: https://github.com/npm/npm/issues/12039)
Now, I can tar the current directory and write to stdout like so:
tar --exclude='./node_modules/' -cv . | gzip > archive.tar.gz
however, when you extract npm pack tarballs, all the contents of a your package should be in a directory called 'package'.
So my question is - how can I tarball the current directory, but put the current directory inside a directory called 'package' before it gets archived?
Is there some tar -flag that lets you do that?
I did some legwork and as far as my testing goes, npm will accept a tarball with everything in the root, or everything in a subdirectory called 'package'.
To test the above theory, you can tar a NPM project directory with:
tar --exclude='node_modules' -c . > archive.tar
then install it somewhere else with
npm install /path/to/archive.tar
you can't install in the same project though, NPM will complain about circular deps, so install it in another project.

Correct way to use npm-cache VS Team Services

I'm currently using npm install to install all packages for an Angular2 project CI project, however this step has started to increase the time it takes build agents to build the project. I would like to get build run times down for our CI pipeline by using the npm-cache command, however I keep getting failed attempts at running the command through the build agent.
I have tried using a Command Line step with the following:
This result in the error File name doesn't indicate a full path to a executable file.
I have also tried added an npm step:
Which results in the following:
2017-06-14T13:01:13.2274922Z [command]C:\Program Files\nodejs\npm.cmd cache install
2017-06-14T13:01:19.8609829Z npm ERR! Usage: npm cache add tarball file
2017-06-14T13:01:19.8630626Z npm ERR! npm cache add folder
2017-06-14T13:01:19.8630626Z npm ERR! npm cache add tarball url
2017-06-14T13:01:19.8640981Z npm ERR! npm cache add git url
2017-06-14T13:01:19.8651863Z npm ERR! npm cache add name#version
2017-06-14T13:01:19.8661607Z npm ERR! npm cache ls path
2017-06-14T13:01:19.8672137Z npm ERR! npm cache clean pkg#version
What is the correct way to call npm cache from Team Services?
You need to install npm-cache package first, then call npm-cache command via command line task.
Add npm task (npm command: install; arguments: npm-cache -g)
Add Command Line task (Tool: C:\\Windows\\ServiceProfiles\\NetworkService\\AppData\\Roaming\\npm\\npm-cache; Arguments:install; Working folder: $(Build.SourcesDirectory))
Node: Previous steps uses the private build agent that running as service with NetworkService account. You can add C:\\Windows\\ServiceProfiles\\NetworkService\\AppData\\Roaming\\npm to system environment variable (restart machine is needed) and call npm-cache command directly.
You also can change build agent account to use your account (by default C:\Users\[user name]\AppData\Roaming\npm has added to user environment variable), then call npm-cache command after install npm-cache package with -g argument)
On the other hand, host agent won’t cache the packages, so it is useless if you are using hosted agent.

NPM publish will copy the all files to NPM server or not?

I am successfully publish a package by following this official tutorial:
Does it mean NPM copy all my files to NPM server? Because the command I use is:
npm publish
No other params.
Whereas in Bower, the command I use is:
bower register raphael.backbone git://github.com/tomasAlabes/backbone.raphael.git
It is clearly the source is git repo, so bower is clearly a registry, not copying the project files.
Question:
"NPM publish " will copy my files to NPM server or not?
npm publish compresses your current working directory into a tarball and uploads it to the npm registry; so yes, npm publish will copy your files to npm server.
You actually don't have to use git for an npm package, though using it is a good idea to use version control for any software development.

Getting npm install issues when build angular2 source

I want to build angular2 source on my windows 8.1 x64 dev PC.
Node version 5.1.0
Npm version 3.3.12
1) cloned the repo - OK
2) run bower install - OK
3) run npm install - Errors:
npm ERR! tar.unpack unzip error C:\Windows\Temp\npm-11016-2e7afc52\registry.npmj
s.org\fsevents\-\fsevents-1.0.2.tgz
npm ERR! tar.unpack untar error C:\Windows\Temp\npm-11016-2e7afc52\registry.npmj
s.org\fsevents\-\fsevents-1.0.2.tgz
npm ERR! tar.unpack unzip error C:\Windows\Temp\npm-11016-2e7afc52\registry.npmj
s.org\firefox-profile\-\firefox-profile-0.3.9.tgz
npm ERR! tar.unpack untar error C:\Windows\Temp\npm-11016-2e7afc52\registry.npmj
s.org\firefox-profile\-\firefox-profile-0.3.9.tgz
Do you have any ideas how can I fix that?
I tried to cleanup the cache and also make sure that it is actually did by lookin in to npm cache folder C:\Users\Me\AppData\Roaming\npm-cache. Also I was trying to reclone the repo an run a precess again multipe times but npm install just continue to throw the same exeptions again and again
Use the following process
Create C:\Windows\Temp\ if it does not exist
Create a dummy folder within C:\Windows\Temp\ if it does exist
Run npm config ls -l
Temporary files are stored by default in the folder specified by the tmp config, which defaults to the TMPDIR, TMP, or TEMP environment variables, or /tmp on Unix and c:\windows\temp on Windows.
Temp files are given a unique folder under this root for each run of the program, and are deleted upon successful exit.
Change the tmp var to a new path by editing one of the .npmrc files:
The four relevant files are:
per-project config file (/path/to/my/project/.npmrc)
per-user config file (~/.npmrc)
global config file ($PREFIX/etc/npmrc)
npm builtin config file (/path/to/npm/npmrc)
References
npmrc man page
npm folders: temp files

Download a package from npm as a tar (not installing it to a module)

Is there some URL from which I can download a given package from npm (as a tarball or something)? I need the exact files that were originally uploaded to npm.
Using npm install gets a different, generated package.json for example. I want the exact original set of files that was published.
You can use npm view to get the URL to the registry's tarball (in this example for the module level):
$ npm view level dist.tarball
And to download tarball, you can use npm pack:
$ npm pack level
Just run the command
npm view [package name] dist.tarball
It will return a tar url.
Running npm pack PACKAGE_NAME will download a tarball of any package on npm.
To extract it, just run tar -xzf DOWNLOADED_FILE.tgz
Example:
npm pack react
then extract:
tar -xzf react-16.6.3.tgz
If you need to get the tarball without having npm installed, you can fetch the package information using curl and use jq to get the right information from the JSON:
curl https://registry.npmjs.org/PACKAGE-NAME/ \
| jq '.versions[."dist-tags".latest].dist.tarball'
This is for instance useful if you're building a Docker container that requires one npm package, and don't want to install npm just for that.
Yes, you can npm install <git remote URL> to download the full repository into node_modules. This will be directly from the repository's host, rather than via npm, though. See the npm install docs for more information.