When I build my Electron app for production I still get a node_modules folder with the dependencies. The folder is constituted by:
The dependencies which are installed via package.json I already noticed that I can just delete them from the folder (since their code is inside webpack bundle.js )
ffprobe-static, which actually occupies the largest amount with 40Mb
The nodejs modules such as ajv,deferential, debug, decamelize, etc (158 folders total, while I don't even know most of them, let alone use them directly)
Regarding 2: Is it mandatory to have the binary for ffprobe-static? Can I use ffprobe-static with the ffmpeg.dll given alongside Electron binary?
Regarding 3: Why do I need these and how can I get rid of them? Also, Electron binary already comes with an 18.9Mb node.dll file. Again, can't I use this instead of having again the node_modules?
Related
NPM donwloads a lot of files needed for the webpack/libraries. From what i understand, webpack generates a one single bundle file, that contains all code for script working. After that, when i finish building my app, do i need to keep all those jquery/react files and webpack itself? Or should i just delete them?
It's common practice to make a project portable/shareable by following these steps;
Create a package.json and ensure to capture all dependencies,devDependencies and/or peerDependencies.
Add/commit this package.json and package-lock.json files to your version control
Create a .gitignore file and add node_modules to it (in essence, this cuts out that baggage)
For production purpose (e.g. to be shared with client finished product), build the project (which often results into a small files, often within /build or dist). And then you can always push that build file to AWS or Heroku or the clients' servers.
What does the above help you achieve?
You can easily start the project using any machine, as long as you run npm install which reads from your package.json.
I am using VS 2019 for Core 3.1 development and I installed Font Awesome whith Yarn:
yarn add #fortawesome/fontawesome-free
However, whenI try to reference it in my HEAD section like this:
<script defer src="~/lib/#fortawesome/fontawesome-free/js/all.js"></script>
I get the following error:
the 'fortawesome' doesnt exist in actual context
Package managers like yarn, npm, etc. just add the packages to your project. They generally aren't ready to deploy directly at that point, but rather require a build pipeline to create the actual JS/CSS resources. The #fortawesome/fontawesome repo is an exception in that 1) it's not actually a package and 2) the files are already "built". Even then, though, they still won't be in the right location.
I'm not overly familiar with yarn, but npm, for example, puts everything in a node_modules directory. That directory is not served by default, and should not be served, because it contains "raw" stuff. You'd need a separate build pipeline (using npm scripts, webpack, gulp, etc.) to build/copy assets into a directory that is served (i.e. wwwroot). That is likely the piece you are missing.
For this, since there's no build actually required, you just need to have the assets copied to something like wwwroot/lib and then you'll be able to reference it via the script tag on the page. That can be done with any tool like npm, webpack, gulp, grunt, etc. so you'll just have to research and pick the one that fits your needs best.
For large web apps npm install resp. yarn install does take a lot of time, mostly in a step called Linking Dependencies. What is happening here? Is it fetching the dependencies of the dependencies? Or something completely different? Which files are created during this step?
When you call yarn install, the following things happen in order:
Resolution: Yarn starts resolving dependencies by making requests to the registry and recursively looking up each dependency.
Downloading/Fetching: Next, Yarn looks in a global cache directory to see if the package needed has already been downloaded. If it hasn't, Yarn fetches the tarball for the package and places it in the global cache so it can work offline and won't need to download dependencies more than once. Dependencies can also be placed in source control as tarballs for full offline installs.
Linking: Finally, Yarn links everything together by copying all the files needed from the global cache into the local node_modules directory after identifying what's already there and what's not there.
yarn install does take a lot of time, mostly in a step called Linking Dependencies
You should notice that Step 3: Linking is taking more time than Step 1: Resolution and Step 2: Fetching where the actual download happens. During by this step we already have things that we need ready and downloaded, then why is it taking long, did we miss anything?
Yes, COPY to local project into node_modules folder...! The reason for this is that this copy is not equivalent to copying one large 4.7GB ISO file. Instead it's multiple super small files (Don't take it light when I say multiple, it can be 15k+ files :P ), hence take a lot of time to copy. (Also, it is important to note that when you download the packages, you download one large tar file per package, whose contents should then be extracted into the cache which also takes time)
It is slower due to
Anti-virus: Your antivirus is sitting in the middle and doing a quick inspect (in addition to our yarn checking if it already exists) on every single file yarn is trying to copy cutting its speed by so much. If you are on Windows, try adding your project's parent folder as exception to Windows Defender.
Storage medium's transfer rate: SSDs can improve this speed hugely (Sorry, SSHDs and FireCudas will not help either, this is gonna be one time).
But is this efficient? Can I have it taken from the global node_modules (after creating one)?
Nope for both questions. Because of the way node works each package finds its dependencies only relative to its own location. Also because each project may want to use different versions of the same package to ensure its working properly and not broken by package updates.
Ideally, the project folder should be lean. An efficient way of doing this would be to have a global node_modules folder. Any and all requested packages are downloaded if not already present AND used from this location. Actually Ruby does it this way. Here's my global Ruby's equivalent of node_modules folder. Notice the presence of different versions of the same package for use in different projects.
But keep in mind that it would reduce project portability. It's a trade-off that any manager (be it rubygems or node modules) has to make. I can just copy the node project folder (which in fact may take hours because you will be copying the (local) node_modules folder as well, but I can expect it to work if I have just that project folder, as opposed to copying a ruby project would only some seconds to few minutes, as there is no local packages (or gems as they call them) folder, but running the project on different system would require those packages to be present on the global gems folder.
The documentation for yarn install can be found here.
You can use the command
yarn install --verbose
Show additional logs while installing dependencies
The output will show what the yarn/npm install is doing.
It's good for debugging in case the process is failing or taking a long time.
The linking phase works in essentially 3 big steps:
Find every file that need to be in node_modules
Check this list versus what is already there and find what need to be copied around
from cache to node_modules
Do the copy
Maybe this issue on Github will help you out.
https://github.com/yarnpkg/yarn/issues/1496
I am new, and when i first created my app based on the documentation using npm create-react-app i found it there were a lot of package included inside folder node_module when i code and i only use react and react DOM and etc from the basic.
node_modules
acorn
timer
ansi
and many more
I wonder if anyone can help my how to understand each use inside the node_module or where can i find the documentation for each use?
or how can i just reduce to what i want to use only to decrease the app size?
The answers are 2:
because you're using an automated scaffolding tool, which essentially does everything for you, and, you have just to code, it is supposed to locally deploy all the packages it needs to work (for example webpack is needed to bundle your code, babel to transpile it, ...
under node_modules you will find all the packages of the whole app. That's means you will find both your dependencies and the dependencies of your dependencies (this rule has some exceptions and you can find them in the npm documentation.
example:
// your code depends on A
var dependency = require('A');
// but then, inside your A dependency you can also find something similar to:
var b = require('B');
how can i just reduce to what i want to use only to decrease the app size?
You basically can't do it. They are all needed.
Most of the libraries that we pull from npm have dependencies. You may use only react and react-dom but there are react-scripts that require lots of stuff. I don't think that you have to worry about the size of the node_modules. That's not what you are suppose to ship in production.
If you want to see what are these all modules about you may open their folder and fine README.md file.
When I download a package in bower or npm, many irrelevant/not-useful files also
get included especially in npm packages. I generally only include .css or .js files of that package for e.g. in bootstrap e.t.c. But for package like polymer or Anjular.js or Electron we need multiple other file too. So when I deploy my project should I include the complete folder or just copy the required files separately or is there any other approach ?
When you deploy to your production server, a good approach would be to only include the files you are actually using. You should also create an optimized file to serve to the client.
To achieve that, you normally have a deploy script that generates one or more minified .css or .js files (for example, vendor.js for libraries and bundle.js for your code).
This script can be build with tools like grunt, gulp or you can use a module bundler like Webpack or Jspm.
You can create your own Grunt or Gulp script like the ones in this website.
When using a module bundler and modern javascript to organize your code, you'll build, for example, an entry javascript file (bundle.js) that will look like
import {MyLibrary} from 'my-library';
import {AnOtherLibrary} from 'an-other-library';
console.log(MyLibrary.saySomething());
In this case, Webpack handles the node_modules imports and the creation / minification / concatenation of the production version of .js files.
As you said, there are multiple files to be included (js, css or even fonts) so configuring your scripts may take some time. Try finding a Yeoman generator that uses your frameworks, and use the deploy scripts others already created for this purpose. This one uses AngularJS, Bower and Grunt. This one uses Webpack and ReactJS.