Is there a way to check what packages are installed? I would expect something like dbt list packages?
The context is:
Until I run dbt deps the content of packages.yml gives me nothing. And there are some situations when the models could be triggered without running dbt deps
I would like to check the packages in runtime
I searched over google and dbt --help but I didn't find anything.
No, there is no dbt list packages command; there is no --list option for dbt deps; packages are not listed in dbt debug; installed packages are not listed in manifest.json; nor is there another officially supported analog of pip list to determine all of the currently-installed packages in your runtime.
You should open an issue for this!
Your current options (v1.3.0) are:
Run dbt clean and dbt deps before any other dbt command, to ensure your installed dependencies match what is in your packages.yml file
Run dbt compile and inspect manifest.json to ensure it has the macro(s) and/or model(s) you need from a package (note that dbt compile will fail if packages are listed in packages.yml and not instealled)
Inspect your project's dbt_packages subdirectory. Each installed package will have its own directory; if that package has any dependencies, they will be nested under subsequent dbt_packages directories
dbt clean and dbt deps are pretty fast, so the cost of (1) is low, and you don't have to manage the logic/complexity of various bad states if you go that route. But I think you should absolutely open an issue for this!
You should have a packages.yml file in your repo that contains all the packages that are installed via dbt deps.
Following the dbt deps, a directory called /dbt_modules (by default, can be changed in your dbt_project.yml) will appear in your repo (it is included, also by default, in your .gitignore file to avoid duplicating the source code for the package).
Recently I had the opportunity to work with packages closely and I published an article The Practical Guide to Utilizing DBT Packages for Data Transformation which explains dbt packages and its underhood
Here is the quote:
To verify that a package is installed in your dbt project, you can check the packages.yml file and run the dbt deps command.
Check the packages.yml file: This file lists all of the packages that are installed in your dbt project. Look for the name of the package you want to verify. If it is listed in the packages list, then it is installed.
Run the dbt deps command:
This command will show you a list of all of the packages that are installed in your dbt project. Look for the name of the package you want to verify. If it is listed, then it is installed.
In the root dbt project dir, you observe a new dir dbt_modules/ which contains the compiled packages that are ready to be used. NOTE: dir dbt_modules/ has to be added to .gitignore.
>>> tree -L 1 .
.
├── data
├── dbt_modules
├── dbt_project.yml
├── macros
├── models
├── packages
├── packages.yml
├── profiles.yml
├── snapshots
└── target
If your packages.yml file contains package that is not installed then you would not be able to run any dbt command:
>>> dbt list
Encountered an error:
Compilation Error
dbt found 1 package(s) specified in packages.yml, but only 0 package(s) installed in dbt_modules. Run dbt deps to install package dependencies.
So this is our guarantee that in runtime we would not have any issues related to the package installation.
Related
This is probably a pretty basic question, but I can't find an answer:
If I have a project with a dependency in package.json listed as foobar: ^3.2.1, what version of that dependency will be installed when I run vite build, assuming that the latest available version of the package is 3.4.5?
First thing first, vite build won't change anything to your dependencies. I won't install ones nor update them. It will only build your project (i.e. compile / transpile / minify / bundle etc.) using your source code and the code it imports (likely within the node_modules).
It will build locally, so using your local dependencies in the node_modules folder.
To check the current package version you have installed, you can run:
npm list --depth=0 | grep foobar
(The grep part is optional)
You can also open your package-lock.json or yarn.lock file and search for your package to know to what version your package has been fixed to.
To understand about the semantic version with npm, read this documentation: https://docs.npmjs.com/about-semantic-versioning
I am trying to configure dbt_utils for dbt with github. I have the following in my packages.yml file:
packages:
- package: dbt-labs/dbt_utils
version: 0.8.0
https://github.com/actonmarketing/actondata/blob/main/packages.yml
And I have removed packages from my gitignore file.
When I run dbt deps I get a message saying no packages found in packages.yml.
What am I doing wrong?
packages.yml needs to be at the same hierarchy level as dbt_project.yml
moving packages.yml from project_name/dbt_packages/packages.yml to project_name/packages.yml worked for me.
--
If you want it to live in a subdirectory, see the dbt docs here :
https://docs.getdbt.com/docs/building-a-dbt-project/package-management#project-subdirectories
You might just need to save your packages.yml file
I'm working on creating a local repository that will contain all packages I use in my project, so I can have those packages installed on a machine that does not have access to the internet. I think of the repository that I could clone on the machine and run yarn install to have all the packages available in the project from the local repository. How can I do that? Similar question was asked here Using npm how can I download a package as a zip with all of its dependencies included in the package
There's not enough information in your question to fully understand your situation, but if you commit your node_modules directory to the repository, the modules will be there without the user having to run npm or yarn to install them. This assumes the user will run code from the repo workspace and that there aren't any modules that require a compilation step or other build step that may be platform-specific. But if they're all plain ol' JavaScript modules, you should be fine.
If you want to have all the modules as a separate repo rather than checking in node_modules, I can offhand think of two ways this might work.
Have the packages repo be a check-in of a fully installed node_modules directory. Then make that repo a Git submodule of the main repo that gets cloned as node_modules in the main repo.
Use npm pack to create .tgz files for each package you need. Store those files in the packages repo. Clone that repo into a known path on your target machine. Have the main repo install via path names. For example, if you run npm install /var/packages/foo-1.0.0.tgz, it will add a line to your package.json that might look something like this: "foo": "file:../../../var/packages/foo-1.0.0.tgz". In that case, npm install will install from that path rather than over the network.
I have a custom node module, that I can't publish on NPM.
I'd like to use it as any module I have on NPM.
Is it possible without the awful thing of copying the folder into the node_modules?
The answer is: YES, IT's POSSIBLE.
Let's assume your module has a dist folder, with the built source
(for example I run gulp on my src folder and procude the dist folder).
you simply got to run npm pack ./dist in your library.
this will produce a tgz archive with your library named your-library-version
then you can install your module in your project by simply running
npm i path-to/your-library-version.tgz
And you're done.
Let's say my library fodler is C:\ngx-mat-lib
so my tgz will be in this folder, since the dist folder should be a child of ngx-mat-lib.
In my project I'll run
npm i C:/ngx-mat-lib/ngx-mat-lib-0.0.1.tgz
Note: using forward slashes to avoid doubling them
I am using Yarn v0.19.1 to install some dependencies. I deleted my node_modules folder completely and did a fresh yarn install.
I am trying to install the dependency leaflet using yarn add leaflet. The module installs successfully, except during the Cleaning Modules... phase, Yarn removes the images folder which would typically live within leaflet/dist/images. When I do a npm install leaflet this folder does not get removed.
During a yarn install, the images folder is present until the Cleaning modules phase happens.
Who/what is doing this? Is this something yarn does? Or is this something specified within the leaflet library? How could I resolve this?
I checked the package.json in the leaflet library and nothing seemed out of the ordinary there. It runs a jake file, but even within that file nothing is being deleted related to images.
Here is what the folder looks like, within my node_modules folder, for both package manager installs:
yarn
npm
There was a .yarnclean file in my project. This added some files/folders to ignore when installing dependencies. Running yarn clean added this file and I wasn't aware until I saw this issue. Viewing the docs also gave the same info.
I resolved this by removing images from the set of ignored directories.
Here is my .yarnclean file for an example:
# test directories
__tests__
test
tests
powered-test
# asset directories
docs
doc
website
assets
# examples
example
examples
# code coverage directories
coverage
.nyc_output
# build scripts
Makefile
Gulpfile.js
Gruntfile.js
# configs
.tern-project
.gitattributes
.editorconfig
.*ignore
.eslintrc
.jshintrc
.flowconfig
.documentup.json
.yarn-metadata.json
# misc
*.gz
*.md