Blackduck Synopsys Yarn Detector cannot find project version name - gitlab-ci

I'm using Blackduck version 5.6.2 on a Create-React-App application with dependencies installed using yarn v1.22.11.
Blackduck executes as a job in a GitLab CI pipeline. Previously, I used npm to install the packages in the blackduck step before running the scan. Blackduck scanner was able to pick up the project name and version number without any explicit configuration.
I've recently changed from NPM to yarn to take advantage of selective dependency resolution feature in attempts to overcome nested dependencies reported by Blackduck scanner. Once I converted the Blackduck step to use yarn, Blackduck picked up on yarn.lock and package.json files in my repository to use Yarn detector this was seen from the outputs (and the expected behaviour according to
Synopsys' documentation). However, it is no longer able to pick up the project version name correctly.
Here are the messages that indicate this:
2021-09-27 19:13:47 INFO [main] --- Determining project info.
2021-09-27 19:13:47 INFO [main] --- A project name could not be decided. Using the name of the source path.
2021-09-27 19:13:47 INFO [main] --- A project version name could not be decided. Using the default version text.
Once the test completes and I visit the blackduck web console for test results, I now see "Default Detect Version" as the version name instead of the version number that I had before.
I've tried passing -detect.project.version.notes=./package.json argument into the step that starts the blackduck scanner shell script, but it's still falling back onto "Default Detect Version" as the version name.
The docs for yarn detector stops after specifying arguments to include and exclude workspaces, the path to yarn binary, and if production dependencies should be included.
Does anyone have any idea why the Yarn detector is unable to pick up the version number from package.json and how I can address this issue without passing a hardcoded value as the argument to the scanner shell script?

I was able to solve both the project name and version issue as well as blackduck scanner reporting vulnerabilities within nested dependency by performing the following
Switch from using yarn to npm for package installation.
This allowed the NPM detector to utilize package.json and package-lock.json to automatically determine the project name and version.
Add the following flags to blackduck scanner when starting the shell script.
--detect.npm.include.dev.dependencies=false
--detect.npm.include.peer.dependencies=false
Example:
bash <(curl -s https://detect.synopsys.com/detect.sh) --blackduck.url="${BLACKDUCK_HUB_URL}" --blackduck.username="${BLACKDUCK_HUB_USER}" --blackduck.password="${BLACKDUCK_HUB_PASS}" --blackduck.trust.cert=true --detect.policy.check=true --detect.hub.signature.scanner.paths=./ --detect.npm.include.dev.dependencies=false --detect.npm.include.peer.dependencies=false
This will instruct blackduck scanner to not look at dev dependency and nested dependencies.
Additional notes
It is not necessary to use npm-forced-resolutions or yarn's selective dependency resolution; For future reference, npm's "override" function was not mentioned because it has not been implemented yet at the time this answer was written.
If you are using create-react-app, make sure react-scripts is moved from dependencies to devDependencies (as per Dan Abramov's recommendation to address this issue). You can do this by running npm uninstall react-scripts, then running npm install --save-dev react-scripts or npm install --save-dev react-scripts#<specific version>

first off let me point you to the Synopsys Community portal which is the best and quickest place to ask these questions and get answers https://community.synopsys.com/s/. Full disclose, I work at Synopsys and am involved with the Black Duck product.
Can I also ask if the version "5.6.2" you listed refers to the Synopsys Detect scanning tool or Black Duck itself? I ask as the latest version of Black Duck 2021.8.2 and detect 7.5.0 does support extracting the project version name from the yarn.lock and/or package.json, and you should not need to provide any additional parameters to have detect automatically pick this up.
One other thing to note is that generally you should be running the yarn command on your project prior to running detect. This produces more accurate analysis results within Black Duck and resolves the dependencies using your normal package manager tools. Which would mean detect will extract the project name and version information from the yarn.lock file not the package.json file. For more details, you can see this page here https://synopsys.atlassian.net/wiki/spaces/INTDOCS/pages/631275892/Detectors

Related

How to handle package dependencies for some build targets when building with Copr?

I want to have a rpm package build with Copr1. My current build target list is Fedora 35, 36, rawhide and Centos 7 and Stream 8. I have not yet created the copr project.
Compiling on one of my machines, the package builds successfully on the Fedora variants with mock. The problem is that on Centos variants one of the build dependencies and some of its dependencies are not available. I have found appropriate srpm files and compiled them on one of my machines with the Centos Stream 8 (one of them required two custom patches). With those custom dependencies I am able to successfully compile the original package.
So just to be clear, the problem is that the spec file contains for example
BuildRequires: libsomething
where libsomething is available as a plain upstream package in some of the build targets while needs an additional custom repo for some other build targets.
The FAQ says the following about dependencies:
Can I depend on other packages, which are not in Fedora/EPEL?
Yes, they just need to be available in some yum repository. It can either be another Copr repo or a third-party yum repo (e.g jpackage). Click on “Edit” in your project and add the appropriate repositories into the “Repos” field. Packages from your project are available to be used at build time as well, but only for the project you are currently building and not from your other projects.
But this sounds like an all or nothing approach, and I absolutely do not want to override the already existing upstream packages, only provide them when they are missing.
So what strategy do people use to handle this?
Update: I have now created copr projects and made some attempts at building (after resolving dependencies of dependencies in several levels), but the problem is as I describe above. If I add copr://hlovdal/projectname as a build dependency then epel-8-x86_64 compiles fine because it is provided with the missing dependencies while fedora-35-x86_64 fails because the repository does not have any fedora packages. If I remove the repo epel fails while fedora succeeds.
I also attempted to add the base url from the corresponding /etc/yum.d.repo file, and only hardcode epel instead of $distname hoping that the fedora builds would just ignore non-existing/wrong repo setting, but the build does not like that and still fails.
1 Copr is Fedora's freely available build system.

NPM5, What is the difference of package-lock.json with package.json?

After updating NPM to version 5, I found package-lock.json file with package.json.
What is the difference between this two files?
What are the advantages of package-lock.json?
A package.json file: lists the packages that your project depends on. allows you to specify the versions of a package that your project can use using semantic versioning rules.
According to the npm docs,
package-lock.json is automatically generated for any operations where npm modifies either the node_modules tree, or package.json . It describes the exact tree that was generated, such that subsequent installs are able to generate identical trees, regardless of intermediate dependency updates.
This file is intended to be committed into source repositories, and serves various purposes:
Describe a single representation of a dependency tree such that teammates, deployments, and continuous integration are guaranteed to install exactly the same dependencies.
Provide a facility for users to "time-travel" to previous states of node_modules without having to commit the directory itself.
To facilitate greater visibility of tree changes through readable source control diffs.
Basically package-lock.json is used to optimize the installation process by allowing npm to skip repeated metadata resolutions for previously-installed packages.
Before npm 5.x.x, package.json was the source of truth for a project. What lived in package.json was law. npm users liked this model and grew very accustomed to maintaining their package file. However, when package-lock was first introduced, it acted contrary to how many people expected it to. Given a pre-existing package and package-lock, a change to the package.json (what many users considered the source of truth) was not reflected in the package-lock.
Example: Package A, version 1.0.0 is in the package and package-lock. In package.json, A is manually edited to version 1.1.0. If a user who considers package.json to be the source of truth runs npm install, they would expect version 1.1.0 to be installed. However, version 1.0.0 is installed, despite the fact that v1.1.0 is listed is the package.json.
Example: A module does not exist in the package-lock, but it does exist in the package.json. As a user who looks to package.json as the source of truth, I would expect for my module to be installed. However since the module is not present in package-lock, it isn’t installed, and my code fails because it cannot find the module.
Read more about package-lock.json in the Official npm Documentation!
package.json records only your direct dependencies and their versions.
package-lock.json records not only your direct dependencies and exact versions, but also those of all dependencies of your dependencies - the entire dependency tree, in other words, with exact versions.
It's the fact that package-lock.json records the exact versions of all dependencies of a project, including sub-dependencies, that ensures that builds will be identical each time. (This is why npm ci bases its build on package-lock.json, not package.json.)
Builds based on package.json (as with npm i) cannot guarantee that all sub-dependencies are the exact same versions each build (e.g., if the subdependency of one of your dependencies releases an update, but the version of your direct dependency doesn't change), even if exact version numbers for direct dependencies are specified in the package.json.

Why doesn't Yarn install all executables in .bin folder?

I've just started using the Yarn package manager and I downloaded a starter Ionic 2 project.
In this project, we have a lot of help from scripts that compile, minify, lint and bundle our code. All of this is provided by ionic-app-scripts, which has several dependencies which it uses to run commands.
The problem is when I use Yarn to install, the node_modules/.bin/ folder does not contain all the necessary executables, like tslint, which is a dependency of ionic-app-scripts, so it is not directly in my package.json.
The result is that when I use Yarn, ionic-app-scripts doesn't work because it expects that the .bin folder contains a tslint executable!
What can I do? Are the ionic-app-scripts's definitions a problem?
[note]: npm install works, but Yarn is much faster!
This is a known issue, and there's a pull request with more information.
In short, if you want to fix this now, you'll have to explicitly include the packages from which you need binaries in your dependencies.
I had this issue but a different solution.
The solution was from this ticket https://github.com/yarnpkg/yarn/issues/992#issuecomment-318996260
... my workaround is going to file manager, right click on /node_modules main folder, selecting properties, and check-uncheck "read-only". You can do it also using attrib in command line. Then you retry installation and it works.

When to use shrinkwrap, npm-lockdown, or npm-seal

I'm coming from a background much more familiar with composer. I'm getting gulp (etc) going for the build processes and learning node and how to use npm as I go.
It's very odd (again, coming from a composer background) that a composer.lock-like manifest is not included by default. Having said that, I've been reading documentation on [shrinkwrap], [npm-lockdown], and [npm-seal]. ...and the more documentation I read, the more confused I become as to which I should be choosing (everyone thinks their way is the best way). One of the issues I notice is that npm-seal hasn't changed in 4 years and npm-lockdown in 8 months -- this all leads me to wonder if this because it's not needed with the newest version of npm...
What are the benefits / drawbacks of each?
In what cases would I use one over another in Project A, but use a different one in Project B?
How will each impact our development workflow?
PS: Brownie points if you include the most basic implementation example for each. ;)
npm shrinkwrap is the most standard way how to lock your dependencies. And yes, npm install does not create it by default which is a pity and it is something that npm creators definitely should change.
npm-lockdown is trying to do the same things as npm shrinkwrap, there are two minor points in which npm-lockdown is better: it handles optional dependencies better and it validates checksums of the package:
https://www.npmjs.com/package/lockdown#related-tools
Both these features seem not so relevant for me; I'm quite happy with npm shrinkwrap: For example, npmjs guarantees that once you upload certain package at certain version, it stays immutable - so checking sha checksums is not so hot (I've never encountered an error caused by this).
seal is meant to be used together with npm shrinkwrap. It adds the 'checksum checking' aspect. It looks abandoned and quite raw.
Good question - I'm going to skip everything but shrinkwrap because it is the de-facto way to do this, per NPM's docs.
Long story short the npm-shrinkwrap.json file is akin to your lock files you are used to in every other package manager, though NPM allows different versions of the same package to play nice together by isolation - literally scoping and copying different entire versions to node_modules at different levels of the tree. If two projects that are parent-child to each other use the exact same version, NPM will copy the version to only the parent and the child will traverse up the tree to find the package.
Best practice is simply to update package.json for your direct dependencies, run npm install, verify that things are working while developing, then run npm shrinkwrap when you are just about to commit and push. NOTE: make sure to rm npm-shrinkwrap.json before running npm install during active development - if your direct dependencies have changed, you want package.json to be used, and not the lock! Also include node_modules in your .gitignore or equivalent in your source control system. Then, when you are deploying and getting to run the project, run npm install like normal. If npm finds an npm-shrinkwrap.json file, it will use that to recursively pull all locked modules, and it will ignore package.json in both your project and all dependent projects.
You might find shrinkpack useful – it checks in the tarballs which npm install downloads and bundles them into your repository, before finally rewriting npm-shrinkwrap.json to point at that local bundle instead.
This way, your project is totally locked down, completely available offline, and much quicker to install.

Renaming a published NPM module

Is there any way to rename an NPM module that has already been published? I want to change the name of my module to more accurately match the API it exposes but would not like to leave people who have already installed it in the lurch.
There isn't any exposed way to do that. When I've encountered this in the past the approach I took was:
npm deprecate %ProjectName%#"<=put-latest-version-here" "WARNING: This project has been renamed to %NewProjectName%. Install using %NewProjectName% instead."
npm Deprecate instructions
In simple words no you can't. But npm provides you a different solution called npm deprecate.
What it does is it marks a particular version or version ranges of that package as deprecated. So next if someone tries to install this package they get a warning package deprecated along with your custom message, in which you can easily specify your new package name.
Usage:
npm deprecate my-package-name#"< latest-version" "your message"
Your message can be any thing like:
WARNING: This project has been renamed to your-new-package-name. Install using new-package-name instead.
In less than 24 hours i ran following command to delete wrong package.
npm unpublish <wrong package name> --force
From the documentation:
Registry data is immutable, meaning once published, a package cannot
change. We do this for reasons of security and stability of the users
who depend on those packages.
However newly published packages - within 72 hours - can be unpublished by running:
npm unpublish <package_name> -f
This will remove the package from the NPM registry if it was published less than 72 hours ago. Then you can change your package's name and publish it again.
Caution: You need to wait 24 hours if you try to republish package
with the same name
Someone has built a handy little npm plugin for doing this easily 😊
https://www.npmjs.com/package/#tiaanduplessis/pkg-rename
Install the package using npm -g install #tiaanduplessis/pkg-rename
Rename your npm module in the package.json file and save it
run pkg-rename old-package-name
From the documentation:
This will get the latest version of the old package from npm and deprecate this and all previous published versions with a message:
WARNING: This project has been renamed to new-package-name. Install using new-package-name instead.
You can also add the --publish flag to publish the new package name as part of the same action.
pkg-rename old-package-name --publish
Remember, rename the package in package.json first, then run the pkg-rename command.
I once was in this situation. I published a package with the name bowser-or-node instead of browser-or-node.
There's no way to rename a package, you have to deprecate and publish a new package.
Although there's one other option. If you just published your package (less than 24 hours from time of publish) and if you're sure you're okay with deleting the package and publish a new one with the right name, you can go ahead and do it. But NPM won't allow you to delete the package once it's been 24 hours since the time of publish.
Fortunately I figured out that I published with the wrong name in less than 20 minutes. So I just deleted and published again with a new name.
Something marvelous just happened to me: I managed to rename a package. It was originally known as stdout-renderer, but I changed every possible occurence of the name, and republished it after having deprecated the original and voila it shows up under its new name (cli-artist) undeprecated in the newly updated list. I'm not sure which field to change, but I would imagine it be in package.json because that's the only one where the casing matched in my case.
hope that helps!