Why does "npm search" provide incomplete results? - npm

Question

I've noticed, that using the npm search command does not guarantee complete results. Here is an example:
$npm search jasmine
does not list the jasmine-diff, jasmine-diff-reporter packages, while
$npm search jasmine diff
does.
I've read the doc, and there is no mention of any incompleteness, indeed it states
npm search performs a … search through package metadata for all files in the registry
I think, this somehow implies, that search should be consistent and complete. As one can see the jasmine-diff-reporter package does have the term jasmine in its keywords:
And it doesn't matter, that there is no word jasmine in the description section, since other packages like jasmine-diff have that word and are still missing in the $npm search jasmine output.
So could anyone explain this behavior somehow and/or suggest a workaround (except to use Google or something like that)?

Answer 1

The problem is the new "fast endpoint search" for the "npm search" that was implemented in the https://github.com/npm/npm/commit/e3229324d507fda10ea9e94fd4de8a4ae5025c75. I have registered a bug now: https://github.com/npm/cli/issues/1211.
I investigated npm scripts and found that the old search used the URL https://myNpmServer.com/repository/myNpmRegistry/-/all to get the package information while the new search uses https://myNpmServer.com/repository/myNpmRegistry/-/v1/search?text=%2F.*%2F&size=20. This value "20" is hardcoded, but you can change it with the --searchlimit=N parameter for the "npm search" and it is the simpliest workaround.
The only problem is that you never know how the search results are big. There is no value which means "infinity" (I tried to pass -1 and it did not work). If you really need the full search you can either refuse of "npm search" and parse directly the JSON output of the https://myNpmServer.com/repository/myNpmRegistry/-/all or you can hack the file <NodeInstallationDir>/lib/node_modules/npm/lib/search.js and add your own parameter --oldsearch:
if (npm.config.get('oldsearch')) {
allPackageSearch(searchOpts).on('data', function (pkg) {
entriesStream.write(pkg)
}).on('error', function (e) {
entriesStream.emit('error', e)
}).on('end', function () {
entriesStream.end()
})
} else {
esearch(searchOpts).on('data', function (pkg) {
entriesStream.write(pkg)
!esearchWritten && (esearchWritten = true)
}).on('error', function (e) {
if (esearchWritten) {
// If esearch errored after already starting output, we can't fall back.
return entriesStream.emit('error', e)
}
log.warn('search', 'fast search endpoint errored. Using old search.')
allPackageSearch(searchOpts).on('data', function (pkg) {
entriesStream.write(pkg)
}).on('error', function (e) {
entriesStream.emit('error', e)
}).on('end', function () {
entriesStream.end()
})
}).on('end', function () {
entriesStream.end()
})
After that you can say "npm search --oldsearch --registry ... '/regexp/'" and it must display really all packages.
ADDITIONAL NOTE (nice to know it):
Please be aware that during manipulating with .js scripts inside Node installation (adding there your printouts etc.) you can achieve an error message
npm ERR! invalid value written to input stream
After that something gets broken and "npm search" stops working at all or displays really few output. In order to repair this just keep adding other printouts until it fails again with the abovementioned message. Then the next run (only once) you see these messages:
npm WARN all-package-metadata cached-entry-stream Empty or invalid stream
npm WARN Failed to read search cache. Rebuilding
npm WARN Building the local index for the first time, please be patient
and it returns to proper state again. I did not investigate furhter why this happens and did not find a way to force invalidating this search cache.
I hope my investigations can become helpful for somebody.