Inconsistency in number of packages fetched

[divmain]

As outlined in #3, the number of packages reported as having been fetched is not consistent. This might simply be due to inconsistencies in the package manager itself. But I'll leave this issue open until I've had a chance to dig in and either confirm (behavior in package manager) or remediate (issue in sfw).

cc @danielweck

[danielweck]

Today's sfw run:

=== Socket Firewall ===

 - 1969 packages fetched successfully

NPM says added 2145 packages

==> 176 discrepancy.

Note that I always execute npm cache clear --force ; rm -rf node_modules/ ; rm -f package-lock.json before invoking sfw npm install --foreground-scripts (with or without SFW_DEBUG=true)

[danielweck]

FYI, today's sfw run:

=== Socket Firewall ===

 - 1970 packages fetched successfully

NPM still states added 2145 packages

==> 175 discrepancy.

[divmain]

Hey @danielweck can you give this another try when you get the chance? Pushed out a new version today that might be related to what you were seeing here.

[danielweck]

=== Socket Firewall ===

 - 1966 packages fetched successfully

NPM says:

added 2145 packages, and audited 2146 packages in 3m

I noticed this message in the console:

npm warn skipping integrity check for git dependency ssh://git@github.com/electron/node-gyp.git

... which reminded me that not all packages in our application's dependency tree are necessarily hosted at the NPM registry. Perhaps this explains the discrepancy.

EDIT: no, package-lock.json doesn't show any other Git-hosted dependencies.
EDIT: package-lock.json contains 2217 individual "resolved" packages:
jq -S '.. | .resolved? // empty' package-lock.json | wc -l
Example of same package with different incompatible semantic versioning (can't be hoisted and shared globally):

"https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-8.1.0.tgz"
"https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-7.0.0.tgz"

[divmain]

I wouldn't think this would be an issue of multiple versions of the same package - we have some good test coverage there, and it should work fine.

Any chance you'd be willing to share your lock file, even partially redacted? I know that's a lot to ask, but I'm not sure how to reproduce without greater detail.

[divmain]

I have another idea for how to run this down, if you're willing to run a script locally to capture all the network requests to registry.npmjs.org. The lock-file will be best for me, but if that's not possible, let me know and I'll try to pull something together to help diagnose what's going on.

[danielweck]

Thank you for the follow-up.
Here is the NPM package-lock.json as created by rm -rf node_modules && rm -f package-lock.json && npm i:

https://github.com/edrlab/thorium-reader/blob/develop/package-lock.json

Note that to test sfw I actually run this script which temporarily "disables" the Git dependencies not hosted at the NPM registry:

https://github.com/edrlab/thorium-reader/blob/develop/scripts/socket-firewall-npm-install.sh

[danielweck]

To reproduce at your end:

• git clone https://github.com/edrlab/thorium-reader.git
• cd thorium-reader
• ./scripts/socket-firewall-npm-install.sh

[divmain]

You're awesome, thank you! I should be able to do some deep analysis on this. It might take me a few days - we have another event coming up that'll require much of my attention. But I will come back with an explanation for what's going on. Thanks again!