Regression: Linux: (edge case) nyc returns 0% coverage if Node.js binary has permissions to bind to ports < 1024 via setcap #1281

aral · 2020-02-15T19:24:12Z

To reproduce

Install the simple test app.

git clone /~https://github.com/aral/simple-demonstration-of-nyc-bug-1281 bug-1281
cd bug-1281
npm i

Ensure Node.js does not have permissions to access privileged ports:
```
sudo setcap 'cap_net_bind_service=-ep' `which node`
```
npm run coverage succeeds (100%)

Get permissions for Node to bind to ports < 1024:

sudo setcap 'cap_net_bind_service=+ep' `which node`

npm run coverage fails (0%)

Run the second step again to return to being able to use nyc properly.

It appears that using setcap to set permissions has side-effects. e.g., see this: Linux capabilities (setcap) seems to disable LD_LIBRARY_PATH.

What should happen

Coverage should be 100% at Step 5.

What actually happens

Coverage is 0% at Step 5.

Regression

Works with nyc@14.1.1 (with both Tape 4.11.0 and Tape 5.0.0). The issue occurs on nyc@15.0.1.

To reproduce

Check out the git repo at https://source.ind.ie/site.js/app (branch: hugo)

Run ./install

Run npm run coverage

What happens (nyc version 15.0.0)

Incorrectly reports 0% coverage:

> @small-tech/site.js@12.11.0 coverage /home/aral/small-tech/site.js/app > QUIET=true nyc tape test/*.js | tap-nyc ----------|---------|----------|---------|---------|------------------- File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s ----------|---------|----------|---------|---------|------------------- All files | 0 | 0 | 0 | 0 | ----------|---------|----------|---------|---------|------------------- total: 143 passing: 143 duration: 680ms

What happens (nyc version 14.1.1)

Works correctly:

> @small-tech/site.js@12.11.0 coverage /home/aral/small-tech/site.js/app > QUIET=true nyc tape test/*.js | tap-nyc -----------------------|----------|----------|----------|----------|-------------------| File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s | -----------------------|----------|----------|----------|----------|-------------------| All files | 36.68 | 24.61 | 36.76 | 37.5 | | app | 56.76 | 39.44 | 63.08 | 58.5 | | index.js | 56.76 | 39.44 | 63.08 | 58.5 |... 97,898,899,900 | app/bin/commands | 12.24 | 0 | 0 | 12.24 | | serve.js | 12.24 | 0 | 0 | 12.24 |... 50,353,357,359 | app/bin/lib | 15.69 | 12.5 | 5.13 | 15.67 | | RsyncWatcher.js | 7.69 | 0 | 0 | 7.95 |... 91,192,193,195 | cli.js | 100 | 100 | 100 | 100 | | console-timestamp.js | 14.29 | 100 | 0 | 14.29 | 4,5,9,10,11,12 | ensure.js | 8.79 | 0 | 0 | 8.99 |... 97,199,202,203 | runtime.js | 100 | 75 | 100 | 100 | 16 | status.js | 14.29 | 0 | 0 | 14.29 |... 35,36,37,39,42 | sync.js | 9.8 | 0 | 0 | 9.8 |... 15,119,120,126 | app/lib | 53.23 | 44.44 | 38.89 | 57.89 | | Stats.js | 50 | 44 | 35.29 | 54.72 |... 18,120,122,124 | clr.js | 100 | 50 | 100 | 100 | 6 | errors.js | 100 | 100 | 100 | 100 | | -----------------------|----------|----------|----------|----------|-------------------| total: 143 passing: 143 duration: 1.7s

Environment Information

System: OS: Linux 5.3 Pop!_OS 19.10 CPU: (8) x64 Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz Memory: 1.15 GB / 15.35 GB Binaries: Node: 12.14.1 - ~/.nvm/versions/node/v12.14.1/bin/node Yarn: 1.21.1 - /usr/bin/yarn npm: 6.13.7 - ~/.nvm/versions/node/v12.14.1/bin/npm npmPackages: nyc: ^14.1.1 => 14.1.1

The text was updated successfully, but these errors were encountered:

AndyOGo · 2020-04-25T18:39:58Z

I have a similar issue, but with both versions of nyc.

nyc just does not see my covered code paths :(
AndyOGo/stylelint-declaration-strict-value#64

ljharb · 2020-04-25T20:17:57Z

Have you compared tape v4 vs tape v5?

AndyOGo · 2020-04-25T21:21:49Z

@ljharb
Thank you very much for your input.

No, I get tape ^4.9.0 through stylelint-test-rule-tape
/~https://github.com/stylelint/stylelint-test-rule-tape/blob/6279a6512269720e737aa06490a6bf8f4d27fa31/package.json#L26

ljharb · 2020-04-25T22:20:22Z

In that case it’s unlikely to be a regression in tape itself; please let me know if there’s anything actionable on tape’s side.

AndyOGo · 2020-04-26T06:34:10Z

Alright.
@ljharb @aral
For my case it's not working with esm, but with babel-register (though slower).

My updated .nycrc:

{
  "require": [
    "babel-register"
  ],
  "all": true,
  "exclude": ["node_modules", "**/coverage/**", "commitlint.config.js", "test", "lint-staged.config.js"],
  "include": ["src/**/*"],
  "check-coverage": true,
  "per-file": true,
  "branches": 85,
  "lines": 85,
  "functions": 85,
  "statements": 85
}

According to the docs esm should work 😕
/~https://github.com/istanbuljs/nyc#require-additional-modules

I think those issues maybe be related:

AndyOGo · 2020-04-26T06:39:17Z

Alright downgrading esm to 3.2.20 works on my local machine, but no in Travis CI...

coreyfarrell · 2020-04-26T15:07:29Z

@AndyOGo OP does not appear to be directly using babel or esm and they did not experience the issue with nyc 14.1.1 so your issue seems unrelated. Please do not comment on existing issues when you have a similar issue, create a new issue.

@aral sorry I haven't had a chance to dig deep into your issue yet. By chance would you be able to create a simplified branch? In particular a demonstration using less (or even no) extra dependencies beyond tape and nyc would be useful, as well not using a custom ./install script.

As for tape + nyc 15 I know for sure of projects successfully using this combination, I use them in node-preload for example. Additional note: I just noticed an active PR to upgrade node-preload to use tape@5 which passed CI so both tape@4 and tape@5 are confirmed to be compatible with nyc 15.

AndyOGo · 2020-04-26T15:42:52Z

Thanks for your answer @coreyfarrell
For my case it turned out to be a problem with esm required by nyc -r esm.
I think the relevant issue maybe this:
standard-things/esm#852

Anyway I switched to babel-register and it works.

aral · 2020-05-18T15:51:56Z

@coreyfarrell Sorry it’s taken me so long to get back to you.

Here’s the simplest possible demonstration of the bug:

git clone /~https://github.com/aral/simple-demonstration-of-nyc-bug-1281 bug-1281
cd bug-1281
npm i
npm run coverage

What should happen

Coverage should be 100%.

What actually happens

Coverage is 0%.

Further details (by version)

nyc	tape	result
14.1.1	4.11.0	100% (correct)
15.0.1	4.11.0	0% (wrong)
15.0.1	5.0.0	0% (wrong)

The code

package.json (relevant bits)

…
  "scripts": {
    "test": "tape test/*.js",
    "coverage": "nyc tape test/*.js"
  },
…  
"dependencies": {
    "nyc": "^15.0.1",
    "tape": "^5.0.0"
  }

index.js (file being tested)

module.exports = function () {
  return true
}

test/index.js (tape tests)

const app = require('..')
const test = require('tape')

test('app',t => {
  t.ok(app())
  t.end()
})

coreyfarrell · 2020-05-18T16:03:11Z

I see 100% coverage when I run the demo repository you just posted. Tested with node.js 8, 10, 12 and 14 under Fedora 30 x86_64.

aral · 2020-05-18T16:33:00Z

@coreyfarrell Right, I just tested on a Mac and a PC (running Node v12.16.2) and I got 100% coverage also.

Tried it on my main ~~Linux~~ Ubuntu derivative dev box (x86_64) and… got 0%.

So it’s a ~~Linux~~ Ubuntu derivative distribution error! Curiouser and curiouser :)

(I’m running Pop!_OS 20.04; based on Ubuntu 20.04.)

Update: Actually read your response properly and saw that you were running it under Fedora. So it’s a Debian/Ubuntu/Pop!_OS issue?

coreyfarrell · 2020-05-18T16:55:14Z

I have no idea, this issue does not make sense. You are using npm to install and run (not pnpm)? Also note I used nvm to install node so I'm running official node.js binaries, not distro binaries.

aral · 2020-05-18T17:08:15Z

@coreyfarrell Agree, it doesn’t :)

I’m using npm to install and nvm to install node also.

Just tried:

Fedora 31: works
Ubuntu 18.04: works
Pop!_OS 19.04: works
Ubuntu 20.04 (which Pop!_OS 20.04 is based on): works

So it’s definitely either something that is unique to my system or to Pop!_OS 20.04. Now that I’ve narrowed it down to that, will continue to look into it and let you know if I find anything.

aral · 2020-05-18T17:38:04Z

@coreyfarrell Wow, I figured out what it is… talk about a niche case: nyc returns 0% coverage if the Node.js binary has permissions to bind to ports < 1024 on Linux via setcap.

To reproduce (with the simple example)

Ensure Node.js does not have permissions to access privileged ports:
```
sudo setcap 'cap_net_bind_service=-ep' `which node`
```
npm run coverage succeeds (100%)

Get permissions for Node to bind to ports < 1024:

sudo setcap 'cap_net_bind_service=+ep' `which node`

npm run coverage fails (0%)

Run the first command again to return to being able to use nyc properly.

It appears that using setcap to set permissions has side-effects. e.g., Linux capabilities (setcap) seems to disable LD_LIBRARY_PATH.

Thanks again for your time.

coreyfarrell · 2020-05-18T18:23:37Z

This looks to be a node.js behavior intentionally implemented for security reasons, apparently any setcap will block nyc from working due to node.js not honoring NODE_OPTIONS or NODE_PATH. See nodejs/node#22648 for details. I'm not sure this warrants a note in the nyc documentation, running setcap on the node binary is not recommended by node.js itself and I suspect is very rarely done.

Thanks for following-up on this issue, I'm closing this as I don't believe any further action is needed from nyc.

aral · 2020-05-19T19:10:36Z

In case anyone else comes across this, one workaround is to disable privileged ports on Linux. Privileged ports made sense in the 1970s but they don’t in 2020. The option to remove them or to reduce the number was added in version 4.11 of the Linux kernel.

I outline what I’ll be doing in the next version of Site.js here:
https://source.small-tech.org/site.js/app/-/issues/169

aral changed the title ~~Regression: tape coverage works with 14.1.1, breaks with 15.0.0~~ Regression: tape coverage works with 14.1.1, breaks with 15.0.0 (reports 0% coverage) Feb 15, 2020

aral changed the title ~~Regression: tape coverage works with 14.1.1, breaks with 15.0.0 (reports 0% coverage)~~ Regression: coverage using tape works on nyc 14.1.1, breaks on nyc 15.0.0 (reports 0% coverage) Feb 15, 2020

AndyOGo mentioned this issue Apr 25, 2020

Collect test coverage for reports tape-testing/tape#504

Closed

AndyOGo mentioned this issue Apr 25, 2020

Chore/add coverage report AndyOGo/stylelint-declaration-strict-value#64

Merged

aral changed the title ~~Regression: coverage using tape works on nyc 14.1.1, breaks on nyc 15.0.0 (reports 0% coverage)~~ Linux: nyc returns 0% coverage if Node.js binary has permissions to bind to ports < 1024 via setcap (edge case) May 18, 2020

aral changed the title ~~Linux: nyc returns 0% coverage if Node.js binary has permissions to bind to ports < 1024 via setcap (edge case)~~ Regression: Linux: (edge case) nyc returns 0% coverage if Node.js binary has permissions to bind to ports < 1024 via setcap May 18, 2020

coreyfarrell closed this as completed May 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Regression: Linux: (edge case) nyc returns 0% coverage if Node.js binary has permissions to bind to ports < 1024 via setcap #1281

Regression: Linux: (edge case) nyc returns 0% coverage if Node.js binary has permissions to bind to ports < 1024 via setcap #1281

aral commented Feb 15, 2020 •

edited

Loading

To reproduce

What happens (nyc version 15.0.0)

What happens (nyc version 14.1.1)

Environment Information

AndyOGo commented Apr 25, 2020 •

edited

Loading

ljharb commented Apr 25, 2020

AndyOGo commented Apr 25, 2020 •

edited

Loading

ljharb commented Apr 25, 2020

AndyOGo commented Apr 26, 2020 •

edited

Loading

AndyOGo commented Apr 26, 2020 •

edited

Loading

coreyfarrell commented Apr 26, 2020 •

edited

Loading

AndyOGo commented Apr 26, 2020

aral commented May 18, 2020 •

edited

Loading

coreyfarrell commented May 18, 2020

aral commented May 18, 2020 •

edited

Loading

coreyfarrell commented May 18, 2020

aral commented May 18, 2020

aral commented May 18, 2020 •

edited

Loading

coreyfarrell commented May 18, 2020

aral commented May 19, 2020

Regression: Linux: (edge case) nyc returns 0% coverage if Node.js binary has permissions to bind to ports < 1024 via setcap #1281

Regression: Linux: (edge case) nyc returns 0% coverage if Node.js binary has permissions to bind to ports < 1024 via setcap #1281

Comments

aral commented Feb 15, 2020 • edited Loading

To reproduce

What should happen

What actually happens

Regression

To reproduce

What happens (nyc version 15.0.0)

What happens (nyc version 14.1.1)

Environment Information

AndyOGo commented Apr 25, 2020 • edited Loading

ljharb commented Apr 25, 2020

AndyOGo commented Apr 25, 2020 • edited Loading

ljharb commented Apr 25, 2020

AndyOGo commented Apr 26, 2020 • edited Loading

AndyOGo commented Apr 26, 2020 • edited Loading

coreyfarrell commented Apr 26, 2020 • edited Loading

AndyOGo commented Apr 26, 2020

aral commented May 18, 2020 • edited Loading

What should happen

What actually happens

Further details (by version)

The code

package.json (relevant bits)

index.js (file being tested)

test/index.js (tape tests)

coreyfarrell commented May 18, 2020

aral commented May 18, 2020 • edited Loading

coreyfarrell commented May 18, 2020

aral commented May 18, 2020

aral commented May 18, 2020 • edited Loading

To reproduce (with the simple example)

coreyfarrell commented May 18, 2020

aral commented May 19, 2020

aral commented Feb 15, 2020 •

edited

Loading

AndyOGo commented Apr 25, 2020 •

edited

Loading

AndyOGo commented Apr 25, 2020 •

edited

Loading

AndyOGo commented Apr 26, 2020 •

edited

Loading

AndyOGo commented Apr 26, 2020 •

edited

Loading

coreyfarrell commented Apr 26, 2020 •

edited

Loading

aral commented May 18, 2020 •

edited

Loading

aral commented May 18, 2020 •

edited

Loading

aral commented May 18, 2020 •

edited

Loading