CI: add Node.js 15

[nstepien]

Shamelessly based on this commit: edf3fe2

[nstepien]

From the changelog:

build: drop support for VS2017

[nstepien]

I guess it's still too early.
BTW I would suggest giving github actions a try, should be much faster than Travis/AppVeyor.

[coveralls]

Coverage remained the same at 100.0% when pulling d09eb0f on nstepien:patch-1 into fbe48d7 on lovell:master.

[lovell]

Node.js 15 ships with npm v7, which now assumes the user permission of the working directory when running install scripts. This means we need to ensure the working directory is owned by root via sudo chown 0.0 ${PWD} - see 345b374 for an example of this.

[nstepien]

This PR depends on this now: docker-library/official-images#8933

[nstepien]

@lovell Looks like we're good to go now, the FreeBSD CI is still failing, but it is failing on master as well so I guess it's expected.

[lovell]

Merci beaucoup!

[nstepien]

Thanks! When can we expect v0.26.3 with the prebuilt binaries for Node 15?

[lovell]

The use of N-API means the existing binaries already work with Node.js 15.

$ nvm i 15
Downloading and installing node v15.0.1...
Downloading https://nodejs.org/dist/v15.0.1/node-v15.0.1-linux-x64.tar.xz...
######################################################################### 100.0%
Computing checksum with sha256sum
Checksums matched!
now using node v15.0.1 (npm v7.0.3)

$ npm i sharp

added 71 packages, and audited 71 packages in 16s

6 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

$ node -p "require('sharp').format"
{
  jpeg: {
    id: 'jpeg',
    input: { file: true, buffer: true, stream: true },
    output: { file: true, buffer: true, stream: true }
  },
...

[nstepien]

Mmh, unfortunately it's not working in our CI. Any idea why the prebuilt binaries might not be used?

C:\Jenkins\...>npm install 
npm ERR! code 1
npm ERR! path C:\Jenkins\...\node_modules\sharp
npm ERR! command failed
npm ERR! command C:\Windows\system32\cmd.exe /d /s /c "(node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)"
npm ERR! info sharp Using cached C:\Windows\system32\config\systemprofile\AppData\Local\npm-cache\_libvips\libvips-8.10.0-win32-x64.tar.br
npm ERR! info sharp Creating C:\Jenkins\...\node_modules\sharp\build\Release
npm ERR! info sharp Copying DLLs from C:\Jenkins\...\node_modules\sharp\vendor\8.10.0\lib to C:\Jenkins\...\node_modules\sharp\build\Release
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | win32 | x64
npm ERR! gyp ERR! find Python 
npm ERR! gyp ERR! find Python Python is not set from command line or npm configuration
npm ERR! gyp ERR! find Python Python is not set from environment variable PYTHON
npm ERR! gyp ERR! find Python checking if "python3" can be used
npm ERR! gyp ERR! find Python - "python3" is not in PATH or produced an error
npm ERR! gyp ERR! find Python checking if "python" can be used
npm ERR! gyp ERR! find Python - "python" is not in PATH or produced an error
npm ERR! gyp ERR! find Python checking if "python2" can be used
npm ERR! gyp ERR! find Python - "python2" is not in PATH or produced an error
npm ERR! gyp ERR! find Python checking if Python is C:\Python37\python.exe
npm ERR! gyp ERR! find Python - "C:\Python37\python.exe" could not be run
npm ERR! gyp ERR! find Python checking if Python is C:\Python27\python.exe
npm ERR! gyp ERR! find Python - "C:\Python27\python.exe" could not be run
npm ERR! gyp ERR! find Python checking if the py launcher can be used to find Python
npm ERR! gyp ERR! find Python - "py.exe" is not in PATH or produced an error
npm ERR! gyp ERR! find Python 
npm ERR! gyp ERR! find Python **********************************************************
npm ERR! gyp ERR! find Python You need to install the latest version of Python.
npm ERR! gyp ERR! find Python Node-gyp should be able to find and use Python. If not,
npm ERR! gyp ERR! find Python you can try one of the following options:
npm ERR! gyp ERR! find Python - Use the switch --python="C:\Path\To\python.exe"
npm ERR! gyp ERR! find Python   (accepted by both node-gyp and npm)
npm ERR! gyp ERR! find Python - Set the environment variable PYTHON
npm ERR! gyp ERR! find Python - Set the npm configuration variable python:
npm ERR! gyp ERR! find Python   npm config set python "C:\Path\To\python.exe"
npm ERR! gyp ERR! find Python For more information consult the documentation at:
npm ERR! gyp ERR! find Python https://github.com/nodejs/node-gyp#installation
npm ERR! gyp ERR! find Python **********************************************************
npm ERR! gyp ERR! find Python 
npm ERR! gyp ERR! configure error 
npm ERR! gyp ERR! stack Error: Could not find any Python installation to use
npm ERR! gyp ERR! stack     at PythonFinder.fail (C:\ProgramData\nvm\v15.0.0\node_modules\npm\node_modules\node-gyp\lib\find-python.js:302:47)
npm ERR! gyp ERR! stack     at PythonFinder.runChecks (C:\ProgramData\nvm\v15.0.0\node_modules\npm\node_modules\node-gyp\lib\find-python.js:136:21)
npm ERR! gyp ERR! stack     at PythonFinder.<anonymous> (C:\ProgramData\nvm\v15.0.0\node_modules\npm\node_modules\node-gyp\lib\find-python.js:200:18)
npm ERR! gyp ERR! stack     at PythonFinder.execFileCallback (C:\ProgramData\nvm\v15.0.0\node_modules\npm\node_modules\node-gyp\lib\find-python.js:266:16)
npm ERR! gyp ERR! stack     at exithandler (node:child_process:316:5)
npm ERR! gyp ERR! stack     at ChildProcess.errorhandler (node:child_process:328:5)
npm ERR! gyp ERR! stack     at ChildProcess.emit (node:events:327:20)
npm ERR! gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:275:12)
npm ERR! gyp ERR! stack     at onErrorNT (node:internal/child_process:465:16)
npm ERR! gyp ERR! stack     at processTicksAndRejections (node:internal/process/task_queues:80:21)
npm ERR! gyp ERR! System Windows_NT 10.0.14393
npm ERR! gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\ProgramData\\nvm\\v15.0.0\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
npm ERR! gyp ERR! cwd C:\Jenkins\...\node_modules\sharp
npm ERR! gyp ERR! node -v v15.0.0
npm ERR! gyp ERR! node-gyp -v v7.1.1
npm ERR! gyp ERR! not ok

[lovell]

Node.js 15 ships with npm v7, which now assumes the user permission of the working directory when running install scripts.

I suspect you've run into this breaking change of npm v7 - check the user permissions on all directories involved.

[nstepien]

We've done some investigation, it seems like install/libvips.js works, but install/dll-copy.js fails silently. It does work when ran manually though.
https://ci.appveyor.com/project/lovell/sharp/builds/35915090/job/hudb6bnchusaicyg
https://ci.appveyor.com/project/lovell/sharp/builds/35915090/job/7lqf5ivbvwgp4vbp
It seems like the same thing happens in your own appveyor builds (see Copying DLLs from ...), except there you have the necessary build tools to make it work.
I don't understand why libvips.js works but dll-copy.js doesn't. Or is it an issue with prebuild-install?

[lovell]

npm v7 is far less verbose during installation - please can you trying adding a non-zero exit code here to see if it might be swallowing an error message:

diff --git a/install/dll-copy.js b/install/dll-copy.js
index 1e88b1d..0fcadda 100644
--- a/install/dll-copy.js
+++ b/install/dll-copy.js
@@ -33,5 +33,6 @@ if (platform === 'win32') {
       });
   } catch (err) {
     npmLog.error('sharp', err.message);
+    process.exit(1);
   }

[medianick]

I've been combing through Process Monitor-captured events captured while running npm install sharp, and it appears that dll-copy.js does in fact work -- i.e., the build\Release folder is getting created and populated -- but then prebuild-install is failing, and as a logical result, node-gyp rebuild is being invoked, which removes the just-created build\Release folder and its contents. With node-gyp then failing due to build tools we don't have present on our Jenkins agents (unlike AppVeyor), the overall install fails, as @nstepien reported.

I haven't figured out why prebuild-install is failing. I can see the parent process querying for and locating ...node_modules\.bin\prebuild-install.CMD as the executable to use, followed by what appears to be its execution. If I invoke it manually from ...node_modules\sharp (i.e., running ..\.bin\prebuild-install.cmd), I do indeed get an %ERRORLEVEL% of 1, as I also do if I invoke node ..\prebuild-install\bin.js from the same location (which is ultimately what the cmd file is doing).

[medianick]

Running ..\prebuild-install\bin.js through the debugger, this is where it exits:

 55 } else if (!(typeof pkg._from === 'string')) {
 56   log.info('install', 'installing standalone, skipping download.')
>57   process.exit(1)

Running the npm install sharp --loglevel verbose, I see the same thing:

425 error command C:\Windows\system32\cmd.exe /d /s /c "(node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)"
426 error info sharp Using cached C:\Users\njones\AppData\Local\npm-cache\_libvips\libvips-8.10.0-win32-x64.tar.br
426 error info sharp Creating C:\Users\njones\rando\node_modules\sharp\build\Release
426 error info sharp Copying DLLs from C:\Users\njones\rando\node_modules\sharp\vendor\8.10.0\lib to C:\Users\njones\rando\node_modules\sharp\build\Release
426 error prebuild-install info begin Prebuild-install version 5.3.6
426 error prebuild-install info install installing standalone, skipping download.

Given that the install is running C:\Windows\system32\cmd.exe /d /s /c "(node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)", that process.exit(1) from prebuild-install would lead to node-gyp rebuild being run (which also fails, as we lack Python, etc.), which then leads to the overall failure.

It looks like prebuild-install added npm 7 support in 6.0.0. I'll add a PR.