Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow non-conda builds and pip install #70

Merged
merged 24 commits into from
Apr 8, 2022
Merged

Allow non-conda builds and pip install #70

merged 24 commits into from
Apr 8, 2022

Conversation

richfitz
Copy link
Contributor

@richfitz richfitz commented Apr 1, 2022

This PR adds a non-conda build, proven to work with a docker image which we build on buildkite.

To do this:

  • HDF5 was put under cmake control
  • Had a fight with pybind11's detection
  • The HighFive library, removed as a submodule in 0750219 is now included as a manually updated lib (not a submodule); see scripts/ for the details
  • added a MANIFEST.in which seems to be this month's way of controlling what gets included

Note that I can't work out how to get pybind11 to install before pip tries to install the package. It turns out that if that doesn't happen pip installs a hopelessly broken package with almost no warning! I tried both install_requires and setup_requires but neither seem to behave. If there's any documentation apart from random SO threads we might be able to find out the correct way of making this work.

UPDATE: this pybind11 dependency resolution thing is extremely annoying in practice. There must be some way of declaring a dependency in setup.py that setup.py actually respects? If you know it John, please share. This is not promising (pybind/python_example#20) - my quick read "you can't do this thing, build a wheel instead, enjoy shipping broken source packages"

@johnlees; you will need to update the Azure pipelines to point at the moved repo - they have not run here.

@richfitz richfitz marked this pull request as ready for review April 1, 2022 12:58
@richfitz richfitz requested a review from johnlees April 1, 2022 12:59
@richfitz richfitz changed the title Allow non-conda builds Allow non-conda builds and pip install Apr 1, 2022
@richfitz
Copy link
Contributor Author

richfitz commented Apr 1, 2022

The sad story of a failed install:
Step 4/6 : RUN pip install -r /tmp/requirements.txt && rm -f /tmp/requirements.txt
 ---> Running in a6e96251d148
Collecting Cython>=0.26.1
  Downloading Cython-0.29.28-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 3.2 MB/s eta 0:00:00
Collecting pybind11
  Downloading pybind11-2.9.2-py2.py3-none-any.whl (213 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 213.6/213.6 KB 3.3 MB/s eta 0:00:00
Collecting dendropy>=4.4.0
  Downloading DendroPy-4.5.2.tar.gz (15.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.2/15.2 MB 3.2 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting flask
  Downloading Flask-2.1.1-py3-none-any.whl (95 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.2/95.2 KB 3.4 MB/s eta 0:00:00
Collecting flask-cors
  Downloading Flask_Cors-3.0.10-py2.py3-none-any.whl (14 kB)
Collecting gunicorn
  Downloading gunicorn-20.1.0-py3-none-any.whl (79 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.5/79.5 KB 3.2 MB/s eta 0:00:00
Collecting h5py
  Downloading h5py-3.6.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 3.3 MB/s eta 0:00:00
Collecting hdbscan
  Downloading hdbscan-0.8.28.tar.gz (5.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.2/5.2 MB 3.3 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting matplotlib
  Downloading matplotlib-3.5.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.9/11.9 MB 3.2 MB/s eta 0:00:00
Collecting networkx
  Downloading networkx-2.7.1-py3-none-any.whl (2.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 3.3 MB/s eta 0:00:00
Collecting numpy
  Downloading numpy-1.22.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.8/16.8 MB 3.2 MB/s eta 0:00:00
Collecting pandas
  Downloading pandas-1.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.7/11.7 MB 3.3 MB/s eta 0:00:00
Collecting pp-sketchlib>=1.7.0
  Downloading pp_sketchlib-1.7.6.tar.gz (367 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 367.5/367.5 KB 3.2 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting requests
  Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.1/63.1 KB 2.6 MB/s eta 0:00:00
Collecting scikit-learn>=0.24
  Downloading scikit_learn-1.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 26.5/26.5 MB 3.2 MB/s eta 0:00:00
Collecting scipy
  Downloading scipy-1.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (42.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.3/42.3 MB 3.1 MB/s eta 0:00:00
Collecting tqdm
  Downloading tqdm-4.63.1-py2.py3-none-any.whl (76 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 76.6/76.6 KB 3.0 MB/s eta 0:00:00
Collecting treeswift
  Downloading treeswift-1.1.26-py2.py3-none-any.whl (32 kB)
Collecting tzlocal<3.0
  Downloading tzlocal-2.1-py2.py3-none-any.whl (16 kB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.10/site-packages (from dendropy>=4.4.0->-r /tmp/requirements.txt (line 3)) (58.1.0)
Collecting click>=8.0
  Downloading click-8.1.2-py3-none-any.whl (96 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.6/96.6 KB 3.3 MB/s eta 0:00:00
Collecting Jinja2>=3.0
  Downloading Jinja2-3.1.1-py3-none-any.whl (132 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 132.6/132.6 KB 3.1 MB/s eta 0:00:00
Collecting itsdangerous>=2.0
  Downloading itsdangerous-2.1.2-py3-none-any.whl (15 kB)
Collecting Werkzeug>=2.0
  Downloading Werkzeug-2.1.0-py3-none-any.whl (224 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 224.6/224.6 KB 3.2 MB/s eta 0:00:00
Collecting Six
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting joblib>=1.0
  Downloading joblib-1.1.0-py2.py3-none-any.whl (306 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 307.0/307.0 KB 3.4 MB/s eta 0:00:00
Collecting cycler>=0.10
  Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting pillow>=6.2.0
  Downloading Pillow-9.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.3/4.3 MB 3.3 MB/s eta 0:00:00
Collecting python-dateutil>=2.7
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 KB 3.2 MB/s eta 0:00:00
Collecting kiwisolver>=1.0.1
  Downloading kiwisolver-1.4.2-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 3.3 MB/s eta 0:00:00
Collecting fonttools>=4.22.0
  Downloading fonttools-4.31.2-py3-none-any.whl (899 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 899.5/899.5 KB 3.2 MB/s eta 0:00:00
Collecting pyparsing>=2.2.1
  Downloading pyparsing-3.0.7-py3-none-any.whl (98 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.0/98.0 KB 3.3 MB/s eta 0:00:00
Collecting packaging>=20.0
  Downloading packaging-21.3-py3-none-any.whl (40 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.8/40.8 KB 2.7 MB/s eta 0:00:00
Collecting pytz>=2020.1
  Downloading pytz-2022.1-py2.py3-none-any.whl (503 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 503.5/503.5 KB 3.3 MB/s eta 0:00:00
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.0/139.0 KB 3.3 MB/s eta 0:00:00
Collecting charset-normalizer~=2.0.0
  Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
  Downloading idna-3.3-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 KB 3.6 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
  Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 149.2/149.2 KB 3.3 MB/s eta 0:00:00
Collecting threadpoolctl>=2.0.0
  Downloading threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Building wheels for collected packages: dendropy, hdbscan, pp-sketchlib
  Building wheel for dendropy (setup.py): started
  Building wheel for dendropy (setup.py): finished with status 'done'
  Created wheel for dendropy: filename=DendroPy-4.5.2-py3-none-any.whl size=453171 sha256=9851715ce2690cf4f7212339ea0af264172b41de9eb5ebb9dc26c4ba22f38443
  Stored in directory: /root/.cache/pip/wheels/2c/c8/56/a077856a8b16234966f2f32da8c60d8083bd8e131f4171aca9
  Building wheel for hdbscan (pyproject.toml): started
  Building wheel for hdbscan (pyproject.toml): finished with status 'done'
  Created wheel for hdbscan: filename=hdbscan-0.8.28-cp310-cp310-linux_x86_64.whl size=3655262 sha256=bf21dfbb851da4abd2a5e16dceca35d5c2d4809268e9aaa5af45037c5182abc3
  Stored in directory: /root/.cache/pip/wheels/d8/bf/4e/27087150384893aa010d585e01cdf6016d6b750c02cb7d4dee
  Building wheel for pp-sketchlib (setup.py): started
  Building wheel for pp-sketchlib (setup.py): finished with status 'error'
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [70 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.10
      creating build/lib.linux-x86_64-3.10/pp_sketch
      copying pp_sketch/matrix.py -> build/lib.linux-x86_64-3.10/pp_sketch
      copying pp_sketch/__init__.py -> build/lib.linux-x86_64-3.10/pp_sketch
      copying pp_sketch/__main__.py -> build/lib.linux-x86_64-3.10/pp_sketch
      running build_ext
      -- The C compiler identification is GNU 10.2.1
      -- The CXX compiler identification is GNU 10.2.1
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5_cpp.so;/usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.10.6") found components: CXX
      CMake Error at CMakeLists.txt:54 (find_package):
        Could not find a package configuration file provided by "pybind11"
        (requested version 2.6) with any of the following names:
      
          pybind11Config.cmake
          pybind11-config.cmake
      
        Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set
        "pybind11_DIR" to a directory containing one of the above files.  If
        "pybind11" provides a separate development package or SDK, be sure it has
        been installed.
      
      
      -- Configuring incomplete, errors occurred!
      See also "/tmp/pip-install-ue8jii4y/pp-sketchlib_5073a44a03ff4fb3ae4f5290f963aa80/build/temp.linux-x86_64-3.10/CMakeFiles/CMakeOutput.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-ue8jii4y/pp-sketchlib_5073a44a03ff4fb3ae4f5290f963aa80/setup.py", line 101, in <module>
          setup(
        File "/usr/local/lib/python3.10/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/usr/local/lib/python3.10/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/usr/local/lib/python3.10/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/usr/local/lib/python3.10/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/usr/local/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 299, in run
          self.run_command('build')
        File "/usr/local/lib/python3.10/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/usr/local/lib/python3.10/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/usr/local/lib/python3.10/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/usr/local/lib/python3.10/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/usr/local/lib/python3.10/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/tmp/pip-install-ue8jii4y/pp-sketchlib_5073a44a03ff4fb3ae4f5290f963aa80/setup.py", line 54, in run
          self.build_extension(ext)
        File "/tmp/pip-install-ue8jii4y/pp-sketchlib_5073a44a03ff4fb3ae4f5290f963aa80/setup.py", line 92, in build_extension
          subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
        File "/usr/local/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/tmp/pip-install-ue8jii4y/pp-sketchlib_5073a44a03ff4fb3ae4f5290f963aa80', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/pip-install-ue8jii4y/pp-sketchlib_5073a44a03ff4fb3ae4f5290f963aa80/build/lib.linux-x86_64-3.10', '-DPYTHON_EXECUTABLE=/usr/local/bin/python', '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pp-sketchlib
  Running setup.py clean for pp-sketchlib
Successfully built dendropy hdbscan
Failed to build pp-sketchlib
Installing collected packages: treeswift, pytz, certifi, Werkzeug, urllib3, tzlocal, tqdm, threadpoolctl, Six, pyparsing, pybind11, pillow, numpy, networkx, MarkupSafe, kiwisolver, joblib, itsdangerous, idna, gunicorn, fonttools, dendropy, Cython, cycler, click, charset-normalizer, scipy, requests, python-dateutil, packaging, Jinja2, h5py, scikit-learn, pp-sketchlib, pandas, matplotlib, flask, hdbscan, flask-cors
  Running setup.py install for pp-sketchlib: started
  Running setup.py install for pp-sketchlib: finished with status 'done'
  DEPRECATION: pp-sketchlib was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. A possible replacement is to fix the wheel build issue reported above. Discussion can be found at https://github.com/pypa/pip/issues/8368
Successfully installed Cython-0.29.28 Jinja2-3.1.1 MarkupSafe-2.1.1 Six-1.16.0 Werkzeug-2.1.0 certifi-2021.10.8 charset-normalizer-2.0.12 click-8.1.2 cycler-0.11.0 dendropy-4.5.2 flask-2.1.1 flask-cors-3.0.10 fonttools-4.31.2 gunicorn-20.1.0 h5py-3.6.0 hdbscan-0.8.28 idna-3.3 itsdangerous-2.1.2 joblib-1.1.0 kiwisolver-1.4.2 matplotlib-3.5.1 networkx-2.7.1 numpy-1.22.3 packaging-21.3 pandas-1.4.1 pillow-9.1.0 pp-sketchlib-1.7.6 pybind11-2.9.2 pyparsing-3.0.7 python-dateutil-2.8.2 pytz-2022.1 requests-2.27.1 scikit-learn-1.0.2 scipy-1.8.0 threadpoolctl-3.1.0 tqdm-4.63.1 treeswift-1.1.26 tzlocal-2.1 urllib3-1.26.9
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
  1. pybind11 is collected
  2. installation of pp-sketchlib fails because pip just did not install pybind11 (yet - it would work a second time around)
  3. rather than reporting the failure, pip let's us know quietly that it messed up and has installed a junk package instead
  4. further installation of pp-sketchlib would fail because it's "already satisfied"
  note: This error originates from a subprocess, and is likely not a problem with pip.

I beg to differ

@richfitz
Copy link
Contributor Author

richfitz commented Apr 1, 2022

OK, this suggests that we should be using pyproject.toml https://gitlab.cern.ch/slac_itk/external/pybind11/-/blob/04dd3262f0420d48531a82021fa4895ba14bdcb2/docs/compiling.rst#id2

that is related to some degree to poetry? Something to try next week

@richfitz
Copy link
Contributor Author

richfitz commented Apr 4, 2022

Here's the actual docs https://pybind11.readthedocs.io/en/stable/compiling.html

@richfitz
Copy link
Contributor Author

richfitz commented Apr 4, 2022

pypa/pip#8368 - this is an interesting view into how pip and python installations are such an awful mess


PACKAGE_ROOT=$(realpath $HERE/..)
PACKAGE_NAME=pp-sketchlib
PACKAGE_ORG=mrcide
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we do have a poppunk org on dockerhub, but we could change this. We could also make a bacpop one/ What do you think is best?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docker has changed their licencing in annoying ways, leading to rate limiting on pushes/pulls. We have applied for and got some level of exception for mrcide which is why I put it here. Very happy to use the poppunk one, but we'll need to get our robot user added to that org (there are also organisation size/user limits) and it might end up causing a few issues if we hit rate limits. But we can cross that bridge when we get to it, including getting you to organise an exception for poppunk

@@ -3,4 +3,4 @@

'''PopPUNK sketching functions'''

__version__ = '1.7.4'
__version__ = '1.7.6.2'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

v2 PR has been pending one final fix (this only changes the CLI options, and will make the python API more robust by making everything support a kwarg, but not change it).

Anyway, I will sort out that merge after this, as that's still some time away from being ready (#67)

Copy link
Member

@johnlees johnlees left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, but I will try and include the azure fix in this before merging

@johnlees johnlees merged commit e5b11eb into master Apr 8, 2022
@johnlees johnlees deleted the bacpop-19 branch April 8, 2022 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants