numpy 2.0

[bmiszalski]

I ran a pip install today of the pipeline and it grabbed numpy 2.0 as part of the requirements for the installation as listed in setup.py.

Trying to run the pipeline resulted in the following exception being raised:

File "/home/dcadmin/.local/lib/python3.11/site-packages/pywifes/wifes_calib.py", line 571, in derive_wifes_calibration
temp_fvals = savitzky_golay(temp_full_y,101,1,0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dcadmin/.local/lib/python3.11/site-packages/pywifes/wifes_calib.py", line 394, in savitzky_golay
b = numpy.mat([[k**i for i in order_range] for k in range(-half_window, half_window+1)])
^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/numpy/init.py", line 397, in getattr
raise AttributeError(
AttributeError: np.mat was removed in the NumPy 2.0 release. Use np.asmatrix instead.

There may be other incompatible functions than the example given above, but I haven't tried to take this any further to find them.

It would be a good idea to update the numpy functions to be 2.0 compatible.

Otherwise a temporary fix could be specifying in setup.py:
"numpy < 2.0",

[conken]

That's a good idea to resolve any numpy conflicts. In this particular case, it would be worth considering replacement of the hand-crafted S-G filtering with the more feature-rich scipy version. (I suspect it was added manually because the S-G filter didn't exist in scipy when pywifes was first written.)

[felipeji]

We have a similar issue with scipy, so in the setup.py is set as scipy==1.9.1. As numpy 2 is not compatible with scipy 1.9.1, setting the pipeline properly to work with numpy 2 would means that scipy incompatibilities need to be fixed as well. So for now, I just set numpy < 2.0 in the setup.py. I'll leave the issue open, including a note in the section Known Issues and Suggestions for Improvement in the readme for future improvement.

[bmiszalski]

I also had to set scipy >= 1.9.1 and photutils>=1.8.0 to get it to build properly on debian:12. I'll try and reproduce the error I get - essentially just build requirements for scipy conflicting with other things.

[bmiszalski]

`Collecting scipy==1.9.1
Downloading scipy-1.9.1.tar.gz (42.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.0/42.0 MB 10.1 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [66 lines of output]
The Meson build system
Version: 0.62.2
Source dir: /tmp/pip-install-imwn5tuy/scipy_f18e470b43ba4ae78bad061282c133fe
Build dir: /tmp/pip-install-imwn5tuy/scipy_f18e470b43ba4ae78bad061282c133fe/.mesonpy-1fer50xt/build
Build type: native build
Project name: SciPy
Project version: 1.9.1
C compiler for the host machine: cc (gcc 12.2.0 "cc (Debian 12.2.0-14) 12.2.0")
C linker for the host machine: cc ld.bfd 2.40
C++ compiler for the host machine: c++ (gcc 12.2.0 "c++ (Debian 12.2.0-14) 12.2.0")
C++ linker for the host machine: c++ ld.bfd 2.40
Host machine cpu family: x86_64
Host machine cpu: x86_64
Compiler for C supports arguments -Wno-unused-but-set-variable: YES
Library m found: YES
Fortran compiler for the host machine: gfortran (gcc 12.2.0 "GNU Fortran (Debian 12.2.0-14) 12.2.0")
Fortran linker for the host machine: gfortran ld.bfd 2.40
Program cython found: YES (/tmp/pip-build-env-hz8r7lna/overlay/local/bin/cython)
Program pythran found: YES (/tmp/pip-build-env-hz8r7lna/overlay/local/bin/pythran)
Program cp found: YES (/usr/bin/cp)
Program python found: YES (/usr/bin/python3)
Found pkg-config: /usr/bin/pkg-config (1.8.1)
Library npymath found: YES
Library npyrandom found: YES
Found CMake: /usr/bin/cmake (3.25.1)
Run-time dependency openblas found: NO (tried pkgconfig and cmake)
Run-time dependency openblas found: NO (tried pkgconfig and cmake)

  ../../scipy/meson.build:130:0: ERROR: Dependency "OpenBLAS" not found, tried pkgconfig and cmake
  
  A full log can be found at /tmp/pip-install-imwn5tuy/scipy_f18e470b43ba4ae78bad061282c133fe/.mesonpy-1fer50xt/build/meson-logs/meson-log.txt
  + meson setup --native-file=/tmp/pip-install-imwn5tuy/scipy_f18e470b43ba4ae78bad061282c133fe/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 --prefix=/usr /tmp/pip-install-imwn5tuy/scipy_f18e470b43ba4ae78bad061282c133fe /tmp/pip-install-imwn5tuy/scipy_f18e470b43ba4ae78bad061282c133fe/.mesonpy-1fer50xt/build
  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/usr/lib/python3/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
             ^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-hz8r7lna/overlay/local/lib/python3.11/dist-packages/mesonpy/__init__.py", line 969, in get_requires_for_build_wheel
      with _project(config_settings) as project:
    File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
      return next(self.gen)
             ^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-hz8r7lna/overlay/local/lib/python3.11/dist-packages/mesonpy/__init__.py", line 948, in _project
      with Project.with_temp_working_dir(
    File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
      return next(self.gen)
             ^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-hz8r7lna/overlay/local/lib/python3.11/dist-packages/mesonpy/__init__.py", line 777, in with_temp_working_dir
      yield cls(source_dir, tmpdir, build_dir)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-hz8r7lna/overlay/local/lib/python3.11/dist-packages/mesonpy/__init__.py", line 682, in __init__
      self._configure(reconfigure=bool(build_dir) and not native_file_mismatch)
    File "/tmp/pip-build-env-hz8r7lna/overlay/local/lib/python3.11/dist-packages/mesonpy/__init__.py", line 713, in _configure
      self._meson(
    File "/tmp/pip-build-env-hz8r7lna/overlay/local/lib/python3.11/dist-packages/mesonpy/__init__.py", line 696, in _meson
      return self._proc('meson', *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-hz8r7lna/overlay/local/lib/python3.11/dist-packages/mesonpy/__init__.py", line 691, in _proc
      subprocess.check_call(list(args))
    File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['meson', 'setup', '--native-file=/tmp/pip-install-imwn5tuy/scipy_f18e470b43ba4ae78bad061282c133fe/.mesonpy-native-file.ini', '-Ddebug=false', '-Doptimization=2', '--prefix=/usr', '/tmp/pip-install-imwn5tuy/scipy_f18e470b43ba4ae78bad061282c133fe', '/tmp/pip-install-imwn5tuy/scipy_f18e470b43ba4ae78bad061282c133fe/.mesonpy-1fer50xt/build']' returned non-zero exit status 1.
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
`

[bmiszalski]

Anyhow, adding continuous testing / integration will help sort out these packaging problems (for different dependencies and different versions of Python)

[felipeji]

Yeah, in the last version (the one we deliver) we have in the setup.py:

    install_requires=[
        "wheel",
        "setuptools",
        "astropy",
        "scipy==1.9.1",
        "numpy",
        "matplotlib",
        "photutils==1.8.0",
        "pandas",
]

I'm now added the condition in numpy <2

[bmiszalski]

Right, specifically I need to have >= for scipy (and probably photutils). The equality alone was not sufficient.