Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[package] libpq/15.4: broken symlinks in include/catalog/ #24895

Open
scandyna opened this issue Aug 11, 2024 · 2 comments
Open

[package] libpq/15.4: broken symlinks in include/catalog/ #24895

scandyna opened this issue Aug 11, 2024 · 2 comments
Assignees
Labels
bug Something isn't working

Comments

@scandyna
Copy link

Description

Problem

Uploading libpq/15.4 to a conan remote fails, because the package contains broken symlinks in the include/catalog/ subfolder.

Expected

There are no broken symlinks in the package and upload works.

Workaround

Set the CONAN_SKIP_BROKEN_SYMLINKS_CHECK environment variable to True fixes the upload.

I think that those headers are not used for my particular use case (building Qt5), but it could be a problem for other cases, meaning the check has sense.
Also, as far as I understand, this will not work anymore with Conan 2.

Some details

In the generated package, under include/catalog/, there are symlinks, like pg_conversion_d.h, pg_foreign_data_wrapper_d.h and others.

For the particular case of my build, pg_foreign_data_wrapper_d.h points to /root/.conan/data/libpq/15.4/_/_/build/f35a0224260f0d753acb9d90beb1f1c501d0d02b/src/src/backend/catalog/pg_foreign_data_wrapper_d.h.

If the package is just downloaded, the build folder does not exist, or could be located in another place.

I also did look into a libpq/14.5 I installed last year on my machine, and those symlinks do not exist, they are files.

Package and Environment Details

  • Package: libpq/15.4 (also 15.5)
  • Operating System+version: Linux Ubuntu 24.04 (also 22.04)
  • Compiler+version: GCC 13
  • Docker image: registry.gitlab.com/scandyna/docker-images-ubuntu/ubuntu-24.04-build-qt-gcc-13 (also the same on a native Ubuntu 22.04 with GCC 11)
  • Conan version: conan 1.64.1
  • Python version: Python 3.12.3

Conan profile

Note: also reproducible with the default profile on Ubuntu 22.04.

In the CI, 2 profiles are used.
The host profile is named linux_ubuntu-24.04_gcc13_x86_64_qt_and_more
The build profile is named linux_gcc13_x86_64

Configuration (profile_host):
[settings]
arch=x86_64
build_type=Debug
compiler=gcc
compiler.cppstd=17
compiler.libcxx=libstdc++11
compiler.version=13
os=Linux
os.distribution=ubuntu-24.04
[options]
boost:header_only=True
icu:shared=False
libalsa:shared=True
qt:gui=True
qt:qtserialport=True
qt:shared=True
qt:widgets=True
qt:with_glib=False
qt:with_mysql=False
qt:with_odbc=True
qt:with_pq=True
qt:with_sqlite3=True
qt:with_vulkan=False
zlib:shared=False
[build_requires]
[env]
[buildenv]
CC=gcc-13
CXX=g++-13
Configuration (profile_build):
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=17
compiler.libcxx=libstdc++11
compiler.version=13
os=Linux
[options]
[build_requires]
[env]
[buildenv]
CC=gcc-13
CXX=g++-13

Steps to reproduce

To see the broken symlinks, on an Ubuntu 22.04:

conan install libpq/15.4@

Here are the commands in the CI:

conan remote add gitlab_test https://gitlab.com/api/v4/projects/16325699/packages/conan
conan config install --source-folder conan https://gitlab.com/api/v4/projects/16304913/packages/generic/conan1_config/0.1.8/conan_config.zip
conan install libpq/15.4@ --profile:build linux_gcc13_x86_64 --profile:host linux_ubuntu-24.04_gcc13_x86_64_qt_and_more --settings:build build_type=Release --settings:host build_type=Debug --build missing --update
conan user --password ${CI_JOB_TOKEN} --remote gitlab_test ci_user
conan upload "*" --remote gitlab_test --all -c

Logs

Click to expand log
Running with gitlab-runner 17.0.0~pre.88.g761ae5dd (761ae5dd)
  on green-2.saas-linux-small-amd64.runners-manager.gitlab.com/default ns46NMmJ, system ID: s_85d7af184313
Resolving secrets
Preparing the "docker+machine" executor 00:21
Using Docker executor with image registry.gitlab.com/scandyna/docker-images-ubuntu/ubuntu-24.04-build-qt-gcc-13:latest ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.gitlab.com/scandyna/docker-images-ubuntu/ubuntu-24.04-build-qt-gcc-13:latest ...
Using docker image sha256:9500ee95e2e02672198e82ef8f494f55aa34684fdfdc5f1d3dce09eec3345581 for registry.gitlab.com/scandyna/docker-images-ubuntu/ubuntu-24.04-build-qt-gcc-13:latest with digest registry.gitlab.com/scandyna/docker-images-ubuntu/ubuntu-24.04-build-qt-gcc-13@sha256:c4a1519fdb497e23c944c6162e0179a8483233c9daebe5e3475633438dcc5cfb ...
Preparing environment 00:06
Running on runner-ns46nmmj-project-16325699-concurrent-0 via runner-ns46nmmj-s-l-s-amd64-1723369727-4c40f6db...
Getting source from Git repository 00:02
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/scandyna/conan-qt-builds/.git/
Created fresh repository.
Checking out eaf500b1 as detached HEAD (ref is testing)...
Skipping Git submodules setup
$ git remote set-url origin "${CI_REPOSITORY_URL}"
Executing "step_script" stage of the job script 00:06
Using docker image sha256:9500ee95e2e02672198e82ef8f494f55aa34684fdfdc5f1d3dce09eec3345581 for registry.gitlab.com/scandyna/docker-images-ubuntu/ubuntu-24.04-build-qt-gcc-13:latest with digest registry.gitlab.com/scandyna/docker-images-ubuntu/ubuntu-24.04-build-qt-gcc-13@sha256:c4a1519fdb497e23c944c6162e0179a8483233c9daebe5e3475633438dcc5cfb ...
$ conan --version
Conan version 1.64.1
$ conan remote add gitlab_test https://gitlab.com/api/v4/projects/16325699/packages/conan
WARN: Remotes registry file missing, creating default one in /root/.conan/remotes.json
$ conan config install --source-folder conan https://gitlab.com/api/v4/projects/16304913/packages/generic/conan1_config/0.1.8/conan_config.zip
Trying to download  https://gitlab.com/api/v4/projects/16304913/packages/generic/conan1_config/0.1.8/conan_config.zip
Unzipping 19.3KB
Installing settings.yml
Copying file windows_gcc8_x86_64_qt_widgets_modules_boost to /root/.conan/profiles
Copying file windows_gcc8_x86_64_qt_widgets_modules to /root/.conan/profiles
Copying file linux_ubuntu-18.04_clang10_x86_64_libc++_tsan_qt_widgets_modules to /root/.conan/profiles
Copying file linux_gcc8_x86_64_tsan_qt_core to /root/.conan/profiles
Copying file windows_gcc7_x86_64 to /root/.conan/profiles
Copying file linux_clang10_x86_64_libc++ to /root/.conan/profiles
Copying file linux_ubuntu-20.04_clang11_x86_64_libc++_tsan_qt_widgets_modules to /root/.conan/profiles
Copying file linux_gcc8_x86_64_tsan to /root/.conan/profiles
Copying file linux_ubuntu-18.04_clang10_x86_64_libc++_qt_widgets_modules to /root/.conan/profiles
Copying file linux_gcc7_x86_64_tsan to /root/.conan/profiles
Copying file linux_ubuntu-20.04_clang11_x86_64_libc++ to /root/.conan/profiles
Copying file linux_ubuntu-24.04_gcc13_x86_64 to /root/.conan/profiles
Copying file windows_msvc16_x86 to /root/.conan/profiles
Copying file windows_msvc15_x86_64 to /root/.conan/profiles
Copying file boost to /root/.conan/profiles
Copying file windows_gcc8_x86_64 to /root/.conan/profiles
Copying file linux_clang6.0_x86_64_libc++_qt_widgets_modules_boost to /root/.conan/profiles
Copying file windows_msvc16_x86_64 to /root/.conan/profiles
Copying file linux_gcc7_x86_64_qt_core to /root/.conan/profiles
Copying file linux_clang6.0_x86_64_libc++ to /root/.conan/profiles
Copying file linux_gcc7_x86_64 to /root/.conan/profiles
Copying file linux_ubuntu-20.04_clang11_x86_64_libc++_qt_widgets_modules_boost to /root/.conan/profiles
Copying file qt_core to /root/.conan/profiles
Copying file linux_ubuntu-18.04_gcc8_x86_64_qt_widgets_modules_boost to /root/.conan/profiles
Copying file linux_to_windows_gcc7_x86 to /root/.conan/profiles
Copying file linux_ubuntu-18.04_gcc8_x86_64 to /root/.conan/profiles
Copying file windows_msvc16_x86_qt_widgets_modules to /root/.conan/profiles
Copying file linux_clang6.0_x86_64_libc++_tsan_qt_core to /root/.conan/profiles
Copying file qt_gui_modules to /root/.conan/profiles
Copying file linux_ubuntu-24.04_gcc13_x86_64_qt_and_more to /root/.conan/profiles
Copying file qt_widgets_modules to /root/.conan/profiles
Copying file linux_clang6.0_x86_64_libc++_tsan_qt_gui_modules to /root/.conan/profiles
Copying file linux_to_windows_gcc7_x86_64 to /root/.conan/profiles
Copying file linux_clang6.0_x86_64_libc++_tsan to /root/.conan/profiles
Copying file windows_gcc7_x86_qt_widgets_modules to /root/.conan/profiles
Copying file linux_gcc7_x86_64_qt_core_modules to /root/.conan/profiles
Copying file windows_gcc7_x86 to /root/.conan/profiles
Copying file windows_msvc15_x86_64_qt_widgets_modules to /root/.conan/profiles
Copying file linux_clang11_x86_64_libc++ to /root/.conan/profiles
Copying file linux_ubuntu-18.04_clang10_x86_64_libc++_tsan to /root/.conan/profiles
Copying file windows_msvc16_x86_64_qt_widgets_modules_boost to /root/.conan/profiles
Copying file windows_gcc7_x86_64_qt_widgets_modules to /root/.conan/profiles
Copying file linux_clang6.0_x86_64_libc++_tsan_qt_core_modules to /root/.conan/profiles
Copying file linux_clang6.0_x86_64_libstdc++11 to /root/.conan/profiles
Copying file linux_clang6.0_x86_64_libc++_qt_widgets_modules to /root/.conan/profiles
Copying file linux_clang6.0_x86_64_libc++_qt_gui_modules to /root/.conan/profiles
Copying file linux_gcc8_x86_64 to /root/.conan/profiles
Copying file linux_ubuntu-20.04_clang11_x86_64_libc++_qt_widgets_modules to /root/.conan/profiles
Copying file linux_gcc7_x86_64_qt_gui_modules to /root/.conan/profiles
Copying file qt_core_modules to /root/.conan/profiles
Copying file linux_gcc7_x86_64_qt_widgets_modules to /root/.conan/profiles
Copying file common_libraries_options to /root/.conan/profiles
Copying file linux_ubuntu-18.04_gcc8_x86_64_qt_widgets_modules to /root/.conan/profiles
Copying file linux_ubuntu-20.04_clang11_x86_64_libc++_tsan to /root/.conan/profiles
Copying file linux_ubuntu-18.04_clang10_x86_64_libc++ to /root/.conan/profiles
Copying file linux_gcc13_x86_64 to /root/.conan/profiles
Copying file windows_msvc16_x86_64_qt_widgets_modules to /root/.conan/profiles
Copying file linux_clang6.0_x86_64_libc++_tsan_qt_widgets_modules to /root/.conan/profiles
Copying file linux_ubuntu-18.04_clang10_x86_64_libc++_qt_widgets_modules_boost to /root/.conan/profiles
$ conan remote list
conancenter: https://center.conan.io/ [Verify SSL: True]
gitlab_test: https://gitlab.com/api/v4/projects/16325699/packages/conan [Verify SSL: True]
$ conan install ${PACKAGE_NAME}/${PACKAGE_VERSION}@ --profile:build $CONAN_PROFILE_BUILD --profile:host $CONAN_PROFILE_HOST --settings:build build_type=Release --settings:host build_type=$BUILD_TYPE --build missing --update
WARN: *** Conan 1 is legacy and on a deprecation path ***
WARN: *** Please upgrade to Conan 2 ***
Auto detecting your dev setup to initialize the default profile (/root/.conan/profiles/default)
Found gcc 13
gcc>=5, using the major as version
************************* WARNING: GCC OLD ABI COMPATIBILITY ***********************
 
Conan detected a GCC version > 5 but has adjusted the 'compiler.libcxx' setting to
'libstdc++' for backwards compatibility.
Your compiler is likely using the new CXX11 ABI by default (libstdc++11).
If you want Conan to use the new ABI for the default profile, run:
    $ conan profile update settings.compiler.libcxx=libstdc++11 default
Or edit '/root/.conan/profiles/default' and set compiler.libcxx=libstdc++11
************************************************************************************
Default settings
	os=Linux
	os_build=Linux
	arch=x86_64
	arch_build=x86_64
	compiler=gcc
	compiler.version=13
	compiler.libcxx=libstdc++
	build_type=Release
*** You can change them in /root/.conan/profiles/default ***
*** Or override with -s compiler='other' -s ...s***
Configuration (profile_host):
[settings]
arch=x86_64
build_type=Debug
compiler=gcc
compiler.cppstd=17
compiler.libcxx=libstdc++11
compiler.version=13
os=Linux
os.distribution=ubuntu-24.04
[options]
boost:header_only=True
icu:shared=False
libalsa:shared=True
qt:gui=True
qt:qtserialport=True
qt:shared=True
qt:widgets=True
qt:with_glib=False
qt:with_mysql=False
qt:with_odbc=True
qt:with_pq=True
qt:with_sqlite3=True
qt:with_vulkan=False
zlib:shared=False
[build_requires]
[env]
[buildenv]
CC=gcc-13
CXX=g++-13
Configuration (profile_build):
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=17
compiler.libcxx=libstdc++11
compiler.version=13
os=Linux
[options]
[build_requires]
[env]
[buildenv]
CC=gcc-13
CXX=g++-13
libpq/15.4: Not found in local cache, looking in remotes...
libpq/15.4: Trying with 'conancenter'...
Downloading conanmanifest.txt
Downloading conanfile.py
Downloading conan_export.tgz
libpq/15.4: Downloaded recipe revision a3354d556287c3d091c9f4d578b66ec8
Installing package: libpq/15.4
Requirements
    libpq/15.4 from 'conancenter' - Downloaded
Packages
    libpq/15.4:f35a0224260f0d753acb9d90beb1f1c501d0d02b - Download
Installing (downloading, building) binaries...
libpq/15.4: Retrieving package f35a0224260f0d753acb9d90beb1f1c501d0d02b from remote 'gitlab_test' 
Downloading conanmanifest.txt
Downloading conaninfo.txt
Downloading conan_package.tgz
libpq/15.4: Package installed f35a0224260f0d753acb9d90beb1f1c501d0d02b
libpq/15.4: Downloaded package revision 0
Aggregating env generators
$ conan user --password ${CI_JOB_TOKEN} --remote gitlab_test ci_user
Changed user of remote 'gitlab_test' from 'None' (anonymous) to 'ci_user'
$ conan upload "*" --remote gitlab_test --all -c
Uploading to remote 'gitlab_test':
Uploading libpq/15.4 to remote 'gitlab_test'                                             
Downloading conan_sources.tgz
Recipe is up to date, upload skipped
Uploading package 1/1: f35a0224260f0d753acb9d90beb1f1c501d0d02b to 'gitlab_test'         
ERROR: libpq/15.4:f35a0224260f0d753acb9d90beb1f1c501d0d02b: Upload package to 'gitlab_test' failed: The file is a broken symlink, verify that you are packaging the needed destination files: '/root/.conan/data/libpq/15.4/_/_/package/f35a0224260f0d753acb9d90beb1f1c501d0d02b/include/catalog/pg_foreign_data_wrapper_d.h'.You can skip this check adjusting the 'general.skip_broken_symlinks_check' at the conan.conf file.
ERROR: Errors uploading some packages
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit code 1
@scandyna scandyna added the bug Something isn't working label Aug 11, 2024
@perseoGI perseoGI self-assigned this Aug 14, 2024
@scandyna
Copy link
Author

I have downloaded the postgresql sources that the recipe takes, and tried to build it by steps on an Ubuntu-18.04 with gcc.
Note: I did out of source build, like documented here:
https://www.postgresql.org/docs/15/install-procedure.html

The various make install that are done in the conanfile.py does not bring symlinks in the install path.
It also not installs the catalog header files.

In the source tree, in src/include/catalog, we have headers like pg_aggregate.h.

In the build tree, in src/include/catalog, there are symlinks, like pg_aggregate_d.h, with an absolute path to the source tree.

In PR 24755, line 247-248 have been fixed to copy the catalog from the source tree:

copy(self, "*.h", src=os.path.join(self.source_folder, "src", "include", "catalog"),
                  dst=os.path.join(self.package_folder, "include", "catalog"))

Because we do in source build, I think that source_folder and build_folder are the same.
This will bring the broken symlinks to the package.

In the build() method, configure and make should be called from the build tree:

def build(self):
  ...
  autotools = Autotools(self)
    with chdir(self, os.path.join(self.build_folder)):
      autotools.configure()
    with chdir(self, os.path.join(self.build_folder, "src", "backend")):
      autotools.make(target="generated-headers")
  ...

The same should be done for the package() method:

def package(self):
  ...
  autotools = Autotools(self)
    with chdir(self, os.path.join(self.build_folder, "src", "common")):
      autotools.install()
  ...

Now, the package will not have the broken symlinks anymore.
The headers like pg_aggregate.h will be present in include/catalog,
but headers like pg_aggregate_d.h are missing.

This is because they are not in the build tree in src/backend/catalog (only a symlink to a Makefile is present in my case).

In line 254-255, the catalog should also be copied from the source tree:

copy(self, "*.h", src=os.path.join(self.source_folder, "src", "backend", "catalog"),
                            dst=os.path.join(self.package_folder, "include", "catalog"))

I did try to update the recipe locally, and it works (builds, no symlinks and catalog files present, test package works).
Because this seems to be an important change, I think I should test it with various configurations. I also completely ignored the MSVC part.

scandyna added a commit to scandyna/conan-center-index that referenced this issue Sep 22, 2024
To avoid broken symlinks, do out of source build.
Change `build()` and `package()` method to do out of source build.
Copy the backend catalog from the source tree.
Note: the `MSVC` part is ignored in this commit.
@scandyna
Copy link
Author

I work on this on my fork.

Tests are here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants