Build script improvements #149
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
These were needed for the cmake install testing which is disabled. We will add a separate job for testing the install.
… on action runner.
This is no longer needed because the build scripts do it automatically.
This reverts commit 2cbc501.
|
@senior-zero @trxcllnt I could use your 👀 on this to let me know if this setup with using devcontainer specific build directories is going to be compatible with clangd. |
1 task
miscco
approved these changes
Jun 30, 2023
|
@trxcllnt @senior-zero my thinking with this is that we can add a config to the devcontainer.json like this: |
ci/build_common.sh
Outdated
| @@ -23,11 +23,44 @@ readonly GPU_ARCHS=$(echo $3 | tr ' ,' ';') | |||
|
|
|||
| readonly PARALLEL_LEVEL=${PARALLEL_LEVEL:=$(nproc)} | |||
|
|
|||
| BUILD_DIR=${BUILD_DIR:-../build/${DEVCONTAINER_NAME:-}} | |||
There was a problem hiding this comment.
If no DEVCONTAINER_NAME I would go with local or unknown instead of nothing. That way all builds are in subdirectories of build/ and you don't have this edge case
There was a problem hiding this comment.
I've updated it so building outside a container will build to cccl/build/local. Furthermore, I've also made it so the most recent build will always be symlinked to cccl/build/latest so there is a consistent entry point to the most recent build.
Make it so builds outside of devcontainer build to build/local. Also, always make it so the most recent build is symlinked to build/latest.
jrhemstad
added a commit
that referenced
this pull request
Jul 11, 2023
* Simplify some genexes and update compiler ID where needed. * Remove unnecessary NVC++ tweaks. * Add guards against unsupported arch options. * Enable PIC for RDC targets. * Make os in matrix more general. * Update os docs. * Test build-and-test with windows support. * Syntax * Factor into env. * Update to use format(). * Make the whole container line optionally empty. * All in format(). * format * Don't specify option. * test specifying workflow as input. * quotes * `<complex>` Enable users to disable normalization for division too. (#474) * `<complex>` Enable users to disable normalization for division too. Also add some documentation for those. * Bump CUB. * Can't dynamically specify "uses:" * Set image name in env variable. * Try specifying options even if image is empty. * Remove stale "workflow:" input * Actually remove the workflow: input. * Don't change the name of the Thrust job. * Enable thrust test job. * Don't update git. * Remove windows from the matrix for now. * Update git version * Get rid of outdated apt package. * Add NVIDIA_VISIBLE_DEVICES env to test job container. * Update more CUDA compiler id checks. * Move dev build toggle to the cccl project level. This allows the install rules to be generated for all projects even when the dev builds are disabled. * Turn on install rules for everything. * Introduce SM90 tuning policy into scan * Factor common logic out of build_thrust.sh * Add CUB build/test and enable in matrix. * Remove install tests from Thrust/CUB. These are now handled at the CCCL level, using newer and more robust techniques. * Initial build system updates for monorepo. * Update CMake option docs. * Allow TOPLEVEL_PROJECT logic to be force enabled. * Version lock Thrust to other CCCL components. * Remove leftover RDC flag tomfoolery. * Remove temporary warning suppression. * Add `LANGUAGES` keyword to CMake project calls. * Simplify some genexes and update compiler ID where needed. * Remove unnecessary NVC++ tweaks. * Add guards against unsupported arch options. * Turn on PIC for RDC targets. * Update more CUDA compiler id checks. * Move dev build toggle to the cccl project level. This allows the install rules to be generated for all projects even when the dev builds are disabled. * Remove install tests from Thrust/CUB. These are now handled at the CCCL level, using newer and more robust techniques. * Rebase submodules on upstream main branches. * only build for sm70 for now * Add libcu++ test script and enable libcu++ test jobs. * Update how RDC flags are applied to Catch2 utilities. * Add a step to calculate sccache hit rate. * Update sccache script to print file contents. * Add sccache to job summary. * Fix writing the hit rate to job summary. * Remove set -x to only write hit_rate to stderr * Add final "CI" job that can be used as status check. * Delete old workflow files. * Delete old CI scripts. * Remove devcontainers from this branch. * Add devcontainers. * Make thrust/cub tests timeout after 15 seconds. * Workaround three-way partition compilation issue * Fix link to std::binary_semaphore documentation (#486) Previous link points at empty page as cppreference documentation was merged with the page for std::counting_semaphore. * Fix reduce by key tile state for Pascal * Tune select and partition for SM90 * Refactor SM90 tuning * Update permissions according to AJ's suggestion. * Add .cache to gitignore. * Add overview of CI pipeline. * Add mermaid diagram. * Better diagram * Update readme. * Fix link. * Add links to matrix file. * Tune RLE for SM90 * Pull long constraint out of template header We have nvbug4019931 where the large number of constraints coupled with fold expressions in the remplate header causes nvcc to reject this strange albeit valid code. To work around this, we pull it out into a separate concept definition. * Rename `test::input_iterator` to `test::cpp17_input_iterator` * Implement C++20 `default_sentinel` * Fix warp collectives docs to require syncwarp * Include tuning docs into doc tree * Modernize warp load docs * Format warp load * Modernize warp merge sort docs * Format warp merge sort * Modernize warp reduce docs * Format warp reduce * Update CI doc and link to it from readme. * Fix bullet indentation. * Modernize warp scan docs * Format warp scan * Expand build/test. * Add examples. * Modernize warp store docs * Format warp store * Ensure that any CMake re-rooting doesn't break our find_file. * Echo command to clone PR on failure. * Factor image name into environment variable. * Pass build and test job images as inputs. * Fix notice syntax. * Update CUDA11.0 to 11.1 in matrix. * Explicitly check exit code from script. * Update to exit with the exit code on error. * Add repro instructions. * Update repro instructions. * Update clone instruction to recurse submodules. * Fix building with gcc-6 and make the test suite pass (#485) * Add gcc-6 image * Avoid build breaks due to gcc-6 * Disable failing libcxx tests for gcc-6 * Make tests pass with gcc-6 and c++14 * Make tests pass with gcc-6 and c++17 * Don't use host networking. * Disable test jobs for now. * Update devcontainers. * Update devcontainers. * Add launch script and repro instructions. * Update repro message. * Add repro instructions. * Add libcu++ build job. * typo * Update the libcu++ submodule * Disable failing test for gcc7 as that is broken (#493) * Initial commit of new build system for monorepo. * Restore semantic version checks. * Allow TOPLEVEL_PROJECT logic to be force enabled. * Bump version to 2.2.0 to match Thrust/CUB/CCCL. * Leave `LIBCUDACXX_COMPUTE_ARCHS` alone. This should not be set to `CMAKE_CUDA_ARCHITECTURES`. * Fix some paths. * Remove `CMAKE_CUDA_HOST_COMPILER` handling. This lets us remove a bunch of hacky work-arounds since changing that variable is fragile. * Remove some NVC++ hacks. * Add a special case for version 1 -> 2 compat checks. * Add `LANGUAGES` keyword to CMake project calls. * Update more CUDA compiler ids. * Move dev build toggle to the cccl project level. This allows the install rules to be generated for all projects even when the dev builds are disabled. * Support the `no_execute` keyword for lit tests * For `no_execute` check whether the test should have failed at runtime * Fix broken deprecated attribute for old nvcc * Add fix fo gcc-7 * Initial port to new monorepo build system. * Update CMake option docs. * Allow TOPLEVEL_PROJECT logic to be force enabled. * Version lock CUB with other CCCL libraries. * Remove temporary warning suppression. * Add `LANGUAGES` keyword to CMake project calls. * Simplify some genexes and update compiler ID where needed. * Remove unnecessary NVC++ tweaks. * Add guards against unsupported arch options. * Enable PIC for RDC targets. * Update more CUDA compiler ID checks. * Move dev build toggle to the cccl project level. This allows the install rules to be generated for all projects even when the dev builds are disabled. * Remove install tests from Thrust/CUB. These are now handled at the CCCL level, using newer and more robust techniques. * Update how RDC settings are applied to Catch2 utilities. * Update CUB to prepare for repo unification. * Prepare libcudacxx for CCCL monorepo. * Prepare CUB for CCCL monorepo. * Prepare Thrust for CCCL monorepo. * Remove subproject submodules to prepare for repo merge. * Fix use of uninitialized data in histogram * WAR GNU style of line directive * Make Catch2 runner a C++ target * Fix ambiguous comparisons for C++20 * Fix missing return statement for C++17 * Remove submodules from CI workflow (#115) * Remove recursive checkout. * Remove updating git version. * WAR GNU style of line directive * Missing new line * Missing copyright * try and fix common type for old CTK * Disable use of certain builtins for old nvcc We actually check whether the builtin is available, but it seems that this is not enough * Fix async scan / counting iterator tests * Make sccache work locally (#113) * Remove submodule stuff from workflow. * Update sccache bucket to sccache-devs * Update workflow to move source files to coder home. * Fix path to composite action for sccache. * Fix path. * Use composite action before moving cloned repo. * Reference composite action from moved dir. * Copy instead of move. * Reference cccl dir. * Fix path for sccache hit rate script. * Move sccache stats after cd to home dir. * Don't write to step summary. * cd to coder home dir for sccache script. * Debug sccache hit rate step. * Debug * cat sccache files. * Update sccache hitrate script to write output to stdout. * Use notice workflow command for hit rate. * Get rid of sccache debug stuff. * Write the no new compile requests to stdout. * Write hit rate to step summary. * Wrap hitrate script stderr output in collapsed group. * Don't write to step summary as we don't have permission as the coder user. * Fix redirection. * Fix end group. * Make hit rate script more robust to allow spaces at start. * Debug. * Write awk output. * Quotes around file paths. * Use old style regex. awk on Ubuntu 18.04 doesn't support [[:space:]] or [[:digit:]] * Fix cub::CachingDeviceAllocator * Actually build the whole libcu++ tests * Tune scan by key for SM90 * Visualize variant space coverage * Benchmark batched memcpy * Extract warp and block level threshold into tuning * Extract look-back tuning * Use macro to suppress warnings Older nvcc does not know `nv_diag_supppress` but only `diag_suppress` so special case it * Fix `_LIBCUDACXX_UNREACHABLE` for old nvcc * Do not use constexpr globals on device Old nvcc does not know constexpr variables on device, so move to `const` in those cases * Do not use `<bit>` related builtins for older nvcc * Do not use nodiscard with a friend comparison operator Move to a member function * Do not use `__builtin_assume` with old nvcc * Supress warnings from `pipeline` tests * Work around issues with `expected` for older nvcc In particular, constrained friend functions are critical * Old nvcc does not like how `construct_at` is defined The constrain breaks in earlier standard modes, so as a workaround remove the constraint from our internal `__construct_at` prior to c++20 and switch back to the proper `construct_at` with c++20 onwards * Disable broken tests for new features * Do not rely on builtins that are not host device for complex * Restrict usage of `no_unique_address` to slightly more modern nvcc * Make `mdspan` work with older nvcc and clang * Make `expected` work with older nvcc and clang * Make `optional` compile with old clang and nvcc * Cleanup NVCC version filtering * Properly qualify `forward_as_tuple` inside `tuple_cat` Fixes [BUG]: forward_as_tuple from cuda/std/tuple clashes with one from tuple if std:: type is inside cuda::std::tuple #142 * Add pair plot * WAR nvbench helper lib issue * Remove extra headers * Randomize input * Add `sccache` to lit (#147) * Properly querry `CMAKE_CUDA_COMPILER_LAUNCHER` for ccache support (#152) * Introduce look-back tuning in three-way partition * Three-way partition benchmark * Port three-way partition to use Catch2 * Add gcc-6 to the test matrix * Improve implementation of `destructible` We did not apply this for old gcc compilers, but it turns out that they will fall over more complex ranges concepts later. We can alleviate this by reusing `is_destructible` and generally clean the implementation up. * Implement new traits This implements the updates to iterator_traits as well as the new readable_traits and incrementable_traits. It further adds the related iter_value_t and iter_difference_t aliases * Tune reduce by key on H100 * Tune unique by key on H100 * Remove unused code (#145) We never run the original libcxx benchmarks nor do we use our old docker images * Fix documentation of `atomic_ref` * Build script improvements (#149) * Add DEVCONTAINER_NAME env variable to devcontainers. * Remove enabling install rules. These were needed for the cmake install testing which is disabled. We will add a separate job for testing the install. * Factor out common build/configure logic. * Add compile commands to common cmake options. * Update sccache_stats to selectively add "notfice" prefix when running on action runner. * Remove manual sccache stats calculation from workflow. This is no longer needed because the build scripts do it automatically. * Remove devcontainer name env variable. * Revert "Remove devcontainer name env variable." This reverts commit 2cbc501. * Use devcontainer specific build directory. * Add build/latest. Make it so builds outside of devcontainer build to build/local. Also, always make it so the most recent build is symlinked to build/latest. * Fix icpc / denormals * Decrypted no-ftz / nodaz options * Enable tests (#167) * Add DEVCONTAINER_NAME env variable to devcontainers. * Remove enabling install rules. These were needed for the cmake install testing which is disabled. We will add a separate job for testing the install. * Factor out common build/configure logic. * Add compile commands to common cmake options. * Update sccache_stats to selectively add "notfice" prefix when running on action runner. * Remove manual sccache stats calculation from workflow. This is no longer needed because the build scripts do it automatically. * Remove devcontainer name env variable. * Revert "Remove devcontainer name env variable." This reverts commit 2cbc501. * Use devcontainer specific build directory. * Get rid of stale Windows runs-on handling. * Make test job conditional on new input. * Add simple matrix for test. * Enable Thrust test script. * Update thrust test script to use build_common. * Remove redundant build job. * Run nvidia-smi step in test job. * Update CUB test script. * Update libcu++ test script. * Enable CUB and libcu++ test jobs. * Hack to disable verbose CUB test output. * Exclude device_radix_sort CUB tests for now. * Update matrix to full matrix with tests on latest host compiler. * Run tests for all C++ dialects on latest host compiler. * Limit libcu++ tests parallel execution. * Add back set +x to build_common * Disable Thrust and CUB jobs for now. * Hardcode 8 workers for libcu++ tests. * Re-enable Thrust/CUB builds and tests. * Add build/latest. Make it so builds outside of devcontainer build to build/local. Also, always make it so the most recent build is symlinked to build/latest. --------- Co-authored-by: Jake Hemstad <[email protected]> Co-authored-by: Michael Schellenberger Costa <[email protected]> Co-authored-by: Georgy Evtushenko <[email protected]> Co-authored-by: pauleonix <[email protected]> Co-authored-by: Robert Maynard <[email protected]> Co-authored-by: Wesley Maxey <[email protected]>
1 task
alliepiper
pushed a commit
to alliepiper/cccl
that referenced
this pull request
Mar 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #148
Description
Updates the build scripts to:
In order to avoid cmake cache conflicts across devcontainers, I updated the build scripts to use a devcontainer-specific build directory of
build/DEVCONTAINER_NAME, e.g.,build/cuda12.1-gcc7.When running outside of a devcontainer (meaning
DEVCONTAINER_NAMEis not set), then it will build tobuild/local.Every time a build is initiated (either inside or out of a devcontainer), it will create a symlink of the current build directory to
build/latest.Checklist