run edenfs integration tests in github CI #14
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ahornby
force-pushed
the
eden_oss_tests
branch
2 times, most recently
from
68e3a15
to
f934869
Compare
Summary: Connect up the sapling *.t tests so that github CI has visible cli test status. To get them to run: * some needed fbpython on path. I included a shim for that in the test makefile target. * test-rust-hooks: Command not found message, added a glob * test-identity.t: add a glob for the sapling version * test-eolfilename.t: output order was unstable, added sorts to make stable * helpers-testrepo.sh: fix assumption that system hg would be able to read test repo, check if its Sapling first. * added a manifest for the hexdump utility some of the tests required * excluded a few remaining tests (see comments in Makefile for reason) * fixed getdeps support for generating actions steps for test only dependencies NB the tests run as "hg". The expectations would need to be updated if we were to run as "sl" Test Plan: enter ubuntu 22.04 toolbox: `toolbox enter ubuntu-toolbox-22.04 ` make sure system packages are installed `./build/fbcode_builder/getdeps.py install-system-deps --recursive sapling` build sapling (and any depedencieds it requires) `./build/fbcode_builder/getdeps.py build --allow-system-packages --src-dir=. sapling` run the tests. 64 concurrent jobs worked well on my 64GB machine and took 2m10s. ``` ./build/fbcode_builder/getdeps.py test --allow-system-packages --src-dir=. sapling --num-jobs=64 ---------------------------------------------------------------------- Skipped 2 tests (missing feature: Mononoke server available): test-fb-ext-remotefilelog-log.t test-fb-ext-remotefilelog-worker.t Skipped 3 tests (missing feature: running tests with fsmonitor): test-fsmonitor-filemerge.t test-fsmonitor-nonutf8-path.t test-rust-hgevents.t Skipped 46 tests (skipped): test-atexit-epipe.t test-casefolding.t test-check-clang-format.t test-check-code.t test-check-fix-code.t test-check-help.t test-checkserverbookmark.t test-chg.t test-debugbacktrace.t test-debugshell-namespace.t test-eden-clone.t test-eden-commands.t test-eden-du.t test-eden-glob.t test-eden-sparse.t test-eden-watchman-edenapi-glob.t test-eden-watchman-noedenapi-glob.t test-fb-ext-fbconduit.t test-fb-ext-scm-prompt-compat.t test-fb-ext-scm-prompt-git.t test-fb-ext-tweakdefaults-grep.t test-fsmonitor-casefolding.t test-fsmonitor-fallback.t test-fsmonitor-metadata.t test-fsmonitor-refreshclock.t test-fsmonitor-status.t test-fsmonitor-warn-fresh.t test-histedit-reorder.t test-matcher-expand-globs.t test-mutation-infinitepush.t test-non-working-symlink.t test-progress-rust-renderer.t test-progressfile.t test-purge-fsmonitor.t test-remotenames-journal.t test-runlog.t test-rust-status-in-transaction.t test-sparse-track-ignored.t test-status-case-insensitive.t test-status-fresh-instance.t test-status-root-ignored-py.t test-status-watchman.t test-symlink-migration.t test-tracing-under-chg.t test-treestate-fresh-instance.t test-treestate-trackignore.t # Ran 729 tests, 51 skipped, 0 failed. passed on try 0 ``` generate github actions CI (using 16 concurrent jobs to speed them up a little) ``` ./build/fbcode_builder/getdeps.py --allow-system-packages generate-github-actions --free-up-disk --os-type=linux --src-dir=. --output-dir=.github/workflows --job-name "Sapling CLI Getdeps " --job-file-prefix=sapling-cli-getdeps_ sapling --project-install-prefix sapling:/ --num-jobs=16 ``` debugged actions locally with: ``` act -r -j build -W .github/workflows/sapling-cli-getdeps_linux.yml ```
Summary: Ubuntu 24.04 and Fedora 40 are both python 3.12 based. Update sapling's OSS getdeps build to allow build/run/test on python 3.12 while keeping 3.10 support. * distutils is deprecated in python 3.12 stdlib, but fortunately setuptools is pretty similar and includes a vendored distutils, so updated to use setuptools's version and removed deprecated usage like `find_executable` in favor of `shutil.which()`. * added a manifest so we get python3-setuptools installed. * found some tests needed bunzip2, so update manifest for sapling and bz2 to install it * util.py makedate(): datetime.utcfromtimestamp is deprecated in 3.12, so updated to use non-deprecated methods.. Tested with test-command-template.t which shows tz offset. * Makefile: disabled a few tests on 3.12 where updating the expectation to run on 3.12 and 3.10 is tricky * tests modified to run on 3.12 and 3.10: * test-import-eol.t: updated for invalid escape sequence warnings * test-install.t: updated to filter out message that appears in different order on 3.12 vs earlier versions * test-sign-commit.t: updated to add --yes for newer gpg that python 3.12 using distros have Test Plan: enter ubuntu 24.04 toolbox: `toolbox enter ubuntu-toolbox-24.04 ` make sure system packages are installed `./build/fbcode_builder/getdeps.py install-system-deps --recursive sapling` check oss non-thrift build: ``` make oss $ ./sl --version Fri Oct 4 17:14:19 BST 2024 Sapling 4.4.2_20241004_170203_06bd09bd1dd2 (see https://sapling-scm.com/ for more information) ``` build sapling getdepts `./build/fbcode_builder/getdeps.py build --allow-system-packages --no-facebook-internal --src-dir=. sapling --project-install-prefix sapling:/` run the tests locally to validate python 3.12 operation: ``` ./build/fbcode_builder/getdeps.py --allow-system-packages test --src-dir=. sapling --num-jobs=64 --project-install-prefix sapling:/' ``` generate github actions CI to pick up python3-setuptools ``` ./build/fbcode_builder/getdeps.py --allow-system-packages generate-github-actions --free-up-disk --os-type=linux --src-dir=. --output-dir=.github/workflows --job-name "Sapling CLI Getdeps " --job-file-prefix=sapling-cli-getdeps_ sapling --num-jobs=16 --project-install-prefix sapling:/ ``` github CI
ahornby
force-pushed
the
eden_oss_tests
branch
from
f934869
to
1221861
Compare
Summary: Now that we have sapling manifest we can start to run EdenFS tests in github CI. To get them working: * add an OSS stub for missing import of cli.py par_telemetry which was failing import. * eden tests segfaulted on ubuntu 22.04 and 24.04 with CXX_STANDARD set to 17, due to g++ not enablng coroutines by default with C++17. Enabled them Regenerated CI actions with: ``` ./build/fbcode_builder/getdeps.py --allow-system-packages generate-github-actions --free-up-disk --os-type=linux --ubuntu-version=24.04 --src-dir=. --output-dir=.github/workflows --job-name "EdenFS " --job-file-prefix=edenfs_ eden --num-jobs=8 --project-install-prefix sapling:/usr/local ``` Test Plan: enter toolbox: ``` toolbox enter ubuntu-toolbox-24.04 ``` build sapling and eden: `` ./build/fbcode_builder/getdeps.py install-system-deps --recursive eden ./build/fbcode_builder/getdeps.py build --allow-system-packages --src-dir=. sapling ./build/fbcode_builder/getdeps.py build --allow-system-packages --src-dir=. eden ``` run eden tests locally: ``` ./build/fbcode_builder/getdeps.py test --allow-system-packages --num-jobs=8 --src-dir=. eden ```
ahornby
force-pushed
the
eden_oss_tests
branch
from
1221861
to
145684e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Now that we have sapling manifest we can start to run EdenFS tests in github CI
Added a stub for missing import of cli.py par_telemetry which was failing tests.
Fixed getdeps support for test only dependencies in github action generation, then regenerated CI actions with:
Test Plan:
enter toolbox:
build sapling and eden:
``
./build/fbcode_builder/getdeps.py install-system-deps --recursive eden ./build/fbcode_builder/getdeps.py build --allow-system-packages --no-facebook-internal --src-dir=. sapling ./build/fbcode_builder/getdeps.py build --allow-system-packages --no-facebook-internal --src-dir=. eden
./build/fbcode_builder/getdeps.py test --allow-system-packages --no-facebook-internal --num-jobs=8 --src-dir=. eden