-
Notifications
You must be signed in to change notification settings - Fork 280
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
restore mononoke getdeps integration tests (#951)
Summary: X-link: facebookincubator/zstrong#995 Bring back mononoke getdeps integration tests. This is a refesh of the previously working export-D34186407 branch from 2022 Main changes since: * depend on restored sapling manifest from previous commit * bring back selected manifests needed by mononoke that were deleted in D51869247. I added the sqlite binary packages from this to the sqlite3 manifest rather than bring back the sqllite3-bin manifest * add manifests for new tools used: ripgrep, git-lfs, and zstd cli (we already had the zstd libs) * fix a few test expecations that where too closely tied to git cli or TLS version etc (ubuntu 22.04 is on older version) * getdeps MakefileBuilder.run_tests() improvements * fix error status reporting, was not failing if tests failed * pass of --num-jobs to tests to stop it OOMing my machine * pass of --filter to tests so can iterate on one test more easily Can to iterate local execution for one test with: ``` python3 ./build/fbcode_builder/getdeps.py test --num-jobs 4 --allow-system-packages --no-facebook-internal --src-dir=. mononoke_integration --retry 0 --filter server/test-gettreepack.t ``` Pull Request resolved: #951 Test Plan: Setup a local ubuntu 22.04 toolbox with: ``` toolbox create --distro ubuntu --release 22.04 toolbox enter ubuntu-toolbox-22.04 sudo apt update sudo apt install g++ libssl-dev pkg-config sudo locale-gen en_US.UTF-8 curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ``` Build dependencies: ``` python3 ./build/fbcode_builder/getdeps.py install-system-deps --recursive mononoke_integration python3 ./build/fbcode_builder/getdeps.py build --allow-system-packages --no-facebook-internal --src-dir=. --no-tests sapling python3 ./build/fbcode_builder/getdeps.py build --allow-system-packages --no-facebook-internal --src-dir=. --no-tests mononoke ``` then test locally: ``` ./build/fbcode_builder/getdeps.py build --no-deps --num-jobs 16 --allow-system-packages --no-facebook-internal --src-dir=. mononoke_integration ./build/fbcode_builder/getdeps.py test --num-jobs 6 --allow-system-packages --no-facebook-internal --src-dir=. mononoke_integration ``` Generate the github actions yaml ``` ./build/fbcode_builder/getdeps.py generate-github-actions --allow-system-packages --test --os-type linux --job-file-prefix mononoke-integration_ --job-name-prefix "Mononoke Integration " --output-dir .github/workflows mononoke_integration --num-jobs 4 ``` --- [//]: # (BEGIN SAPLING FOOTER) Stack created with [Sapling](https://sapling-scm.com). Best reviewed with [ReviewStack](https://reviewstack.dev/facebook/sapling/pull/951). * __->__ #951 * #950 Reviewed By: quark-zju Differential Revision: D62978526 Pulled By: ahornby fbshipit-source-id: 0070a67d798bb23ee9e78e1a5149ba5364d548c9
- Loading branch information
1 parent
6eedd75
commit d6daa83
Showing
30 changed files
with
904 additions
and
23 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,214 @@ | ||
# This file was @generated by getdeps.py | ||
|
||
name: Mononoke Integration Linux | ||
|
||
on: | ||
push: | ||
branches: | ||
- main | ||
pull_request: | ||
branches: | ||
- main | ||
|
||
permissions: | ||
contents: read # to fetch code (actions/checkout) | ||
|
||
jobs: | ||
build: | ||
runs-on: ubuntu-22.04 | ||
steps: | ||
- uses: actions/checkout@v4 | ||
- name: Update system package info | ||
run: sudo apt-get update | ||
- name: Install system deps | ||
run: sudo python3 build/fbcode_builder/getdeps.py --allow-system-packages install-system-deps --recursive mononoke_integration | ||
- name: Install packaging system deps | ||
run: sudo python3 build/fbcode_builder/getdeps.py --allow-system-packages install-system-deps --recursive patchelf | ||
- name: Install Rust Stable | ||
uses: dtolnay/rust-toolchain@stable | ||
- name: Fetch git-lfs | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests git-lfs | ||
- name: Fetch ripgrep | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests ripgrep | ||
- name: Fetch tree | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests tree | ||
- name: Fetch bz2 | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests bz2 | ||
- name: Fetch ninja | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests ninja | ||
- name: Fetch cmake | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests cmake | ||
- name: Fetch zstd | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests zstd | ||
- name: Fetch python-click | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests python-click | ||
- name: Fetch sqlite3 | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests sqlite3 | ||
- name: Fetch gflags | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests gflags | ||
- name: Fetch glog | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests glog | ||
- name: Fetch fmt | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests fmt | ||
- name: Fetch googletest | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests googletest | ||
- name: Fetch python-six | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests python-six | ||
- name: Fetch autoconf | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests autoconf | ||
- name: Fetch automake | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests automake | ||
- name: Fetch libtool | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests libtool | ||
- name: Fetch jq | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests jq | ||
- name: Fetch libsodium | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests libsodium | ||
- name: Fetch boost | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests boost | ||
- name: Fetch double-conversion | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests double-conversion | ||
- name: Fetch fast_float | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests fast_float | ||
- name: Fetch libdwarf | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests libdwarf | ||
- name: Fetch libevent | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests libevent | ||
- name: Fetch libunwind | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests libunwind | ||
- name: Fetch lz4 | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests lz4 | ||
- name: Fetch snappy | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests snappy | ||
- name: Fetch xz | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests xz | ||
- name: Fetch zlib | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests zlib | ||
- name: Fetch libiberty | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests libiberty | ||
- name: Fetch folly | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests folly | ||
- name: Fetch libffi | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests libffi | ||
- name: Fetch ncurses | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests ncurses | ||
- name: Fetch python | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests python | ||
- name: Fetch nmap | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests nmap | ||
- name: Fetch openssl | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests openssl | ||
- name: Fetch liboqs | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests liboqs | ||
- name: Fetch fizz | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests fizz | ||
- name: Fetch mvfst | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests mvfst | ||
- name: Fetch wangle | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests wangle | ||
- name: Fetch fbthrift | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests fbthrift | ||
- name: Fetch fb303 | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests fb303 | ||
- name: Fetch rust-shed | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests rust-shed | ||
- name: Build git-lfs | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests git-lfs | ||
- name: Build ripgrep | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests ripgrep | ||
- name: Build tree | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests tree | ||
- name: Build bz2 | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests bz2 | ||
- name: Build ninja | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests ninja | ||
- name: Build cmake | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests cmake | ||
- name: Build zstd | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests zstd | ||
- name: Build python-click | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests python-click | ||
- name: Build sqlite3 | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests sqlite3 | ||
- name: Build gflags | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests gflags | ||
- name: Build glog | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests glog | ||
- name: Build fmt | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests fmt | ||
- name: Build googletest | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests googletest | ||
- name: Build python-six | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests python-six | ||
- name: Build autoconf | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests autoconf | ||
- name: Build automake | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests automake | ||
- name: Build libtool | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests libtool | ||
- name: Build jq | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests jq | ||
- name: Build libsodium | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests libsodium | ||
- name: Build boost | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests boost | ||
- name: Build double-conversion | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests double-conversion | ||
- name: Build fast_float | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests fast_float | ||
- name: Build libdwarf | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests libdwarf | ||
- name: Build libevent | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests libevent | ||
- name: Build libunwind | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests libunwind | ||
- name: Build lz4 | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests lz4 | ||
- name: Build snappy | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests snappy | ||
- name: Build xz | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests xz | ||
- name: Build zlib | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests zlib | ||
- name: Build libiberty | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests libiberty | ||
- name: Build folly | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests folly | ||
- name: Build libffi | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests libffi | ||
- name: Build ncurses | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests ncurses | ||
- name: Build python | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests python | ||
- name: Build nmap | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests nmap | ||
- name: Build openssl | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests openssl | ||
- name: Build liboqs | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests liboqs | ||
- name: Build fizz | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests fizz | ||
- name: Build mvfst | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests mvfst | ||
- name: Build wangle | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests wangle | ||
- name: Build fbthrift | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests fbthrift | ||
- name: Build fb303 | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests fb303 | ||
- name: Build rust-shed | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests rust-shed | ||
- name: Build mononoke | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --src-dir=. --no-tests mononoke | ||
- name: Build sapling | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --src-dir=. --no-tests sapling | ||
- name: Build mononoke_integration | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-deps --src-dir=. mononoke_integration --project-install-prefix mononoke_integration:/usr/local | ||
- name: Copy artifacts | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fixup-dyn-deps --strip --src-dir=. mononoke_integration _artifacts/linux --project-install-prefix mononoke_integration:/usr/local --final-install-prefix /usr/local | ||
- uses: actions/upload-artifact@v4 | ||
with: | ||
name: mononoke_integration | ||
path: _artifacts | ||
- name: Test mononoke_integration | ||
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages test --num-jobs 4 --src-dir=. mononoke_integration --project-install-prefix mononoke_integration:/usr/local |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -11,3 +11,6 @@ | |
# Rust libraries and extensions | ||
Cargo.lock | ||
target/ | ||
|
||
# Getdeps | ||
/eden/mononoke/tests/integration/getdeps_build.log |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
[manifest] | ||
name = git-lfs | ||
|
||
[rpms] | ||
git-lfs | ||
|
||
[debs] | ||
git-lfs | ||
|
||
[homebrew] | ||
git-lfs | ||
|
||
# only used from system packages currently | ||
[build] | ||
builder = nop |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
[manifest] | ||
name = jq | ||
|
||
[rpms.distro=fedora] | ||
jq | ||
|
||
[homebrew] | ||
jq | ||
|
||
[download.not(os=windows)] | ||
# we use jq-1.7+ to get fix for number truncation https://github.com/jqlang/jq/pull/1752 | ||
url = https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-1.7.1.tar.gz | ||
sha256 = 478c9ca129fd2e3443fe27314b455e211e0d8c60bc8ff7df703873deeee580c2 | ||
|
||
[build.not(os=windows)] | ||
builder = autoconf | ||
subdir = jq-1.7.1 | ||
|
||
[build.os=windows] | ||
builder = nop | ||
|
||
[autoconf.args] | ||
# This argument turns off some developers tool and it is recommended in jq's | ||
# README | ||
--disable-maintainer-mode |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
[manifest] | ||
name = mononoke_integration | ||
fbsource_path = fbcode/eden | ||
shipit_project = eden | ||
shipit_fbcode_builder = true | ||
|
||
[git] | ||
repo_url = https://github.com/facebook/sapling.git | ||
|
||
[build.not(os=windows)] | ||
builder = make | ||
subdir = eden/mononoke/tests/integration | ||
|
||
[build.os=windows] | ||
# building Mononoke on windows is not supported | ||
builder = nop | ||
|
||
[make.build_args] | ||
build-getdeps | ||
|
||
[make.install_args] | ||
install-getdeps | ||
|
||
[make.test_args] | ||
test-getdeps | ||
|
||
[shipit.pathmap] | ||
fbcode/eden/mononoke/tests/integration = eden/mononoke/tests/integration | ||
|
||
[shipit.strip] | ||
^.*/facebook/.*$ | ||
^.*/fb/.*$ | ||
|
||
[dependencies] | ||
git-lfs | ||
jq | ||
mononoke | ||
nmap | ||
python | ||
python-click | ||
ripgrep | ||
sapling | ||
tree | ||
zstd | ||
|
||
[dependencies.os=linux] | ||
sqlite3 | ||
|
||
[dependencies.os=darwin] | ||
gnu-bash | ||
gnu-coreutils | ||
gnu-grep | ||
gnu-sed |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
[manifest] | ||
name = nmap | ||
|
||
[rpms] | ||
nmap | ||
nmap-ncat | ||
|
||
[debs] | ||
nmap | ||
|
||
# 18.04 combines ncat into the nmap package, newer need the separate one | ||
[debs.not(all(distro=ubuntu,distro_vers="18.04"))] | ||
ncat | ||
|
||
[download.not(os=windows)] | ||
url = https://api.github.com/repos/nmap/nmap/tarball/ef8213a36c2e89233c806753a57b5cd473605408 | ||
sha256 = eda39e5a8ef4964fac7db16abf91cc11ff568eac0fa2d680b0bfa33b0ed71f4a | ||
|
||
[build.not(os=windows)] | ||
builder = autoconf | ||
subdir = nmap-nmap-ef8213a | ||
build_in_src_dir = true | ||
|
||
[build.os=windows] | ||
builder = nop | ||
|
||
[autoconf.args] | ||
# Without this option the build was filing to find some third party libraries | ||
# that we don't need | ||
enable_rdma=no |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
[manifest] | ||
name = python-click | ||
|
||
[download] | ||
url = https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl | ||
sha256 = dacca89f4bfadd5de3d7489b7c8a566eee0d3676333fbb50030263894c38c0dc | ||
|
||
[build] | ||
builder = python-wheel |
Oops, something went wrong.