pants python handling moves to goals #2

jsirois · 2013-07-30T16:31:10Z

Brian has changes that bring the python handling via commands (build, py) into goals

(sapling split of 50491bc04eaf3085d9e8b7f018edfbce68d92da1)

This includes getting rid of command registration magic, since it's complicated and commands are going away entirely anyway. (sapling split of 426a4766355a6b14bed36e0c31cbbdaa32a30d69)

(sapling split of 50491bc04eaf3085d9e8b7f018edfbce68d92da1)

This includes getting rid of command registration magic, since it's complicated and commands are going away entirely anyway. (sapling split of 426a4766355a6b14bed36e0c31cbbdaa32a30d69)

ericzundel · 2015-02-03T22:09:57Z

Commands are gone, long live goals!

…n test. (#4407) ### Problem Currently, on Linux the first thin client call to the daemon can deadlock just after the pantsd->fork->pantsd-runner workflow. Connecting to the process with `gdb` reveals a deadlock in the following stack in the `post_fork` `drop` of the `CpuPool`: ``` #0 0x00007f63f04c31bd in __lll_lock_wait () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f63f04c0ded in pthread_cond_signal@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #2 0x00007f63d3cfa438 in notify_one () at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/sys/unix/condvar.rs:52 No locals. #3 notify_one () at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/sys_common/condvar.rs:39 No locals. #4 notify_one () at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/sync/condvar.rs:208 No locals. #5 std::thread::{{impl}}::unpark () at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/thread/mod.rs:633 No locals. #6 0x00007f63d3c583d1 in crossbeam::sync::ms_queue::{{impl}}::push<futures_cpupool::Message> (self=<optimized out>, t=...) at /home/kwilson/.cache/pants/rust-toolchain/registry/src/github.com-1ecc6299db9ec823/crossbeam-0.2.10/src/sync/ms_queue.rs:178 guard = <optimized out> self = <optimized out> #7 0x00007f63d3c588ed in futures_cpupool::{{impl}}::drop (self=<optimized out>) at /home/kwilson/.cache/pants/rust-toolchain/git/checkouts/futures-rs-a4f11d094efefb0a/f7e6bc8/futures-cpupool/src/lib.rs:236 self = 0x37547a0 #8 0x00007f63d3be871c in engine::fs::{{impl}}::post_fork (self=0x3754778) at /home/kwilson/dev/pants/src/rust/engine/src/fs.rs:355 self = 0x3754778 #9 0x00007f63d3be10e4 in engine::context::{{impl}}::post_fork (self=0x37545b0) at /home/kwilson/dev/pants/src/rust/engine/src/context.rs:93 self = 0x37545b0 #10 0x00007f63d3c0de5a in {{closure}} (scheduler=<optimized out>) at /home/kwilson/dev/pants/src/rust/engine/src/lib.rs:275 scheduler = 0x3740580 #11 with_scheduler<closure,()> (scheduler_ptr=<optimized out>, f=...) at /home/kwilson/dev/pants/src/rust/engine/src/lib.rs:584 scheduler = 0x3740580 scheduler_ptr = 0x3740580 #12 engine::scheduler_post_fork (scheduler_ptr=0x3740580) at /home/kwilson/dev/pants/src/rust/engine/src/lib.rs:274 scheduler_ptr = 0x3740580 #13 0x00007f63d3c1be8c in _cffi_f_scheduler_post_fork (self=<optimized out>, arg0=0x35798f0) at src/cffi/native_engine.c:2234 _save = 0x34a65a0 x0 = 0x3740580 datasize = <optimized out> #14 0x00007f63f07b5a62 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 ``` This presents as a hang in the thin client, because the pailgun socket is left open in the pantsd-runner. ### Solution Add pre-fork hooks and tear down the `CpuPool` instances prior to forking and rebuilding them. ### Result Can no longer reproduce the hang.

…n test. (pantsbuild#4407) ### Problem Currently, on Linux the first thin client call to the daemon can deadlock just after the pantsd->fork->pantsd-runner workflow. Connecting to the process with `gdb` reveals a deadlock in the following stack in the `post_fork` `drop` of the `CpuPool`: ``` #0 0x00007f63f04c31bd in __lll_lock_wait () from /lib64/libpthread.so.0 No symbol table info available. pantsbuild#1 0x00007f63f04c0ded in pthread_cond_signal@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. pantsbuild#2 0x00007f63d3cfa438 in notify_one () at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/sys/unix/condvar.rs:52 No locals. pantsbuild#3 notify_one () at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/sys_common/condvar.rs:39 No locals. pantsbuild#4 notify_one () at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/sync/condvar.rs:208 No locals. pantsbuild#5 std::thread::{{impl}}::unpark () at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/thread/mod.rs:633 No locals. pantsbuild#6 0x00007f63d3c583d1 in crossbeam::sync::ms_queue::{{impl}}::push<futures_cpupool::Message> (self=<optimized out>, t=...) at /home/kwilson/.cache/pants/rust-toolchain/registry/src/github.com-1ecc6299db9ec823/crossbeam-0.2.10/src/sync/ms_queue.rs:178 guard = <optimized out> self = <optimized out> pantsbuild#7 0x00007f63d3c588ed in futures_cpupool::{{impl}}::drop (self=<optimized out>) at /home/kwilson/.cache/pants/rust-toolchain/git/checkouts/futures-rs-a4f11d094efefb0a/f7e6bc8/futures-cpupool/src/lib.rs:236 self = 0x37547a0 pantsbuild#8 0x00007f63d3be871c in engine::fs::{{impl}}::post_fork (self=0x3754778) at /home/kwilson/dev/pants/src/rust/engine/src/fs.rs:355 self = 0x3754778 pantsbuild#9 0x00007f63d3be10e4 in engine::context::{{impl}}::post_fork (self=0x37545b0) at /home/kwilson/dev/pants/src/rust/engine/src/context.rs:93 self = 0x37545b0 pantsbuild#10 0x00007f63d3c0de5a in {{closure}} (scheduler=<optimized out>) at /home/kwilson/dev/pants/src/rust/engine/src/lib.rs:275 scheduler = 0x3740580 pantsbuild#11 with_scheduler<closure,()> (scheduler_ptr=<optimized out>, f=...) at /home/kwilson/dev/pants/src/rust/engine/src/lib.rs:584 scheduler = 0x3740580 scheduler_ptr = 0x3740580 pantsbuild#12 engine::scheduler_post_fork (scheduler_ptr=0x3740580) at /home/kwilson/dev/pants/src/rust/engine/src/lib.rs:274 scheduler_ptr = 0x3740580 pantsbuild#13 0x00007f63d3c1be8c in _cffi_f_scheduler_post_fork (self=<optimized out>, arg0=0x35798f0) at src/cffi/native_engine.c:2234 _save = 0x34a65a0 x0 = 0x3740580 datasize = <optimized out> pantsbuild#14 0x00007f63f07b5a62 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 ``` This presents as a hang in the thin client, because the pailgun socket is left open in the pantsd-runner. ### Solution Add pre-fork hooks and tear down the `CpuPool` instances prior to forking and rebuilding them. ### Result Can no longer reproduce the hang.

…n-source-code

This speeds things up: ``` $ hyperfine --warmup 2 './dist/pants.2.3.0.dev3+gitddc75f42..pex -V' '../pants/dist/pants.2.3.0.dev3+gitddc75f42..pex -V' Benchmark #1: ./dist/pants.2.3.0.dev3+gitddc75f42..pex -V Time (mean ± σ): 636.7 ms ± 23.3 ms [User: 441.7 ms, System: 35.9 ms] Range (min … max): 618.9 ms … 694.6 ms 10 runs Benchmark #2: ../pants/dist/pants.2.3.0.dev3+gitddc75f42..pex -V Time (mean ± σ): 1.170 s ± 0.017 s [User: 959.8 ms, System: 56.2 ms] Range (min … max): 1.155 s … 1.212 s 10 runs Summary './dist/pants.2.3.0.dev3+gitddc75f42..pex -V' ran 1.84 ± 0.07 times faster than '../pants/dist/pants.2.3.0.dev3+gitddc75f42..pex -V' ```

This lets you compile a package without needing to run tests or package a binary. Note that you can only directly compile a first-party package - to check if a third-party package is buildable, it must be a dependency of a first-party package. Works around #13175. This does not yet have an optimal UX. It's overly chatty: ``` ❯ ./pants check testprojects/src/go:: 13:36:38.57 [INFO] Initializing scheduler... 13:36:39.20 [INFO] Scheduler initialized. 13:36:39.71 [ERROR] Completed: pants.backend.go.goals.check.check_go - go failed (exit code 1). Partition #1 - github.com/pantsbuild/pants/testprojects/src/go/pants_test: ./testprojects/src/go/pants_test/foo.go:3:1: syntax error: non-declaration statement outside function body Partition #2 - github.com/pantsbuild/pants/testprojects/src/go/pants_test/bar: 𐄂 go failed. ``` We also should be streaming the result of each compilation as it happens, which has an added benefit of that output happening when compilation happens as a side effect of `run`, `package`, and `test`. Those fixes will be in a followup. [ci skip-rust] [ci skip-build-wheels]

Before: ``` ❯ ./pants check testprojects/src/go:: 14:51:39.15 [INFO] Completed: pants.backend.go.goals.check.check_go - go succeeded. Partition #1 - github.com/pantsbuild/pants/testprojects/src/go/pants_test: Partition #2 - github.com/pantsbuild/pants/testprojects/src/go/pants_test/bar: ✓ go succeeded. ``` ``` ❯ ./pants check testprojects/src/go:: 14:52:54.54 [ERROR] Completed: pants.backend.go.goals.check.check_go - go failed (exit code 1). Partition #1 - github.com/pantsbuild/pants/testprojects/src/go/pants_test: ./testprojects/src/go/pants_test/foo.go:3:1: syntax error: non-declaration statement outside function body Partition #2 - github.com/pantsbuild/pants/testprojects/src/go/pants_test/bar: 𐄂 go failed. ``` After: ``` ❯ ./pants check testprojects/src/go:: 14:20:40.79 [INFO] Completed: Check Go compilation - go succeeded. ✓ go compile succeeded. ``` ``` ❯ ./pants check testprojects/src/go:: 14:23:16.18 [ERROR] Completed: Compile with Go - github.com/pantsbuild/pants/testprojects/src/go/pants_test failed (exit code 1). ./testprojects/src/go/pants_test/foo.go:3:1: syntax error: non-declaration statement outside function body 14:23:16.18 [ERROR] Completed: Check Go compilation - go failed (exit code 1). 𐄂 go compile failed. ``` That is, we now: - Stream results for each package, rather than waiting to batch at the very end. - Don't log the `Partition #1` part, which is verbose and confusing. - Log success at DEBUG level, rather than INFO level, for less noise. (Reminder: these workunits will show up in the dynamic UI still) [ci skip-rust]

Before: ``` ❯ ./pants check testprojects/src/go:: 14:51:39.15 [INFO] Completed: pants.backend.go.goals.check.check_go - go succeeded. Partition #1 - github.com/pantsbuild/pants/testprojects/src/go/pants_test: Partition #2 - github.com/pantsbuild/pants/testprojects/src/go/pants_test/bar: ✓ go succeeded. ``` ``` ❯ ./pants check testprojects/src/go:: 14:52:54.54 [ERROR] Completed: pants.backend.go.goals.check.check_go - go failed (exit code 1). Partition #1 - github.com/pantsbuild/pants/testprojects/src/go/pants_test: ./testprojects/src/go/pants_test/foo.go:3:1: syntax error: non-declaration statement outside function body Partition #2 - github.com/pantsbuild/pants/testprojects/src/go/pants_test/bar: 𐄂 go failed. ``` After: ``` ❯ ./pants check testprojects/src/go:: 14:20:40.79 [INFO] Completed: Check Go compilation - go succeeded. ✓ go compile succeeded. ``` ``` ❯ ./pants check testprojects/src/go:: 14:23:16.18 [ERROR] Completed: Compile with Go - github.com/pantsbuild/pants/testprojects/src/go/pants_test failed (exit code 1). ./testprojects/src/go/pants_test/foo.go:3:1: syntax error: non-declaration statement outside function body 14:23:16.18 [ERROR] Completed: Check Go compilation - go failed (exit code 1). 𐄂 go compile failed. ``` That is, we now: - Stream results for each package, rather than waiting to batch at the very end. - Don't log the `Partition #1` part, which is verbose and confusing. - Log success at DEBUG level, rather than INFO level, for less noise. (Reminder: these workunits will show up in the dynamic UI still) [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

#13388) Before: ``` ❯ ./pants check testprojects/src/go:: 14:51:39.15 [INFO] Completed: pants.backend.go.goals.check.check_go - go succeeded. Partition #1 - github.com/pantsbuild/pants/testprojects/src/go/pants_test: Partition #2 - github.com/pantsbuild/pants/testprojects/src/go/pants_test/bar: ✓ go succeeded. ``` ``` ❯ ./pants check testprojects/src/go:: 14:52:54.54 [ERROR] Completed: pants.backend.go.goals.check.check_go - go failed (exit code 1). Partition #1 - github.com/pantsbuild/pants/testprojects/src/go/pants_test: ./testprojects/src/go/pants_test/foo.go:3:1: syntax error: non-declaration statement outside function body Partition #2 - github.com/pantsbuild/pants/testprojects/src/go/pants_test/bar: 𐄂 go failed. ``` After: ``` ❯ ./pants check testprojects/src/go:: 14:20:40.79 [INFO] Completed: Check Go compilation - go succeeded. ✓ go compile succeeded. ``` ``` ❯ ./pants check testprojects/src/go:: 14:23:16.18 [ERROR] Completed: Compile with Go - github.com/pantsbuild/pants/testprojects/src/go/pants_test failed (exit code 1). ./testprojects/src/go/pants_test/foo.go:3:1: syntax error: non-declaration statement outside function body 14:23:16.18 [ERROR] Completed: Check Go compilation - go failed (exit code 1). 𐄂 go compile failed. ``` That is, we now: - Stream results for each package, rather than waiting to batch at the very end. - Don't log the `Partition #1` part, which is verbose and confusing. - Log success at DEBUG level, rather than INFO level, for less noise. (Reminder: these workunits will show up in the dynamic UI still) [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

) Add a `jvm_lockfile` pytest fixture and related custom mark `pytest.mark.jvm_lockfile` for defining lockfiles for use in tests. This allows avoiding having to replicate lockfile content in code using `TestCoursierWrapper`, which can become unwiedly once a resolve in complicated enough. Lockfiles are generated by running `./pants internal-generate-test-lockfile-fixtures ::` which collects all tests marked with `pytest.mark.jvm_lockfile`, resolves their requirements, and writes out the resulting lockfiles. A custom Pants goal is used so that the plugin code gets access to the Pants source tree and targets. (An earlier iteration of this PR tried to use a `RuleRunner` in a script under `build-support/bin`, but `RuleRunner` is used for testing in an isolated directory and has no access to the Pants repository sources.)

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

The Pants native client which was introduced in #11922 has so far only been usable in the Pants repo when the `USE_NATIVE_PANTS` environment variable was set. To make the native client available to end users, this change begins distributing the native-client binary in Pants wheels. A corresponding change in the `scie-pants` repo (pantsbuild/scie-pants#172) uses the native client to run `pants`. To reduce the surface area of `scie-pants` (rather than having it be responsible for handling the special `75` exit code similar to the `pants` script integration), this PR also moves to having the native-client execute its own fallback (via `execv`) to the Pants entrypoint. In future, the `pantsbuild/pants` `pants` script could also use that facility (e.g. by specifying a separate `pants_server` bash script as the entrypoint to use as the `_PANTS_SERVER_EXE`). ---- As originally demonstrated on #11831, the native client is still much faster than the legacy client. Using pantsbuild/scie-pants#172, the timings look like: ``` Benchmark #1: PANTS_NO_NATIVE_CLIENT=true PANTS_SHA=836cceb74e6af042e7c82677f3ceb4927efce20e scie-pants-macos-x86_64 help Time (mean ± σ): 1.161 s ± 0.067 s [User: 830.6 ms, System: 79.2 ms] Range (min … max): 1.054 s … 1.309 s 10 runs Benchmark #2: PANTS_SHA=836cceb74e6af042e7c82677f3ceb4927efce20e scie-pants-macos-x86_64 help Time (mean ± σ): 271.0 ms ± 30.6 ms [User: 8.9 ms, System: 6.9 ms] Range (min … max): 241.5 ms … 360.6 ms 12 runs Summary 'PANTS_SHA=836cceb74e6af042e7c82677f3ceb4927efce20e scie-pants-macos-x86_64 help' ran 4.29 ± 0.54 times faster than 'PANTS_NO_NATIVE_CLIENT=true PANTS_SHA=836cceb74e6af042e7c82677f3ceb4927efce20e scie-pants-macos-x86_64 help' ``` Fixes #11831.

benjyw pushed a commit that referenced this issue Apr 1, 2014

Attempt #2 at proper multiplatform distillation for pants.pex.

7fd8c14

(sapling split of 50491bc04eaf3085d9e8b7f018edfbce68d92da1)

benjyw pushed a commit that referenced this issue Apr 8, 2014

Attempt #2 at proper multiplatform distillation for pants.pex.

dc0bf04

(sapling split of 50491bc04eaf3085d9e8b7f018edfbce68d92da1)

ericzundel closed this as completed Feb 3, 2015

areitz mentioned this issue Mar 11, 2015

./pants --changed .. does not work for deleted targets #382

Closed

jsirois mentioned this issue Aug 19, 2015

Conversion between Targets and their BUILD aliases. #2024

Closed

kwlzn mentioned this issue Sep 22, 2015

nailgun client error handling isn't clear to users when nailgun isn't connectable #2238

Closed

kwlzn mentioned this issue Apr 6, 2016

Flaky UnpicklingError in test_engine.py #3149

Closed

kwlzn mentioned this issue May 13, 2016

inconsistent result for dependencies #3428

Closed

kwlzn mentioned this issue May 26, 2016

implement unhandled globs exclude types #3505

Closed

kwlzn mentioned this issue Jun 9, 2017

Add support for local test caching. #4660

Merged

kwlzn mentioned this issue Sep 12, 2017

depmap beautification #4860

Closed

stuhood mentioned this issue Oct 18, 2017

Working implementation of jacoco. #4978

Merged

kwlzn mentioned this issue Nov 17, 2017

release pants as a pex #4896

Closed

kwlzn mentioned this issue May 18, 2018

Declare v2 products on v1 Tasks #4769

Closed

kwlzn mentioned this issue Jul 16, 2018

Non-cachable products for @console_rules #6146

Closed

Eric-Arellano mentioned this issue Jan 13, 2019

Allow Pants to run with Python 3 via ./pants3 script #6959

Merged

Eric-Arellano mentioned this issue Mar 12, 2019

Update docs to explain how to use Python 3 #7368

Closed

Eric-Arellano referenced this issue in Eric-Arellano/pants Mar 13, 2019

Pass #2 - explain option after Pants version

8764c6e

Eric-Arellano referenced this issue in Eric-Arellano/pants Apr 26, 2019

Improvement #2: show all discovered interpreters

9729ed4

Eric-Arellano referenced this issue in Eric-Arellano/pants Apr 26, 2019

Improvement #2: show all discovered interpreters

1449a72

Eric-Arellano mentioned this issue Apr 29, 2019

Mark build-support Python files as Pants targets to lint build-support #7633

Merged

Eric-Arellano mentioned this issue May 8, 2019

WIP: In target adaptors, populate attributes with a None value #7680

Closed

blorente mentioned this issue May 10, 2019

Add shellcheck.py and apply first round of fixes #7698

Merged

Eric-Arellano referenced this issue in Eric-Arellano/pants May 11, 2019

Approach #2: DRY filter_files()

d0a5047

jsirois mentioned this issue May 18, 2019

OOM vulnerability in protobuf 2.0.6: RUSTSEC-2019-0003 #7760

Closed

olafurpg added a commit to olafurpg/pants that referenced this issue Sep 25, 2020

Merge pull request pantsbuild#2 from tpasternak/fastpass-executable-i…

ca318ed

…n-source-code

This was referenced Oct 1, 2020

use libCST to stop having "the versions of python you can parse" depend on "pants's python version" for good #10894

Closed

Teach project introspection goals about special-cased dependencies fields #10888

Closed

This was referenced Oct 9, 2020

Rule graph fails when codegen uses DependenciesRequest or UnparsedAddressInputs #10917

Closed

make ImmutableValue Hashable, and add more type safety to fields #10932

Closed

Eric-Arellano mentioned this issue Oct 22, 2020

Pants peek goal #4861

Closed

Eric-Arellano mentioned this issue Nov 10, 2020

add support for binary build with pyinstaller + staticx #10684

Closed

jsirois mentioned this issue Nov 12, 2020

Pants reports a ModuleNotFoundError when attempting to import from a file within the same project #11118

Closed

Eric-Arellano mentioned this issue Nov 25, 2020

Add support for Shiv as package target #11207

Closed

This was referenced Feb 24, 2021

Add --stats-log to log metrics at end of run #11599

Merged

Add metrics for time saved from local and remote caching #11601

Merged

Eric-Arellano mentioned this issue Mar 29, 2021

Add Poetry macros to generate Python requirements and constraints #10655

Closed

kaos mentioned this issue Aug 27, 2021

wip: unions using protocol classes. #12577

Closed

Eric-Arellano mentioned this issue Dec 2, 2021

Add test_timeout fields to go_package #13707

Merged

Eric-Arellano mentioned this issue Mar 8, 2022

Build args changes do not trigger changed since determination for docker images #14724

Open

thejcannon mentioned this issue Apr 10, 2022

Clean PID-specific exception file if empty on process exit #15086

Merged

Eric-Arellano mentioned this issue May 10, 2022

Treat imports under "if TYPE_CHECKING" as weak. #15384

Closed

benjyw mentioned this issue May 11, 2022

Prepare the 2.13.0.dev1 release. #15393

Merged

tdyas pushed a commit to tdyas/pants that referenced this issue Jul 6, 2022

add REAPI batch APIs, take pantsbuild#2

0e6d2a5

tdyas mentioned this issue Jul 6, 2022

[internal] use CAS BatchReadBlobs API for small blob reads, take #2 #16076

Closed

tdyas pushed a commit to tdyas/pants that referenced this issue Jul 12, 2022

add REAPI batch APIs, take pantsbuild#2

06c66f6

tdyas pushed a commit to tdyas/pants that referenced this issue Jul 26, 2022

add REAPI batch APIs, take pantsbuild#2

082eb08

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

kuza55 mentioned this issue Apr 2, 2024

Pants should release it's lock on run targets #20642

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pants python handling moves to goals #2

pants python handling moves to goals #2

jsirois commented Jul 30, 2013

ericzundel commented Feb 3, 2015

pants python handling moves to goals #2

pants python handling moves to goals #2

Comments

jsirois commented Jul 30, 2013

ericzundel commented Feb 3, 2015