Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve UX with CA derivations #875

Open
wants to merge 82 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
00c2e2c
Fix with ca derivations
thufschmitt Dec 21, 2020
030d72e
Add some tests for the content-addressed derivations
thufschmitt Feb 18, 2021
111abeb
Properly register the unresolved derivation
thufschmitt Apr 12, 2021
22a638e
Fix a localStore/destStore confusion
thufschmitt Apr 20, 2021
ddfece5
Check for previous builds using the drvPath rather than the output paths
thufschmitt Apr 20, 2021
fad8839
Don't call queryPartialDerivationOutputMap on the dest store
thufschmitt Apr 20, 2021
410d7a0
getBuildOutput: Don't query the dest store for the outputs
thufschmitt Apr 22, 2021
68d18b2
Correctly handle the non-ca case when building
thufschmitt Apr 22, 2021
af8f070
Allow gc-ing the destination store
thufschmitt Apr 22, 2021
70402e5
Actually use the `content-addressed` test
thufschmitt Apr 28, 2021
d1e41cf
Update the ca test
thufschmitt Apr 28, 2021
c5a092e
Clarify what happens with the out paths in the db
thufschmitt Apr 28, 2021
d243424
Move mkDerivation-for-ca to config.nix.in
thufschmitt Apr 28, 2021
4a6a342
eval-jobs: Don't strip the output path for ia derivations
thufschmitt Apr 28, 2021
89a55b9
Add the output paths of CA derivations to the database
thufschmitt Apr 28, 2021
34aaa0b
Add more ca tests
thufschmitt Apr 28, 2021
2a2aa72
Test evaluating a ca drv without the experimental feature
thufschmitt Apr 28, 2021
a07b1db
Move the ca tests to a subdirectory
thufschmitt Apr 28, 2021
3d64cfb
Fix a potential queue-runner crash
thufschmitt Apr 28, 2021
9f2b576
Prevent enabling ca-derivations from breaking substitution
thufschmitt Apr 28, 2021
a79afe0
Document the state of ca derivations
thufschmitt Apr 28, 2021
f676dc8
Also register the ca outputs on the dest store
thufschmitt May 3, 2021
79e5e73
Fix the resolving of derivations
thufschmitt May 31, 2021
529c28f
Make non-ca-depending-on-ca derivations work
thufschmitt Jul 28, 2021
c6005f4
Correctly handle “real” input-addressed derivations
thufschmitt Jul 29, 2021
1036075
Make perlcritic happy
thufschmitt Oct 18, 2021
a084efc
Extract some parts of `State::buildRemote` into auxiliary functions
thufschmitt Oct 20, 2021
46ae574
Fix build with Nix 2.6pre
thufschmitt Jan 19, 2022
cdff806
Flake: Update
thufschmitt Jan 20, 2022
77df6b4
Merge remote-tracking branch 'thufschmitt/nix-ca' into nix-ca4
aciceri Jun 25, 2022
8b34b2f
Don't query outputs in case of CA derivations
aciceri Jun 25, 2022
d8639ac
Using new libstore method and fixes due to CA derivations
aciceri Jun 25, 2022
de2fc58
`string` -> `std::string`
aciceri Jun 25, 2022
0ef435e
Forgot to remove old comment in the merge
aciceri Jun 27, 2022
db89bbe
Fixed regression to #1126
aciceri Jun 27, 2022
8d52640
DB model changed
aciceri Jun 30, 2022
954e41f
Keep track if step build outputs are CA or not
aciceri Jun 30, 2022
3fdc9e4
Improved UI to correctly handle CA derivations
aciceri Jun 30, 2022
391ada9
"Content addressed" field in builds details
aciceri Jul 2, 2022
8311b49
Merge branch 'NixOS:master' into aciceri/ca-derivations
aciceri Aug 30, 2022
a9f1bc4
Merge remote-tracking branch 'upstream/master' into ca-derivations-re…
SuperSandro2000 Nov 26, 2023
d549a17
Fix compilation against Nix 2.16 and 2.17: ca-derivations followup
t184256 Aug 2, 2023
d1c1eaa
Copy the derivation input closure to the remote builder and the deriv…
DarkKirb Jul 2, 2022
b4dd816
require buildoutputs to be present before displaying it
DarkKirb May 30, 2023
c257abe
make it work
DarkKirb May 30, 2023
9e528b2
Add realisations to binary cache for ca-derivations
SuperSandro2000 Dec 4, 2023
02e3537
Merge commit 'f216bce0e62654ab4809003ec0756ee3e40d5172' into ca-deriv…
Ericson2314 Dec 4, 2023
82ded22
Merge commit '4dc8fe0b08edc421c251270ccd4be3e5bf9d66b4' into ca-deriv…
Ericson2314 Dec 4, 2023
91cdfe5
Merge commit '874fcae1e895c0884f3694e2f5ec1749cb23e5d6' into ca-deriv…
Ericson2314 Dec 4, 2023
a0e854b
Merge remote-tracking branch 'upstream/master' into ca-derivations
Ericson2314 Dec 4, 2023
040d734
Restore accidentally deleted blank line
Ericson2314 Dec 4, 2023
5357d88
Remove stray whitespace in test
Ericson2314 Dec 4, 2023
3b3050c
Forgot to remove this duplicated code
Ericson2314 Dec 4, 2023
972f995
Style, indenting
Ericson2314 Dec 4, 2023
513f584
Two more style nits
Ericson2314 Dec 4, 2023
96eaec0
Merge branch 'ca-derivations-prep' into nix-ca
Ericson2314 Dec 4, 2023
32f34e2
Merge branch 'ca-derivations-prep' into nix-ca
Ericson2314 Dec 4, 2023
30c0619
Fix indentation
Ericson2314 Dec 4, 2023
897a921
Merge branch 'ca-derivations-prep' into nix-ca
Ericson2314 Dec 6, 2023
5b5f295
Merge branch 'ca-derivations-prep' into nix-ca
Ericson2314 Dec 7, 2023
049f608
Revert "Do not copy for both stores for now"
Ericson2314 Dec 7, 2023
4ba5179
Merge remote-tracking branch 'upstream/master' into nix-ca
Ericson2314 Dec 10, 2023
b755784
Tighten logic in build success case
Ericson2314 Dec 10, 2023
7128f39
Clean up for loop slightly
Ericson2314 Dec 10, 2023
ea0f14d
Simplify build-remote logic a little bit
Ericson2314 Dec 10, 2023
9c6c5ec
Clean up `queue-monitor.cc` a bit
Ericson2314 Dec 10, 2023
2359db8
flake.lock: Update to include unrelease 2.19 backports
Ericson2314 Dec 11, 2023
0b36532
Use `destStore` realisations to resolve CA derivations
Ericson2314 Dec 10, 2023
17c8d51
Don't write things to the local store that don't belong
Ericson2314 Dec 10, 2023
7d7a2a3
Beef of content addressing test
Ericson2314 Dec 10, 2023
ba70b4e
Merge branch 'ca-derivations-prep' into nix-ca
Ericson2314 Dec 11, 2023
25a5e4e
Merge branch 'ca-derivations-prep' into nix-ca
Ericson2314 Dec 11, 2023
bfe3567
Revert "Revert query -- those columns don't exist yet!"
Ericson2314 Dec 11, 2023
7f3e2c8
Merge branch 'ca-derivations-prep' into nix-ca
Ericson2314 Dec 11, 2023
002a77a
Merge branch 'master' into nix-ca
Ericson2314 Jan 24, 2024
0b632ec
Merge branch 'master' into nix-ca
Ericson2314 Jan 25, 2024
ee6c465
Merge branch 'master' into nix-ca
Ericson2314 Jan 26, 2024
c87a066
Merge branch 'ca-no-new-col' into nix-ca
Ericson2314 Jan 26, 2024
91c641d
Merge branch 'ca-no-new-col' into nix-ca
Ericson2314 Jan 26, 2024
2b1d277
Fix mistake in merge
Ericson2314 Jan 26, 2024
d7e7bee
Merge branch 'master' into nix-ca
Ericson2314 Jan 26, 2024
b207f20
Merge remote-tracking branch 'upstream/master' into nix-ca
Ericson2314 May 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions src/hydra-queue-runner/hydra-queue-runner.cc
Original file line number Diff line number Diff line change
Expand Up @@ -333,11 +333,12 @@ unsigned int State::createBuildStep(pqxx::work & txn, time_t startTime, BuildID

for (auto & [name, output] : getDestStore()->queryPartialDerivationOutputMap(step->drvPath, &*localStore))
txn.exec_params0
("insert into BuildStepOutputs (build, stepnr, name, path) values ($1, $2, $3, $4)",
("insert into BuildStepOutputs (build, stepnr, name, path, contentAddressed) values ($1, $2, $3, $4, $5)",
buildId, stepNr, name,
output
? std::optional { localStore->printStorePath(*output)}
: std::nullopt);
: std::nullopt,
step->drv->type().isCA());

if (status == bsBusy)
txn.exec(fmt("notify step_started, '%d\t%d'", buildId, stepNr));
Expand Down Expand Up @@ -409,9 +410,10 @@ int State::createSubstitutionStep(pqxx::work & txn, time_t startTime, time_t sto
if (r.affected_rows() == 0) goto restart;

txn.exec_params0
("insert into BuildStepOutputs (build, stepnr, name, path) values ($1, $2, $3, $4)",
("insert into BuildStepOutputs (build, stepnr, name, path, contentAddressed) values ($1, $2, $3, $4, $5)",
build->id, stepNr, outputName,
localStore->printStorePath(storePath));
localStore->printStorePath(storePath),
drv.type().isCA());

return stepNr;
}
Expand Down
12 changes: 12 additions & 0 deletions src/lib/Hydra/Controller/Build.pm
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,18 @@ sub build_GET {

$c->stash->{steps} = [$build->buildsteps->search({}, {order_by => "stepnr desc"})];

$c->stash->{contentAddressed} = 0;
# Hydra marks single outputs as CA but currently in Nix only derivations
# can be CA (and *all* their outputs are CA).
# So the next check (which assumes that if a step's output is CA then
# all the other outptus and the whole derivation are CA) is safe.
foreach my $step (@{$c->stash->{steps}}) {
if ($step->buildstepoutputs->search({contentaddressed => 1})->count > 0) {
$c->stash->{contentAddressed} = 1;
last;
}
}

$c->stash->{binaryCachePublicUri} = $c->config->{binary_cache_public_uri};
}

Expand Down
7 changes: 7 additions & 0 deletions src/lib/Hydra/Schema/Result/BuildStepOutputs.pm
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,11 @@ __PACKAGE__->table("buildstepoutputs");
data_type: 'text'
is_nullable: 1

=head2 contentaddressed

data_type: 'boolean'
is_nullable: 0

=cut

__PACKAGE__->add_columns(
Expand All @@ -68,6 +73,8 @@ __PACKAGE__->add_columns(
{ data_type => "text", is_nullable => 0 },
"path",
{ data_type => "text", is_nullable => 1 },
"contentaddressed",
{ data_type => "boolean", is_nullable => 0 },
);

=head1 PRIMARY KEY
Expand Down
29 changes: 23 additions & 6 deletions src/root/build.tt
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,13 @@ END;
%]

[% BLOCK renderOutputs %]
[% start=1; FOREACH output IN outputs %]
[% IF !start %],<br/>[% END; start=0; output.path %]
[% start=1; FOREACH output IN step.buildstepoutputs %]
[% IF !start %],<br/>[% END; start=0; %]
[% IF step.status != 0 && output.contentaddressed %]
[% output.name %]
[% ELSE %]
[% output.path %]
[% END %]
[% END %]
[% END %]

Expand All @@ -40,9 +45,9 @@ END;
<td>[% step.stepnr %]</td>
<td>
[% IF step.type == 0 %]
Build of <tt>[% INCLUDE renderOutputs outputs=step.buildstepoutputs %]</tt>
Build of <tt>[% INCLUDE renderOutputs step=step %]</tt>
[% ELSE %]
Substitution of <tt>[% INCLUDE renderOutputs outputs=step.buildstepoutputs %]</tt>
Substitution of <tt>[% INCLUDE renderOutputs step=step %]</tt>
[% END %]
</td>
<td>
Expand Down Expand Up @@ -382,9 +387,21 @@ END;
<td><tt>[% build.drvpath %]</tt></td>
</tr>
<tr>
<th>Output store paths:</th>
<td><tt>[% INCLUDE renderOutputs outputs=build.buildoutputs %]</tt></td>
<th>Content addressed:</th>
<td><tt>
[% IF contentAddressed %]
Yes
[% ELSE %]
No
[% END %]
</tt></td>
</tr>
[% IF !contentAddressed || step.status == 0 %]
<tr>
<th>Output store paths:</th>
<td><tt>[% INCLUDE renderOutputs step=step %]</tt></td>
</tr>
[% END %]
[% chartsURL = c.uri_for('/job' build.project.name build.jobset.name build.job) _ "#tabs-charts" %]
[% IF build.finished && build.closuresize %]
<tr>
Expand Down
15 changes: 8 additions & 7 deletions src/sql/hydra.sql
Original file line number Diff line number Diff line change
Expand Up @@ -300,13 +300,14 @@ create table BuildSteps (


create table BuildStepOutputs (
build integer not null,
stepnr integer not null,
name text not null,
path text,
primary key (build, stepnr, name),
foreign key (build) references Builds(id) on delete cascade,
foreign key (build, stepnr) references BuildSteps(build, stepnr) on delete cascade
build integer not null,
stepnr integer not null,
name text not null,
path text,
contentAddressed boolean not null,
primary key (build, stepnr, name),
foreign key (build) references Builds(id) on delete cascade,
foreign key (build, stepnr) references BuildSteps(build, stepnr) on delete cascade
);


Expand Down
Loading