Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: tolerate workflow that needs a missing job (#1595) #1619

Merged
merged 3 commits into from
Feb 16, 2023

Conversation

jsoref
Copy link
Contributor

@jsoref jsoref commented Feb 8, 2023

Try to fix #1595...

Commit Message

{{title}} (#1619)

Change planner functions to return errors

This enables createStages to return unable to build dependency graph

Fix PlanEvent to properly report errors relating to events/workflows

@jsoref jsoref requested a review from a team as a code owner February 8, 2023 16:55
@codecov
Copy link

codecov bot commented Feb 8, 2023

Codecov Report

Merging #1619 (b14fb1b) into master (4989f44) will increase coverage by 0.81%.
The diff coverage is 69.26%.

@@            Coverage Diff             @@
##           master    #1619      +/-   ##
==========================================
+ Coverage   61.22%   62.03%   +0.81%     
==========================================
  Files          46       46              
  Lines        7141     7239      +98     
==========================================
+ Hits         4372     4491     +119     
+ Misses       2462     2440      -22     
- Partials      307      308       +1     
Impacted Files Coverage Δ
pkg/container/docker_cli.go 82.23% <ø> (ø)
pkg/container/docker_logger.go 52.08% <ø> (ø)
pkg/container/docker_pull.go 33.33% <ø> (ø)
pkg/container/docker_run.go 14.08% <ø> (+0.49%) ⬆️
pkg/container/docker_volume.go 0.00% <ø> (ø)
pkg/container/file_collector.go 37.30% <0.00%> (ø)
pkg/container/host_environment.go 0.00% <0.00%> (ø)
pkg/model/planner.go 44.71% <21.27%> (-4.12%) ⬇️
pkg/container/docker_images.go 27.02% <29.41%> (-4.13%) ⬇️
pkg/model/github_context.go 61.06% <32.00%> (-19.40%) ⬇️
... and 23 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@jsoref jsoref marked this pull request as draft February 8, 2023 17:44
@jsoref
Copy link
Contributor Author

jsoref commented Feb 9, 2023

With this, I'm able to run act -l for the vector repository and get something reasonable

output
jsoref@jsoref-mbp vector % ../nektos/act/dist/local/act -l
WARN  ⚠ You are using Apple M1 chip and you have not specified container architecture, you might encounter issues while running act. If so, try running it with '--container-architecture linux/amd64'. ⚠
WARN[0000] unable to build dependency graph for Test Harness (test-harness.yml)
Stage  Job ID                                         Job name                                                                                                                Workflow name                  Workflow file            Events
0      release-build-normal                           Release Build (normal)                                                                                                  Baseline Timings               baseline.yml             workflow_dispatch,schedule
0      debug-build                                    Debug Build                                                                                                             Baseline Timings               baseline.yml             workflow_dispatch,schedule
0      debug-rebuild                                  Debug Rebuild                                                                                                           Baseline Timings               baseline.yml             schedule,workflow_dispatch
0      check                                          Cargo Check                                                                                                             Baseline Timings               baseline.yml             workflow_dispatch,schedule
0      release-build-optimized                        Release Build (optimized)                                                                                               Baseline Timings               baseline.yml             workflow_dispatch,schedule
0      publish-new-environment                        publish-new-environment                                                                                                 Environment Suite              environment.yml          pull_request,push,workflow_dispatch
0      add-to-project                                 Add issue to Gardener project board                                                                                     gardener_open_issue.yml        gardener_open_issue.yml  issues
0      add-dependabot-to-project                      Add dependabot PR to Gardener project board                                                                             gardener_open_pr.yml           gardener_open_pr.yml     pull_request_target
0      add-contributor-to-project                     Add contributor PR to Gardener project board                                                                            gardener_open_pr.yml           gardener_open_pr.yml     pull_request_target
0      sync-install                                   sync-install                                                                                                            Update install.sh Suite        install-sh.yml           push,workflow_dispatch
0      test-integration                               Integration - Linux, ${{ matrix.test }}                                                                                 Integration Test Suite         integration-test.yml     workflow_dispatch,push,pull_request
0      build-x86_64-unknown-linux-gnu                 Build - x86_64-unknown-linux-gnu                                                                                        K8S E2E Suite                  k8s_e2e.yml              workflow_dispatch,push,pull_request
0      compute-k8s-test-plan                          Compute K8s test plan                                                                                                   K8S E2E Suite                  k8s_e2e.yml              workflow_dispatch,push,pull_request
0      label                                          label                                                                                                                   Pull Request Labeler           labeler.yml              pull_request_target
0      Nightly                                        Nightly                                                                                                                 Nightly                        nightly.yml              workflow_dispatch,schedule
0      generate-publish-metadata                      Generate Publish-related Metadata                                                                                       Publish                        publish.yml              workflow_call
0      cancel-previous                                cancel-previous                                                                                                         Regression Detector            regression.yml           pull_request
0      compute-metadata                               Compute metadata for regression experiments                                                                             Regression Detector            regression.yml           pull_request
0      compute-metadata                               Compute metadata for regression experiments                                                                             Regression Detector (trusted)  regression_trusted.yml   workflow_run
0      Release                                        Release                                                                                                                 Release Suite                  release.yml              push
0      spelling                                       Check Spelling                                                                                                          Check Spelling                 spelling.yml             push,pull_request_target
0      changes                                        changes                                                                                                                 Test Suite                     test.yml                 pull_request,push
1      test-install                                   test-install                                                                                                            Update install.sh Suite        install-sh.yml           push,workflow_dispatch
1      test-integration-check                         test-integration-check                                                                                                  Integration Test Suite         integration-test.yml     push,pull_request,workflow_dispatch
1      test-e2e-kubernetes                            K8s ${{ matrix.kubernetes_version.version }} / ${{ matrix.container_runtime }} (${{ matrix.kubernetes_version.role }})  K8S E2E Suite                  k8s_e2e.yml              push,pull_request,workflow_dispatch
1      build-x86_64-pc-windows-msvc-packages          Build Vector for x86_64-pc-windows-msvc (.zip)                                                                          Publish                        publish.yml              workflow_call
1      build-armv7-unknown-linux-gnueabihf-packages   Build Vector for armv7-unknown-linux-gnueabihf (.tar.gz)                                                                Publish                        publish.yml              workflow_call
1      build-aarch64-unknown-linux-gnu-packages       Build Vector for aarch64-unknown-linux-gnu (.tar.gz)                                                                    Publish                        publish.yml              workflow_call
1      build-x86_64-unknown-linux-musl-packages       Build Vector for x86_64-unknown-linux-musl (.tar.gz)                                                                    Publish                        publish.yml              workflow_call
1      build-x86_64-apple-darwin-packages             Build Vector for x86_64-apple-darwin (.tar.gz)                                                                          Publish                        publish.yml              workflow_call
1      build-aarch64-unknown-linux-musl-packages      Build Vector for aarch64-unknown-linux-musl (.tar.gz)                                                                   Publish                        publish.yml              workflow_call
1      build-armv7-unknown-linux-musleabihf-packages  Build Vector for armv7-unknown-linux-musleabihf (.tar.gz)                                                               Publish                        publish.yml              workflow_call
1      build-x86_64-unknown-linux-gnu-packages        Build Vector for x86_64-unknown-linux-gnu (.tar.gz, DEB, RPM)                                                           Publish                        publish.yml              workflow_call
1      build-baseline                                 Build baseline Vector container                                                                                         Regression Detector            regression.yml           pull_request
1      build-comparison                               Build comparison Vector container                                                                                       Regression Detector            regression.yml           pull_request
1      transmit-metadata                              Transmit metadata to trusted workflow                                                                                   Regression Detector            regression.yml           pull_request
1      confirm-valid-credentials                      Confirm AWS credentials are minimally valid                                                                             Regression Detector (trusted)  regression_trusted.yml   workflow_run
1      test-linux                                     Unit - x86_64-unknown-linux-gnu                                                                                         Test Suite                     test.yml                 pull_request,push
1      test-misc                                      Miscellaneous - Linux                                                                                                   Test Suite                     test.yml                 pull_request,push
1      test-windows                                   Unit - Windows                                                                                                          Test Suite                     test.yml                 pull_request,push
1      checks                                         Checks                                                                                                                  Test Suite                     test.yml                 pull_request,push
1      check-component-features                       Component Features - Linux                                                                                              Test Suite                     test.yml                 pull_request,push
1      test-cli                                       CLI - Linux                                                                                                             Test Suite                     test.yml                 pull_request,push
1      cross-linux                                    Cross - ${{ matrix.target }}                                                                                            Test Suite                     test.yml                 push,pull_request
1      test-mac                                       Unit - Mac                                                                                                              Test Suite                     test.yml                 push,pull_request
1      check-msrv                                     Check minimum supported Rust version                                                                                    Test Suite                     test.yml                 pull_request,push
1      test-vrl-wasm                                  VRL WASM - Linux                                                                                                        Test Suite                     test.yml                 pull_request,push
1      test-vrl                                       VRL - Linux                                                                                                             Test Suite                     test.yml                 pull_request,push
2      install-shell-failure                          install-shell-failure                                                                                                   Update install.sh Suite        install-sh.yml           push,workflow_dispatch
2      master-failure                                 master-failure                                                                                                          Integration Test Suite         integration-test.yml     workflow_dispatch,push,pull_request
2      master-failure                                 master-failure                                                                                                          K8S E2E Suite                  k8s_e2e.yml              workflow_dispatch,push,pull_request
2      deb-verify                                     Verify DEB Packages                                                                                                     Publish                        publish.yml              workflow_call
2      macos-verify                                   Verify macOS Package                                                                                                    Publish                        publish.yml              workflow_call
2      rpm-verify                                     Verify RPM Packages                                                                                                     Publish                        publish.yml              workflow_call
2      upload-baseline-image-to-ecr                   Upload images to ECR                                                                                                    Regression Detector (trusted)  regression_trusted.yml   workflow_run
2      upload-comparison-image-to-ecr                 Upload images to ECR                                                                                                    Regression Detector (trusted)  regression_trusted.yml   workflow_run
2      master-failure                                 master-failure                                                                                                          Test Suite                     test.yml                 pull_request,push
2      cross-linux-check                              Cross - Linux                                                                                                           Test Suite                     test.yml                 pull_request,push
3      publish-github                                 Publish to GitHub                                                                                                       Publish                        publish.yml              workflow_call
3      publish-cloudsmith                             Publish to Cloudsmith                                                                                                   Publish                        publish.yml              workflow_call
3      publish-docker                                 Publish to Docker                                                                                                       Publish                        publish.yml              workflow_call
3      publish-s3                                     Publish to S3                                                                                                           Publish                        publish.yml              workflow_call
3      submit-job                                     Submit regression job                                                                                                   Regression Detector (trusted)  regression_trusted.yml   workflow_run
4      publish-homebrew                               Publish to Homebrew                                                                                                     Publish                        publish.yml              workflow_call
4      publish-failure                                Send Publish Failure Notification                                                                                       Publish                        publish.yml              workflow_call
4      detect-regression                              Determine regression status                                                                                             Regression Detector (trusted)  regression_trusted.yml   workflow_run
4      analyze-experiment                             Download regression analysis & upload report                                                                            Regression Detector (trusted)  regression_trusted.yml   workflow_run

Detected multiple jobs with the same job name, use `-W` to specify the path to the specific workflow.

@jsoref jsoref marked this pull request as ready for review February 9, 2023 03:20
@ChristopherHX
Copy link
Contributor

With this change, you are allowing to run https://github.com/jsoref/necktos-act-issue-1595/actions/runs/4126583173/workflow.
I don't think exit code 0 while running a workflow with a missing dependency is correct.

Copy link
Member

@KnisterPeter KnisterPeter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine, just the debug output seems to be wrong

pkg/model/planner.go Outdated Show resolved Hide resolved
Copy link
Member

@KnisterPeter KnisterPeter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with this now, but @ChristopherHX has a valid point.
If that's fixed I'll approve

@mergify mergify bot requested a review from a team February 14, 2023 07:56
@jsoref jsoref force-pushed the issue-1595 branch 2 times, most recently from ebbfc42 to 38fc5f0 Compare February 14, 2023 08:30
@mergify
Copy link
Contributor

mergify bot commented Feb 14, 2023

@jsoref this pull request has failed checks 🛠

@mergify mergify bot added the needs-work Extra attention is needed label Feb 14, 2023
@ChristopherHX
Copy link
Contributor

Seems like you fixed the exit code for the exact case I wrote above, but depending on the order of the workflows the exit code is still 0.

_ ➜ /workspaces/act (issue-1595) $ ./act -W necktos-act-issue-1595/.github/workflows/bad-dependency.yml -W necktos-act-issue-1595/.github/workflows/pick-me.yml
[pick-me/pick-me] 🚀  Start image=catthehacker/ubuntu:act-latest
[pick-me/pick-me]   🐳  docker pull image=catthehacker/ubuntu:act-latest platform= username= forcePull=true
[pick-me/pick-me]   🐳  docker create image=catthehacker/ubuntu:act-latest platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[]
[pick-me/pick-me]   🐳  docker run image=catthehacker/ubuntu:act-latest platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[]
[pick-me/pick-me] ⭐ Run Main Run a one-line script
[pick-me/pick-me]   🐳  docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/0] user= workdir=
| Hello, world!
[pick-me/pick-me]   ✅  Success - Main Run a one-line script
[pick-me/pick-me] 🏁  Job succeeded
_ ➜ /workspaces/act (issue-1595) $ ./act -W necktos-act-issue-1595/.github/workflows/pick-me.yml -W necktos-act-issue-1595/.github/workflows/bad-dependency.yml 
WARN[0000] unable to build dependency graph for bad-dependency (bad-dependency.yml) 
WARN[0000] unable to build dependency graph for bad-dependency (bad-dependency.yml) 
Error: no reachable stages for push event for 1 workflow(s)

I mean it is ok for me to run all valid workflows, but once one is an invalid workflow I expect exit code 1.

So the first case

  • running pick-me is completely fine
  • however I still expect exit code 1, because bad-dependency is invalid.

@jsoref
Copy link
Contributor Author

jsoref commented Feb 14, 2023

  • You can't pass more than one -W flag (last one wins -- this is admittedly surprising, and it'd be nice if cobra at least hinted to users about this behavior)
  • Based on the current model, it isn't possible for the planner to keep workflows that it would hate in time to hate them in time to talk about them when running runnable jobs. (This is if you use -W necktos-act-issue-1595/.github/workflows instead)
    • Reworking it to be able to handle that is really too much to ask.

@ChristopherHX
Copy link
Contributor

You are right about that flag..., silently dropping previous flags is unexpected behavior for me. Other cli parsers I know are throwing.

Reworking it to be able to handle that is really too much to ask.

Hmm, can't we return both err and the partial result? Then just return the planner error if execution of the workflow succeeded.

@ChristopherHX
Copy link
Contributor

Seems to work for me, that is what I expect.

➜ /workspaces/act (issue-1595) $ ./act -W necktos-act-issue-1595/.github/workflows/
WARN[0000] unable to build dependency graph for bad-dependency (bad-dependency.yml) 
WARN[0000] unable to build dependency graph for half working (missing.yml) 
WARN[0000] unable to build dependency graph for bad-dependency (bad-dependency.yml) 
WARN[0000] unable to build dependency graph for half working (missing.yml) 
[pick-me/pick-me] 🚀  Start image=catthehacker/ubuntu:act-latest
[pick-me/pick-me]   🐳  docker pull image=catthehacker/ubuntu:act-latest platform= username= forcePull=true
[pick-me/pick-me]   🐳  docker create image=catthehacker/ubuntu:act-latest platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[]
[pick-me/pick-me]   🐳  docker run image=catthehacker/ubuntu:act-latest platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[]
[pick-me/pick-me] ⭐ Run Main Run a one-line script
[pick-me/pick-me]   🐳  docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/0] user= workdir=
| Hello, world!
[pick-me/pick-me]   ✅  Success - Main Run a one-line script
[pick-me/pick-me] 🏁  Job succeeded
Error: unable to build dependency graph for half working (missing.yml)
This patch make this PR what I expect, however other maintainer and the owner can outvote me
diff --git a/cmd/root.go b/cmd/root.go
index a9dc891..3619c7b 100644
--- a/cmd/root.go
+++ b/cmd/root.go
@@ -366,26 +366,35 @@ func newRunCommand(ctx context.Context, input *Input) func(*cobra.Command, []str
 			filterEventName = events[0]
 		}
 
+		var plannerErr error
 		if jobID != "" {
 			log.Debugf("Preparing plan with a job: %s", jobID)
-			filterPlan, err = planner.PlanJob(jobID)
+			filterPlan, plannerErr = planner.PlanJob(jobID)
 		} else if filterEventName != "" {
 			log.Debugf("Preparing plan for a event: %s", filterEventName)
-			filterPlan, err = planner.PlanEvent(filterEventName)
+			filterPlan, plannerErr = planner.PlanEvent(filterEventName)
 		} else {
 			log.Debugf("Preparing plan with all jobs")
-			filterPlan, err = planner.PlanAll()
+			filterPlan, plannerErr = planner.PlanAll()
 		}
-		if err != nil {
-			return err
+		if filterPlan == nil && plannerErr != nil {
+			return plannerErr
 		}
 
 		if list {
-			return printList(filterPlan)
+			err = printList(filterPlan)
+			if err != nil {
+				return err
+			}
+			return plannerErr
 		}
 
 		if graph {
-			return drawGraph(filterPlan)
+			err = drawGraph(filterPlan)
+			if err != nil {
+				return err
+			}
+			return plannerErr
 		}
 
 		// plan with triggered jobs
@@ -413,13 +422,13 @@ func newRunCommand(ctx context.Context, input *Input) func(*cobra.Command, []str
 		// build the plan for this run
 		if jobID != "" {
 			log.Debugf("Planning job: %s", jobID)
-			plan, err = planner.PlanJob(jobID)
+			plan, plannerErr = planner.PlanJob(jobID)
 		} else {
 			log.Debugf("Planning jobs for event: %s", eventName)
-			plan, err = planner.PlanEvent(eventName)
+			plan, plannerErr = planner.PlanEvent(eventName)
 		}
-		if err != nil {
-			return err
+		if plan == nil && plannerErr != nil {
+			return plannerErr
 		}
 
 		// check to see if the main branch was defined
@@ -507,14 +516,22 @@ func newRunCommand(ctx context.Context, input *Input) func(*cobra.Command, []str
 		if watch, err := cmd.Flags().GetBool("watch"); err != nil {
 			return err
 		} else if watch {
-			return watchAndRun(ctx, r.NewPlanExecutor(plan))
+			err = watchAndRun(ctx, r.NewPlanExecutor(plan))
+			if err != nil {
+				return err
+			}
+			return plannerErr
 		}
 
 		executor := r.NewPlanExecutor(plan).Finally(func(ctx context.Context) error {
 			cancel()
 			return nil
 		})
-		return executor(ctx)
+		err = executor(ctx)
+		if err != nil {
+			return err
+		}
+		return plannerErr
 	}
 }
 
diff --git a/pkg/model/planner.go b/pkg/model/planner.go
index c7f4eb0..40bc67c 100644
--- a/pkg/model/planner.go
+++ b/pkg/model/planner.go
@@ -172,14 +172,15 @@ type workflowPlanner struct {
 func (wp *workflowPlanner) PlanEvent(eventName string) (*Plan, error) {
 	plan := new(Plan)
 	if len(wp.workflows) == 0 {
-		log.Debug("no workflows found by planner")
+		log.Debugf("no workflows found by planner")
 		return plan, nil
 	}
+	var lastErr error
 
 	for _, w := range wp.workflows {
 		events := w.On()
 		if len(events) == 0 {
-			log.Debugf("no events found for workflow: %s", w.File)
+			log.Warn("no events found for workflow: %s", w.File)
 			continue
 		}
 
@@ -188,16 +189,14 @@ func (wp *workflowPlanner) PlanEvent(eventName string) (*Plan, error) {
 				stages, err := createStages(w, w.GetJobIDs()...)
 				if err != nil {
 					log.Warn(err)
+					lastErr = err
 				} else {
 					plan.mergeStages(stages)
 				}
 			}
 		}
 	}
-	if len(plan.Stages) == 0 {
-		return nil, fmt.Errorf("no reachable stages for %s event for %d workflow(s)", eventName, len(wp.workflows))
-	}
-	return plan, nil
+	return plan, lastErr
 }
 
 // PlanJob builds a new run to execute in parallel for a job name

@jsoref
Copy link
Contributor Author

jsoref commented Feb 14, 2023

I'm happy to apply most of that. This seems unnecessary:

-		log.Debug("no workflows found by planner")
+		log.Debugf("no workflows found by planner")

@ChristopherHX
Copy link
Contributor

Feel free to drop this, unneeded change. I will approve once the behavior similar to my patch.

You can also drop this part.

-			log.Debugf("no events found for workflow: %s", w.File)
+			log.Warn("no events found for workflow: %s", w.File)

@jsoref
Copy link
Contributor Author

jsoref commented Feb 14, 2023

(I'm just trying to convince my computer to run some tests in VSCode before I amend.)

@mergify mergify bot removed the needs-work Extra attention is needed label Feb 14, 2023
ChristopherHX
ChristopherHX previously approved these changes Feb 14, 2023
Copy link
Contributor

@ChristopherHX ChristopherHX left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All in all it's fine from my side.

PlanAll and PlanJob should also return err for the planner

diff --git a/pkg/model/planner.go b/pkg/model/planner.go
index fc98684..1769b73 100644
--- a/pkg/model/planner.go
+++ b/pkg/model/planner.go
@@ -205,16 +205,18 @@ func (wp *workflowPlanner) PlanJob(jobName string) (*Plan, error) {
 	if len(wp.workflows) == 0 {
 		log.Debugf("no jobs found for workflow: %s", jobName)
 	}
+	var lastErr error
 
 	for _, w := range wp.workflows {
 		stages, err := createStages(w, jobName)
 		if err != nil {
 			log.Warn(err)
+			lastErr = err
 		} else {
 			plan.mergeStages(stages)
 		}
 	}
-	return plan, nil
+	return plan, lastErr
 }
 
 // PlanAll builds a new run to execute in parallel all
@@ -224,17 +226,19 @@ func (wp *workflowPlanner) PlanAll() (*Plan, error) {
 		log.Debug("no workflows found by planner")
 		return plan, nil
 	}
+	var lastErr error
 
 	for _, w := range wp.workflows {
 		stages, err := createStages(w, w.GetJobIDs()...)
 		if err != nil {
 			log.Warn(err)
+			lastErr = err
 		} else {
 			plan.mergeStages(stages)
 		}
 	}
 
-	return plan, nil
+	return plan, lastErr
 }
 
 // GetEvents gets all the events in the workflows file

@mergify
Copy link
Contributor

mergify bot commented Feb 15, 2023

@jsoref this pull request has failed checks 🛠

@mergify mergify bot added the needs-work Extra attention is needed label Feb 15, 2023
This enables createStages to return `unable to build dependency graph`

Fix PlanEvent to properly report errors relating to events/workflows
@mergify mergify bot removed the needs-work Extra attention is needed label Feb 15, 2023
@mergify mergify bot requested a review from a team February 15, 2023 03:22
Copy link
Contributor

@ChristopherHX ChristopherHX left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✔️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unable to build dependency graph
3 participants