Move charm building from pytest tests to separate workflow jobs to increase CI stability

[ca-scribner]

Context

We've encountered a lot of instability in the kfp CI of late, including oddly flaky charmcraft builds for some of the charms. For some unknown reasons, standalone charmcraft seems to build things more reliably than building them in the pytest file directly using the pytest-operator.

Proposed here is that we move the building for these charms out of the pytest files and into their own jobs, then pass the .charm files as artifacts to the tests/publish jobs. This would hopefully add stability just from using charmcraft, but also we can have retry logic around failed build jobs (build jobs could auto-retry X times, so flakes could be mitigated)

What needs to get done

1. refactor the pytest tests to optionally build the charms or accept their artifacts as inputs. We might be able to use some data platform CI as inspiration here
2. refactor the CI workflows to build the charms separately and pass them to integration tests

Definition of Done

1. kfp integration test CI does not build its own charms, but instead uses those built in separate jobs

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5618.

This message was autogenerated

[ca-scribner]

canonical/bundle-kubeflow#1005 discusses instability that feels related to this issue