pulp ostree: compose job is finished but importing to pulp is not finished

[yih-redhat]

Describe the bug
When build edge image with pulp enabled, the compose job show as finished, but the importing to pulp is not finished yet, need to wait until import finished. (wait time is usually several minutes, depends on network or pulp server location)
From customer's point of view, if compose job finished, it means they can use image content in pulp.

Environment

• OS version RHEL 9.3

To Reproduce
Steps to reproduce the behavior:

1. setup pulp server
2. create pulp.toml file:
   provider = "pulp.ostree"

[settings]
server_address = "http://localhost:9090"
repository = "commit-repo"
basepath = "commit-path"
username = "admin"
password = "foobar"
3. build image with command "composer-cli compose start-ostree firstcommit edge-commit --ref rhel/9/x86_64/edge ./pulp.toml"
4. After some time, check compose job status, it is finished. Check pulp content, it's empty.
5. Then after another several minutes, check pulp content, the image is imported.

Expected behavior
If compose job is finished, the image should be imported into pulp already.

Additional context
Add any other context about the problem here.

[achilleas-k]

Pulling in some quotes from the PR where this issue was discussed:

Comment from @teg #3636 (comment)

This is what happens with AMIs today, AFAIK. A potential solution for both AMIs and ostree commits is to split the job in two pieces: 1) build+upload; and 2) import. Then importing could be done in parallel with future builds, so there is no blocking.

Response from @thozza #3636 (comment)

I had a very similar idea, I would just make the import async and add a new job type which would wait for import to finish in parallel with any osbuild job. Basically in the same way as we do for depsolving.

The tricky part is that such a change would be very disruptive, since composer assumes that the UUID of the osbuild job is the actual compose UUID and we use that to determine status, logs, etc in API endpoints. We already have a special case for Koji where the last job is the KojiFinalize and its UUID is used as the compose UUID. We would have to do a similar thing for new non-Koji composes and only for some of them depending on the used target. I'm sure someone can figure out an elegant and backward incompatible way to make it work, I just wanted to point out that this is not a trivial task.

[achilleas-k]

Upload function now waits for async tasks to finish: #3636 (comment)

[yih-redhat]

Verified, fixed.
Closing this issue.