chore: Stream S3 files directly to zip

[DafyddLlyr]

What does this PR do?

• A further step to try and improve the submissions process to Uniform
• Stream S3 files directly to the zip, no need to write and then delete locally
• I'll try this on the Pizza to check if it works, but will need to promote to staging to checkout Fargate health
• Regardless of the actual affect on Fargate CPU usage this also seems like a sensible small simplification here I think

[github-actions]

Removed vultr server and associated DNS entries

[Mike-Heneghan]

What does this PR do?

• A further step to try and improve the submissions process to Uniform
• Stream S3 files directly to the zip, no need to write and then delete locally
• I'll try this on the Pizza to check if it works, but will need to promote to staging to checkout Fargate health
• Regardless of the actual affect on Fargate CPU usage this also seems like a sensible small simplification here I think

Hey @DafyddLlyr

I think the code change looks good but I was struggling to figure out how to test this in the Pizza 🤔

I've got a submission service where I upload some files and send it to uniform which seems to be successful based on the network tab. Although when I look on the Hasura console for the pizza service I don't see anything in the uniform applications:

I believe the files are uploaded to S3 but I'm trying to see the submission info so I can connect the application to uniform to an S3 bucket folder to check the files are uploaded successfully?

[jessicamcinchak]

Catching up on reviews - @Mike-Heneghan easier way to test this one might be to use "Send to email", which generates the same zip containing user-uploaded files from S3 that gets sent to Uniform!

Here's how I'd do that:

1. Add my own email to teams.submission_email for a given team in this pizza's Hasura console
2. Create a new flow within that team - it will need a file upload & send component with "Send to email" checked at minimum
3. Publish the flow
4. Go through & submit via the /preview link
5. Check your inbox - there should be an email with a /download-application-files endpoint - confirm you can download the zip and that it contains all the files you uploaded

In terms of your prior Uniform test, a couple things could have not been connecting here:

• Only LDCs within Lambeth, Southwark, or Bucks can be submitted successfully to Uniform - even though you can "toggle it on" in any Send component - which flow were you trying?
• Did you check the Hasura events table on the pizza - was there an error thrown for your attempted Uniform submission? These can fail a bit quietly on non-production envs unless you're looking
• The Uniform audit table only stores the sessionId, not the actual zip - if you clone the branch, you can test Uniform-specific zip locally by commenting out this line, then finding the generated zip in temp files to preview

[DafyddLlyr]

Thanks for the helpful comment above @jessicamcinchak 🙌

Testing of this has been a little on the backburner as publishing was failing due to a bug fixed earlier in the week. I've now rebased this which should make things simpler now.

[jessicamcinchak]

Coming back to this one in hopes of merging before you're off next week!

I hit one issue in testing:

• I went through this flow: https://2380.planx.pizza/testing/upload-and-label-demo/preview
• I successfully got sent this download link: https://api.2380.planx.pizza/download-application-files/fc218103-895c-4355-b5c0-38192a725b87?email=jessica%40opensystemslab.io&localAuthority=testing
• But it throws this error "Failed to download application files. Error: Cannot generate BOPS payload: Error: Cannot get flow 7e8f514e-e867-45f3-a1df-baf03190ca9c" even though that flow does indeed exist on the pizza

Might just be a rebase issue? Sorry this one has sat for awhile, happy to make time to test again today if you can similarly re-create!

[DafyddLlyr]

Thanks for the heads up - will also take another look here. No real rush to move this one forward but I'll investigate and fix and then @ you if I need another review 👍

[DafyddLlyr]

@jessicamcinchak Finally got back to this one - sorry for the holdup.

The issue was that the flow wasn't published yet on the Pizza. I published and then send to email worked as expected 👍

There's a small PR here which might make this slightly easier to work out in future - I keep on hitting this issue! theopensystemslab/planx-core#199

[jessicamcinchak]

Eeep sorry for silly mistake 🙈 working now, happy for this to go in!

[DafyddLlyr]

Eeep sorry for silly mistake 🙈 working now, happy for this to go in!

No zero bother at all - I keep hitting the same issue! Appreciate the reviews this morning - sorry if me tagging these for review has bothered you on a day ay off 🙏