FE-232 fix dev env for manifest dot py

[bahill]

FE-232

Problem:

The first step in the HCA ingest process requires the creation and deployment of a manifest of the data to be ingested. This is accomplished via a python script - manifest.py - which has a number of dependencies. To expedite the setup of a dev env for this script and any HCA ingest work, I had created a Docker Compose dev env. The latest version of the image had a known poetry issue whereby not using a virtualenv (as is suggested for Poetry in Docker) ran into an instance of Poetry removing a previously installed library (cffi) which was a dependency of several other required libraries and dependencies used by manifest.py. As such, attempting to run manifest.py in the Docker Compose dev env failed - and the work around of setting up a local dev takes 10-12 hours - which is prohibitive, to say the least.

Solution:

Added cffi to the pyproject.toml, so that it is required and cannot be removed.

To Test:

• clone this repository and check out the branch for this PR FE-232-fix-dev-env-for-manifest-dot-py
• from the root of the project invoke the Docker Compose dev env docker compose run -w /hca-ingest app bash
  • Note that if you are not already logged in to gcloud, you will need to do so before running
    the docker compose command, as this will pull the latest image from Artifact Registry.
• Authenticate with gcloud using your Broad credentials gcloud auth login
• Then set up your billing project gcloud config set project PROJECT_ID
• Then set up your default login for applications gcloud auth application-default login
• From orchestration/:
  • Run poetry install to set up a local python virtual environment and install needed dependencies
• From the root of the project, run the port forwarding script in the background ./ops/helmfiles/dagster/forward_ports.sh dev &
• Download the manifest file attached to this PR and put it in the orchestration directory for this project.
  • dcp99_manifest.csv
• From orchestration/ run 'poetry run python3 hca_manage/manifest.py load -e dev -c dcp99_manifest.csv -r dcp99`

The script should complete without error.

More context in the Ops Playbook and the top level README and Ops README in this project.

Checklist

• Documentation has been updated as needed.