DPO3DPKRT-763/generate download scenes workaround #552
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Stakeholders require puiblishing capability but an issue with the geenrate downloads process pollutes the system with duplicates. this temporary fix is only for while that development is wrapped up (active)
- didn't carry over from previous commit.
EMaslowskiQ
deleted the
DPO3DPKRT-763/hotfix-generate-download-scenes-workaround
branch
EMaslowskiQ
added a commit
that referenced
this pull request
Oct 11, 2023
[DPO3DPKRT-746] fix/large file upload failure (#543) * switched to streaming approach for file uploads * removed body size check for Nginx to remove filesize cap when uploading * added 'Name' to WorkflowReport to assist with identification and file generation * Updated GLTF dependencies to v2.5.1 (current: 3.5.1) [DPO3DPKRT-760] new edan publishing api (#549) EDAN has new endpoints that need to be reflected deprecating previous configurations. new: https://console.si.edu/apis/3d-api-dev/api/v1.0/admin/upsertResource Note: all developers need to update their environment variables and related scripts (e.g. env.DEV.bat) to point to the new EDAN server. PACKRAT_EDAN_3D_API = https://console.si.edu/apis/3d-api-dev/ Additionally, if doing development against EDAN update the dev environment variables for the EDAN server. PACKRAT_EDAN_SERVER = https://edandev.si.edu/ [DPO3DPKRT-764] allowing for easier local Proxy operation [DPO3DPKRT-763] generate download scenes workaround (#552) * Temporarily bypass Generate Downloads while development wraps up Stakeholders require publishing capability but an issue with the generate downloads process pollutes the system with duplicates. this temporary fix is only for while that development is wrapped up (active)
EMaslowskiQ
added a commit
that referenced
this pull request
Nov 8, 2023
* [DPO3DPKRT-746] fix/large file upload failure (#543) * switched to streaming approach for file uploads - removed body size check for Nginx to remove filesize cap when uploading - added 'Name' to WorkflowReport to assist with identification and file generation * cleanup of commented prior implementations for UploadWorker * change 'Name' property of WorkflowReport to standard varchar(512) * Updated GLTF dependencies to v2.5.1 (current: 3.5.1) * DPO3DPKRT-760/new edan publishing api (#549) EDAN has new endpoints that need to be reflected deprecating previous configurations. new: https://console.si.edu/apis/3d-api-dev/api/v1.0/admin/upsertResource * Updated GLTF dependencies and config to use new EDAN endpoints * Fixes #760: updating to new EDAN endpoints - updated defaults when environment variables aren't set to the EDAN dev environment for safety. Note: all developers need to update their environment variables and related scripts (e.g. env.DEV.bat) to point to the new EDAN server. PACKRAT_EDAN_3D_API = https://console.si.edu/apis/3d-api-dev/ Additionally, if doing development against EDAN update the dev environment variables for the EDAN server. PACKRAT_EDAN_SERVER = https://edandev.si.edu/ * Fixes DPO3DPKRT-764 allowing for easier local Proxy operation updated nginx-dev.conf to avoid CORS conflicts in modern browsers when running locally in proxy mode. * DPO3DPKRT-763/generate download scenes workaround (#552) * Temporarily bypass Generate Downloads while development wraps up Stakeholders require publishing capability but an issue with the generate downloads process pollutes the system with duplicates. this temporary fix is only for while that development is wrapped up (active) * remove additional reference to generating downloads * cleanup of unused variables due to temporary fix * fixed conditional logic when ingesting and added log output * DPO3DPKRT-768/Update LDAP Environment Vars (#554) LDAP/LDAPS endpoints have been updated to better enforce compliance and correct configurations. All environment variables (.env.dev, .env.prod, etc.) need to be updated to use the new endpoint/port for authentication against ActiveDirectory to work. No code has been committed as the environment files sit outside of source control. PACKRAT_LDAP_SERVER=ldap://si-hdc-usdc01.US.SINET.SI.EDU:389 Note: that this will need to be updated further as the port (389) is not their preferred port (636), but that is currently not working as expected and will require deeper configuration and testing. * DPO3DPKRT-767/Support Multiple Cook Resources (#555) * (new) table for CookResources * (new) default data in db for 4 Cook instances * (new) 'resources/cook' endpoint for getting best fit for job type. looks at stored support for a specific job type, how many jobs are waiting, and how many jobs are running. include 'job' query param. accepted values: inspect, scene_generation, generate_downloads, photogrammetry. * (new) centralized functionality into class 'CookResource' * (new) getCookEndpoint routine for returning ideal endpoint for CookJobs * (new) refactored getCookResources to support returning one/many results * (new) added async 'initialize' method to IJob interface to allow point for pre-execution initialization of a job. (e.g. find the best fit Cook resource for the job) * (new) log statements in OCFL path * (new) detailed statement on resource selection added to workflow report. * (new) exporting CookResourceInfo * (new) additional logging forCook resources and utility routine * (fix) updated 'resources/cook' endpoint to use central class * (fix) improved logging output * (fix) long timeouts for checking state of a resource * (fix) several errors in CookJob were not appended to any reports. * DPO3DPKRT-767/support-multiple-cook-resources : jobs 'created' support (#557) * (new) factoring in jobs created field for better distribution across resources.
EMaslowskiQ
added a commit
that referenced
this pull request
Nov 30, 2023
* [DPO3DPKRT-746] fix/large file upload failure (#543) * switched to streaming approach for file uploads - removed body size check for Nginx to remove filesize cap when uploading - added 'Name' to WorkflowReport to assist with identification and file generation * cleanup of commented prior implementations for UploadWorker * change 'Name' property of WorkflowReport to standard varchar(512) * Updated GLTF dependencies to v2.5.1 (current: 3.5.1) * DPO3DPKRT-760/new edan publishing api (#549) EDAN has new endpoints that need to be reflected deprecating previous configurations. new: https://console.si.edu/apis/3d-api-dev/api/v1.0/admin/upsertResource * Updated GLTF dependencies and config to use new EDAN endpoints * Fixes #760: updating to new EDAN endpoints - updated defaults when environment variables aren't set to the EDAN dev environment for safety. Note: all developers need to update their environment variables and related scripts (e.g. env.DEV.bat) to point to the new EDAN server. PACKRAT_EDAN_3D_API = https://console.si.edu/apis/3d-api-dev/ Additionally, if doing development against EDAN update the dev environment variables for the EDAN server. PACKRAT_EDAN_SERVER = https://edandev.si.edu/ * Fixes DPO3DPKRT-764 allowing for easier local Proxy operation updated nginx-dev.conf to avoid CORS conflicts in modern browsers when running locally in proxy mode. * DPO3DPKRT-763/generate download scenes workaround (#552) * Temporarily bypass Generate Downloads while development wraps up Stakeholders require publishing capability but an issue with the generate downloads process pollutes the system with duplicates. this temporary fix is only for while that development is wrapped up (active) * remove additional reference to generating downloads * cleanup of unused variables due to temporary fix * fixed conditional logic when ingesting and added log output * DPO3DPKRT-768/Update LDAP Environment Vars (#554) LDAP/LDAPS endpoints have been updated to better enforce compliance and correct configurations. All environment variables (.env.dev, .env.prod, etc.) need to be updated to use the new endpoint/port for authentication against ActiveDirectory to work. No code has been committed as the environment files sit outside of source control. PACKRAT_LDAP_SERVER=ldap://ldaps.si.edu:636 * DPO3DPKRT-767/Support Multiple Cook Resources (#555) * (new) table for CookResources * (new) default data in db for 4 Cook instances * (new) 'resources/cook' endpoint for getting best fit for job type. looks at stored support for a specific job type, how many jobs are waiting, and how many jobs are running. include 'job' query param. accepted values: inspect, scene_generation, generate_downloads, photogrammetry. * (new) centralized functionality into class 'CookResource' * (new) getCookEndpoint routine for returning ideal endpoint for CookJobs * (new) refactored getCookResources to support returning one/many results * (new) added async 'initialize' method to IJob interface to allow point for pre-execution initialization of a job. (e.g. find the best fit Cook resource for the job) * (new) log statements in OCFL path * (new) detailed statement on resource selection added to workflow report. * (new) exporting CookResourceInfo * (new) additional logging forCook resources and utility routine * (fix) updated 'resources/cook' endpoint to use central class * (fix) improved logging output * (fix) long timeouts for checking state of a resource * (fix) several errors in CookJob were not appended to any reports. * DPO3DPKRT-767/support-multiple-cook-resources : jobs 'created' support (#557) * (new) factoring in jobs created field for better distribution across resources. * Updated to LDAPS and ldaps.si.edu. (#559)
EMaslowskiQ
added a commit
that referenced
this pull request
Jan 24, 2024
* Temporarily bypass Generate Downloads while development wraps up Stakeholders require publishing capability but an issue with the generate downloads process pollutes the system with duplicates. this temporary fix is only for while that development is wrapped up (active) * remove additional reference to generating downloads * cleanup of unused variables due to temporary fix * fixed conditional logic when ingesting and added log output
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Temporary workaround bypassing generate downloads to allow users to ingest/publish while development wraps on a generate downloads issue.