DPO3DPKRT-767/Support Multiple Cook Resources #555
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(new) default data in db for 4 Cook instances (new) 'resources/cook' endpoint for getting best fit for job type. looks at stored support for a specific job type, how many jobs are waiting, and how many jobs are running. include 'job' query param. accepted values: inspect, scene_generation, generate_downloads, photogrammetry.
(fix) updated 'resources/cook' endpoint to use central class (fix) improved logging output
(new) refactored getCookResources to support returning one/many results (new) added async 'initialize' method to IJob interface to allow point for pre-execution initialization of a job. (e.g. find the best fit Cook resource for the job) (new) log statements in OCFL path (fix) log messages not using correct quotes (fix) si-voyager-scene incorrectly labeled (fix) long timeouts for checking state of a resource
(new) exporting CookResourceInfo (fix) several errors in CookJob were not appended to any reports.
EMaslowskiQ
added a commit
that referenced
this pull request
Nov 8, 2023
* [DPO3DPKRT-746] fix/large file upload failure (#543) * switched to streaming approach for file uploads - removed body size check for Nginx to remove filesize cap when uploading - added 'Name' to WorkflowReport to assist with identification and file generation * cleanup of commented prior implementations for UploadWorker * change 'Name' property of WorkflowReport to standard varchar(512) * Updated GLTF dependencies to v2.5.1 (current: 3.5.1) * DPO3DPKRT-760/new edan publishing api (#549) EDAN has new endpoints that need to be reflected deprecating previous configurations. new: https://console.si.edu/apis/3d-api-dev/api/v1.0/admin/upsertResource * Updated GLTF dependencies and config to use new EDAN endpoints * Fixes #760: updating to new EDAN endpoints - updated defaults when environment variables aren't set to the EDAN dev environment for safety. Note: all developers need to update their environment variables and related scripts (e.g. env.DEV.bat) to point to the new EDAN server. PACKRAT_EDAN_3D_API = https://console.si.edu/apis/3d-api-dev/ Additionally, if doing development against EDAN update the dev environment variables for the EDAN server. PACKRAT_EDAN_SERVER = https://edandev.si.edu/ * Fixes DPO3DPKRT-764 allowing for easier local Proxy operation updated nginx-dev.conf to avoid CORS conflicts in modern browsers when running locally in proxy mode. * DPO3DPKRT-763/generate download scenes workaround (#552) * Temporarily bypass Generate Downloads while development wraps up Stakeholders require publishing capability but an issue with the generate downloads process pollutes the system with duplicates. this temporary fix is only for while that development is wrapped up (active) * remove additional reference to generating downloads * cleanup of unused variables due to temporary fix * fixed conditional logic when ingesting and added log output * DPO3DPKRT-768/Update LDAP Environment Vars (#554) LDAP/LDAPS endpoints have been updated to better enforce compliance and correct configurations. All environment variables (.env.dev, .env.prod, etc.) need to be updated to use the new endpoint/port for authentication against ActiveDirectory to work. No code has been committed as the environment files sit outside of source control. PACKRAT_LDAP_SERVER=ldap://si-hdc-usdc01.US.SINET.SI.EDU:389 Note: that this will need to be updated further as the port (389) is not their preferred port (636), but that is currently not working as expected and will require deeper configuration and testing. * DPO3DPKRT-767/Support Multiple Cook Resources (#555) * (new) table for CookResources * (new) default data in db for 4 Cook instances * (new) 'resources/cook' endpoint for getting best fit for job type. looks at stored support for a specific job type, how many jobs are waiting, and how many jobs are running. include 'job' query param. accepted values: inspect, scene_generation, generate_downloads, photogrammetry. * (new) centralized functionality into class 'CookResource' * (new) getCookEndpoint routine for returning ideal endpoint for CookJobs * (new) refactored getCookResources to support returning one/many results * (new) added async 'initialize' method to IJob interface to allow point for pre-execution initialization of a job. (e.g. find the best fit Cook resource for the job) * (new) log statements in OCFL path * (new) detailed statement on resource selection added to workflow report. * (new) exporting CookResourceInfo * (new) additional logging forCook resources and utility routine * (fix) updated 'resources/cook' endpoint to use central class * (fix) improved logging output * (fix) long timeouts for checking state of a resource * (fix) several errors in CookJob were not appended to any reports. * DPO3DPKRT-767/support-multiple-cook-resources : jobs 'created' support (#557) * (new) factoring in jobs created field for better distribution across resources.
EMaslowskiQ
added a commit
that referenced
this pull request
Nov 30, 2023
* [DPO3DPKRT-746] fix/large file upload failure (#543) * switched to streaming approach for file uploads - removed body size check for Nginx to remove filesize cap when uploading - added 'Name' to WorkflowReport to assist with identification and file generation * cleanup of commented prior implementations for UploadWorker * change 'Name' property of WorkflowReport to standard varchar(512) * Updated GLTF dependencies to v2.5.1 (current: 3.5.1) * DPO3DPKRT-760/new edan publishing api (#549) EDAN has new endpoints that need to be reflected deprecating previous configurations. new: https://console.si.edu/apis/3d-api-dev/api/v1.0/admin/upsertResource * Updated GLTF dependencies and config to use new EDAN endpoints * Fixes #760: updating to new EDAN endpoints - updated defaults when environment variables aren't set to the EDAN dev environment for safety. Note: all developers need to update their environment variables and related scripts (e.g. env.DEV.bat) to point to the new EDAN server. PACKRAT_EDAN_3D_API = https://console.si.edu/apis/3d-api-dev/ Additionally, if doing development against EDAN update the dev environment variables for the EDAN server. PACKRAT_EDAN_SERVER = https://edandev.si.edu/ * Fixes DPO3DPKRT-764 allowing for easier local Proxy operation updated nginx-dev.conf to avoid CORS conflicts in modern browsers when running locally in proxy mode. * DPO3DPKRT-763/generate download scenes workaround (#552) * Temporarily bypass Generate Downloads while development wraps up Stakeholders require publishing capability but an issue with the generate downloads process pollutes the system with duplicates. this temporary fix is only for while that development is wrapped up (active) * remove additional reference to generating downloads * cleanup of unused variables due to temporary fix * fixed conditional logic when ingesting and added log output * DPO3DPKRT-768/Update LDAP Environment Vars (#554) LDAP/LDAPS endpoints have been updated to better enforce compliance and correct configurations. All environment variables (.env.dev, .env.prod, etc.) need to be updated to use the new endpoint/port for authentication against ActiveDirectory to work. No code has been committed as the environment files sit outside of source control. PACKRAT_LDAP_SERVER=ldap://ldaps.si.edu:636 * DPO3DPKRT-767/Support Multiple Cook Resources (#555) * (new) table for CookResources * (new) default data in db for 4 Cook instances * (new) 'resources/cook' endpoint for getting best fit for job type. looks at stored support for a specific job type, how many jobs are waiting, and how many jobs are running. include 'job' query param. accepted values: inspect, scene_generation, generate_downloads, photogrammetry. * (new) centralized functionality into class 'CookResource' * (new) getCookEndpoint routine for returning ideal endpoint for CookJobs * (new) refactored getCookResources to support returning one/many results * (new) added async 'initialize' method to IJob interface to allow point for pre-execution initialization of a job. (e.g. find the best fit Cook resource for the job) * (new) log statements in OCFL path * (new) detailed statement on resource selection added to workflow report. * (new) exporting CookResourceInfo * (new) additional logging forCook resources and utility routine * (fix) updated 'resources/cook' endpoint to use central class * (fix) improved logging output * (fix) long timeouts for checking state of a resource * (fix) several errors in CookJob were not appended to any reports. * DPO3DPKRT-767/support-multiple-cook-resources : jobs 'created' support (#557) * (new) factoring in jobs created field for better distribution across resources. * Updated to LDAPS and ldaps.si.edu. (#559)
EMaslowskiQ
added a commit
that referenced
this pull request
Dec 13, 2023
* [DPO3DPKRT-746] fix/large file upload failure (#543) - switched to streaming approach for file uploads - removed body size check for Nginx to remove filesize cap when uploading - added 'Name' to WorkflowReport to assist with identification and file generation - change 'Name' property of WorkflowReport to standard varchar(512) - updated GLTF dependencies to v2.5.1 (current: 3.5.1) * DPO3DPKRT-760/new edan publishing api (#549) - EDAN has new endpoints that need to be reflected deprecating previous configurations. new: https://console.si.edu/apis/3d-api-dev/api/v1.0/admin/upsertResource Note: all developers need to update their environment variables and related scripts (e.g. env.DEV.bat) to point to the new EDAN server. PACKRAT_EDAN_3D_API = https://console.si.edu/apis/3d-api-dev/ Additionally, if doing development against EDAN update the dev environment variables for the EDAN server. PACKRAT_EDAN_SERVER = https://edandev.si.edu/ * Fixes DPO3DPKRT-764 allowing for easier local Proxy operation - updated nginx-dev.conf to avoid CORS conflicts in modern browsers when running locally in proxy mode. * Temporarily bypass Generate Downloads while development wraps up - Stakeholders require publishing capability but an issue with the generate downloads process pollutes the system with duplicates. this temporary fix is only for while that development is wrapped up (active) * DPO3DPKRT-768/Update LDAP Environment Vars (#554) - LDAP/LDAPS endpoints have been updated to better enforce compliance and correct configurations. All environment variables (.env.dev, .env.prod, etc.) need to be updated to use the new endpoint/port for authentication against ActiveDirectory to work. No code has been committed as the environment files sit outside of source control. PACKRAT_LDAP_SERVER=ldap://si-hdc-usdc01.US.SINET.SI.EDU:389 Note: that this will need to be updated further as the port (389) is not their preferred port (636), but that is currently not working as expected and will require deeper configuration and testing. * DPO3DPKRT-767/Support Multiple Cook Resources (#555) - 'resources/cook' endpoint for getting best fit for job type. looks at stored support for a specific job type, how many jobs are waiting, and how many jobs are running. include 'job' query param. accepted values: inspect, scene_generation, generate_downloads, photogrammetry. Note: developers will need to add the new tables to their local db. Refer to Packrat.ALTER.sql * DPO3DPKRT-770/URI Encoding and Filename Cleanup - improved name sanitizing for ingest titles to avoid conflicts with other systems (e.g. Voyager, Cook, etc.). replaces all special characters with '-' except for -, _, and . - proper use of encodeURI vs. encodeURIComponent for Voyager references
EMaslowskiQ
added a commit
that referenced
this pull request
Jan 24, 2024
* (new) table for CookResources * (new) default data in db for 4 Cook instances * (new) 'resources/cook' endpoint for getting best fit for job type. looks at stored support for a specific job type, how many jobs are waiting, and how many jobs are running. include 'job' query param. accepted values: inspect, scene_generation, generate_downloads, photogrammetry. * (new) centralized functionality into class 'CookResource' * (new) getCookEndpoint routine for returning ideal endpoint for CookJobs * (new) refactored getCookResources to support returning one/many results * (new) added async 'initialize' method to IJob interface to allow point for pre-execution initialization of a job. (e.g. find the best fit Cook resource for the job) * (new) log statements in OCFL path * (new) detailed statement on resource selection added to workflow report. * (new) exporting CookResourceInfo * (new) additional logging forCook resources and utility routine * (fix) updated 'resources/cook' endpoint to use central class * (fix) improved logging output * (fix) long timeouts for checking state of a resource * (fix) several errors in CookJob were not appended to any reports.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added support for dynamic Cook resource assignment in CookJobs. When a job is created, available Cook resources are queried to find the best fit for the current job type looking at the resource's configuration and # of jobs running/waiting. This allows for better utilization of Cook resources and load balancing for concurrent jobs.
Currently supports the following jobs:
Additionally, a public endpoint was created allowing other services (i.e. Cook, Voyager, etc.) to query Packrat to get the best fit Cook resource. This is useful for various automations including Photogrammetry.
Example:
Developers need to add the CookResource table to their local DB. See 'Packrat.ALTER.sql' for details and default data.