Create guidance for memory-intensive use cases, introduce Python memory profiler

[j-opatz]

Replace italics below with details for this issue.

Describe the Enhancement

During the creation of the use case for issue #1004, it was found that Github Actions automated testing has a limited capacity for memory. As a result of this use case requiring a vast memory storage for its Python script, the automated testing fails. Even after adjustments to try and limit the amount of data held in memory, the automated testing continues to exit unsuccessfully.

The goal of this issue is twofold: the first is to document in the Contributor's Guide what to do when this type of scenario happens.

The second goal is the introduction of a Python memory logger/profiler that becomes standard when creating use cases. By introducing this tool, along with a memory "cap", contributors can know what to do if their use case will exceed the Github automated testing before pushing the branch into automated testing.

Time Estimate

Documentation goal: 1 day
Python memory profiler: 2-4 days

Sub-Issues

Consider breaking the enhancement down into sub-issues.

• document in Contributor's Guide what to do for large memory usage in use cases
• Research and introduce Python memory logger/profiler as new standard for use case creation

Relevant Deadlines

None

Funding Source

2792541 and 2702691

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Linked issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[j-opatz]

@georgemccabe can you list the current Github memory limit in the Issue write-up? A link to a Github page detailing that limit would be even better. I'm trying to draft up some language on this.

[georgemccabe]

@j-opatz here is info about the environment that runs in GitHub Actions: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources
We use Linux.

[j-opatz]

Linked PR was approved and successfully manages the issue raised here.