OJ-35758: rework log handlers to use a smart filter

[gavinpitt-jf]

Description

Improve how we divert/filter what logs go to the Console logger and what logs go to the Agent Log file.

Previously, we relied on exclusively filtering DEBUG level messages away from the Console logger, but into the Log file Handler. Now, we rely on the send_to_agent_log_file helper function from jf_ingest.logging_helper, which explicitly tags log records as being for only the agent log file. This allows us to retain the log records levels, instead of flattening them all to logging.DEBUG

This PR must be merged after this ingest PR is merged and deployed

Testing

Here is the Agent Console output, which is very user friendly. Below that is the log file, which contains some extra logging information (NOTE: the logs around "skipping sprints" are now INFO level logs instead of DEBUG :) )

Logging setup complete with handlers for log file, stdout, and streaming.
Will write output files into ./output/20240605_155126
Running ingestion healthcheck validation!
Validating configuration...

Jira details:
  URL:      https://jelly-ai.atlassian.net
  Username: [email protected]
  Password: **********
==> Testing Jira connection...
Authenticating to Jira API at https://jelly-ai.atlassian.net using the username and password secrets for [email protected] of company orthogonal-networks
==> Getting Jira version...
Found Jira version as 1001.0.0-SNAPSHOT
==> Getting Jira deployment type...
Response headers does not contain X-ANODEID! Customer is NOT running Jira Data Center.
==> Getting Jira permissions...
Found granted permissions as ['DELETE_OWN_WORKLOGS', 'CREATE_ISSUES', 'DELETE_OWN_COMMENTS', 'WORK_ON_ISSUES', 'MODIFY_REPORTER', 'EDIT_ISSUES', 'ADD_COMMENTS', 'EDIT_OWN_COMMENTS', 'ASSIGN_ISSUES', 'BROWSE_PROJECTS', 'EDIT_OWN_WORKLOGS', 'EDIT_ALL_WORKLOGS', 'EDIT_ALL_COMMENTS', 'CLOSE_ISSUES', 'SET_ISSUE_SECURITY', 'SCHEDULE_ISSUES', 'USER_PICKER', 'DELETE_ALL_COMMENTS', 'ADMINISTER_PROJECTS', 'RESOLVE_ISSUES', 'DELETE_ISSUES', 'VIEW_READONLY_WORKFLOW', 'MOVE_ISSUES', 'ASSIGNABLE_USER', 'TRANSITION_ISSUES', 'LINK_ISSUES', 'DELETE_ALL_WORKLOGS']
==> Testing Jira user browsing permissions...
Downloading Users...
Done downloading Users! Found 469 users
We can access 469 Jira users.
==> Testing Jira project permissions...
With provided credentials, the following projects are discoverable: {'JFR', 'OJ'}.
Checking project access.
Testing access for project: "JFR"
With provided credentials, we can access issues, versions, and components within project JFR
Testing access for project: "OJ"
With provided credentials, we can access issues, versions, and components within project OJ
Checking access to fields
Checking access to resolutions
Checking access to issue types
Checking access to issue link types
Checking access to priorities
Checking access to boards
Checking access to sprints
. Skipping Git Validation.

Memory & Disk Usage:
  Available memory: 768.25 MB
  Disk usage for jf_agent/output: 301 GB / 460 GB
  Size of jf_agent/output dir:  24K
Attempting to upload healthcheck result to s3...
Successfully uploaded healthcheck.json
Successfully uploaded jf_agent.log
Successfully uploaded healthcheck result to s3!

Done
Obtained Jira configuration, attempting download...
Attempting to use JF Ingest for Jira Ingestion
Set global value INGESTION_TYPE to AGENT
Beginning load_and_push_jira_to_s3
Using local version of ingest
Authenticating to Jira API at https://jelly-ai.atlassian.net using the username and password secrets for [email protected] of company orthogonal-networks
Data will not be saved locally
Data will be submitted to jellyfish
Downloading Jira Projects...
Done downloading Projects!
Downloading Jira Project Components...
Done downloading Project Components!
Downloading Jira Versions...
Done downloading Jira Versions!
Done downloading Jira Project, Components, and Version. Found 2 projects
Downloading Jira Fields...
Done downloading Jira Fields! Found 214 fields
Downloading Users...
Done downloading Users! Found 469 users
Downloading Jira Resolutions...
Done downloading Jira Resolutions! Found 9 resolutions
Downloading IssueTypes...
Done downloading IssueTypes! found 34 Issue Types
Downloading IssueLinkTypes...
Done downloading IssueLinkTypes! Found 10 Issue Link Types
Downloading Jira Priorities...
Done downloading Jira Priorities! Found 5 priorities
Downloading Jira Statuses...
Done downloading Jira Statuses! Found 122
Downloading Boards...
Done downloading Boards! Found 54 boards
Downloading Sprints...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 54/54 [00:14<00:00,  3.67it/s]
Skipping issues and worklogs bc config.skip_issues is True
Data has not been saved locally, because save_locally was set to false in the ingest config!
Data has been submitted to jellyfish
Starting Git download for 1 provided git configurations
downloading github projects... ✓
downloading github users... ✓
downloading github repos...
downloading repos for Jellyfish-AI: 1repos [00:09,  9.15s/repos]
Done downloading repos!
downloading github commits on included branches...
downloading commits for branch develop in repo jellyfish (Jellyfish-AI): 447commits [00:02, 164.70commits/s]
Done downloading commits for included branches!
downloading github prs...
processing prs for jellyfish (81621092): 419prs [00:26, 15.94prs/s]
Done downloading github PRs!
Compressing ./output/20240605_155126/diagnostics.json
Compressing ./output/20240605_155126/healthcheck.json
Sending data to Jellyfish...
Starting 8 threads
Successfully uploaded diagnostics.json.gz
Successfully uploaded git_8v1crHqhmq/bb_projects.json.gz
Successfully uploaded git_8v1crHqhmq/bb_users.json.gz
Successfully uploaded status.json.gz
Successfully uploaded healthcheck.json.gz
Successfully uploaded git_8v1crHqhmq/bb_repos.json.gz
Successfully uploaded git_8v1crHqhmq/bb_commits.json.gz
Successfully uploaded git_8v1crHqhmq/bb_prs.json.gz
Successfully uploaded config.yml
Agent run succeeded: True
Successfully uploaded jf_agent.log
Successfully uploaded .done
Done!
Closing the agent log stream.
Log stream stopped.

jf_agent.log