Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DPO3DPKRT-869/RecordKeeper System and Logger #632

Merged
merged 14 commits into from
Oct 2, 2024

Conversation

EMaslowskiQ
Copy link
Collaborator

RecordKeeper

  • Centralized system that manages Logging w/ audits and Notifications (future)

Logger

  • Structured JSON log data
  • Rate control and management
  • Guaranteed consistency and FIFO output
  • Supports: critical, error, warning, info, debug
  • Contextual data on each message (user, request, caller, section)
  • System levels utilities for TAIL & LESS formatting
  • Profiler statements for measuring function duration
  • Colorized and clean formatted console output
  • Internal throughput testing routines

Note: Just the system. Existing log statements in the system need to be moved over.

(new) RecordKeeper class
(new) Duplicated Logger class under RecordKeeper w/ colorization
(new) api/sandbox/play endpoint for tests
(new) logger class with support for: critical, error, warn, info, debug
(new) logs stored as structured JSON for processing/analysis
(fix) updated LogSection types to be more descriptive
(new) clean console output in development mode
(new) error handling for Log configuration
(new) coloring of messages sent to console
(new) helper to display all hidden charcters. (showAllCharactersWithEscapes)
(new) track duration of operations with profile & profileEnd
(fix) generalize LogEntry creation
(fix) added support for 'critical' and better colors
(fix) incorrect handling of idUser and idRequest
(new) log and profile wrapper functions for RecordKeeper
(new) added milliseconds to profiler output
(fix) better handling of invalid user and request ids
(fix) native types and complex data breaking console output
(fix) restructured code and comments
(new) Security (SEC) LogSection for security concerns (threat)
(fix) user id not formatted correctly
(new) ops_logLess ingests JSON log into Less w/ cleaned output
(new) ops_logTail handles dynamic JSON log file with Tail + clean output
(new) tracking of log counts of each type and exposing in recordKeeper
(new) test data and routines in sandbox
(new) load balancing support for logging w/ configurable rates
(new) environment variable for log target rate (PACKRAT_LOG_RATE)
(new) stats now automatically tracking total counts
(new) internal log testing methods w/ random messages
(new) debug mode with additional logging during Load Balancing
(fix) posting logs now asynchronous and blocking
(fix) more accurate stats. updating after transports finish
(fix) enforce specific levels
(fix) clean up endpoint implementation
(new) actual log rate, maximum log rate, and rolling average log rate
(new) support for creating new log files if the current exceeds 150MB
(new) all routines checking if logger is running before posting logs
(new) allow for reconfiguring basic parameters at runtime
(new) separate RateManager class for handling log rates
@EMaslowskiQ EMaslowskiQ self-assigned this Oct 2, 2024
@EMaslowskiQ EMaslowskiQ merged commit 4779a9f into develop Oct 2, 2024
3 checks passed
@EMaslowskiQ EMaslowskiQ deleted the DPO3DPKRT-869/foundation-record-keeper-system branch October 2, 2024 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant