Improve Self Diagnostics and support setting configuration in file

[xiang17]

The motivation for this change is to improve troubleshooting the SDK as another attempt for #1981 in a different approach.

I'm proposing an approach for a self diagnostics module that is enabled by modifying a configuration file, rather than an environment variable. For example, the log starts flowing into a file after the SysAdmin or DevOps sets up a configuration in a certain location with contents like this:

{
    "LogDirectory": ".",
    "FileSize": 1024,
    "LogLevel": "Error"
}

There is no need of redeployment/restart to enable and disable the logs. In comparison, the environment variables are set for a process when it starts and are not updated. There are workarounds(on Windows), but they will be OS-specific.

Self Diagnostics Module Design Principles

1. The self diagnostics module captures and write log messages to disk, when enabled.
2. It should have minimal overhead to the current process so the user can feel confident to enable it.
3. It should be accessible to use in most environments/cases. For example, we want to use the least dependencies and/or features of the OS. It doesn't require administrator permissions. It can be enabled/disabled without restarting the process.
4. Bounded resource usage. For example, it should not consume ever-growing memory or generate an ever-growing file.

Implementation Plan

1. Self Diagnostics Module listens to the events with EventListener.
2. Self Diagnostics Module writes the message in the event to a fix-sized local file at a certain location in a circular way. (Overwrite from beginning once write position reaches the end.)
3. Self Diagnostics Module checks a configuration file for log file location and file size periodically (for example, every 10 seconds).
4. By default, the configuration doesn't exist, Self Diagnostics Module will not write to any file. Once the file is created with valid configuration format/values, Self Diagnostics Module will start writing log file according to configuration. After the file is deleted or modified, the log file handle will be updated, either closed (if deleted or modified with invalid configuration) or closed and reopened according to the configuration file modification (if modified with valid configuration).
5. To optimize write performance, Self Diagnostics Module will utilize Memory-Mapped Files feature instead of directly writing to file.

I'm targeting to finish the feature before the 2.18 milestone

[TimothyMothra]

I have some feedback on this:

I'm proposing an approach for a self diagnostics module that is enabled by modifying a configuration file, rather than an environment variable.

The reason an environment variable was suggested is that we have some customers who cannot redeploy their application just to change a config file, or the config file is unavailable. Having an environment variable trigger would allow us to investigate production servers when an error condition is occurring.

Does your proposal offer any new features that we don't already provide with the current config-based File logger?

[cijothomas]

who cannot redeploy their application just to change a config file

If this design follows the Otel self-diagnostics, then the config file is not required to be part of the application, and can be added/modified/removed without re-deploying the app.

[xiang17]

That's right. It doesn't need redeployment to enable/disable the log. The user just need to create/modify/delete a file at a certain location in Prod.