[CT-1550] Make log format specification more flexible

[gshank]

We are in the process of changing the logging system to be less dependent on global variables and more self-contained, and also to make specifying custom log managers possible.

Right now we can only set one "log_format" which applies to both the stdout logger and the file logger. In this new logging world we will almost certainly want to be able to be more granular in settings for the loggers. For example, it seems desirable to enable people to use the text logger for stdout and the json logger for the file log. Once we implement a way to produce protobuf logs, people will probably not want to send those binary logs to stdout.

It's an open question about how these configs should be implemented. At the very least we would want to differentiate log formats for existing loggers. But it also feels like we might want to be able to configure different logging output to custom loggers without resorting to Python code.

[DustinMoriarty]

We need this as well. Right now there is no way to turn off logging to the file system. We need to be able to run DBT without write access to the file system for a couple reasons.

1. With the python models, we need to be able to make sure that people are not storing data in the file system. We want to deny the DBT user file system access.
2. File system logging is unnecessary and inefficient. We just need logging to stderr like every other application we put in a container.

Regardless, the whole point of python logging is to leave decisions about logging configuration up to the user. We need a way to provide our own logging configuration. A --log-config-file variable would meet this use case perfectly.

[jtcohen6]

Another related issue: #6615

[jtcohen6]

@DustinMoriarty To your points:

Python model code is shipped up to run remotely in your data warehouse/platform. It isn't possible to access the same file system as the dbt project, unless you're connecting to a data processing engine that runs within the local process (e.g. fal, DuckDB).

Regardless, the whole point of python logging is to leave decisions about logging configuration up to the user. We need a way to provide our own logging configuration.

In v1.4, we have implemented a much more flexible EventManager that should allow programmatic invokers of dbt-core to configure as they want, and register their own custom callbacks if desired. While programmatic invocations (via Python API) are not yet officially supported, we're planning to get this ready for the v1.5 release in April.

[nathaniel-may]

closing in favor of #6639