[CT-430] [Bug] Internal event buffer full. Earliest events will be dropped (FIFO).

[jaklan]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Recently we have noticed a new log when running dbt compile:

Internal event buffer full. Earliest events will be dropped (FIFO).

It's a bit confusing what the warning actually means - is that related to dbt or redshift? What's the actual impact of having dropped events?

Expected Behavior

More clear log message.

Steps To Reproduce

Run dbt compile against a Redshift database having thousands of models.

Relevant log output

10:42:12  Running with dbt=1.0.4
10:42:12  Partial parse save file not found. Starting full parse.
10:43:21  Found 3591 models, 8797 tests, 0 snapshots, 6 analyses, 680 macros, 0 operations, 56 seed files, 1722 sources, 0 exposures, 0 metrics
10:43:22  
10:43:35  Concurrency: 10 threads (target='dev')
10:43:35  
10:45:57  Internal event buffer full. Earliest events will be dropped (FIFO).
10:46:28  Done.
10:46:29  Error sending message, disabling tracking

Environment

- OS: macOS
- Python: 3.9
- dbt: 1.0.4

What database are you using dbt with?

redshift

Additional Context

No response

[iknox-fa]

Thanks for the bug report @jaklan and sorry you found that warning a bit confusing. The context here is that dbt has an internal queue of all of the log events that have occurred since invocation and this is what you see when that buffer gets full. In general it can be ignored unless you're working with the dbt-rpc server which uses the internal queue.

Obviously, we can't put all of that context into a simple log message, but I expanded it a bit in this PR. Let me know if that seems reasonable to you.

[jaklan]

@iknox-fa thanks for the clarification! About the PR - I think it's fine, and if dbt provides the verbose flag one day - imho it could be moved there then.