Skip to content

Commit

Permalink
CDK: Ensure AirbyteLogger is thread-safe using Lock (#9943)
Browse files Browse the repository at this point in the history
* Ensure AirbyteLogger is thread-safe

- Introduce a global lock to ensure `AirbyteLogger` is thread-safe.
- The `logging` module is thread-safe, however `print` is not, and is currently used. This means that messages sent to stdout can clash if connectors use threading. This is obviously a huge problem when the IPC between the source/destination is stdout!
- A `multiprocessing.Lock` could have been introduced however given that `logging` module is not multiprocess-safe I thought that thread-safety should be first goal.
- IMO the `AirbyteLogger` should be a subclass of the `logging.Logger` so you have thread-safety automatically, however I didn't want to make a huge wholesale change here.

* Revert lock and add deprecation warning instead
  • Loading branch information
jdclarke5 authored Feb 28, 2022
1 parent 61ab34c commit be8f124
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion airbyte-cdk/python/airbyte_cdk/logger.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
# Copyright (c) 2021 Airbyte, Inc., all rights reserved.
#

from deprecated import deprecated
import logging
import logging.config
import sys
Expand Down Expand Up @@ -101,7 +102,7 @@ def log_by_prefix(msg: str, default_level: str) -> Tuple[int, str]:

return log_level, rendered_message


@deprecated(version="0.1.47", reason="Use logging.getLogger('airbyte') instead")
class AirbyteLogger:
def log(self, level, message):
log_record = AirbyteLogMessage(level=level, message=message)
Expand Down

0 comments on commit be8f124

Please sign in to comment.