Asynchronous/double-buffering support?

[toddlipcon]

This has come up a few times in the past, but figured I'd re-raise it after the transition from google code to github:

I'm seeing a case in my application where doing LOG() calls from an event loop thread is causing the entire event loop to stall for hundreds of milliseconds. Eventually I captured stacks showing that it's blocking trying to acquire log_mutex while another thread is in the middle of a disk flush. The disk flush can be very slow in the case that the logging disk is in the middle of an ext4 checkpoint (the buffered IO calls file_update_time to update the inode mtime, but the inode is locked for writeback)

A partial solution here would be to add double-buffering within the glog process (instead of relying on fwrite/fflush buffering). Writes would be appended into a buffer, and at "flush" time, we swap the buffer and do a non-buffered write of the old buffer to the file, while not holding the lock. Meanwhile other threads can continue to append to the new buffer without blocking.

A fuller solution would involve deferring the actual logging work to a background thread (eg by a MPSC queue). Either a bounded implementation (which drops messages or blocks when the queue is full) or an unbounded one (which uses a lot of RAM if you can't keep up) could be reasonable choices depending on the workload.

Does anyone have a fork of glog that does this? Would the glog maintainers be willing to accept a pull request if we decided to implement it ourselves?

[Bklyn]

This sounds like a great feature. Holding up any thread to block on expensive I/O operations is a bad idea. There should be a way to set things up so a background thread handles all I/O.

[toddlipcon]

FWIW I ended up implementing this using existing public glog APIs, in case anyone's interested in borrowing the code for their projects (or for distribution as part of glog)

https://github.com/apache/kudu/blob/master/src/kudu/util/async_logger.h
https://github.com/apache/kudu/blob/master/src/kudu/util/async_logger.cc
https://github.com/apache/kudu/blob/master/src/kudu/util/logging.cc#L129 (EnableAsyncLogging)

[steve-o]

An additional enhancement with caveats is to replace the mutex with a lock-free queue, there is discussion by the author of g3log here:

https://kjellkod.wordpress.com/2015/06/30/the-worlds-fastest-logger-vs-g3log/

[tangfu]

@toddlipcon ：hi，have you benchmark kudu's async_logger？with native glog？

[toddlipcon]

I didn't do throughput benchmarks, but the async logging solved a lot of blocking issues that we saw before where critical parts of our code would get "stuck" for several hundred milliseconds. Throughput is likely improved as well, but that's rarely a consideration for our app, so we didn't test it.

[Sasthan-SK]

Hi @sergiud @toddlipcon Since the ticket is closed, did we add Asynchronous/double-buffering support in glog ? It doesn't appear so, but can you please confirm.

[sergiud]

No support was added. The issue was closed due to inactivity. I'll reopen it.