Overview
Features
Download
Documentation
Community
Add-Ons & Services

LogFile always flushes to disk on Windows

A general discussion forum.

LogFile always flushes to disk on Windows

Postby avogel » 20 Sep 2011, 12:08

Hello,

we are using Poco logging (v1.4.1) on Windows and Linux x86-64, and we noticed that, when using a high log level, it's very slow on Windows, which suggested unbuffered I/O. The Poco documentation did not mention any way to change this behavior, so I looked at the Poco source code. It turned out that LogFileImpl for Win32 always first calls WriteFile which directly passes the message to the OS, then FlushFileBuffers which is a flush to disk operation.

On Unix however, the LogFile_std.cpp does
Code: Select all
_str << text << std::endl;

which effectively calls flush() on the underlying std::ostream every time, which is a flush to OS operation. So this would be equivalent to calling WriteFile but not FlushFileBuffers on Win32.

For our purposes, and I guess for most applications, flushing every message to disk is excessive as this is only adds additional safety in the event of a system crash, hard reset, power failure or maybe sudden unplugging of the device. On the other hand, this greatly limits the performance of logging and may affect the performance of other parts of the application if they perform some activity on the same disk. This is a problem when generating debug or trace logs, as our customers will occasionally have to do when there's an issue with our application. As mentioned above, this only affects Windows, on Linux there is no flush to disk but only to the operating system. This means, when the logging call returns, the data is guaranteed to be in the OS cache but probably not the device yet. This is ok for our purposes since we are mostly concerned about application crashes and not system crashes.

For now, we consider creating a custom subclass of Poco::FileLogger and overriding the write method at least to only use WriteFile. Shouldn't there be an easier way though especially given that Win32 and Unix behavior are different? It's quite unusual for a logging framework to perform a flush to disk after every message also.

Best regards,
avogel

EDIT: Completed the post after submitting too early.
avogel
 
Posts: 1
Joined: 20 Sep 2011, 11:57

Re: LogFile always flushes to disk on Windows

Postby victor » 02 Feb 2012, 12:17

In your case, is it okay to use a Poco::AsyncChannel as the first channel to wrap all things into a logging thread?
victor
 
Posts: 5
Joined: 29 Dec 2010, 06:18

Re: LogFile always flushes to disk on Windows

Postby guenter » 02 Feb 2012, 13:52

I guess we really should make the flushing behavior optional (controlled via a property) in the next (patch) release.
Bug report here: https://sourceforge.net/tracker/index.p ... tid=725709
guenter
 
Posts: 1155
Joined: 11 Jul 2006, 16:27
Location: Austria

Re: LogFile always flushes to disk on Windows

Postby codecandy2k » 02 Feb 2012, 20:20

I would also make NOT flushing the default, as the utility of forcing a system flush on every log file write is extremely minimal in all but the most critical systems, where a more robust transaction-based system would probably be better anyway. FlushFileBuffers can be extremely expensive. I've seen it take upwards of 500ms on a high volume server, which would completely kill system performance when used for logging.

It would also be counter-productive if logged from a separate thread where your log entries would end up being badly backed up while waiting for the flush to complete.
codecandy2k
 
Posts: 40
Joined: 03 Dec 2011, 18:48


Return to General Discussion

Who is online

Users browsing this forum: No registered users and 1 guest