www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Creating a logging library and have questions

reply sybrandy <sybrandy gmail.com> writes:
Hello,

Since I'm continuing to write software in D (no, nothing special...just 
pet projects) I felt that I needed to create a library to handle logging 
to a file.  I have a good start and it seems to work pretty well.  I 
even have a couple ideas left to implement to make it better, however 
I'm having trouble with one.  One thing that I did implement, which I 
think is pretty cool, is a set up the class to start a thread as a 
daemon to run in the background and check for updates to the config 
file, much like log4j.  This way the user can change the log level at 
runtime.

One thing I'm thinking that I want to do is have the writing to the 
logfile handled by a separate class.  This way I'm hoping to ensure that 
there's not a lot of unnecessary synchronizing going on and the amount 
of work writing to the logfile won't slow down the rest of the program. 
  My question is, what is the best approach?

1) I use the current core.thread library and put all my messages in a 
buffer that the thread checks periodically, pulls off a new message, and 
then writes it to a file.  There will be some work to make sure nothing 
collides with each other, but I think I can manage it.

2) I wait for the new threading library with message passing to come out 
and just rely on the message passing to handle everything.  It's a much 
cleaner approach based on my experience with Erlang, but there are two 
issues and the major one is I have no idea when it'll be ready. 
Granted, I don't need this capability now, but I'm interested in getting 
it to work.

3) Something else.  I really don't have much experience with threading, 
so I'm being very careful and really want to understand it.  This 
library looks to be a good way to learn, however if it's not the best 
way to do things, then what would be?

Thanks in advance for any input.

Casey
Feb 02 2010
parent reply Rainer Deyke <rainerd eldwood.com> writes:
sybrandy wrote:

 1) I use the current core.thread library and put all my messages in a
 buffer that the thread checks periodically, pulls off a new message, and
 then writes it to a file.  There will be some work to make sure nothing
 collides with each other, but I think I can manage it.


Wouldn't this allow logging messages to be lost before a hard crash?  To
me, that would greatly reduce the utility of the logging system.


 3) Something else.  I really don't have much experience with threading,
 so I'm being very careful and really want to understand it.  This
 library looks to be a good way to learn, however if it's not the best
 way to do things, then what would be?


Global mutex associated with the logging system.  Lock, output, unlock.
 There are scalability issues with that approach, but I don't see why it
wouldn't work in this situation.  (Plus, if you have a message queue,
you probably need to protect that queue with a mutex anyway.)


-- 
Rainer Deyke - rainerd eldwood.com
Feb 02 2010
parent reply sybrandy <sybrandy gmail.com> writes:
On 02/03/2010 12:03 AM, Rainer Deyke wrote:

 sybrandy wrote:

 1) I use the current core.thread library and put all my messages in a
 buffer that the thread checks periodically, pulls off a new message, and
 then writes it to a file.  There will be some work to make sure nothing
 collides with each other, but I think I can manage it.


 Wouldn't this allow logging messages to be lost before a hard crash?  To
 me, that would greatly reduce the utility of the logging system.


If done improperly, yes, it would.  My hope is that if I did go with 
this method, I would try to find a way to ensure no data was lost. 
Oddly enough, I'm currently having that problem with the single-threaded 
version for some reason.  The test program will stop and some of the 
logging statements never make it to the file.

 3) Something else.  I really don't have much experience with threading,
 so I'm being very careful and really want to understand it.  This
 library looks to be a good way to learn, however if it's not the best
 way to do things, then what would be?


 Global mutex associated with the logging system.  Lock, output, unlock.
   There are scalability issues with that approach, but I don't see why it
 wouldn't work in this situation.  (Plus, if you have a message queue,
 you probably need to protect that queue with a mutex anyway.)


Understood.  My goal is that if I do put the writing in another thread, 
I do my best to ensure it will scale.  I have a tendency to put a lot of 
logging statements in code when I'm trying to debug something, so I 
don't want to slow things down too much nor do I want to lose anything. 
  In short: I want the log writing to be as out of the way as possible.

Casey
Feb 03 2010
parent reply Steve Teale <steve.teale britseyeview.com> writes:
sybrandy Wrote:


 On 02/03/2010 12:03 AM, Rainer Deyke wrote:

 sybrandy wrote:

 1) I use the current core.thread library and put all my messages in a
 buffer that the thread checks periodically, pulls off a new message, and
 then writes it to a file.  There will be some work to make sure nothing
 collides with each other, but I think I can manage it.


 Wouldn't this allow logging messages to be lost before a hard crash?  To
 me, that would greatly reduce the utility of the logging system.


 
 If done improperly, yes, it would.  My hope is that if I did go with 
 this method, I would try to find a way to ensure no data was lost. 
 Oddly enough, I'm currently having that problem with the single-threaded 
 version for some reason.  The test program will stop and some of the 
 logging statements never make it to the file.

 3) Something else.  I really don't have much experience with threading,
 so I'm being very careful and really want to understand it.  This
 library looks to be a good way to learn, however if it's not the best
 way to do things, then what would be?


 Global mutex associated with the logging system.  Lock, output, unlock.
   There are scalability issues with that approach, but I don't see why it
 wouldn't work in this situation.  (Plus, if you have a message queue,
 you probably need to protect that queue with a mutex anyway.)


 
 Understood.  My goal is that if I do put the writing in another thread, 
 I do my best to ensure it will scale.  I have a tendency to put a lot of 
 logging statements in code when I'm trying to debug something, so I 
 don't want to slow things down too much nor do I want to lose anything. 
   In short: I want the log writing to be as out of the way as possible.
 
 Casey


I second Rainer. A logging system should commit (at least) error messages
immediately, particularly if the application has multiple threads. Otherwise it
is going to make debugging a crashing system a nightmare.  When I do it I just
stick 'synchronized' in front of the statement that does the write.
Feb 06 2010
parent sybrandy <sybrandy gmail.com> writes:
 I second Rainer. A logging system should commit (at least) error messages
immediately, particularly if the application has multiple threads. Otherwise it
is going to make debugging a crashing system a nightmare.  When I do it I just
stick 'synchronized' in front of the statement that does the write.


Yes, I fully understand that and in the current single-threaded version 
I have, that's exactly what happens: all error and fatal messages cause 
the buffer to flush.

What I'm looking for is the best way to handle having a daemon writer. 
My biggest concern here is multi-threaded applications.  Granted, it 
would be nice to not have the writing in the same thread as the rest of 
the code to try to keep file I/O from affecting performance, but that's 
secondary.

Here's what I know: a variable of type OutputStream cannot be shared.  I 
did not try using __gshared, but regardless while this would work and I 
could easily synchronize the writes, I can see a lot of contention if 
multiple threads trying to write a lot of data to a log file. (E.g. 
trace statements that capture variable states to enhance debugging) 
Granted, this shouldn't be the case in production code, but if I can 
find a better way to do this, I'd love to.  This is why I thought that 
the new threading model with message passing would be good, but I again 
have concerns if there are a lot of messages trying to be written.  I 
know in Erlang you can fill up the message buffer if you're not careful.

Casey
Feb 06 2010