www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - inotify and recursion

reply Hugo Florentino <hugo acdam.cu> writes:
Hi,

By any chance has anyone linked against libnotify so as to monitor 
certain changes in a directory and subdirectories, and act according to 
event?
I am thinking in combining this with fossil [1] to automatically 
document changes in /etc

My main concern is that (if I undestand it correctly), inotify requires 
a file descriptor for every object watched, and if a directory has many 
files this could probably have an impact on performance, or do you know 
if just by using the directory I want to monitor, everything within it 
will also be watched?

Also, I have never linked before against a Linux library and inotify 
does not seem particularly easy because of the file descriptors, so if 
someone has linked against this library and has a sample code, I would 
very much appreciate if you could share it.

Regards, Hugo

[1] www.fossil-scm.org
Dec 26 2013
parent reply "David Eagen" <davideagen mailinator.com> writes:
Here is a first attempt. I'm sure there are much better ways to 
do this but this should at least get you going in the right 
direction.

Some of the key things to know are the undocumented (at least not 
documented on the web site) modules that are available. One of 
them contains the Linux inotify header. If you've downloaded the 
zip file from the website take a look at the src/druntime/src 
directory for these modules.

I took the example from 
http://stackoverflow.com/questions/4062806/inotify-how-to-use-it-linux 
and made some changes to get it to compile and run in D.

import core.sys.linux.sys.inotify;
import core.stdc.stdlib : malloc;
import core.sys.posix.unistd : read;

import std.stdio : writeln;
import std.string: toStringz;

enum PATH_MAX = 256;

void main()
{
     string filename="aaa";

     int inotfd = inotify_init();
     int watch_desc = inotify_add_watch(inotfd, 
toStringz(filename), IN_MODIFY);

     size_t bufsiz = inotify_event.sizeof + PATH_MAX + 1;
     inotify_event* event = cast(inotify_event *) malloc(bufsiz);

     /* wait for an event to occur */
     read(inotfd, event, event.sizeof);

     /* process event struct here */
     writeln("Received inotify event.");
}

As far as monitoring a directory, you can do that with inotify 
the same way you would a file. See 
http://www.ibm.com/developerworks/linux/library/l-ubuntu-inotify/index.html 
for more a C example (Listing 1).
Dec 26 2013
next sibling parent reply Hugo Florentino <hugo acdam.cu> writes:
On Fri, 27 Dec 2013 03:23:00 +0000, David Eagen wrote:

 Here is a first attempt. I'm sure there are much better ways to do
 this but this should at least get you going in the right direction.
 ...


Thanks!

BTW, it it a requirement to use malloc, and if so, when would I need to 
free the memory allocated by it?
Dec 26 2013
next sibling parent "David Eagen" <davideagen mailinator.com> writes:
On Friday, 27 December 2013 at 03:39:58 UTC, Hugo Florentino 
wrote:

 BTW, it it a requirement to use malloc, and if so, when would I 
 need to free the memory allocated by it?


Inotify is expecting you to read into a buffer. You could 
allocate that buffer from the GC with core.memory.malloc() 
instead.

You want to free the memory once you finish processing the event.
Dec 26 2013
prev sibling parent reply "Gary Willoughby" <dev nomad.so> writes:
On Friday, 27 December 2013 at 03:39:58 UTC, Hugo Florentino 
wrote:

 BTW, it it a requirement to use malloc, and if so, when would I 
 need to free the memory allocated by it?


I use a static ubyte array.

I've been using inotify quite a bit and found it to be very good 
but there are a few things to keep in mind though. First it can 
block further program execution when watching files. To avoid 
this use the select function in 'core.sys.posix.sys.select'. 
Second, if the file doesn't exist or is moved once watched, the 
inotifiy instance or the watch descriptor will be invalid and 
need to be re-initialised. Tip: moving can occur if edited with a 
text editor. As i found out trying to test inotify by changing a 
file using vim! That was debug pain!

Here is a snippet of code to show how i used it. It's not 
complete but it shows you how to use the select function and how 
all the bits fit together. I used an infinite loop because this 
was part of a daemon that runs forever so you may want to handle 
that better.

/**
  * Module.
  */
module common.file.watcher.inotifyengine;

/**
  * Imports.
  */
import core.sys.linux.sys.inotify;
import core.sys.posix.sys.select;
import core.sys.posix.unistd;
import common.file.logger;
import common.file.watcher.engine;
import std.string;

/**
  * A class to watch for changes in a file.
  *
  * Uses the linux inotify subsystem.
  */
class INotifyEngine : Engine
{
	/**
	 * The event buffer length.
	 *
	 * This gives us for 1024 events which should be more than 
enough.
	 */
	private enum eventBufferLength = (inotify_event.sizeof + 16) * 
1024;

	/**
	 * The notification flags.
	 *
	 * These are what inotify uses to descriminate on which events 
to nofify us about.
	 */
	private enum notificationFlags  = IN_MODIFY | IN_ATTRIB | 
IN_DELETE_SELF | IN_MOVE_SELF | IN_IGNORED;

	/**
	 * The inotify instance.
	 */
	private int _inotifyInstance;

	/**
	 * The watch descriptor.
	 */
	private int _watchDescriptor;

	/**
	 * The file descriptor set for the select call.
	 */
	private fd_set _fileDescriptorSet;

	/**
	 * The timeout for the select call.
	 */
	private timeval _timeout;

	/**
	 * Constructor.
	 *
	 * Params:
	 *     logger = The logger object used to log messages.
	 *
	 * See_Also:
	 *     Engine
	 */
	public this(Logger logger)
	{
		super(logger);
	}

	/**
	 * Initialize inotify.
	 *
	 * Throws:
	 *     Exception if inotify fails to initialize.
	 */
	private void initInotify()
	{
		this._inotifyInstance = inotify_init();

		if (this._inotifyInstance < 0)
		{
			this._logger.error("Inotify failed to initialize.");
		}

		this._watchDescriptor = 
inotify_add_watch(this._inotifyInstance, 
cast(char*)this._lastFileName.toStringz(), notificationFlags);
	}

	/**
	 * Stop inotify.
	 */
	private void closeInotify()
	{
		inotify_rm_watch(this._inotifyInstance, this._watchDescriptor);
		close(this._inotifyInstance);
	}

	/**
	 * Change the watcher if the file name changes.
	 */
	private void checkFileNameChange()
	{
		string currentFileName = this.getFileName();

		if (currentFileName != this._lastFileName)
		{
			this._logger.warning("Watched file name changed to '%s'", 
currentFileName);

			this.notify(this._lastFileName);
			this._lastFileName = currentFileName;
			this.notify(this._lastFileName);

			this.closeInotify();
			this.initInotify();

			this._logger.info("Starting to watch '%s' for alerts (%d, 
%d)", this._lastFileName, this._inotifyInstance, 
this._watchDescriptor);
		}
	}

	/**
	 * Retry watching if there was a problem e.g. the file doesn't 
exist yet.
	 */
	private void checkWatchStatus()
	{
		if (this._inotifyInstance == -1 || this._watchDescriptor == -1)
		{
			this._logger.error("Failed watching '%s' for alerts, 
retrying...", this._lastFileName);

			this.closeInotify();
			this.initInotify();

			this._logger.info("Starting to watch '%s' for alerts (%d, 
%d)", this._lastFileName, this._inotifyInstance, 
this._watchDescriptor);
		}
	}

	/**
	 * Check the file for any changes.
	 *
	 * If changes occur then execute the action if one has been 
assigned.
	 * We are using the select call to perform non blocking event 
handling waiting for inotify.
	 * If inotify detects a change in the file, select returns 
immediately.
	 *
	 * See_Also:
	 *     IEngine
	 */
	public void start()
	{
		this._lastFileName = this.getFileName();

		this.notify(this._lastFileName);

		this.initInotify();

		this._logger.info("Starting to watch '%s' for alerts (%d, %d)", 
this._lastFileName, this._inotifyInstance, this._watchDescriptor);

		ubyte[eventBufferLength] eventBuffer;
		void* eventPointer;

		while (true)
		{
			FD_ZERO(&this._fileDescriptorSet);
			FD_SET(this._inotifyInstance, &this._fileDescriptorSet);

			this._timeout.tv_usec = 0;
			this._timeout.tv_sec  = this._interval;

			if (select(FD_SETSIZE, &this._fileDescriptorSet, null, null, 
&this._timeout))
			{
				auto bytesRead = read(this._inotifyInstance, eventBuffer.ptr, 
eventBuffer.length);

				for (eventPointer = eventBuffer.ptr; eventPointer < 
eventBuffer.ptr + bytesRead; null)
				{
					inotify_event* event = cast(inotify_event*)eventPointer;

					if (event.mask & IN_MODIFY || event.mask & IN_ATTRIB)
					{
						this.notify(this._lastFileName);
					}

					if (event.mask & IN_DELETE_SELF || event.mask & IN_MOVE_SELF 
|| event.mask & IN_IGNORED)
					{
						this.notify(this._lastFileName);

						this.closeInotify();
						this.initInotify();

						this._logger.warning("Alert log moved or deleted, 
re-initialized to watch '%s' again (%d, %d)", this._lastFileName, 
this._inotifyInstance, this._watchDescriptor);
					}

					eventPointer += inotify_event.sizeof + event.len;
				}
			}

			this.checkFileNameChange();
			this.checkWatchStatus();
		}
	}

	/**
	 * Destructor.
	 */
	~this()
	{
		this.closeInotify();
	}
}
Dec 27 2013
parent Hugo Florentino <hugo acdam.cu> writes:
On Fri, 27 Dec 2013 12:56:25 +0000, Gary Willoughby wrote:

 On Friday, 27 December 2013 at 03:39:58 UTC, Hugo Florentino wrote:

 BTW, it it a requirement to use malloc, and if so, when would I need 
 to free the memory allocated by it?


 I use a static ubyte array.

 I've been using inotify quite a bit and found it to be very good but
 there are a few things to keep in mind though. First it can block
 further program execution when watching files. To avoid this use the
 select function in 'core.sys.posix.sys.select'.


Thanks. Have you ever tried using epoll?
I have read that for asynchronous things it works much better than 
select.
Dec 27 2013
prev sibling parent reply Artur Skawina <art.08.09 gmail.com> writes:
On 12/27/13 04:23, David Eagen wrote:

 void main()
 {
     string filename="aaa";
 
     int inotfd = inotify_init();
     int watch_desc = inotify_add_watch(inotfd, toStringz(filename), IN_MODIFY);
 
     size_t bufsiz = inotify_event.sizeof + PATH_MAX + 1;
     inotify_event* event = cast(inotify_event *) malloc(bufsiz);
 
     /* wait for an event to occur */
     read(inotfd, event, event.sizeof);


You probably meant

   read(inotfd, event, (*event).sizeof);

but in this case the inotify_event structure contains an optional
trailing buffer, so it should be

   read(inotfd, event, bufsiz);

artur
Dec 27 2013
parent reply "David Eagen" <davideagen mailinator.com> writes:
On Friday, 27 December 2013 at 10:56:55 UTC, Artur Skawina wrote:

 You probably meant

    read(inotfd, event, (*event).sizeof);

 but in this case the inotify_event structure contains an 
 optional
 trailing buffer, so it should be

    read(inotfd, event, bufsiz);

 artur


Yes, thanks for the correction.

I had trouble getting the file name from the event. I think it's 
because the inotify module has name defined as char[0]. So I did 
this, which prints the name by using the extra data beyond the 
inotify_event struct itself:


     void* buf = GC.malloc(bufsiz);

     /* wait for an event to occur */
     size_t readlen = read(inotfd, buf, bufsiz);
     inotify_event* event = cast(inotify_event*) (buf);

     /* process event struct here */
     writeln("Received inotify event:");
     writeln("Bytes read: ", readlen);
     writeln("Length: ", event.len);
     writeln("Name:", cast(char[])(buf[event.len..readlen]));
Dec 27 2013
next sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 12/27/13 14:28, David Eagen wrote:

 I had trouble getting the file name from the event. I think it's because the
inotify module has name defined as char[0]. So I did this, which prints the
name by using the extra data beyond the inotify_event struct itself:
 
 
     void* buf = GC.malloc(bufsiz);
 
     /* wait for an event to occur */
     size_t readlen = read(inotfd, buf, bufsiz);
     inotify_event* event = cast(inotify_event*) (buf);
 
     /* process event struct here */
     writeln("Received inotify event:");
     writeln("Bytes read: ", readlen);
     writeln("Length: ", event.len);
     writeln("Name:", cast(char[])(buf[event.len..readlen]));


    
   writeln("Name:", (cast(char*)&event.name)[0..event.len-1]); // 2do: strip
any extra trailing \0s.

It's probably easier (and safer) if you do it like this:

   struct MyInotifyEvent(size_t BS) {
      inotify_event event;
      char[BS] buffer;
      alias event this;
   }
   [...]
   enum bufsiz = inotify_event.sizeof + PATH_MAX + 1;
   auto event = cast(MyInotifyEvent!bufsiz*)malloc(bufsiz);
   [...]
   writeln("Name:", event.buffer[0..event.len-1]); // 2do: strip any extra
trailing \0s.
  
artur
Dec 27 2013
prev sibling next sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 12/27/13 15:13, Artur Skawina wrote:

    struct MyInotifyEvent(size_t BS) {
       inotify_event event;
       char[BS] buffer;
       alias event this;
    }
    [...]
    enum bufsiz = inotify_event.sizeof + PATH_MAX + 1;
    auto event = cast(MyInotifyEvent!bufsiz*)malloc(bufsiz);
    [...]
    writeln("Name:", event.buffer[0..event.len-1]); // 2do: strip any extra
trailing \0s.


Well, that could get awkward, as the inotify api doesn't really support
disabling
event batching. A saner design would be something like the code below.

artur

struct INotify {
   Fd fd;
   
   import std.exception;
   import core.sys.posix.unistd;
   
   this(cint flags) {
      fd = inotify_init(flags);
      errnoEnforce(fd!=-1);
   }
   ~this() {
      close(fd);
   }
   
   Wd add(const char* name, uint mask) { return inotify_add_watch(fd, name,
mask); }
   Fd remove(Wd wd)                    { return inotify_rm_watch(fd, wd); }
   
   bool get(CB)(CB cb) {
      void[4*1024] buffer = void;
      
      auto got = read(fd, &buffer, buffer.sizeof);
      
      if (got<=inotify_event.sizeof)
         return false;
         
      size_t off = 0;
         
      while (off<=got-inotify_event.sizeof) {
         auto evp = cast(inotify_event*)&buffer[off];
         auto namebuf = evp.len ? (cast(char*)&evp.name)[0..evp.len-1] : null;

         while (namebuf.length && !namebuf[namebuf.length-1])
            --namebuf.length;

         cb(evp, namebuf);
         
         off += inotify_event.sizeof + evp.len;
      }
      
      assert(off==got);
      return true;
   }
}

void main(string argv[]) {
   import std.string;
   auto ntf = INotify(0);
   
   foreach (arg; argv[1..$])
      ntf.add(toStringz(arg), IN_MODIFY);
   
   void myCallback(/*scope*/ const inotify_event* ev, /*scope*/ char[] name) {
      /* Don't even think about escaping any of the args. Dup if necessary. */
      import std.stdio;
      writeln(*ev, " \"", name, "\"");
   }
   
   while (1)
      ntf.get(&myCallback);
}


// libc i/f follows:

alias uint32_t = uint;
alias cint = int;
alias Fd = cint;

extern(C) {
   Fd inotify_init();
   Fd inotify_init1(cint flags);
   Wd inotify_add_watch(Fd fd, const char* pathname, uint mask);
   Fd inotify_rm_watch(Fd fd, Wd wd);
}

Fd inotify_init(cint flags) { return inotify_init1(flags); }

enum IN_MODIFY = 0x00000002; /* DEFINEME */

struct Wd { Fd fd; alias fd this; }

struct inotify_event {
   Wd       wd;       /* Watch descriptor */
   uint32_t mask;     /* Mask of events */
   uint32_t cookie;   /* Unique cookie associating related events (for
rename(2)) */
   uint32_t len;      /* Size of name field */
   char     name[0];   /* Optional null-terminated name */
   
   void toString(DG, FT)(scope DG sink, FT fmt) const  trusted {
      import std.format;
      foreach (I, E; this.tupleof)
         static if (I<this.tupleof.length-1)
            formatValue(sink, E, fmt), sink(" ");
   }
}
Dec 27 2013
prev sibling parent reply Hugo Florentino <hugo acdam.cu> writes:
Hi,

A moment ago I was reading a reply to one of my posts, and noticed 
Artur Skawina did this:


 void main(string argv[]) {
    import std.string;
 ...


I didn't know D allowed importing modules from a function.
Now, my question is: what is the advantage of this over importing the 
regular way?

Regards, Hugo
Dec 28 2013
parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 28 December 2013 at 14:23:17 UTC, Hugo Florentino 
wrote:

 I didn't know D allowed importing modules from a function.
 Now, my question is: what is the advantage of this over 
 importing the regular way?

 Regards, Hugo


Several advantages:

1) It is easier to maintain the code as origins of imported 
symbols become obvious immediately - less chance to get unused 
imports hanging around after some refactoring.

2) Considerably smaller chance of name clash between symbols from 
different modules (can only clash if used within same function / 
scope)

3) If it is templated function / scope, it won't be imported 
unless template is instantiated. "lazy" import effect. That may 
improve compilation times quite a lot in certain style of code.
Dec 28 2013