Way back in August 2005 a really interesting file tracking feature was added to the Linux kernel (v2.6.13) and given all the interest today in data governance and security I thought it’d be nice to put a spotlight on it, it’s called inotify.

After a quick bit of research it looks like the feature was developed at Novell by engineers John McCutchan and Robert Love and it basically lets you put your filesystem under the microscope. Everything that happens from file access to read/write/modify to creation, deletion of files (and more) are all turned into events that you can subscribe to so that you can watch a play-by-play of everything that’s happening.

Here’s an example of what it reports when I copy the file “Picture1.png” (467K) into my /export/mynotes folder using SMB while monitoring with inotify.

2015/12/04 20:00:14;/export/mynotes/;OPEN,ISDIR
2015/12/04 20:00:14;/export/mynotes/;CLOSE_NOWRITE,CLOSE,ISDIR
2015/12/04 20:00:14;/export/mynotes/;OPEN,ISDIR
2015/12/04 20:00:14;/export/mynotes/;CLOSE_NOWRITE,CLOSE,ISDIR
2015/12/04 20:00:14;/export/mynotes/;CLOSE_NOWRITE,CLOSE,ISDIR
2015/12/04 20:00:14;/export/mynotes/;OPEN,ISDIR
2015/12/04 20:00:14;/export/mynotes/;ACCESS,ISDIR
2015/12/04 20:00:14;/export/mynotes/;CLOSE_NOWRITE,CLOSE,ISDIR
2015/12/04 20:00:14;/export/mynotes/Picture1.png;CREATE
2015/12/04 20:00:14;/export/mynotes/Picture1.png;OPEN
2015/12/04 20:00:14;/export/mynotes/Picture1.png;ATTRIB
2015/12/04 20:00:14;/export/mynotes/Picture1.png;MODIFY
2015/12/04 20:00:14;/export/mynotes/Picture1.png;ATTRIB
2015/12/04 20:00:14;/export/mynotes/Picture1.png;MODIFY
2015/12/04 20:00:14;/export/mynotes/Picture1.png;MODIFY
2015/12/04 20:00:14;/export/mynotes/Picture1.png;MODIFY
2015/12/04 20:00:14;/export/mynotes/Picture1.png;MODIFY
2015/12/04 20:00:14;/export/mynotes/Picture1.png;MODIFY
2015/12/04 20:00:14;/export/mynotes/Picture1.png;MODIFY
2015/12/04 20:00:14;/export/mynotes/Picture1.png;MODIFY
2015/12/04 20:00:14;/export/mynotes/Picture1.png;MODIFY
2015/12/04 20:00:14;/export/mynotes/Picture1.png;MODIFY
2015/12/04 20:00:14;/export/mynotes/Picture1.png;MODIFY
2015/12/04 20:00:14;/export/mynotes/Picture1.png;MODIFY
2015/12/04 20:00:14;/export/mynotes/Picture1.png;MODIFY
2015/12/04 20:00:14;/export/mynotes/Picture1.png;MODIFY
2015/12/04 20:00:14;/export/mynotes/Picture1.png;MODIFY
2015/12/04 20:00:14;/export/mynotes/Picture1.png;MODIFY
2015/12/04 20:00:14;/export/mynotes/Picture1.png;ATTRIB
2015/12/04 20:00:14;/export/mynotes/Picture1.png;CLOSE_WRITE,CLOSE
2015/12/04 20:00:14;/export/mynotes/;OPEN,ISDIR
2015/12/04 20:00:14;/export/mynotes/;OPEN,ISDIR
2015/12/04 20:00:14;/export/mynotes/;CLOSE_NOWRITE,CLOSE,ISDIR
2015/12/04 20:00:14;/export/mynotes/;OPEN,ISDIR
2015/12/04 20:00:14;/export/mynotes/;OPEN,ISDIR
2015/12/04 20:00:14;/export/mynotes/;CLOSE_NOWRITE,CLOSE,ISDIR
2015/12/04 20:00:14;/export/mynotes/;CLOSE_NOWRITE,CLOSE,ISDIR
2015/12/04 20:00:14;/export/mynotes/;OPEN,ISDIR
2015/12/04 20:00:14;/export/mynotes/;CLOSE_NOWRITE,CLOSE,ISDIR
2015/12/04 20:00:14;/export/mynotes/;OPEN,ISDIR
2015/12/04 20:00:14;/export/mynotes/;OPEN,ISDIR
2015/12/04 20:00:14;/export/mynotes/;CLOSE_NOWRITE,CLOSE,ISDIR
2015/12/04 20:00:14;/export/mynotes/;CLOSE_NOWRITE,CLOSE,ISDIR

Yes, it’s noisy. There’s some great information in there on what and when a file was created/modified but the one big gap in inotify is the lack of information about who is accessing the files/directories. I think that that’s due to the “who” information not being readily available deep down in the kernel where inotify is implemented without making significant changes to pass along UID/GID information.

Unfortunately that’s a big gap when trying to do data governance but the basics that inotify does provide are still really useful for audit logging. For example, you could put a watch on sensitive directories that shouldn’t be modified except very infrequently like /etc/ and you can do interesting usage analysis to figure out what types of files are being accessed most often and at what times of day.

If you’d like to try out inotify on your Linux server here’s the command to get it installed on a debian/ubuntu system:

sudo apt-get install inotify-tools

And here’s how to get it to monitor or “watch” a specific directory and all it’s subdirectories, for example the /export directory.

inotifywait -r -m --timefmt "%Y/%m/%d %H:%M:%S" 
   --format "%T;%w%f;%e" /export >> 
   /var/log/inotify_export.log &

Given the amount of information it logs you’re going to want to be careful with inotifywait as it could easily fill the drive after a few days on an actively used system. So if you are going to leave it running as a background process think about adding a log rotation entry in /etc/logrotate.conf like the one below which will rotate your inotify events log file every time it reaches 100MB in size.

/var/log/inotify_export.log {
    rotate 10
    size 100M
    compress
    copytruncate
    weekly
    missingok
}

You could also combine this with Splunk or other tools to generate some nice alerts and reports.

I used a QuantaStor v3 storage appliance to do the above noted hacking with inotify. The underlying filesystem I used is ZFS (ZoL) and the /exports/mynotes is a bind point that QuantaStor automatically creates for every Network Share you create. In this case one I created a share called “mynotes” in the web UI and then did the rest at the command line. But you could do all the above on just about any server that’s Linux based. Hope you enjoyed the article!
-Steve

Categories: Storage Appliance Hardware