Use inotify to monitor open pages for changes

I just found out that the monitoring provided by gio does not allow monitoring the full directory tree, but just one level at a time :( An inotify is linux only, so would like to avoid it if possible.

However I was wondering if it would be sufficient to check e.g. the current page and parents on some regular timing. In that case we could just check timestamps on a regular interval without the need of any real kernel based filesystem monitoring and it would not be as heavy as checking the whole notebook. Still it would detect most changes in the visible part of the page index.

Well, wouldn't that gio monitoring be sufficient for monitoring the
current page for changes? It might be more battery-friendly than a
polling loop, depending on the platform. Either way would be nice,
though. :)

On Sun, Jun 5, 2011 at 16:50, Jaap Karssenberg
<email address hidden> wrote:
> I just found out that the monitoring provided by gio does not allow
> monitoring the full directory tree, but just one level at a time :( An
> inotify is linux only, so would like to avoid it if possible.
>
> However I was wondering if it would be sufficient to check e.g. the
> current page and parents on some regular timing. In that case we could
> just check timestamps on a regular interval without the need of any real
> kernel based filesystem monitoring and it would not be as heavy as
> checking the whole notebook. Still it would detect most changes in the
> visible part of the page index.
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/792058
>
> Title:
>  Use inotify to monitor open pages for changes
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/zim/+bug/792058/+subscribe
>

I would be interested in this too. As it stands I usually have to fully close a notebook and then reopen it. It can be problematic because, for example, if I changed notebook 1 on computer A, go to computer B, forget to refresh and write something else in notebook 1, then I overwrite what I did on computer A.

On Wed, Aug 24, 2011 at 5:53 PM, AT <email address hidden> wrote:

> I would be interested in this too. As it stands I usually have to fully
> close a notebook and then reopen it. It can be problematic because, for
> example, if I changed notebook 1 on computer A, go to computer B, forget
> to refresh and write something else in notebook 1, then I overwrite what
> I did on computer A.
>

This overwrite is unlikely. Zim always checks the actual file when you open
a page. So when you open a page that was modified elsewhere the new content
will be shown. Also zim checks the modification time on the file before
overwriting. So if the page was already open and then modified elsewhere it
will give an error as soon as you start modifying it.

The only part that is not updated automatically is what pages are shown in
the index. That is the topic of this bug report.

-- Jaap

Attached a plugin that does a check on changes in the folder + parent folders every time a page is opened or reloaded. Please test it and report here if it works and especially how it works on large notebooks. I deliberately made it so that it will introduce some lag when scaling is not good - so you will notice.

1/ only detects new / missing files, but not changes in content (content is detected when the changed page is opened or reloaded)
2/ completely passive, so no timer or notification used, just triggered on the user action
3/ checks current page namespace and parents - so updates in root should always be detected, sub-trees only when accessed

Anyone testing this fix ? If so please leave some feedback about the performance.

Thanks,

Jaap

Hi Jaap,

On Fri, Jan 06, 2012 at 01:47:12PM -0000, Jaap Karssenberg wrote:
> Anyone testing this fix ? If so please leave some feedback about the
> performance.

yes, I am using it with my office PC. So far, I did not experience any
difference in the performance between zim sessions with or without plugin.
But my largest notebook contains only around 200 pages.

cheers,
 Stefan

INFO: Open page: <FileStorePage: Tickets:013000 Test> (<IndexPath: Tickets:013000 Test>)
!! DOING BACKGROUND CHECK ON <IndexPath: >
!! OK
!! DOING BACKGROUND CHECK ON <IndexPath: Tickets>
!! new child <FileStorePage: Tickets:013001 Test>
Traceback (most recent call last):
  File "/mnt/home/home/zabuch/install/zim/zim-0.54/zim/plugins/checkindex.py", line 122, in on_open_page
    check_pagelist(index, p)
  File "/mnt/home/home/zabuch/install/zim/zim-0.54/zim/plugins/checkindex.py", line 22, in check_pagelist
    _schedule_updates(index, path)
  File "/mnt/home/home/zabuch/install/zim/zim-0.54/zim/plugins/checkindex.py", line 56, in _schedule_updates
    _insert_child(index, page)
  File "/mnt/home/home/zabuch/install/zim/zim-0.54/zim/plugins/checkindex.py", line 69, in _insert_child
    (page.basename, path.id, page.hascontent, False))
NameError: global name 'path' is not defined

Tomek

Comment from mailing list:

On Mon, Jan 23, 2012 at 6:18 AM, Adam Porter <email address hidden> wrote:
> I use Dropbox with Zim. I am, of course, aware that Zim doesn't
> automatically detect new pages or changes in directory structure, but
> I have always thought that Zim would automatically load the newer
> version of a page if it had been saved to disk while Zim was on a
> different page. Just now, though, I loaded a page on my netbook that
> had been modified earlier today on my laptop, and it loaded the old
> version of the page, as if it was still in memory and it didn't check
> the mtime on the disk. I hit Ctrl+R and it showed the new version
> right away.

Strange. I emailed this comment yesterday but it hasn't shown up.
----
Hi Jaap,

Sorry for taking so long to test it. I tried it just now by enabling
the plugin, then copying and renaming a page in the filesystem, and
then clicking sibling and parent pages over and over in Zim. The new
page never appeared. Then I ran Update Index from the menu, and it
appeared.

It doesn't work for me either (there is a bug in the plugin) - see the traceback from my previous comment: https://bugs.launchpad.net/zim/+bug/792058/comments/8

Personally I think "live monitoring" isn't needed. If the "Update Index" command simply re-executed the same file-reading routines as what is done at first start up, that to me would be a terrific improvement.

The fact that it's user-initiated will prevent normal performance being impacted. People with huge wikis are already used to Zim taking some time at initial startup, as long as the rebuild doesn't take longer than that it should be fine.

I also feel that this type of functionality is redundant with "View Reload" (just refresh that one page/file?), personally this could include the "Update Index" functionality.

@HansBKK: I'm sorry, but I disagree with your assessment - indexing is
already done in the background, so users should never have to wait for
this. And it is not unreasonable to expect a desktop application to detect
changes to the local file system.

Yes: the whole point is that, if I have Zim on two systems, synced with Dropbox or whatever, Zim should detect when the page is updated underneath it so I don't edit an older version and get a conflict or data loss. If I have to remember whether I updated a page on another system and remember to manually refresh it, I guarantee that I will forget sometimes. :)

Jaap, if you are still interested in this plugin you posted, let me know how I can help debug it.

Thanks for all your work on Zim. It has become indispensable to me, and I hope to use it for many years to come.

On Fri, Feb 17, 2012 at 12:04 AM, Adam Porter <email address hidden>wrote:

> Jaap, if you are still interested in this plugin you posted, let me know
> how I can help debug it.
>

I think I have sufficient info to merge this feature, but lack the time at
this moment to get it in the next release. Still it is on my list, so it
will show up in a release or two.

-- Jaap

Jaap, is this feature (auto-detect changes to files on filesystem and automatic page reload) still forthcoming? I am using Zim 0.56 on Dropbox and am losing data when I forget to quit Zim and restart it before using it on my second computer. If you still need testers, please let me know how to help. Thank you!

And Roland Mas asked me to apply this on the Debian package to fix this issue too:
http://pastebin.com/16FX2Pka

It would be nice to see this issue fixed once for all... sharing a notebook via dropbox/owncloud is a popular use case (I do it as well).

Until this is properly implemented, it would be nice if zim could just stop caching the pages. Here's a simple patch for this, courtesy of Roland Mas and myself. It simply flushes the pages before each try to open/load a page.

I have applied this patch to the Debian package in version 0.65-1.

Caching issue will be fixed in next release. Patch in testing to check file on disk on each page load.

Has Raphaël's patch been incorporated into mainline (lp:zim)? I tried installing a zim snapshot release from the PPA, but it doesn't seem to reload pages from disk upon opening them. Thanks.

Would this be appropriate functionality for a plugin? I'm sure many of us would happily embrace a quick and dirty auto-reload implementation, such as polling at a configurable interval. The opt-in nature of a plugin would mitigate the performance considerations, and a more efficient solution could still be integrated into the core feature set as time permits.

Thank you for a great tool!
Chris