Implement storage monitor (devices and connectivity)

Jason can u look at this issue if u can work with GIO to solve it?

2009/11/27 Mikkel Kamstrup Erlandsen <email address hidden>

> Public bug reported:
>
> This is a reminder bug.
>
> We need to write a "storage monitor" that manage the various storage
> mediums we can have event subjects stored on. The storage monitor should
> write any changes back to the 'storage' table in the DB.
>
> What we need to track is connection and disconnections of storage
> volumes (probably using gio) - ie. track whether or not files on USB
> thumb drives are available or not etc. Also the network connection
> state should be updated in the storage table so that we can filter out
> remote files and website on request.
>
> ** Affects: zeitgeist
> Importance: High
> Status: Confirmed
>
> ** Changed in: zeitgeist
> Milestone: None => 0.3.1
>
> ** Changed in: zeitgeist
> Status: New => Confirmed
>
> ** Changed in: zeitgeist
> Importance: Undecided => High
>
> --
> Implement storage monitor (devices and connectivity)
> https://bugs.launchpad.net/bugs/489194
> You received this bug notification because you are subscribed to The
> Zeitgeist Project.
>
> Status in Zeitgeist Framework: Confirmed
>
> Bug description:
> This is a reminder bug.
>
> We need to write a "storage monitor" that manage the various storage
> mediums we can have event subjects stored on. The storage monitor should
> write any changes back to the 'storage' table in the DB.
>
> What we need to track is connection and disconnections of storage volumes
> (probably using gio) - ie. track whether or not files on USB thumb drives
> are available or not etc. Also the network connection state should be
> updated in the storage table so that we can filter out remote files and
> website on request.
>
>

Ok, let me jot some thoughts donw on this so we have it....

 * All monitoring should be done async, without polling. Ie. using signals and callback, not sleep(5); check_drives();
 * I think we can use gio.VolumeMonitor() to listen for connect/disconnect of volumes. Then we use gio.Volume.get_uuid() as the key in the 'storage' table.
 * I have not checked NetworkManager or ConnMan, but I am pretty sure that both provide some DBus API to get async notifications when the connection goes away and comes up. We should use the key 'inet' to store the connectivity state in the 'storage' table.
 * All online subjects requiring a connection to be available, should have their storage attribute set to 'inet'

2009/12/3 Mikkel Kamstrup Erlandsen <email address hidden>:
>  * All online subjects requiring a connection to be available, should have their storage attribute set to 'inet'

Doesn't really matter as it's only a minor detail, but why "inet" and
not "internet"/"online"/whatever?

> but why "inet" and not "internet"/"online"/whatever?

Because the wind was in the west and it was a full moon when I wrote it :-) Or perhaps I don't have a good reason.

One thing though, I don't know if we should take into consideration or not is tunneling or VPN-like setups. So fx. some resources are only available when I am online via my company's VPN... This might get a bit tricky to handle gracefully, and I am not sure it is really convenient anyway...

guys i think this goes into ontologies again
basically u have "local" data and "network" data
whereareas each one of thus has its sub categories such as LAN/WAN,
reomvable storage, etc
I will look if nepomuk covers this aspect

2009/12/3 Mikkel Kamstrup Erlandsen <email address hidden>

> > but why "inet" and not "internet"/"online"/whatever?
>
> Because the wind was in the west and it was a full moon when I wrote it
> :-) Or perhaps I don't have a good reason.
>
> One thing though, I don't know if we should take into consideration or
> not is tunneling or VPN-like setups. So fx. some resources are only
> available when I am online via my company's VPN... This might get a bit
> tricky to handle gracefully, and I am not sure it is really convenient
> anyway...
>
> --
> Implement storage monitor (devices and connectivity)
> https://bugs.launchpad.net/bugs/489194
> You received this bug notification because you are subscribed to The
> Zeitgeist Project.
>
> Status in Zeitgeist Framework: Confirmed
>
> Bug description:
> This is a reminder bug.
>
> We need to write a "storage monitor" that manage the various storage
> mediums we can have event subjects stored on. The storage monitor should
> write any changes back to the 'storage' table in the DB.
>
> What we need to track is connection and disconnections of storage volumes
> (probably using gio) - ie. track whether or not files on USB thumb drives
> are available or not etc. Also the network connection state should be
> updated in the storage table so that we can filter out remote files and
> website on request.
>
>

Seif; I am not sure we need an ontology here. What We really care about is the storage ids. Zeitgeist should not care about what is what. Just that we have "some storage medium" that can be available or not.

Going into the realm of semantics brings a lot of complexity with it, and I think we should only do that where it makes absolute sense. And I think that we can prolly avoid it here... I am not 100% sure though..?

FYI I started hacking on this. I have found a few issues we need to talk about design wise...

Apps will probably want to display human readable names for devices even though the devices might not be connected to your system. Eg. "In order to read CoolArticle.pdf you need to plug in the USB drive named 'Kingston 2GB'". For this we need to store the type of the storage medium and a human readable label - we have neither right now.

This info could go in a separate table and be exposed on a separate object+interface on DBus (like the blacklist extension does). THis way we'd be backwards and forwards compatible... I need to write a mockup for this

I think I will dynamically update the structure of the 'storage' table, adding two new columns, 'type' and 'display_name' via ALTER TABLE statements: http://www.sqlite.org/lang_altertable.html. This should be forwards- and backwards compatible. Comments?

2010/1/16 Mikkel Kamstrup Erlandsen <email address hidden>:
> I think I will dynamically update the structure of the 'storage' table,
> adding two new columns, 'type' and 'display_name' via ALTER TABLE
> statements: http://www.sqlite.org/lang_altertable.html. This should be
> forwards- and backwards compatible. Comments?

Go for it, that's much better than adding a new table.

+1 from me

2010/1/16 Siegfried Gevatter <email address hidden>

> 2010/1/16 Mikkel Kamstrup Erlandsen <email address hidden>:
> > I think I will dynamically update the structure of the 'storage' table,
> > adding two new columns, 'type' and 'display_name' via ALTER TABLE
> > statements: http://www.sqlite.org/lang_altertable.html. This should be
> > forwards- and backwards compatible. Comments?
>
> Go for it, that's much better than adding a new table.
>
> --
> Implement storage monitor (devices and connectivity)
> https://bugs.launchpad.net/bugs/489194
> You received this bug notification because you are subscribed to The
> Zeitgeist Project.
>
> Status in Zeitgeist Framework: Confirmed
>
> Bug description:
> This is a reminder bug.
>
> We need to write a "storage monitor" that manage the various storage
> mediums we can have event subjects stored on. The storage monitor should
> write any changes back to the 'storage' table in the DB.
>
> What we need to track is connection and disconnections of storage volumes
> (probably using gio) - ie. track whether or not files on USB thumb drives
> are available or not etc. Also the network connection state should be
> updated in the storage table so that we can filter out remote files and
> website on request.
>
>
>
>

I stumbled across a small block on the road. Apparently there is no neat way to get the "type" of storage device in GIO from a gio.Drive or gio.Volume. It can certainly give me the right icon, but for some reasson the categorizing logic is not exposed as public API.

Not having the storage type in the log would suck pretty bad. Without it we can only show a dialog like:

  In order to open the file foobar.jpg please insert the volume FOOBAR

Which seems pretty useless to me. If we had proper storage types stored then we could say:

  In order to open the file foobar.jpg please insert the CD labelled FOOBAR

The solutions:
1) Implement our own heuristics via *ugly* hacks. Like inferring the type from the icon name (volume.get_icon().to_string())
2) Get the entry under /dev/* and figure out what it is
3) Patch GIO

The purist in me would dive directly at 3). But depending on a trunk GLIB is a big one to swallow for new coming devs... Any ideas?

Created bug upstream with a nice bug number: https://bugzilla.gnome.org/show_bug.cgi?id=607600

My original plan on patching glib and then using a hacky solution until the patch was in stable glib is not progressing very well. Upstream has valid concerns about the feasibility of canonically categorizing media. It's probably not impossible, but a very big task.

The alternatives would then be:

1) Implement our own heuristics via *ugly* hacks. Like inferring the type from the icon name (volume.get_icon().to_string())
2) Get the entry under /dev/* and figure out what it is
3) (SCRAPPED) Patch GIO
4) (NEW) Don't store the medium type. Simply the name of the stock icon. That way GAJ can still show dialogs like: "<Image of CD> in order to view resume.pdf please insert 'Backup CD 2010-02-16'"

4) comes with the added cost that we can not sensible log media mount/unmounts in the activity log which is a shame... Comments?

2010/2/16 Mikkel Kamstrup Erlandsen <email address hidden>:
> 1) Implement our own heuristics via *ugly* hacks. Like inferring the type from the icon name (volume.get_icon().to_string())

How does GIO (or whatever) choose the icon?

That's a bit complicated. But in a nutshell it already internally does what we need. Basically it has an enumeration of GUnixMountType of which it finds the right one using some heuristics and then converting the GUnixMountType to a stock icon id.

If you are asking what they do in <heuristics> I can't tell you off my mind - I don't have the glib sources available where I am...

I don't think we will have this done for the 0.7 release thus it won't be assigned any milestone for now

This bug applies to the Zeitgeist project only, not the Unity project.