Dynamic Library (Do not display unavailable VOLUMES/DRIVES)

I'm still here :)

What is your suggestion for determining whether or not a part of the filesystem is part of a mounted drive?

Hi!

I'm suffering the same problem. In addition, you have that problem if you mount parts of your collection from the network (e.g. when you're at home) and have that drive unavailable sometimes (on the road).

Currently Exail seems to mess up, if some files are occasionally missing from the known collection. I can't find several titles anymore after running exail without having m external drive not mounted. Others are kept in the collection.

Without knowing too much of Exailes code base (sorry, I'm not a Pythoneer) I would suggest the following solution: Allow users to define which directories of a collection are "dynamic".

In this case users can define which parts of a collection can disappear suddenly, due to dynamically mounted drives. This can happen in the library manager and while importing a directory initially. If Exaile determines that the defined directory does not exist (by frequent checking during runtime), I'd suggest to mark the affected titles grey and simply ignore them for playing. Removing them completly from the collection/playlists view sounds not like a good solution to me, since meta-data is still available from the DB and the user gets a clear view on which files he currently misses.

Knowing that a library directory is completly missing (empty or simply gone) also allows Exaile to determine if a song is "gone" (in terms of being deleted) or just "temporarely unavailable" (in terms of residing on a not mounted drive). If a whole base-library-directory is empty or unavailable this means that the dynamic drive is not mounted. If just a single song is missing, this means the song is deleted. Removal of dynamic library directories can still work through the library manager.

Exaile would need to check the state of its dynamic library directories from time to time during runtime (e.g. every 10 secs) to update the song states in the database. The SQL for marking songs as "available" and "unavailable" should be easy and efficient. I did not check, but it might be possible to determine dynamic directories via DBUS and also assign to DBUS signals to be informed of the appearance and disappearance of dynamic drives/directories.

Thanks to anybody working on this! Hope my design ideas could help.

I also suffer from the same problem when my external had drive with almost all my music is not attached to my laptop, this would seem a very convenient feature/plugin for exaile

Hello,

I see this bug is in 'triaged' status. As far as I can gather, this is the information collection and cleanup phase. I think people could help better if one of the developers could tell us what other information is necessary.

Without knowing much of the details of implementation this looks like a fairly easy and useful feature. (But I am biased obviously) Please let us know if we're missing something.

Thanks.

Well, basically its a question of how to sanely detect and handle the presence/removal of such drives, while also automatically handling actual file deletions. The current system is based exclusively on file paths. So while we could detect the separate filesystem when its mounted, once its unmounted there's no way to distinguish between deletion and being unmounted. This means that, for EVERY FILE we would have to store some identifier for what filesystem its on, and be able to know whether that fs is mounted/unmounted, and check all of those at start. Not very ideal, to say the least.

Another way of handling it could be to just say that if you want a feature like this it must be registered as a separate library, which could be given a "not always present" mark or something like that. Then we just check files to be sure that their library is mounted (easy as the library path will be the same as the start of the file's path), and only if it is, should we attempt to handle the file. This is doable, but a fair chunk of work.

Also, this is a use case that only affects a very small percentage of users and it is thus very low on the list of things to do. It's a legitimate concern, and is likely to be implemented eventually, but don't hold your breath. However, patches are, as always, welcome, as are additional ideas on how to best handle the detection and handling of mounts.

I don't think it's necessary to store such an ID for every song.

You should go for a configuration setting with which the user can indicate which directiories contain dynamic content. If a file in such a location is attempted to be played, you need to check and store in the back if the given location contains files at all (is mounted or not). After that, you can automatically mark every file under the given path as "temporarely not available" in it's display logic and simply don't attempt to play files below this location anymore.

I'd suggest to have a little background task running which checks every X seconds if a location that is marked as dynamic is available to correctly mark the songs while displaying. If a file is not available, but it's dynamic location is marked "mounted", you can asume it has been deleted.

This way of realization also allows you to mark parts of a library as dynamic.

If you go a step further, you could ask DBus about the info instead of simply checking that a dynamic location is empty (not mounted).

HTH!

I am not sure about the database structure, but making the following
assumptions:
1) Library path is included as a field
2) A filter can be written such that an sql select statement with the
library path as a constraint would make those files invisible to all
operations (add to playlist, play, random etc).

Then the user can be given the option to select which libraries are dynamic.
During startup exaile checks to see if this library destination is mounted.
If it is, display the contents, if not use the above proposed filter (sql).
Since if both 1 and 2 above are true, there would not be a need for tracking
deletions of already invisible material. This task can be deferred until the
next start up.

The issue I can immediately see is what happens if the library path was
mounted during start up but then goes away (like removel of storage device
of NFS failure) but that issue is already present I think. It would not be
an additional problem due to adding this feature.

I disagree with tobyS's idea of running a daemon in the background which I
think is a bit too much for this minor feature. However if the "filter" idea
can be loaded with more features, this idea might be revisited.

Also, I do agree this is a minor feature - but one that nobody is
implementing as far as I know. So it's a "nice to have" feature.

Best regards,

On Thu, Oct 16, 2008 at 1:28 PM, Aren Olson <email address hidden> wrote:

> Well, basically its a question of how to sanely detect and handle the
> presence/removal of such drives, while also automatically handling
> actual file deletions. The current system is based exclusively on file
> paths. So while we could detect the separate filesystem when its
> mounted, once its unmounted there's no way to distinguish between
> deletion and being unmounted. This means that, for EVERY FILE we would
> have to store some identifier for what filesystem its on, and be able to
> know whether that fs is mounted/unmounted, and check all of those at
> start. Not very ideal, to say the least.
>
> Another way of handling it could be to just say that if you want a
> feature like this it must be registered as a separate library, which
> could be given a "not always present" mark or something like that. Then
> we just check files to be sure that their library is mounted (easy as
> the library path will be the same as the start of the file's path), and
> only if it is, should we attempt to handle the file. This is doable, but
> a fair chunk of work.
>
> Also, this is a use case that only affects a very small percentage of
> users and it is thus very low on the list of things to do. It's a
> legitimate concern, and is likely to be implemented eventually, but
> don't hold your breath. However, patches are, as always, welcome, as are
> additional ideas on how to best handle the detection and handling of
> mounts.
>
> --
> Dynamic Library (Do not display unavailable VOLUMES/DRIVES)
> https://bugs.launchpad.net/bugs/153054
> You received this bug notification because you are a direct subscriber
> of the bug.
>

"You should go for a configuration setting with which the user can indicate which directiories contain dynamic content."
That is essentially what my second suggestion does, except it places the indication in an existing place well suited to handling it - the library system.

"I'd suggest to have a little background task running which checks every X seconds..."
I'd rather not, polling is an ugly solution, especially when it causes disk activity. 0.3 will have pyinotify support at least for this, and as you suggest later we could just listen to HAL or GVFS over DBUS for the mount/unmount... except not all mounts are necessarily handled by them. Polling/listening also has the problem that it only works while Exaile is running. If you unmount the drive while Exaile is closed, when exaile is started again how does it know that that folder is now unmounted, instead of just deleted? I guess you could look at /etc/mtab, but that's both hackish and linux-dependent.

Really I think the simplest solution would be to make the Mass Storage device plugin capable of everything you can do with the normal collection and then just use that for any kind of removable disk like this. You'd have to cache the DB on the disk to avoid huge startup times, but that's trivial to do. This would also have the advantage that you could then plug the same drive into another system with Exaile and have the entire same collection at hand, complete with ratings/playcounts/etc.

I also would like to have this feature included, it would benefit many users who just cant' have their whole music collection available and/or have it stored somewhere that isnt always available...

speaking today with Aren (reacocard), he came with an interesting solution:

"im thinking perhaps however since we support multiple library path we might be able to make a work around with that, ie have a mount detection on the lib folder itself, and if its a mounted dir and it becomes unmounted, jus thide everything in it instead of deleting it. that's doable, and would likely handle the vast majority of cases affected by this issue

There's now been a substantial number of users who have reported an issue like this, so I think we definitely need to do something about this within the next couple releases.

gio.VolumeMonitor is probably a good way to detect these cases.

I think a solution is needed. I have always had an issue with providing access to networked music storage and mobile devices as they are not consistently accessible. I now use Ampache to get access to such resources but there is no plugin for Exaile to act as a client for it. Android has a Amdroid which is great, ideal for a phone device but too simple as a desktop client.

I am a long term supporter and user of Exaile, and would love to see a way of extending ways of using it. I suspect a plugin is the way to go for reliable external library use. I think you are right, a series of libraries all based on a presumption of possible absence would be a uniform mechanism to help here.

I'm running into the same Problem as my Music-Database doesn't fit my Laptop SSD.
My Idea is if including a Path-existence-detection (via mtab or whatever) takes to long a good option for the beginning would be if the user could easily select via checkbox which library path should be shown and which not. I'm thinking that would be easy if every (dynamic) Library path gets an own Database(/Table) and these Path-Libraries are dynamically joined or excluded in the main static Database, by checking or unchecking a checkbox, that concept could then later be extendet to an automatic or scripted detection.

Simon, if you are experiencing problems with a current version of Exaile, please file bugs on github instead of here. Pull requests that implement this functionality will be gladly accepted.