Incredibly slow import/High cpu usage

After talking to a few people in the irc that also experienced, they mentioned that it seems to happen when the music is in a separate partition. My /home partition is a separate ext4 partition so that may be relevant. Its a normal ext4 partition. The music folder is 100gb, mostly mp3's.

Happens to me with the music in the same partition as everything else, as in, I have only one partition.

30gb of music, same partition. CPU jumps up to 100% when importing or opening noise.

Yeah I've installed luna beta on one of my other computer's today and have seen the same issue as in the machine in my OP. As I'm typing this its been ~3 hours and its only about 75% done with the import. its 100gb of mp3s, other players like banshee/rhythmbox/clementine take one hour tops (usually far less) to import the whole library.

If I open noise via the terminal and let it import I just see a bunch of this:

[_LOG_LEVEL_WARN 20:16:23.204002] GStreamerTagger.vala:331: Could not find embedded image for 'file:///home/brandon/Music/Duff%20McKagan/05%20-%20The%20Slide.mp3'
[_LOG_LEVEL_WARN 20:16:23.215084] GStreamerTagger.vala:331: Could not find embedded image for 'file:///home/brandon/Music/Duff%20McKagan/06%20-%20Translucent.mp3'
[_LOG_LEVEL_WARN 20:16:23.225690] GStreamerTagger.vala:331: Could not find embedded image for 'file:///home/brandon/Music/Duff%20McKagan/07%20-%20Mothers%20Day.mp3'
[_LOG_LEVEL_WARN 20:16:23.236500] GStreamerTagger.vala:331: Could not find embedded image for 'file:///home/brandon/Music/Duff%20McKagan/08%20-%20I%20See%20Through%20You.mp3'

From the very beginning of the import those show up for every song (which is to be expected since none of them have embedded album art). The strange thing is it imports the first 3000 or so songs very quickly while giving the same warnings so I'm not sure that the warnings necessarily have anything to do with it, for some reason after around 3000 it just slows to an absolute crawl, sucks all cpu, and becomes unresponsive. Its still importing, just very, very slowly.

All of my album art is done via a folder.jpg image in the album directory.

I've launched noise via noise -d this time and logged all output, but I don't see any odd output :/. The end of the log is around when the imported started crawling, but I don't really see any difference between what I'm seeing early in the log and late in the log, no idea why noise just slows to a crawl after importing around 3000 or so songs fairly quickly.

Same problem over here.

Using noise 1.0, on elemetary 0.2 luna, 64bit, updated, with music folder in a ext4 separated partition, with over 30 gb of mp3.

Takes a lot of time to import library, to open, and crash a lot of times.

The log of initialization is:
[_LOG_LEVEL_INFO 14:54:04.552094] Application.vala:79: Noise version: 1.0
[_LOG_LEVEL_INFO 14:54:04.552184] Application.vala:81: Kernel version: 3.2.0-34-generic
[_LOG_LEVEL_INFO 14:54:04.568831] AudioPlayer.vala:34: Activating AudioPlayer Device plugin
[_LOG_LEVEL_INFO 14:54:04.568913] CDRom.vala:34: Activating CD-Rom Device plugin
[_LOG_LEVEL_INFO 14:54:04.568963] LastFM.vala:54: Activating Last.fm plugin
[_LOG_LEVEL_INFO 14:54:04.569014] Plugin.vala:43: Activating MPRIS plugin
[_LOG_LEVEL_INFO 14:54:04.569223] iPod.vala:34: Activating iPod Device plugin
[_LOG_LEVEL_FATAL 14:54:52.524424] noise_library_window_get_treeviewsetup_from_playlist: assertion `self != NULL' failed
[_LOG_LEVEL_FATAL 14:54:52.524487] Noise will not function properly.
[_LOG_LEVEL_FATAL 14:54:52.723892] granite_widgets_source_list_tree_start_editing_item: assertion `_tmp2_' failed
[_LOG_LEVEL_FATAL 14:54:52.723963] Noise will not function properly.
[_LOG_LEVEL_FATAL 14:54:52.724817] granite_widgets_source_list_tree_start_editing_item: assertion `_tmp2_' failed
[_LOG_LEVEL_FATAL 14:54:52.724885] Noise will not function properly.
[_LOG_LEVEL_FATAL 14:54:53.253661] granite_widgets_source_list_tree_start_editing_item: assertion `_tmp2_' failed
[_LOG_LEVEL_FATAL 14:54:53.253732] Noise will not function properly.
[_LOG_LEVEL_INFO 14:54:54.039127] TreeViewSetup.vala:57: Creating a new TreeViewSetup for NOISE_VIEW_WRAPPER_HINT_MUSIC
[_LOG_LEVEL_INFO 14:54:54.162682] TreeViewSetup.vala:57: Creating a new TreeViewSetup for NOISE_VIEW_WRAPPER_HINT_READ_ONLY_PLAYLIST
[_LOG_LEVEL_INFO 14:54:54.221734] TreeViewSetup.vala:57: Creating a new TreeViewSetup for NOISE_VIEW_WRAPPER_HINT_READ_ONLY_PLAYLIST
[_LOG_LEVEL_INFO 14:54:54.286116] TreeViewSetup.vala:57: Creating a new TreeViewSetup for NOISE_VIEW_WRAPPER_HINT_SMART_PLAYLIST
[_LOG_LEVEL_INFO 14:54:54.349518] TreeViewSetup.vala:57: Creating a new TreeViewSetup for NOISE_VIEW_WRAPPER_HINT_SMART_PLAYLIST
[_LOG_LEVEL_INFO 14:54:54.773599] AlbumInfo.vala:121:
[_LOG_LEVEL_INFO 14:56:10.122720] TreeViewSetup.vala:57: Creating a new TreeViewSetup for NOISE_VIEW_WRAPPER_HINT_SMART_PLAYLIST
[_LOG_LEVEL_INFO 14:56:10.186846] TreeViewSetup.vala:57: Creating a new TreeViewSetup for NOISE_VIEW_WRAPPER_HINT_SMART_PLAYLIST
[_LOG_LEVEL_INFO 14:56:10.249923] TreeViewSetup.vala:57: Creating a new TreeViewSetup for NOISE_VIEW_WRAPPER_HINT_SMART_PLAYLIST
[_LOG_LEVEL_INFO 14:56:10.313148] TreeViewSetup.vala:57: Creating a new TreeViewSetup for NOISE_VIEW_WRAPPER_HINT_SMART_PLAYLIST
[_LOG_LEVEL_INFO 14:56:10.375687] TreeViewSetup.vala:57: Creating a new TreeViewSetup for NOISE_VIEW_WRAPPER_HINT_READ_ONLY_PLAYLIST
[_LOG_LEVEL_INFO 14:56:10.434883] DataBaseUpdater.vala:137: -- Saving playlists and device preferences DB.
[_LOG_LEVEL_INFO 14:56:10.490448] DataBaseUpdater.vala:143: -- Finished saving playlists and device preferences DB.
[_LOG_LEVEL_INFO 14:56:11.986997] AlbumInfo.vala:121:
[_...

After some investigation, I find out that takes too long to create a specific smart playlist: recently added. Because I've imported all my music once, when Noise instantiate the TreeViewSetup, it load all my library itens (~5000).

Excluding the smart playlist makes Noise open quickly.

It will not solve the problem, since the importation process takes too long too.

It should be fixed now, can someone try to reproduce it ?

Hey Corentin,

thanks for your efforts. I tested it but the behaviour was still the same. After having a look at your recent changes I saw that there is a small bug in them (i.e. still the complete media is scanned through by the smart playlists when adding new media). I opened a merge request with a fix for it.

This fix improves the overall situation, i.e. the import now finishes after 20 minutes for my library (>10.400 files) compared to several hours before, but still there seem to be other issues. E.g. the CPU load still increases over time and also sometimes there are crashes.

One thing that I saw (and might be a candidate for a separate bug) is that after the import (with GSt.Discoverer) finished there are hundreds of threads/tasks starting/running. Without having had a detailed look into it it seems like this has to do with scrobbling/getting album info via LastFM(?). Sometimes there are error messages about too many open sockets and/or that some HTTP requests failed.

Sometimes shortly after the import finished the app crashes. After a restart it is useable though (with a working library).

It still behaves the way several have reported, for me. Reasonably fast for the first two or three thousand. Slows to a crawl after that. After an hour it was at 9 000.

Hundreds of intermittent reports of

[_LOG_LEVEL_FATAL 23:53:43.062491] noise_grid_view_real_val_func: assertion `_tmp2_ != NULL' failed
[_LOG_LEVEL_FATAL 23:53:43.062512] Noise will not function properly.

(after one another) then hundreds of

[_LOG_LEVEL_WARN 23:56:26.947355] GStreamerTagger.vala:335: Could not find embedded image for 'file:///home/eirik/Music/Charanjit%20Singh%20-%201982%20-%20Ten%20Ragas%20To%20A%20Disco%20Beat/A1%20-%2033rpm%20-%20Raga%20Bhairav.mp3'

for... I guess every track? (It really shouldn't warrant a warning.) Most of these are within milliseconds of each other. Then there are insignificant random pauses (see timecodes)

[_LOG_LEVEL_WARN 22:54:38.982158] GStreamerTagger.vala:335: Could not find embedded image for 'file:///home/eirik/Music/Grimes%20-%202012%20-%20Visions/10%20Symphonia%20IX%20(My%20Wait%20Is%20U).mp3'
[_LOG_LEVEL_WARN 22:54:39.010027] GStreamerTagger.vala:335: Could not find embedded image for 'file:///home/eirik/Music/Grimes%20-%202012%20-%20Visions/08%20Be%20A%20Body%20(%E4%BE%98%E5%AF%82).mp3'
[_LOG_LEVEL_WARN 22:54:41.655833] GStreamerTagger.vala:335: Could not find embedded image for 'file:///home/eirik/Music/Grimes%20-%202012%20-%20Visions/11%20Nightmusic%20(feat.%20Majical%20Cloudz).mp3'
[_LOG_LEVEL_WARN 22:54:41.698443] GStreamerTagger.vala:335: Could not find embedded image for 'file:///home/eirik/Music/Grimes%20-%202012%20-%20Visions/07%20Visiting%20Statue.mp3'

A second and a half per track could add up, but is probably to be expected on a mechanical disk. However, there are also pauses, where it times out trying to import, e.g.

[_LOG_LEVEL_WARN 23:46:25.767288] GStreamerTagger.vala:335: Could not find embedded image for 'file:///home/eirik/Music/Blue%20Cheer%20-%201968%20-%20Vincebus%20Eruptum/Blue%20Cheer%20-%20Vincebus%20Eruptum%20-%2001%20-%20Summertime%20Blues.mp3'
[_LOG_LEVEL_WARN 23:48:49.376350] GStreamerTagger.vala:117: GStreamer could not import 'file:///home/eirik/Music/Blue%20Cheer%20-%201968%20-%20Vincebus%20Eruptum/Blue%20Cheer%20-%20Vincebus%20Eruptum%20-%2002%20-%20Rock%20Me%20Baby.mp3': Discovery timed out.

Two and a half minutes!

This is on a separate NTFS partition, by the way (through a symlink in my home folder).

Hope this helps.

The "could not import" pauses do not necessarily get longer over time, some are a few seconds, others take minutes, btw.

I have the same issue. Only 50GB of music and an incredible cpu usage while importing.

I have same issue. Other problem is slowly opening noise (~30 secs)

(sorry my poor english)

I still see the problem here too, I've noticed no improvement at all. 100gb of music on a normal ext4 partition. it takes HOURS upon hours to import, noise is basically not usable here :(

I just installed beatbox, which as far as I can gather noise is based upon, and that imported my library just dandy, very quickly too (5 miniutes vs 3+ hours). No massive cpu freezes and ui freezes either. perhaps noise should rebase on a newer version of beatbox or something, or merge whatever fixes beatbox has?

I've pushed a fix this night, I should be a lot faster then, please confirm the improvement.

working much better now

on elementary OS luna, I tried to import my whole music directory from my mounted NTFS 1TB hard drive - 9548 songs / 104GB of mp3s/FLACs, in windows winamp/foobar it takes up to 2 minutes, music/noise player takes about 15 minutes. the import progress bar was extremely slow, the program was unusable the whole time, the hard drive was working for first 5minutes and then it was idle the whole time, but the files are imported successfuly. now sometimes, when i doubleclick a song in the library, the player freezes for about 5 seconds, then starts playing.

is there anything i can provide to help debugging this?

Help is on the way.

I'm working on improving noise responsiveness for large music libraries :)

Hi! I think we can merge this two reports. It's possible to do so?

(sorry for my bad English)

https://bugs.launchpad.net/noise/+bug/1325361