Track Cache does not Save to Database

I can reproduce the bug: The only thing I can find in the log is:

WARNING: Inconsistent state in TrackDAO. Track is clean while TrackDAO thinks it is dirty. Correcting

rryan: Could that be the problem?

On 1.9.0, I do not see that in my log. I literally just see:

Debug: [Main]: Track Played: ###
Debug: [Main]: Track Played: ###
Debug: [Main]: Track Played: ###

etc., etc.

Hmm, yes we were seeing assertions even in 1.8.0 of problems where the state
was inconsistent. At the time I trusted the track's dirty-flag because it
was marked clean, but it seems that the track should be dirty and the
TrackDAO is correct. This would result in the tracks not getting saved and
being deleted from the cache.

On Fri, Mar 11, 2011 at 11:58 AM, RAFFI TEA <email address hidden>wrote:

> I can reproduce the bug: The only thing I can find in the log is:
>
> WARNING: Inconsistent state in TrackDAO. Track is clean while TrackDAO
> thinks it is dirty. Correcting
>
> rryan: Could that be the problem?
>
> ** Changed in: mixxx
> Importance: Critical => High
>
> ** Changed in: mixxx
> Milestone: None => 1.9.1
>
> --
> You received this bug notification because you are subscribed to Mixxx.
> https://bugs.launchpad.net/bugs/733376
>
> Title:
> Track Cache does not Save to Database
>

Just tested 1.9 r2767. A couple problems remain:
- The main CUE point does not get saved at all, no matter if the track is in the cache or not. (To reproduce: load a track, set the main CUE point somewhere new. Load another track, load the first track. Notice that the cue point is back to where it was originally.) Hot cues seem to save fine though.
- I do get occasional glitches on track load on a 2.2GHz Athlon64 at 21ms but _only_ if the track hasn't been analyzed before, so this may be related to the analysis check code instead.
- You already know, but the side effect of resetting the table to the top on track load is very annoying. (While you're fixing this, can you make the position persistent when switching library views as well?)

Just captured some gprof output in various scenarios (attached.)

In all of them, I varied only the cache size. (Tried 4,5 and 20.) In each case, I loaded the same 20 pre-analyzed tracks in the same order one right after another (didn't wait for the detail waveform to display) and closed Mixxx right after loading the last one. (For comparison, I did the same thing with none of the tracks having been pre-analyzed also with a cache size of 5 to try to spot any reason why there can be a glitch when a non-pre-analyzed track is loaded.)

Let me know if you'd like me to profile any other scenarios.

Can't reproduce the cue point issue in trunk or 1.9. Hmm

"While you're fixing this, can you make the position persistent when switching library views as well?"

I don't think so -- it's much harder than it sounds. File a separate bug?

I've got a potential fix coming up. I rewrote a good deal of BaseSqlTableModel to no longer use QtSql table models. It's mostly working now, and actually may improve library performance.

For anyone who's interested, here is my rewrite of BaseSqlTableModel to not be a QSqlTableModel.

http://pastebin.com/V02LDva4

RJ, could you provide a diff against trunk? bzr patch does not work!

Ahh right, I forgot to mention. This is against 1.9.x because I intend for
the fix to be released in 1.9.1. There will be a lot of work to do on
merging because there are more BaseSqlTableModel children in trunk than in
1.9.x.

On Tue, Mar 22, 2011 at 4:38 AM, RAFFI TEA <email address hidden>wrote:

> RJ, could you provide a diff against trunk? bzr patch does not work!
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/733376
>
> Title:
> Track Cache does not Save to Database
>

Here it is in a .patch. I think pastebin.com is mangling my patches somehow.

OK, another update.. this time cleaner and with fewer bugs

I'm getting close to committing my work to the 1.9.x branch. I'm happy to report that it appears that these changes have significant performance improvements for the library:

For my 4500 track library, here is the time taken to do the work for a bunch of sorts and searches executed one after the other with half a second or so in between using 1.9.0:

http://pastebin.com/4dhEq9bQ

And here is using 1.9.0 + my changes:

http://pastebin.com/MqAjwM49

These numbers are with the CPU set to the "performance" profile and the same optimizations given to GCC. Basically everything was constant except the code change. I'm not bothering with statistical significance -- this is pretty clearly a win ;). I won't attempt to formally quantify it, but it looks like an order of magnitude improvement on sort / search speed.

Now we need TheresaJayne to test this out...

Ok, just pushed my stuff to the 1.9.x branch. To summarize:

- Tracks save now. (Woo?)
- The cache size is much smaller, around 5. I'm scheming to fully remove it.
- The track table updates correctly when tracks are saved and does NOT reset or cause any massive queries to happen when tracks are marked clean
- Sorting and searching shows their up-to-date data (Bug #700802)

Sean : I never saw your cue point bug. I tried it with both fresh and dirty libraries. Shrug.. could you try again w/ the 1.9.x branch?

Marking this committed. I want to get it in the hands of the dev-team and trunk/1.9.x from-source testers ASAP since it wasn't a simple fix.

I can do some extensive testing of trunk starting tomorrow. I'll also try my hand at merging the changes with features_xwax2 if Owen doesn't beat me to it...

--
Joseph Colosimo
http://jwcxz.com
(Mobile)

----- Reply message -----
From: "RJ Ryan" <email address hidden>
To: <email address hidden>
Subject: [Bug 733376] Re: Track Cache does not Save to Database
Date: Thu, Mar 24, 2011 01:49

Ok, just pushed my stuff to the 1.9.x branch. To summarize:

- Tracks save now. (Woo?)
- The cache size is much smaller, around 5. I'm scheming to fully remove it.
- The track table updates correctly when tracks are saved and does NOT reset or cause any massive queries to happen when tracks are marked clean
- Sorting and searching shows their up-to-date data (Bug #700802)

Sean : I never saw your cue point bug. I tried it with both fresh and
dirty libraries. Shrug.. could you try again w/ the 1.9.x branch?

Marking this committed. I want to get it in the hands of the dev-team
and trunk/1.9.x from-source testers ASAP since it wasn't a simple fix.

--
You received this bug notification because you are a direct subscriber
of the bug.
https://bugs.launchpad.net/bugs/733376

Title:
  Track Cache does not Save to Database

Status in Mixxx:
  Fix Committed
Status in Mixxx 1.9 series:
  Fix Committed

Bug description:
  Running 1.9 on Linux. The problem continues to appear in trunk.

  Steps to reproduce:
  1. Open Mixxx.
  2. Start anywhere on your playlist and load a track. It doesn't matter how you load the track. It doesn't matter whether you actually play the track or not.
  3. Now load another track. For the sake of convenience, load the next one on your playlist.
  4. Continue to load tracks for a while. In fact, load 20 tracks total.
  5. Now, load the 21st. Examine the "played" checkbox column. You should see that the checkboxes for your previous tracks have become unchecked!

  But there's more. This problem is *NOT* limited to just the played
  checkboxes -- it is related to *ANY* modification of the database. An
  important example is cues. If you had set cues for each of those 20
  tracks above and then loaded a 21st track, all of your cues that you
  had set before in those 20 tracks are now lost.

  The cache also fails in another interesting way too. Sometimes, I was
  able to load successive tracks and watch individual checkboxes from
  previous tracks get unchecked one-by-one (like the game Snake) or
  perhaps in small grops of 2 or 3.

  There is no useful debug information in mixxx.log. It only shows
  lines saying tracks were played.

To unsubscribe from this bug, go to:
https://bugs.launchpad.net/mixxx/+bug/733376/+subscribe

I tested trunk last night, the library seems faster but i have some comments,

1. if you play any tracks by dragging them to the track list while scanning and the played tracks are in the folders being scanned, at the end you get an error saying the track is already in the list and you get no items at all added to the database.

2. if you then tell it to rescan, it will ADD nothing

(thats 3.5 hours wasted in my case guys)
3. If you select a large number of tracks right click and "Add to Auto DJ" the whole of mixxx locks up and greys out (Ubuntu LTS) but recovers in a bit.

apart from that testing still going on .

Mixxx now uses GitHub for bug tracking. This bug has been migrated to:
https://github.com/mixxxdj/mixxx/issues/5808