after fix for #215728 - Committing to urlclassifier3.sqlite still causes excessive CPU usage and disk I/O (the 2nd)

we should take care that upstream takes a look

i keep importance "High" for now ... until we understand the real impact better.

reed: do you know if there exists such a follow-up bug upstream yet?

My urlclassifier3.sqlite has now grown to 20MB and I haven't used Firefox that much - it definitely grows out of all proportions.

On Mon, May 12, 2008 at 10:07:21PM -0000, Tormod Volden wrote:
> My urlclassifier3.sqlite has now grown to 20MB and I haven't used
> Firefox that much - it definitely grows out of all proportions.

20MB is still in the bounds of expected. it will grow even further,
but at some point it shouldn't increase its size in that dimensions
anymore ...

 - Alexander

stek79, can you please re-confirm the xulrunner-1.9 package version you used when taking the IO stats you posted in comment https://bugs.edge.launchpad.net/ubuntu/+bug/215728/comments/120?

> 20MB is still in the bounds of expected.
Expected? But not sane IMHO. It must be an error, in code or design. It must use some 100kB per web page I have been to.

> It must use some 100kB per web page I have been to.
It doesn't grow based on what pages you've been on. It's a database of known "bad" URLs and the growth you see is simply the db being downloaded incrementally. Once it's complete only updates will be added and old entries removed which shouldn't result in the same growing pattern as in the beginning.

In my case, urlclassifier3.sqlite had grown to 33 MB in a couple of days. Then I switched off the phishing / attack warnings alltogether.

Although the problem is greatly diminished since the xulrunner update, this security function still eats up too much resources (occasional high CPU load plus I/O activity). Which makes surfing with Firefox less pleasant than it should be.

The strange thing is, that Firefox 2 has a similar function, which did not cause any such problems.

There is noticeable loss of functionality during a minute or two of intensive writing, but it is bearable now.

My urlclassifier3.sqlite is now 52752384 bytes.

It will help to know when the growth in size will taper off so that the file updates go largely unnoticed.

Alexander,
      here are my xulrunner packages:

ste@ste-ubuntu:~$ COLUMNS=200 dpkg -l | grep xulrunner
ii xulrunner-1.9 1.9~b5+nobinonly-0ubuntu4~8.04.0mt1 XUL + XPCOM application runner
ii xulrunner-1.9-gnome-support 1.9~b5+nobinonly-0ubuntu4~8.04.0mt1 Support for Gnome in xulrunner-1.9 applications

Am I up-to-date?

BTW, what is this second bug about?

I have had recent loss of responsiveness in fifefox-3.0b5. Today I experienced about three minutes of non-repsonsiveness with high CPU load of about 70% and disk I/O. It seems that this time the file

places.sqlite

is being written to. It size is currently

8626176 2008-05-26 14:13 places.sqlite

FYI, urlclassifier3.sqlite has been stable and still is

52752384 2008-05-26 13:58 urlclassifier3.sqlite

If I do

dpkg -l | grep xulrunner

I get

ii xulrunner 1.8.1.13+nobinonly-0ubuntu1 XUL + XPCOM application runner
ii xulrunner-1.9 1.9~b5+nobinonly-0ubuntu4~8.04.0mt1 XUL + XPCOM application runner

Do I need to remove the first one by any chance.

Are there any workarounds for this?

Thanks.

i think the bug is fixed in commit 118 on our xulrunner-1.9.dev branch. Please test if this issue is gone for you with xulrunner-1.9 (1.9~rc1) and firefox-3.0 (3.0~rc1) in mozillateam hardy ppa https://edge.launchpad.net/~mozillateam/+archive

not a ffox bug -> xulrunner

This is a good link that explains what is happening to give this lockup:

shaver » fsyncers and curveballs
http://shaver.off.net/diary/2008/05/25/fsyncers-and-curveballs/

My places.sqlite file has grown to 12107776 bytes and there is still periodic loss of responsiveness with high CPU usage and concentrated I/O.

My urlclassifier3.sqlite file is growing only slowly at is 54878208 bytes and does not seem to be the problem.

Perhaps this blog explains what I am experiencing:

Christopher Blizzard » Blog Archive » system components, fsync and distribution-specific changes: a cautionary tale
http://www.0xdeadbeef.com/weblog/?p=368

I checked my libsqlite3-0 and found it was version was 3.4.2-2.

I have now used prevu to compile version3.5.9-2~8.04prevu1. I do not know if it will help because the firefox I am using is already compiled and version 3.0~rc1+nobinonly-0ubuntu0.8.04.1.

But I am sure that it is places.sqlite that is causing loss of responsiveness.

Any help or light shedded on this is most appreciated.

Thanks.

On Sat, Jun 14, 2008 at 12:43:49PM -0000, R (Chandra) Chandrasekhar wrote:
> My places.sqlite file has grown to 12107776 bytes and there is still
> periodic loss of responsiveness with high CPU usage and concentrated
> I/O.
>
> My urlclassifier3.sqlite file is growing only slowly at is 54878208
> bytes and does not seem to be the problem.
>
> Perhaps this blog explains what I am experiencing:
>
> Christopher Blizzard » Blog Archive » system components, fsync and distribution-specific changes: a cautionary tale
> http://www.0xdeadbeef.com/weblog/?p=368
>
> I checked my libsqlite3-0 and found it was version was 3.4.2-2.

This isn't an issue in our builds. we have all the patches in rc1 that
were about those fsync issues and we dont use system-sqlite, so we are
not affected by a version mismatch causing performance.

>
> I have now used prevu to compile version3.5.9-2~8.04prevu1. I do not
> know if it will help because the firefox I am using is already compiled
> and version 3.0~rc1+nobinonly-0ubuntu0.8.04.1.
>
> But I am sure that it is places.sqlite that is causing loss of
> responsiveness.

Why are you sure about this?

 - Alexander

Alexander Sack wrote:
> On Sat, Jun 14, 2008 at 12:43:49PM -0000, R (Chandra) Chandrasekhar wrote:

>> But I am sure that it is places.sqlite that is causing loss of
>> responsiveness.
>
> Why are you sure about this?
>
> - Alexander

Because when the computer is unresponsive and I run

top

the first listed process is firefox.

If I then go to ~/.mozilla/firefox/<something> and do

ls -altr

I see the file

places.sqlite

as the last listed and it changes in size as I view it progressively by
repeating the command. If there is a more sophisticated and foolproof way to
confirm this, perhaps by running some other commands, do let me know and I will
try it and post results.

Thanks.

Chandra

Hi,
     some weeks of testing, up to me the problem is almost fixed.

I never experienced any grey issue since the first post, now the IO activity is not so overwhelming like it was before.

Perhaps we can close this bug?

Thanks

stek79 wrote:

> I never experienced any grey issue since the first post, now the IO
> activity is not so overwhelming like it was before.
>
> Perhaps we can close this bug?

I am hesitant to say the problem has been resolved.

I still experience periods where the program does not respond while I/O activity
is intense. There might be two possible causes, specific to my installation:

1. Something in my user profile from earlier versions of FF which has been
carried over. I know that a mailto fix in user.js for previous versions of FF
caused me some trouble until I got rid of that file in FF-3.0.

2. Some FF addon is causing loss of response.

If I were to switch to a new profile to see if the problem goes away, what files
can I safely import from my existing profile? More specifically, is there a web
link to change profile cleanly but without too much hardship, so that existing
bookmarks etc. are preserved?

Thanks.

Chandra

Today, I tried to catch what was going on in my ~/.mozilla/firefox/<something> while there was loss of responsiveness and audible/visible disk I/O.

It appears that places.sqlite-journal is being emptied and filled alternately. This is an extract with timestamps for these two files over time from successive ls -altr commands:

===================
0 2008-07-01 00:18 places.sqlite-journal
12894208 2008-07-01 00:18 places.sqlite

12894208 2008-07-01 00:18 places.sqlite
98488 2008-07-01 00:18 places.sqlite-journal

0 2008-07-01 00:18 places.sqlite-journal
12894208 2008-07-01 00:18 places.sqlite

0 2008-07-01 00:18 places.sqlite-journal
12894208 2008-07-01 00:18 places.sqlite

0 2008-07-01 00:18 places.sqlite-journal
12894208 2008-07-01 00:18 places.sqlite

0 2008-07-01 00:18 places.sqlite-journal
12894208 2008-07-01 00:18 places.sqlite

12894208 2008-07-01 00:18 places.sqlite
143632 2008-07-01 00:18 places.sqlite-journal

0 2008-07-01 00:18 places.sqlite-journal
12894208 2008-07-01 00:18 places.sqlite

===================

It appears that places.sqlite-journal is oscillating in size on disk while places.sqlite does not get changed. The initial and final sizes on disk for these two files are the same.

What is all the disk I/O resulting in?

If it is nothing (initial and final file sizes are the same), why is it happening at all?

Can anyone throw any light on what is going on?

Should I be watching other files?

Is this a result of the "awesome bar" feature?

Or could it be an addon that is causing this?

I really wish for a speedy resolution.

Thanks.

At least one other person seems to have encountered a similar problem on Windows Vista. See post of June 24th, 2008 at:

http://forums.mozillazine.org/viewtopic.php?f=7&p=3612185

entitled

Reducing disk I/O used by Firefox 3.0 • mozillaZine Forums

where the poster writes:

-----Quote----
Looking at the Windows resource monitor, it seemed that places.sqlite and places.sqlite-journal were the main culprits of all the disk I/O.
-----Unquote----

Interesting.

Alexander, do you know if the mozilla developers are working on this?

Thanks

2008/7/1 R (Chandra) Chandrasekhar <email address hidden>:

> At least one other person seems to have encountered a similar problem on
> Windows Vista. See post of June 24th, 2008 at:
>
> http://forums.mozillazine.org/viewtopic.php?f=7&p=3612185
>
> entitled
>
> Reducing disk I/O used by Firefox 3.0 • mozillaZine Forums
>
> where the poster writes:
>
> -----Quote----
> Looking at the Windows resource monitor, it seemed that places.sqlite and
> places.sqlite-journal were the main culprits of all the disk I/O.
> -----Unquote----
>
> --
> after fix for #215728 - Committing to urlclassifier3.sqlite still causes
> excessive CPU usage and disk I/O (the 2nd)
> https://bugs.launchpad.net/bugs/229745
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in The Mozilla Firefox Browser: Incomplete
> Status in "firefox-3.0" source package in Ubuntu: Invalid
> Status in "xulrunner-1.9" source package in Ubuntu: In Progress
> Status in firefox-3.0 in Ubuntu Hardy: Invalid
> Status in xulrunner-1.9 in Ubuntu Hardy: In Progress
> Status in firefox-3.0 in Ubuntu Intrepid: Invalid
> Status in xulrunner-1.9 in Ubuntu Intrepid: In Progress
>
> Bug description:
> Binary package hint: firefox
>
> some follow up comments from bug #215728
>
> [ https://bugs.edge.launchpad.net/ubuntu/+bug/215728/comments/118 ]:
> Yes, there is still some heavy disk I/O going on. It's no where near as
> bad as it was before the update. This patch is a HUGE
> improvement, but it still happens from time to time causing Firefox to
> become unresponsive for a few seconds.
>
>
> [ https://bugs.edge.launchpad.net/ubuntu/+bug/215728/comments/120 ]
> actually I've just measured some firefox heavy IO activity, but as Nick
> reported, it is a matter of seconds and in my case it doesn't make the whole
> system irresponsive as it was before the patch.
>
> Luckily I had disktop running, so here is the output. I can say that the
> the slowdown that I felt was probably due to the large write of 5.5MB in 5
> seconds you can see at 18:28:31 - some random IO activity I guess.
>

R (Chandra) Chandrasekhar, how long does this oscillating happen? one minute? one hour? all the time?

https://bugs.launchpad.net/ubuntu/intrepid/+source/xulrunner-1.9/+bug/229745/comments/15 points to an upstream explanation of the problem with sqlite performance. As I understand it, the problem has been mitigated, but not completely resolved, in firefox 3.0.1, and further resolution is not likely to happen in the intrepid timeframe. I'm therefore marking this bug as 'wontfix' for hardy and intrepid, though of course if a fix becomes available soon upstream that can be considered.

Alexander Sack wrote:
> R (Chandra) Chandrasekhar, how long does this oscillating happen? one
> minute? one hour? all the time?

I did not pay attention to the frequency, but from memory, I would say
it happens from once every half hour to once every hour.

Guys,
    I was having this problem early in the Hardy release cycle, but after
months of usage on multiple machines I can say that now is almost
disappeared.

To me, it would be more correct to say that the bug is resolved, than not.

As usual, great work guys.

2008/9/12 Son <email address hidden>

>
>
> --
> after fix for #215728 - Committing to urlclassifier3.sqlite still causes
> excessive CPU usage and disk I/O (the 2nd)
> https://bugs.launchpad.net/bugs/229745
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in The Mozilla Firefox Browser: Incomplete
> Status in "firefox-3.0" source package in Ubuntu: Invalid
> Status in "xulrunner-1.9" source package in Ubuntu: Triaged
> Status in firefox-3.0 in Ubuntu Hardy: Invalid
> Status in xulrunner-1.9 in Ubuntu Hardy: Won't Fix
> Status in firefox-3.0 in Ubuntu Intrepid: Invalid
> Status in xulrunner-1.9 in Ubuntu Intrepid: Won't Fix
>
> Bug description:
> Binary package hint: firefox
>
> some follow up comments from bug #215728
>
> [ https://bugs.edge.launchpad.net/ubuntu/+bug/215728/comments/118 ]:
> Yes, there is still some heavy disk I/O going on. It's no where near as
> bad as it was before the update. This patch is a HUGE
> improvement, but it still happens from time to time causing Firefox to
> become unresponsive for a few seconds.
>
>
> [ https://bugs.edge.launchpad.net/ubuntu/+bug/215728/comments/120 ]
> actually I've just measured some firefox heavy IO activity, but as Nick
> reported, it is a matter of seconds and in my case it doesn't make the whole
> system irresponsive as it was before the patch.
>
> Luckily I had disktop running, so here is the output. I can say that the
> the slowdown that I felt was probably due to the large write of 5.5MB in 5
> seconds you can see at 18:28:31 - some random IO activity I guess.
>

there have been more and more improvements on this in firefox 3.5 and firefox 3.6 (trunk), to say that this is fixed as good as possible. Some fixes might not land in xulrunner 1.9, but we will have 1.9.1 soon enough. in anycase, if you still have an issue in jaunty you could also test ext4 filesystem which should help to make the filesystem access done by firefox less IO hungry.

Alexander Sack wrote:
> in anycase, if you still have an issue in jaunty you could also
> test ext4 filesystem which should help to make the filesystem access
> done by firefox less IO hungry.

Thanks for that hint. I need clarification on the ext4 filesystem:

1. Will it be an option in jaunty?

2. Is it enough for the root partition / to be ext4 or should the /home
partition also be ext4?

Thank you.

On Intrepid on an AMD 64 running firefox 3.0.8 this is a grep of all firefox related processes when I did

pidstat -d 2 at different times:

Linux 2.6.27-14-generic (hostname) 02/04/09 _x86_64_
===
09:08:05 PID kB_rd/s kB_wr/s kB_ccwr/s Command
09:08:23 9140 0.00 12.00 4.00 firefox
09:08:25 9140 0.00 22.00 0.00 firefox
09:08:27 9140 108.00 102.00 10.00 firefox
09:08:29 9140 0.00 6.00 4.00 firefox
09:08:31 9140 0.00 3.96 3.96 firefox
09:08:33 9140 0.00 22.22 6.06 firefox
09:08:35 9140 0.00 26.00 18.00 firefox
09:08:37 9140 0.00 44.00 0.00 firefox
09:08:39 9140 0.00 4.00 4.00 firefox
09:08:41 9140 0.00 16.00 14.00 firefox
09:08:43 9140 0.00 14.00 6.00 firefox
09:08:45 9140 0.00 16.00 4.00 firefox
09:08:47 9140 6.00 76.00 4.00 firefox
09:08:49 9140 0.00 4.00 4.00 firefox
09:08:51 9140 0.00 26.00 4.00 firefox
09:08:53 9140 0.00 12.00 10.00 firefox
09:08:55 9140 0.00 8.00 6.00 firefox
09:08:57 9140 2.00 16.00 0.00 firefox
09:09:03 9140 0.00 6.00 0.00 firefox
09:09:05 9140 0.00 18.00 4.00 firefox
09:09:07 9140 4.00 126.00 12.00 firefox
09:09:09 9140 0.00 6.00 4.00 firefox
09:09:11 9140 0.00 8.00 2.00 firefox
09:09:13 9140 0.00 10.00 10.00 firefox
09:09:15 9140 0.00 10.00 6.00 firefox
09:28:18 9140 0.00 345.10 0.00 firefox
09:28:20 9140 0.00 352.00 0.00 firefox
09:28:22 9140 0.00 368.00 0.00 firefox
09:28:24 9140 0.00 384.00 0.00 firefox
09:28:26 9140 0.00 418.00 0.00 firefox
09:28:28 9140 0.00 416.00 0.00 firefox
09:28:30 9140 0.00 432.00 0.00 firefox
09:28:32 9140 0.00 448.00 0.00 firefox
09:28:34 9140 0.00 480.00 0.00 firefox
09:28:36 9140 0.00 464.00 0.00 firefox
09:28:38 9140 0.00 512.00 0.00 firefox
09:28:40 9140 0.00 512.00 0.00 firefox
09:28:42 9140 0.00 528.00 0.00 firefox
09:28:44 9140 0.00 544.00 0.00 firefox
===

If this will not be fixed here but only in jaunty, please ensure that this behaviour does not propagate into janunty as well.

Thanks.

This was Firefox idle? Without flash movies, Ajax applications etc.? Mine only shows low one digit kb/s values when using it actively. Nothing when not using it.

Arthur wrote:
> This was Firefox idle? Without flash movies, Ajax applications etc.?
> Mine only shows low one digit kb/s values when using it actively.
> Nothing when not using it.
>

Nothing more than a few open tabs and normal browsing. No flash,
Googlemap, etc.

Ever since FF 3.0.5 beta, I have had this type of problem. It then
decreased and receded into the background when an interim fox with
xulrunner was issued. Now it has suddenly re-emerged into the foreground
as a distraction after the latest security fix in 3.0.8.

On Mon, Mar 23, 2009 at 02:15:00AM -0000, R (Chandra) Chandrasekhar wrote:
> Alexander Sack wrote:
> > in anycase, if you still have an issue in jaunty you could also
> > test ext4 filesystem which should help to make the filesystem access
> > done by firefox less IO hungry.
>
> Thanks for that hint. I need clarification on the ext4 filesystem:
>
> 1. Will it be an option in jaunty?

Yes, from what i can recall its available as an option in the jaunty
installer. However, its still not 100% finished (which is reflected by
the fact that its still not the default for us).

>
> 2. Is it enough for the root partition / to be ext4 or should the /home
> partition also be ext4?

The write operation happens on your /home partition, so migrating that
to ext4 should help. However, remember that with ext4 keeping regular
backups is even more important.

 - Alexander

Hi,
      I've just done that, reinstalled my machine by putting ext4 on / (and
/home) and keeping my data on a separate ext3 partition.

I can only say one word: WOW!

Jaunty coupled with ext4 is super fast. Previoulsy my disk was pretty busy,
especially during startup, with a lot of IO activity. I watch IO activity
all the time, since it is actually my bottleneck.

After the upgrade, I almost _NEVER_ see any IO activity!!! I suspect ext4
delayed allocation is the key here, along with the fact that I reinstalled,
my previous filesystem had certainly some fragmentation (upgraded since
Gutsy).

So I would suggest anybody to deploy a similar approach: ext4 on / and data
in a separate ext3 partition.

Furthermore with Jaunty I noticed that preload is not necessary anymore, it
seems that the filesystem cache is already primed. Many application start up
much faster!

I have a 8-years old PC with 512MB of RAM that now feels like a new machine.

My best congratulations to all the devs :)

2009/4/11 Alexander Sack <email address hidden>

> On Mon, Mar 23, 2009 at 02:15:00AM -0000, R (Chandra) Chandrasekhar wrote:
> > Alexander Sack wrote:
> > > in anycase, if you still have an issue in jaunty you could also
> > > test ext4 filesystem which should help to make the filesystem access
> > > done by firefox less IO hungry.
> >
> > Thanks for that hint. I need clarification on the ext4 filesystem:
> >
> > 1. Will it be an option in jaunty?
>
> Yes, from what i can recall its available as an option in the jaunty
> installer. However, its still not 100% finished (which is reflected by
> the fact that its still not the default for us).
>
>
> >
> > 2. Is it enough for the root partition / to be ext4 or should the /home
> > partition also be ext4?
>
> The write operation happens on your /home partition, so migrating that
> to ext4 should help. However, remember that with ext4 keeping regular
> backups is even more important.
>
> - Alexander
>
> --
> after fix for #215728 - Committing to urlclassifier3.sqlite still causes
> excessive CPU usage and disk I/O (the 2nd)
> https://bugs.launchpad.net/bugs/229745
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in The Mozilla Firefox Browser: Invalid
> Status in “firefox-3.0” source package in Ubuntu: Invalid
> Status in “xulrunner-1.9” source package in Ubuntu: Won't Fix
> Status in firefox-3.0 in Ubuntu Hardy: Invalid
> Status in xulrunner-1.9 in Ubuntu Hardy: Won't Fix
> Status in firefox-3.0 in Ubuntu Intrepid: Invalid
> Status in xulrunner-1.9 in Ubuntu Intrepid: Won't Fix
>
> Bug description:
> Binary package hint: firefox
>
> some follow up comments from bug #215728
>
> [ https://bugs.edge.launchpad.net/ubuntu/+bug/215728/comments/118 ]:
> Yes, there is still some heavy disk I/O going on. It's no where near as
> bad as it was before the update. This patch is a HUGE
> improvement, but it still happens from time to time causing Firefox to
> become unresponsive for a few seconds.
>
>
> [ https://bugs.edge.launchpad.net/ubuntu/+bug/215728/comments/120 ]
> actually I've just measured some firefox heavy IO activity, but as Nick
> reported, it is a matter of seconds and ...