After-the-fact Filters on custom header won't match for IMAP messages

WFM - 2003013108
Reporter: do you still see this problem with a recent build?

I can confirm seeing this bug on Mozilla 1.3 and various nightlies leading up to
its release on my Windows XP machine. I filter based on a variety of custom
headers, some of which always work, and some of which always fail.

I do see the problem for the "List-Id", "Reply-To" and "Return-Path" custom
headers. I don't see the problem for the custom headers "X-BeenThere" and
"X-MirroredBy".

All of these custom header filters have worked for me on older builds.

Also, for what it's worth, I think that bugs #201244 and #184490 are likely to
be duplicates of each other. #20128 may be as well, but it's not clear if the
filter was a custom header or not.

For a guess, is mozilla possibly ignoring custom headers that don't have "X-" as
a prefix? I know that's a fairly common convention for a lot of custom headers
that I see in free mailing lists. Some of the proprietary, or less mainstream
free list servers I see don't use the "X-" prefix, and those are the ones I'm
having problems with.

Frank Widmaier: I am not seeing this problem in 1.3 Final or 1.4RC2; I created a
filter for 'User-Agent contains 1.3' and it worked as expected, for both newly
arrived mail and for running the filter manually (on the Sent folder).

Is this problem still an issue for you? Have you tried upgrading to 1.3 Final
or a 1.4 build?

Did you follow up on Bug 201244, as noted in comment 2? (Ignore the reference
to '20128', that is apparently a typo.) I'm curious if any of the details in
that report jibe with your experience.

=>WFM, no response from reporter.

I also have this issue - I have tried filtering on List-ID to separate mails
based on which mailing list they were sent to.
A view filter utilising this matches no entries, even though there are MANY
(most?) mails in the folder containing the header header
List-ID: <email address hidden>

I had already voted on this issue, but didn't want to spam. I've just retested
with the latest nightly trunk build (2003071704 on Windows 2000), with no change
in behaviour.

I am quite willing to enable debugging (NSPR logging), though a brief look
through the source suggested that there were no useful log points for this problem.

The mail source is IMAP, if that makes a difference...

Considering comment 2, I tested with mail filters (which I don't usually use),
and an attempt to label mails according to the same criteria. This appeared to
succeed (according to the filter logging), but no mails were actually labelled.
I restarted mozilla, and all my view customisations had disappeared. Attempts to
add any new views failed with

Error: uncaught exception: [Exception... "Component returned failure code:
0x80004005 (NS_ERROR_FAILURE) [nsIMsgMailViewList.addMailView]" nsresult:
"0x80004005 (NS_ERROR_FAILURE)" location: "JS frame ::
chrome://messenger/content/mailViewSetup.js :: onOK :: line 89" data: no]

Sorry if I'm mixing up problems here, but I'm adding these items in case they
are relevant.

Reopening

Gary Coady, thanks for your comment -- IMAP does have an issue with customized
headers. See bug 199689, which suggests removing 'custom' and 'body' entirely
from MailViews, simply because they don't work (reliably) for IMAP, at least for
messages in IMAP folders on the server.

*** Bug 233165 has been marked as a duplicate of this bug. ***

Confirming.

xref bug 177294.

See bug 205501 comment 4.

*** Bug 264375 has been marked as a duplicate of this bug. ***

*** Bug 281266 has been marked as a duplicate of this bug. ***

*** Bug 295050 has been marked as a duplicate of this bug. ***

Just a note; I see this behavior with Mozilla 1.7.10, for flags like
X-Spam-Status, X-Spam-Flag, etc. I'm also confused by the claim that
you cannot search for those headers with IMAP, which certainly has the
 HEADER <field-name> <string>
search predicate, at least according to the RFC...

should custom field "sender" work?

Doesn't seem to WFM - TB version 1.6a1 (20060119). This behavior is seriously irritating (and time-wasting) to say the least.

*** Bug 325723 has been marked as a duplicate of this bug. ***

*** Bug 368180 has been marked as a duplicate of this bug. ***

Are we talking about filters or views? I know views don't work, but I have several filters on custom headers that work just fine with my IMAP account.

With current builds, I'm seeing this as a problem with after-the-fact filters ("Run Now") but not for arriving messages. I'm not sure if "after the fact" is what the original reporter meant when he said "I activated the filter." I don't remember whether I had tested incoming mail when I wrote comment 7.

It's also a problem for MailViews (bug 205501), but not for Search Messages.

Something has changed with this from 1.5 to now. In 1.5 a filter on X-Bugzilla-Status contains new. Worked fine in 1.5, doesn't current branch builds. (As per previous dupe.)

*** Bug 369460 has been marked as a duplicate of this bug. ***

Testing further, I see that with TB 1.5.0.9, using a filter on a custom header for an IMAP account works correctly, both for arriving mail and running the filters on the Inbox.

With 2b2, the filters run as expected on arriving mail, but not after-the-fact.

Magnus, is that what you see (per comment 20) or are you seeing a problem with arriving mail?

Yes, that's the same thing I saw - arriving mail seems ok.

*** Bug 372828 has been marked as a duplicate of this bug. ***

*** Bug 374258 has been marked as a duplicate of this bug. ***

*** Bug 377505 has been marked as a duplicate of this bug. ***

*** Bug 377926 has been marked as a duplicate of this bug. ***

can this be the reason why "Trust junk mail header set by SpamAssassin" doesn't work? same story: it used to work on 1.5.0.x, stopped working after upgrade.

*** Bug 381051 has been marked as a duplicate of this bug. ***

Any chance we could fix this asap, the dupes are really filling up on it, not to mention suspected dupes.

The number of dup reports on this would seem to indicate that it is something that should be addressed.

*** Bug 382684 has been marked as a duplicate of this bug. ***

I also have a problem with this exact same behaviour. It is a problem because I want to sort mail already identified as spam that has been used to learn bayes tokens from spam that hasn't. I run my filters on the spam folder where spamassassin puts it. Worked in 1.5, I am now using 2.0.0.0, Linux.

As noted in BZ 205501, it appears that all we need to do to fix this is to make the header caching code cache user-added custom headers (no need to cache all of them) when the messages contain those headers. If someone who knows this code well will point me in the right direction, I will write this patch myself.

These type of filters worked on existing message on version 1.5.x.x but now under version 2.0 they don't work. I wish that Thunderbird had more filter items that were standard instead of custom. -- esp. List-ID as all list providers use this one

Linkify bug pointed by Comment #34. Bug 205501.

*** Bug 387417 has been marked as a duplicate of this bug. ***

It would be great if this problem can be fixed. Without this feature the Spam filtering is very in-effective when server filtering is used.

Hardly likely to happen is it?
It's been 5 years now since Frank Widmaier reported it. I've had to live with this broken feature for several myself. There are lots of reported bugs with IMAP and Thunderbird. Seemingly Thunderbird and IMAP are not meant to work together. Which is ironic with users pushing for portable apps.
Isn't IMAP trendy enough for the programmers to make it work?

I haven't been suffering for that long, but gee it used to work a treat (TB1.5) It should work. I can't understand all this about it not being a bug as some guys seemed to be arguing. Could we get an add-on that will do it, if it can't be done inside???

I decided to re-try the 'Message filters' and 'search messages' using a custom header on my imap account.

I retried it by trying to label emails by the header "X-Bugzilla-Product: Core" and mark them as 'Important' i.e. make them appear red.
I did this on email already in my Inbox and subfolders.

Hey, this works with TB 1.5.0.12 on Windows 2K sp4!

Searching with a custom header using the above header also appears to be working now.

So someone appears to have fixed it and not informed this thread? :-)

This has made my day!

Now if only some smart programmer can make TB upload all its settings, spam filter, etc. into my IMAP account so I can have a truly portable Thunderbird?

It worked in 1.5.0.x, broken in 2.0...

Ah well, I'll not be moving to V2 until this bug and the identities bug is fixed then. :-)

trusting you are all correct => regression. But it seems odd this is also reported as broke in 1.5 per bug 325723

someone can use the builds at http://archive.mozilla.org/pub/thunderbird/nightly/ to narrow the time frame of the regression. I've forgotten at the moment if you should use mozilla1.8.0/ mozilla1.8/ builds (too early in the morning)

between 1.5 and 2.0, we starting running filters after the fact on the local database instead of the imap server, which is what broke this for imap. We did this partly because some imap servers don't support search, or their search support is broken, and people complained that filters worked on incoming mail, but not after the fact...

So what about the rest of us who have working imaps servers?
Do we lose a feature because of faults or shortcomings on broken imap servers?
Seems to me that people using these 'broken' imap servers should be aware of the fault and ask their ISPs for an upgrade.

There were other reasons to switch it - e.g., filters that check if a sender is in an address book only work locally, not on the imap server. Filters run locally tend to be a lot faster, though that's less important than correctness, obviously. Either way, some filters aren't going to work after the fact without some non-trivial re-working of the search backend. Either we could store the custom headers in the .msf files as msg hdr properties and allow local search to work with those properties, or we could figure out how to do hybrid searches, partly locally, partly on the server. I've been leaning to the former, so that things like views on custom headers can work with imap as well.

To me these type of filters worked under 1.5.x and it is 100 giant steps backwards in 2.0. There is NO REASON THAT ANYONE CAN GIVE THAT IS REASONABLE AS TO WHY THEY DON'T WORK UNDER 2.0.X. Since I am the admin of my IMAP server, I can go move the messages else where using the OS then move them back -- Thunderbird thinks they are new messages and will move them but to someone that doesn't have this type of access to their IMAP server, like those who might be using AOL for instance, will have to do something convoluted such as move the IMAP message to a local file then run local filters -- thanks but no thanks as they should be able to be run in place. Or, have a separate instance of 1.5 installed on your system and run that to do after the fact filtering on IMAP.

Probably the best solution is to store all of the headers in the .msf not just some of the headers as one might add a custom header at some later date.

To the person that is using 1.5 -- yes it works under 1.5 but not 2.0.x.

George

I'm sorry, did you not read my previous comment? And funny you should mention AOL - they're one of the IMAP servers that didn't implement SEARCH, though maybe they do now.

The other workaround is to configure your IMAP folders for offline use - then filters run after the fact have access to all the message headers.

Looks like it was bug 278052 that broke this (for the current situation, this bug may have been something else from start...)

(In reply to comment #45)
> between 1.5 and 2.0, we starting running filters after the fact on the local
> database instead of the imap server, which is what broke this for imap. We did
> this partly because some imap servers don't support search, or their search
> support is broken, and people complained that filters worked on incoming mail,
> but not after the fact...
>

I am still using 1.5.0.12. I still see the behavior where it works on incoming, but not after-the-fact. I see others commenting that this broke with 2.x. Or are they saying that with 2.0 filtering on custom headers no longer works even on incoming?

Also: you say that it is searching on the local db rather than the imap server that has broken it ... why would it not work on the local db?

No, they're saying that it still works on incoming mail, but stopped working for filter after the fact. I'm not sure what you're seeing in 1.5.

By default, only some headers are put in the local db. That's why searching/filtering with the local db on imap doesn't work.

To Hartman:

I had no problems with 1.5.0.12 with after fact custom IMAP filters or any of
the 1.5 versions of Thunderbird.

All of my problems with filters started after my upgrade to 2.0.

If I am reading what has taken place, is that filters work on the local
database because of broken IMAP servers -- making the rest of us suffer because
of this... people are still complaining about this issue. I was using Outlook
Express and Outlook 98 until I found Thunderbird and it is filtering
capacity... I now could use one product for reading and filtering email --
Outlook Express will filter IMAP from one IMAP account to another (move from
one IMAP server to another server or account on the same server). Outlook will
do such filter but will not display images inline. Thunderbird does both and
until recently did it very efficiently. Now this issue with filtering being
broken to me has almost made it worth going back to an older version that does
work and forgo the new features of Thunderbird until this is fixed. -- The fix
is simple: include all headers in the local DB.

Turn on offline access would do this but it not only caches the headers, it
caches the body as well and if disk space wasn't a problem this would be a good
solution.

Created an attachment (id=273465)
partial fix

This makes the search code smart enough to look in the msg hdr for arbitrary/custom header values. This works as long as the mailnews.customDBHeaders pref includes the arbitrary/custom header. Now I have to figure out how to make that happen automatically for filters and views on custom headers, or get them in the db somehow...

(In reply to comment #53)
> No, they're saying that it still works on incoming mail, but stopped working
> for filter after the fact. I'm not sure what you're seeing in 1.5.
>
> By default, only some headers are put in the local db. That's why
> searching/filtering with the local db on imap doesn't work.

Can someone explain this term 'after the fact'?

I get my email via imap. It stays on the imap server, I don't download it locally. The custom filtering works for me in 1.5.0.12.

Perhaps 'after the fact' means when its downloaded locally?

Stuart,

When I was using 1.5.x, filtering with IMAP server work very well.

Customized filter broke with Thunderbird 2.0 with "after the fact" filtering but not with "incoming new mail".

There are basically 2 ways that filtering is done:

1) When new mail arrives -- this isn't broken and works fine.

2) After email is in your inbox -- called "after the fact". This type of filtering is done by clicking on "Tools->Message Filters". Next you choose a filter by clicking on it. Then you click on "Run Now". It works perfectly on the "Standards" like "Subject" or "From" but if you have a customized filter such as "List-Id" -- Yahoo Groups and Google Groups use this to identify what group the message originates -- it doesn't work.

Say I joined a Yahoo Group called New_Safari... it would have List-Id of <New_Safari.yahoogroups.com>... and, I didn't create a filter. I soon discovered that I was receiving 75 emails a day from this group. I next created a filter based up on the List-Id (custom filter) so that all messages from this Yahoo Group would now be automatically filter to a folder called "New_Safari". All new incoming email is filter to the above folder. However, if you tried to run the filter on existing email in your inbox, the filter would not work.

Basically there was a change in the way filtering was done from 1.5 to 2.0 which broke customized filters when trying to run them on existing emails. I am not sure who coined the phrase "After the Fact" filtering. To me it isn't a good phrase... the best description, IMHO, is "Running custom filters on existing email is broken". When I was searching for this bug, I could only find it by typing in IMAP Filters because of the title. I would have never know in a million years that this was called "after the fact"

George

Thank you George for this.
It confirms we're talking about the same thing. :-)

If V2 of TB isn't fixed then our company and no doubt others will just not use it. As a business we do a lot of email searching on body, and x-headers via imap.

If the developers don't want to pre-index all the headers of every email, then I can only suggest that when a user adds a custom header to search with, that this action prompts a request to automatically reindex all the email in the persons account. Alternatively just get the program to re-download the headers of each email as it searches.

Hopefully there will be a way for IT developers like myself to install TB 2 with a preset list of custom headers.

I feel we should be user-feature lead rather than server-feature lead.

> I feel we should be user-feature lead rather than server-feature lead.

If the implication is that this was broken on purpose, that's mistaken. We didn't realize at the time that this was going to break filter after the fact for custom headers for IMAP.

Yes, IT developers can install TB 2 with a preset list of custom headers via several mechanisms - from the lightweight: adding a new aaaa.js in the defaults/prefs install dir with the mailnews.customDBHeaders pref set, to heavyweight - using MCD to control users prefs from an http server.

I've always been able to filter messages to a Yahoo or Google group by adding "Reply-To" as a custom header and filtering on "Reply-To: <email address hidden>".

As for the "Newsgroups" header, that belongs to Usenet news, not mail. News filtering is handled separately and doesn't support custom headers yet. You may want to vote for bug 16913 which addresses that problem.

(In reply to comment #60)
> I've always been able to filter messages to a Yahoo or Google group by adding
> "Reply-To" as a custom header and filtering on "Reply-To:
> <email address hidden>".
>
> As for the "Newsgroups" header, that belongs to Usenet news, not mail. News
> filtering is handled separately and doesn't support custom headers yet. You
> may want to vote for bug 16913 which addresses that problem.
>

Ok, so you don't like those examples. I personally use the "Received" header for spam filtering. (e.g. Received header contains "209.104.219."). The spammers "from" is too variable. Subject filtering can't be effective until we get wildcarding implemented (and possibly not even then). But I can identify an IP address range for many spammers. Once identified I can redirect their future messages, but the current inability to apply a filter to the inbox prevents me from being able to clean out the stuff they sent before I identified that IP range.

If you can think of an counter argument to my example, I am sure somebody else can provide yet another use. Regardless of any specific responses to specific examples, there /are/ uses for this capability.

(Personally I don't really see why "Received:" should have to be a custom header anyway...)

(In reply to comment #54)
> To Hartman:
>
> I had no problems with 1.5.0.12 with after fact custom IMAP filters or any of
> the 1.5 versions of Thunderbird.

Nonetheless, I see this issue with 1.5.0.12, exactly as described. Specifically, with the "Received" header, which must be defined as a custom header as it is not (for some inexplicable reason) included among the standard headers.

> Turn on offline access would do this but it not only caches the headers, it
> caches the body as well and if disk space wasn't a problem this would be a good
> solution.
>

It would be nice to have an option to store headers but not body locally, wouldn't it?

> Nonetheless, I see this issue with 1.5.0.12, exactly as described.
> Specifically, with the "Received" header, which must be defined as a custom
> header as it is not (for some inexplicable reason) included among the standard
> headers.

The reason that this bug was re-activated is because of broken IMAP Server Search functions. "Some people" complained that "after the fact" filtering was broken and it turned out that the reason that it was broken is that the IMAP search functions were not implemented or not implemented correctly.

Probably the best way to solve this issue, is not to use the IMAP search functions but to cache the headers in the .MSF databases then let Thunderbird do rest. And, I agree with the statement:

> It would be nice to have an option to store headers but not body locally,
> wouldn't it?

Because some users might be limited by disk drive space.

Until there is a fix, I will do "after the fact filter" one of two ways... since I am the admin of the one of the servers. I have created a folder called
"- temp" so that I can easily move the messages I want filter into that folder. I will create a filter based upon the message. Since my IMAP server stores email in individual files, I then can at the OS level move the files from the temp folder back to the Inbox folder. Now Thunderbird thinks it is a new message and run the filter as if it was new mail. On the server that I don't have admin rights to, I use the offline function.

George

>The reason that this bug was re-activated is because of broken IMAP Server
> Search functions.

Please read #47 again - filtering after the fact on ab presence was also broken in 1.5, and fixed by using local filter evaluation.

Thunderbird 2.0.0.5 can not more than one tag by filter rules.
In Thunderbird 2.0.0.4 it was possible to set more than one flag by filter rules.

*** Bug 380577 has been marked as a duplicate of this bug. ***

Just tried using a custom header filter in TB 3.0a1pre win32 and it appears to still be broken. The header I am filtering on is called "Auto-Submitted" and should have a value of "auto-replied".

The filter works when not run manually, i.e. new incoming mail gets filtered by this rule. This is the case for both 3.0a1pre and 2.0.0.6.

(In reply to comment #50)
> The other workaround is to configure your IMAP folders for offline use - then
> filters run after the fact have access to all the message headers.
>

Thank you, David! This was a painless and effective workaround. Now I can use the Run Now functionality until the bug is resolved.

(In reply to comment #67)
> Just tried using a custom header filter in TB 3.0a1pre win32 and it appears to
> still be broken.

:(

*** Bug 413372 has been marked as a duplicate of this bug. ***

*** Bug 418886 has been marked as a duplicate of this bug. ***

*** Bug 422495 has been marked as a duplicate of this bug. ***

This bug was originally reported in 2002 (!!!) and it appears to be an important usability issue.

Why this bug isn't fixed in such a long time?

That is a darned good q

Perhaps because IMAP is poor cousin to POP3?

I'm been waiting years for Thunderbird to offer user-set default display options.
But no the developers rather build the next sexy version of TB than fix existing problems. :-)

Problem still exists in 2.0.0.14

Cute. I don't think even Microsoft would take six years to fix a big bug like this in Outlook Express.

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; de; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1
Build Identifier: Version 2.0.0.16 (20080708)

Hi,
I'm trying to filter my incomming messages in dependence of the content of an additional email-header "X-Content-Descr". I've go on like described in http://kb.mozillazine.org/Message_Filters#Account_specific_filters , but the rules simply whant to "run"
I've created a new rule, registrade a new special-filter "X-Content-Descr" an add a rule
"X-Content-Descr" content "INFO" => mark as read an add Tag "ISSI-INFO"

Reproducible: Always

Steps to Reproduce:
1. Create a tag "ISSI-INFO"

2. Add a special header, prefs.js should then contain:
 user_pref("mailnews.customHeaders", "X-Content-Descr");

3. Create the rule:
 RootFolderUri=imap://<email address hidden>
 version="8"
 logging="yes"
 name="test_header"
 enabled="yes"
 type="1"
 action="Mark read"
 action="AddTag"
 actionValue="issi-infos"
 condition="OR (\"X-Content-Descr\",contains,INFO)"

4. send/receive an Email with the additional header
5. run "filter"
Actual Results:
Nothing happens.
I've turned the log on and aplied the rule, but nothing happens, there is also nothing in the log.

Expected Results:
The email with the additional header should been marked as read and should also have an Tag "ISSI-INFO"

An Example-Email can be found on http://pastebin.mozilla.org/504808

The Email-Account is an IMAP-Account, but so far I read this should only be a problem if I whant to filter in dependence of the body-content.

I've other "normal" rules, they all run correctly.

Problem was also reported in http://forums.mozillazine.org/viewtopic.php?f=30&t=774115

Thanks for your Help,

Kasko

Created an attachment (id=332340)
Exmple Email for reprodution purpose

Probably a dupe of bug 184490

Is anybody working on solving this issue?

I mean, everybody who seriously uses E-Mail (thus uses IMAP) and seriously needs filtering (which automatically is the case after a few years of seriously using E-Mail) gets hit by this bug and switches to another mail client.

(In reply to comment #64)
> The reason that this bug was re-activated is because of broken IMAP Server
> Search functions. "Some people" complained that "after the fact" filtering was
> broken and it turned out that the reason that it was broken is that the IMAP
> search functions were not implemented or not implemented correctly.
>
Most IMAP servers probably now implement such standards somewhat. Many people make use of these standards-conforming servers and use standards-conforming clients like Seamonkey.

> Probably the best way to solve this issue, is not to use the IMAP search
> functions but to cache the headers in the .MSF databases then let Thunderbird
> do rest.
>
Seamonkey should be able to leverage the standards-compliance/conformance of good IMAP servers. Cannot there be an option in the "Server Settings" section of an IMAP account to choose whether or not to use the server-side IMAP SEARCH command with a warning about nonstandard servers? This would make sense for people who have limited disk space and need mail filtering for already received mail.
I would hope that this issue could finally be solved before Seamonkey 2 is out, as it's been around for a while.

(In reply to comment #79)
> Seamonkey should be able to leverage the standards-compliance/conformance of
> good IMAP servers. Cannot there be an option in the "Server Settings" section
> of an IMAP account to choose whether or not to use the server-side IMAP SEARCH
> command with a warning about nonstandard servers? This would make sense for
> people who have limited disk space and need mail filtering for already received
> mail.

I fully agree. Don't take away *expected* functionality because of quirks in some IMAP servers.

Test result with Tb trunk 2008/8/09 build on MS Win-XP SP3, with Gmail IMAP.

(0) Filter for IMAP account : If User-Agent contains 'Thunderbird' add tag
    No automatic download option is enabled
(1) Initial (Inbox of the IMAP account)
    - Tools/Account Settings/Server Settings/Advanced/Offline
      "Select this folder for offline" : Unchecked
    (1-a) Upon download => Tag was added
          (All headers are downloaded because without header choice upon FETCH)
    (1-b) Manual "Run filters on" => Tag was NEVER added => Remove all tags
          (View source of mail displays the User-Agent: header)
(2) Enable Offline-use, and execute "Download Now"
    - Tools/Account Settings/Server Settings/Advanced/Offline
      "Select this folder for offline" : Checked
      Execute "Download Now"
    - Manual "Run filters on" => Tag was added => Remove all tags
(3) Disable Offline-Use
    - Tools/Account Settings/Server Settings/Advanced/Offline
      "Select this folder for offline" : Unchecked
(4) Restart Tb
(5) Manual "Run filters on" on Inbox => Tag was added

Why Tag was added at step (5) even though offline-use is off?
Tb tried to read downloaded mail data for offline-use when "Run Filters On" at step (1)?

Can anyone confirm that this bug is also in the new Thunderbird 3 (Shredder)?

David, it seems that https://bugzilla.mozilla.org/show_bug.cgi?id=449195 is the same bug as this described here.

Yes, I can confirm that. Just tried with Shredder version 3.0a2 (2008072418). Nothing changed.

*** Bug 380604 has been marked as a duplicate of this bug. ***

*** Bug 449195 has been marked as a duplicate of this bug. ***

Yes, this does look like a dup. Hopefully that one can get resolved soon, as it seems to be a pretty common problem.

*** This bug has been marked as a duplicate of bug 184490 ***

Confirmed again, using Gecko/20080828032200 Shredder/3.0b1pre. Filters using custom headers that work perfectly in TB2.0x fail to work now.

For me this is a blocker, even just for testing... :(

I tried writing an extension that adds a column to the thread pane, giving the user the ability to sort by an (organisation specific) X-Header.

I'm assuming that this is currently not possible though, due to the headers not being available via the gDBView.db.GetMsgHdrForKey(...) function. Is my assumption correct? Is this also what leads to this bug?

As an addition to my comment #86, I've noticed the following:

If I compact all folders (which apparently rebuilds the .msf files), the
filters on *some* custom headers start working!

There's at least one exception, though: filtering on a "Sender:" header.
It looks like maybe there's an internal "Sender", which overrides it? I
vaguely remember another bug report that mentions this, I'll have to go
hunt it down.

In one of such cases, the exact condition in msgFilterRules.dat is:

  condition="OR (\"sender\",contains,<email address hidden>)"

and messages from this particular mailing list truly contain this
header:

  Sender: <email address hidden>

However, TB3 doesn't filter on this condition.

(In reply to comment #88)
> There's at least one exception, though: filtering on a "Sender:" header.
> It looks like maybe there's an internal "Sender", which overrides it? I
> vaguely remember another bug report that mentions this, I'll have to go
> hunt it down.

Right, that's bug 404489. But for TB2.x, it *does* work...

*** Bug 456193 has been marked as a duplicate of this bug. ***

*** Bug 459010 has been marked as a duplicate of this bug. ***

*** Bug 465201 has been marked as a duplicate of this bug. ***

the remaining "Sent" issue sounds like something Kent already fixed.

is this still an issue w/ trunk builds? I'm going to minus this since we now configure imap folders for offline use by default.

David, It might be nice if TB could search imap folders that were configured
for offline use without contacting the server, just as if they were POP3
folders. But I have not found it to be able to do so. Even though all my
imap folders are configured for offline use, I find that my search
capabilities are still limited to the capabilities of the server.

*** Bug 480540 has been marked as a duplicate of this bug. ***

I did some brief testing of this on TB3.0b3pre, and custom headers are working for offline IMAP folders, but not working for online IMAP folders (as expected). The additional code in bug 495529, once landed, allows us to deal with more complex filter attribute enabling situations such as this. The same issues will also exist for Imap body filters, though these custom headers are made more complex by the workaround in attachment 273465 on this bug, which in special circumstances allows them to work even on online IMAP if I understand correctly.

I'm tempted to take this bug, move it to trunk, and morph it into a request to clean up the handling of the custom header filter enabling in TB3 (dependent on bug 495529). The other option is to leave it applying to TB2, and WONTFIX it on mozilla1.8 (if we do such things). The third option is WFM with a new bug to cleanup the enabling issues in the filter editor.

Opinions?

(Doubt you meant bug 495529)

As long as it fixes the remaining part of this bug I think you should do it here.

OK, I morph this into a request to cleanup the enabling of the custom attributes in the filter editor in TB3, using the new capabilities of the filter editor in bug 495529.

Sorry, the correct related bug is bug 495519 (which landed for TB3 beta 3).

After reading the comments on this bug more thoroughly, I've come to the conclusion that the request in this bug is really to enable the use of IMAP server search functionality in offline filters, probably in combination with the offline store. That would allow after-the-fact filters to work reliably where users do not have folders selected for offline use.

I'm not really willing to do that work in the near future, so I am going to remove myself from being assigned to this bug.

I should point out that there are two workarounds for this issue. In the default TB configuration, messages are AFAIK now downloaded for offline use, and then this bug does not apply. For users that do not want messages offline, there is the workaround provided by the existing patch on this bug. That is to add the desired header to the mailnews.customDBHeaders preference, and compact all folders.

So I think what I will do in the short run is to use a new bug, to make sure that when custom headers are defined for filters, they are also automatically added to the message database. (Bug 363238 is close, and for now I'll continue there). That way, users can simply define a custom header in the filter editor, and it will then work on future messages. (To make it work on old messages, the would need to compact old folders manually).

(In reply to comment #101)
That sounds like a good plan. Thanks!!

*** Bug 479682 has been marked as a duplicate of this bug. ***

let's use proper terminology in summary

*** Bug 455680 has been marked as a duplicate of this bug. ***

Is there any news on this bug? I have just run into the same problem, although I am manually running rules on messages after I have read them (to automatically sort them into the correct folders.) I'm trying to sort on the List-Id field, but wiping all the .msf files didn't help. Enabling offline folders on the Inbox then compacting all folders didn't work either. My customHeaders field says:

user_pref("mailnews.customHeaders", "DomainKey-Signature: Received: List-Id");

So I'm not sure whether it's missing a trailing colon there or what. But either way, even though the message is visible in the preview pane when I run the filter, the extra headers still seem to be ignored.

There has been virtually no progress on this bug for years. IMAP tends to be the poor relation compared to POP3 and developers seem to ignore the bugs it has for some reason.

Quite why it's getting ignored is beyond me.

I use TB-3.0.11 now and I didn't see failures with filters for a long time now.
Before I kept getting failures every other day with even fewer filters defined.

Not sure if this means that the problem is fixed or if it only got less likely to happen.

Created attachment 502220
Testcase : Filter rules, with two messages : one which works fine, the other which doesn't work

I'm using Ubuntu 10.10 and Thunderbird 3.1.7 and I have a comparable bug.

In the filter windows, I created a new header : Reply-To. I then tried to use it in a filter and it doesn't work correctly. The attachment 502220 in the comment 109 are the file I used to do my tests.

All my tests were done after the both messages were received. Each time I put the message in the Inbox folder before to run the filters.

I try to filter the both messages in the attachment by three ways : with the Execute button in the filter windows, with Tools --> Execute filters on the message and with Tools --> Execute filters on the folder.
When I applied the filters on the message, nothing happened !
When I used the two others ways, the message which begins by "Filter works :" was well filtered, so moved in the folder I chose (Ubuntu in this case).

I also try this filter on other messages and it doesn't work. In comparison, my other filters (on non-personally created headers such as from or to) works fine with the three ways.

(In reply to comment #110)
I've been filtering on "Reply-To" successfully for years (but not after-the-fact like your tests).
Filtering on "Sender", however, never finds a match.

I just discovered this because "Sender" wouldn't work for me, TB3/Ubuntu; mail was already in a different folder. To force the filter to run, I marked this mail as unread, then dragged it to Inbox - and seemingly, it got filtered correctly (although that rule looks either in Sender or in Reply-To, so I cannot tell what piece actually triggered).. The problem here, also (seemingly) was that the same message according to another filter I have for "To:" should have filtered in to the original folder..

I guess one thing missing from the Filter Log, is actually a note about which header was it that matched a particular rule; currently it just says:

> Applied filter "My filter ONE" to message from Some Person
> <email address hidden> - My Subject at 2011-04-12 15:34:53 moved
> message id = <email address hidden> to
> imap://myaccount%<email address hidden>/INBOX/folderONE
>
> Applied filter "launchpad-bugs" to message from Some Person
> <email address hidden> - My Subject at 2011-04-12 15:34:53 moved
> message id = <email address hidden> to
> imap://myaccount%<email address hidden>/INBOX/folderTWO
>

... and there isn't any information which of the three rules I have in my "launchpad-bugs" rule actually triggered its run... And so, in the end, turns out it works for me (so I'm not really reporting a bug) - but it was just a bit difficult, to determine how to test the rule and see if it works.

I used to get this problem a lot but not recently.
Do you still see it in the latest 8.0?

*** Bug 564189 has been marked as a duplicate of this bug. ***

Still very much broken on 10.02 in IMAP as it has been for many years.

I created a custom header rule:

version="9"
logging="yes"
name="TEST"
enabled="yes"
type="17"
action="Copy to folder"
actionValue="mailbox://nobody@Local%20Folders/TEST"
condition="OR (\"mime-version\",contains,1.0)"

This should have copied at least one email!
But none appeared and the Filter log remained totally blank.

Come December this bug will be 10 years old!
Amazing.

Update:

It did actually work when receiving new mail!

But not when manually run.

The bug is there in TB10.0.2. The filters did work for my IMAP account till today, but when I tried to read filter log (which was 850M at the moment) TB hung, I killed it and restarted, and custom header filters stopped working. But Body filters do work.

Here in my 10.0.2 they only work with new incoming IMAP mail. The manually run filter of custom headers simply doesn't work. So the bug is in there.

And for me (on 10.0.2 too) custom header filter is not working on new incoming mail, but works if run manually.

Are you sure Constantin?

For me it's the opposite and this bug report is for the custom filters not working when run manually.

Yes, I am quite sure. I have List-ID filter. Al message that match that filter drop into my INBOX, but running filters from menu (Tools -> Run filters on folder) in INBOX move them to desired destination.
Moreover, everything was working fine till I tried to see Filter Log, which grew up to 850M to that moment. TB hung, I killed it, manually deleted filterlog.html, and filters did broke.

Maybe I should open separate bug?

This behavior is still present in the most current versions (note message date) of Thunderbird on Ubuntu and SeaMonkey on Windows 7. It does appear to be an issue when applying after the fact to IMAP and POP mail. Someone commented about a search function for AOL. Every webmail account I have, including AOL, has a search function. Many of these *custom* headers are standard headers. The best workaround, in my opinion, is to simply add common headers to future builds. This can also be an issue because occasionally the mail filters cannot find the folder the message is to be moved (another bug?). In my mind, X should imply custom headers since that is the recommended standard. All other common headers should be considered standard.

A suggest list, including those already listed, should contain Authentication-Results, BCC, CC, Content-Type, Content-Transfer-Encoding, Date, Delivered-To, DKIM-Signature, DomainKey-Signature, From, List-Id, List-Subscribe, List-Unsubscribe, Mailing-List, Message-ID, MIME-Version, Precedence, Received, References, Reply-To, Return-Path, Sender, Subject, To.

I'm sure this is not a comprehensive list. Also, some of these may be meaningless and some starting with X might be added. However, the person working on this bug will make the final decision when editing the code. Please offer further suggestions so that we may come to an agreement. I don't think this work around will add bulk or slow down mail.

*** Bug 531483 has been marked as a duplicate of this bug. ***

The bug is still in TB 15.

TB 16.0.1, still broken.

Any link with Bug 678322 ?

Please fix it.

TB 17.0.3/Mac, still broken. And so annoying...

Well I don't know the internal policies of Mozilla, but this bug has been around since 2002 and yet remains unfixed.

So either we are wrong complaining about this or something is a miss with Mozilla and the way they fix bugs?

Any way we can raise the profile of this sticky bug? :)

Thanks for the ping, Stuart.

I have not thought about this for awhile, but jumping from comment 101 to the referenced bug 363238, that bug is only implemented for Thunderbird 19 and later, so the fix would not be in the current shipping product, but would be available in the current beta.

Yet that fix is only automating the workarounds mentioned in bug 101. That is, when you define a new custom header to use in filters, that custom header will only work for future messages unless you rebuild the folder.

But the current situation (that is, what works in the current Thunderbird beta) is that when you define these custom headers, then they will work for all messages received after that point. They can work for past messages if you rebuild the folder. Any other functionality is very difficult to implement given the current design of the backend, and is unlikely to get any additional work for the foreseeable future.

(In reply to Kent James (:rkent) from comment #130)
> Any other functionality is very difficult to implement given the current design of the backend, (snip)

Phenomenon itself is pretty simple;
- If filtering upon fetch of new mail download, custom headers are fetched
  because used custom headers are placed in mailnews.customHeaders by
  message filter definition, and Tb's IMAP code includes the custom headers in
  uid xx fech BODY[HEADER.FIELDS (DATE FROM ... custom_headers ...)],
  then filter can process the custom headers.
- If after the fact filtering, and if custom header is not held in offline-store
  file, Tb doesn't request re-fetch of custom header and try to use locally held
  data(.msf, offline-store file), then filter can't obtain custom header data.

So, any of following can be a solution, although some of them have issue of "Repair Folder is needed".
(note:)
(Offline-Use=On here == auto-sync is enabled && selected for offline use )
(Offline-Use=Off here == auto-sync is disabled || not selected for offline use)
a) Force saving custom header data in .msf by setting in mailnews.customDBHeaders
   in addition to current mailnews.customHeaders upon filter definition.
b) If custom header is used, save custom header data in offline-store file upon
   fetch even when Offlne-use=Off folder, in conjunction with change to "fetch
   body[HEADERS]" from "fetch body[HEADER.FIELDS]", and use the header data in
   offlie-store file even when Offline-use=Off folder.
c) If custom header is used by filter, and if the custom header is not held in
   offline-store file, and if it's Work Online mode,
   issue uid xx fech BODY[HEADER.FIELDS (custom_headers)] according to
   mailnews.customHeaders
d) If custom header is requested, and if IMAP Offline-Use=Off folder, and if
   filter execution request is not "when Checking new mail" only,
   ask user to change the folder to Offline-use=On or not,
   and if answer is no, reject the rule with message of considerate apology.
   In addition to it, even when Offline-Use=On folder, if retention setting for
   offline-store file size limitation is used, ask user,
      Can you promise that you will never complaint about bug 184490?
   and, if answer is no, don't accept the rule.
e) When custom header is fetched upon initial fetch by mailnews.customHeaders,
   if all fetched headers is saved in Disk/Memory cache when Offline-use=Off,
   and if Disk/Memory cache is accessed when custom header is missing,
   problem of this bug may be resolved, but I'm not sure.

I perefer d) if short-term or mid-term solution, and I think d) is kinder to user than current bypass of "hiding Body at filter definition panel" and d) is applcable to "Body filter upon IMAP fetch" case too(see bug 806308 for Body only and bug 199689 for customized header and Body, please).

From perspective of implementation workload only, a) is simplest and easiest. But I don't think it's safe, because it's easily increases memory requirement and may cause performace problem. "setting in mailnews.customDBHeaders", which is known as a simpler workaround of bug 402594 than "filter rule of custom header of Received", should be done by user as a...

Adding some words in bug summary for ease of search.

By the way, Wayne, regression over what?
> <email address hidden> 2007-07-23 Keywords regression
If I understand problem correctly, I think this bug is issue since initial of "message filter on custom header".

(In reply to WADA from comment #132)
> By the way, Wayne, regression over what?

do you mean bug#? no idea.
And I did not confirm via testing.

As written in comment 44, I added regression keyword based on prior comments that it worked in version 1.5

WADA: (re your comment 131): I'm pretty sure that your a) and b) are both implemented by TB 19. It is the alternatives that are quite difficult, for example your c) would require filters to handle async searches, which are quite difficult given the current architecture.

(In reply to Kent James (:rkent) from comment #134)
> I'm pretty sure that your a) and b) are both implemented by TB 19.

Very good news for many bugs!
But one bad... No performane impact by (a)?
(Huge Mbox like [Gmail]/All Mail is very popular, and I saw a bug report by user on slowness with 136MB Junk.msf in the past...)

(In reply to Wayne Mery (:wsmwk) from comment #133)
> As written in comment 44, (snip)

Wayne, thanks for pointing the comment. By you, I could reach comment #45 by David with no effort :-)

(In reply to WADA from comment #135)

> Very good news for many bugs!
> But one bad... No performane impact by (a)?
> (Huge Mbox like [Gmail]/All Mail is very popular, and I saw a bug report by
> user on slowness with 136MB Junk.msf in the past...)

The changes that I am thinking of only expand the size of the mbox for custom headers added by the user. Although there is a performance impact in that case, it applies only to a small minority of users who add a particular custom header. By adding the header, they are saying that it is important to them, so adding it to the database is necessary for them to be able to use it effectively.

(In reply to Kent James (:rkent) from comment #137)
I see, I stop worryng about edge cases by edge users. As Fx 19 is already released, I hope Tb 19 will be available soon. Thanks for your great effort for resolving problems.

> I stop worryng about edge cases by edge users

Are those not the users who help nail down the bug per se?

 #c45 stop imap search

Wouldn't it be better to indicate IMAP server deficiencies to the user and disable features for problematic users and not punish everyone else with dimished functionality? [ditto #46]

> There were other reasons to switch it - e.g., filters that check if a sender is in an address book only work locally

How is that a reason?

Local filters should run as local filters and not attempt to run as server filters.

> or we could figure out how to do hybrid searches, partly locally, partly on the server

yes, please

#48
> Since I am the admin of my IMAP server

That could allow for extensions to manipulate server's EXIM rules [for future messages]

> other workaround is to configure your IMAP folders for offline use

That's not helpful in an ideal way.

#39
> [2007] Isn't IMAP trendy enough for the programmers to make it work?

It should be by now as many smart phones default to IMAP -- as android when microsoft ActiceSync is not available.

Else the option of Z-Push or ZCP or whatever it's been forked in to N months from now.

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Ok, I have just done a complete re-test and confirm that it seems to be working for me now. To be clear, my (previously failing) test was:

1, Receive an email that may have custom headers in it. (eg, X-customheader-set)
2, set up a message filter to look for and action on that customer header (X-customheader-set)
3, Run Manually the new filter

The result was that the filter didnt detect the headers.

HOWEVER.... after reading this I see that Thunderbird needs to have the CUSTOM HEADER already defined in the Message Filter Setup (and therefore begins to record the occurrence) of the header prior to the messages being received. It is because the customer header is being set up AFTER the message arrives that it is not being found.

SO THE SOLUTION IS:

After you set up a new CUSTOMER HEADER (in Message Filter setup) to search for in the Filters List, you need to do a 'Folder Repair' (which rebuilds the MSF file for the folder) on the folders concerned. This then forces Thunderbird to record the existence of the custom header in the MSF file and you will then see that the messages are actioned accordingly to the new filter. Also, any NEW messages received AFTER THE CREATION of your customer header will be actioned on automatically (according to your filter definition)

So, to set up a NEW filter for Custom Headers on existing messages, the process should be:

1, Receive an email that may have custom headers in it. (eg, X-customheader-set)
2, set up a message filter to look for and action on that customer header (X-customheader-set)
3, do FOLDER REBUILD on the folders that you will be running this filter against.
4, (Manually Run the new filter if you need to - depending on your Filter definition)
5, Any NEW messages with X-customheader-set will be actioned accordingly as per your Filter

For me there is now no problem regarding this 'issue'.

Using TB 31.6

Eh?

Surely you can't expect mere mortal humans to do this?

What if they've never read your posting?

No, the solution is for the *program* to prompt the user to perform a Folder rebuild and then do it.

Yes, I saw your comment (Comment 58)

> "If the developers don't want to pre-index all the headers of every email, then I can only suggest that when a user adds a custom header to search with, that this action prompts a request to automatically reindex all the email in the persons account. Alternatively just get the program to re-download the headers of each email as it searches"

This just doesnt seem feasible to me. Imagine an account that has a few hundred thousand emails spread throughout 50 or more folders. And then a user that just added a new custom header with a view of doing a search on the folder (example) "DEALT_2014" is now prompted (or worse, automatically FORCED to wait for) his thunderbird client to cycle through all 50+ folders, rebuilding them and re-downloading those hundreds of thousands emails again. It could take hours (being server, network and machine environment dependant). And after all he just wants to find that ONE or TWO emails he knows exists in 'DEALT_2014' (that contains only 200 emails). It would have been better for him to simply manually choose to Rebuild the folder 'DEALT_2104' - taking a few seconds and getting his search results he is looking for as quick as possible.

Is it really that hard to do 'right-click - properties - Repair Folder'? Im sure that anyone that has the knowledge, or is learning the knowledge, of creating 'custom header-based filters' is also very well adapted to learning the extra part of the procedure (namely "after creating your custom header and the filter, rebuild the desired folder you choose to run the filter on").

(In reply to jimimaseye from comment #143)

> Is it really that hard to do 'right-click - properties - Repair Folder'? Im
> sure that anyone that has the knowledge, or is learning the knowledge, of
> creating 'custom header-based filters' is also very well adapted to learning
> the extra part of the procedure (namely "after creating your custom header
> and the filter, rebuild the desired folder you choose to run the filter on").

Somewhere during the process of creating custom-header, it would be great to show the message above, i.e.,

 "after creating your custom header
  and the filter, rebuild the desired folder you choose to run the filter on"

then at least, someone like me, won't forget to do this (unless a phone call comes in
and I get distracted :-)

TB needs better messages in certain situations IMHO.

Yes, a fair point. Maybe a switchable toggle eg, "Do not show this again" stored in CONFIG.

Anyone interested in fixing this should vote for bug 543956 (Always download All Headers, even for folders *not* set to offline mode)...

*** Bug 1643904 has been marked as a duplicate of this bug. ***

*** Bug 1770440 has been marked as a duplicate of this bug. ***

*** Bug 1787394 has been marked as a duplicate of this bug. ***

*** Bug 1805909 has been marked as a duplicate of this bug. ***