After-the-fact Filters on custom header won't match for IMAP messages

WFM - 2003013108
Reporter: do you still see this problem with a recent build?

I can confirm seeing this bug on Mozilla 1.3 and various nightlies leading up to
its release on my Windows XP machine. I filter based on a variety of custom
headers, some of which always work, and some of which always fail.

I do see the problem for the "List-Id", "Reply-To" and "Return-Path" custom
headers. I don't see the problem for the custom headers "X-BeenThere" and
"X-MirroredBy".

All of these custom header filters have worked for me on older builds.

Also, for what it's worth, I think that bugs #201244 and #184490 are likely to
be duplicates of each other. #20128 may be as well, but it's not clear if the
filter was a custom header or not.

For a guess, is mozilla possibly ignoring custom headers that don't have "X-" as
a prefix? I know that's a fairly common convention for a lot of custom headers
that I see in free mailing lists. Some of the proprietary, or less mainstream
free list servers I see don't use the "X-" prefix, and those are the ones I'm
having problems with.

Frank Widmaier: I am not seeing this problem in 1.3 Final or 1.4RC2; I created a
filter for 'User-Agent contains 1.3' and it worked as expected, for both newly
arrived mail and for running the filter manually (on the Sent folder).

Is this problem still an issue for you? Have you tried upgrading to 1.3 Final
or a 1.4 build?

Did you follow up on Bug 201244, as noted in comment 2? (Ignore the reference
to '20128', that is apparently a typo.) I'm curious if any of the details in
that report jibe with your experience.

=>WFM, no response from reporter.

I also have this issue - I have tried filtering on List-ID to separate mails
based on which mailing list they were sent to.
A view filter utilising this matches no entries, even though there are MANY
(most?) mails in the folder containing the header header
List-ID: <email address hidden>

I had already voted on this issue, but didn't want to spam. I've just retested
with the latest nightly trunk build (2003071704 on Windows 2000), with no change
in behaviour.

I am quite willing to enable debugging (NSPR logging), though a brief look
through the source suggested that there were no useful log points for this problem.

The mail source is IMAP, if that makes a difference...

Considering comment 2, I tested with mail filters (which I don't usually use),
and an attempt to label mails according to the same criteria. This appeared to
succeed (according to the filter logging), but no mails were actually labelled.
I restarted mozilla, and all my view customisations had disappeared. Attempts to
add any new views failed with

Error: uncaught exception: [Exception... "Component returned failure code:
0x80004005 (NS_ERROR_FAILURE) [nsIMsgMailViewList.addMailView]" nsresult:
"0x80004005 (NS_ERROR_FAILURE)" location: "JS frame ::
chrome://messenger/content/mailViewSetup.js :: onOK :: line 89" data: no]

Sorry if I'm mixing up problems here, but I'm adding these items in case they
are relevant.

Reopening

Gary Coady, thanks for your comment -- IMAP does have an issue with customized
headers. See bug 199689, which suggests removing 'custom' and 'body' entirely
from MailViews, simply because they don't work (reliably) for IMAP, at least for
messages in IMAP folders on the server.

*** Bug 233165 has been marked as a duplicate of this bug. ***

Confirming.

xref bug 177294.

See bug 205501 comment 4.

*** Bug 264375 has been marked as a duplicate of this bug. ***

*** Bug 281266 has been marked as a duplicate of this bug. ***

*** Bug 295050 has been marked as a duplicate of this bug. ***

Just a note; I see this behavior with Mozilla 1.7.10, for flags like
X-Spam-Status, X-Spam-Flag, etc. I'm also confused by the claim that
you cannot search for those headers with IMAP, which certainly has the
 HEADER <field-name> <string>
search predicate, at least according to the RFC...

should custom field "sender" work?

Doesn't seem to WFM - TB version 1.6a1 (20060119). This behavior is seriously irritating (and time-wasting) to say the least.

*** Bug 325723 has been marked as a duplicate of this bug. ***

*** Bug 368180 has been marked as a duplicate of this bug. ***

Are we talking about filters or views? I know views don't work, but I have several filters on custom headers that work just fine with my IMAP account.

With current builds, I'm seeing this as a problem with after-the-fact filters ("Run Now") but not for arriving messages. I'm not sure if "after the fact" is what the original reporter meant when he said "I activated the filter." I don't remember whether I had tested incoming mail when I wrote comment 7.

It's also a problem for MailViews (bug 205501), but not for Search Messages.

Something has changed with this from 1.5 to now. In 1.5 a filter on X-Bugzilla-Status contains new. Worked fine in 1.5, doesn't current branch builds. (As per previous dupe.)

*** Bug 369460 has been marked as a duplicate of this bug. ***

Testing further, I see that with TB 1.5.0.9, using a filter on a custom header for an IMAP account works correctly, both for arriving mail and running the filters on the Inbox.

With 2b2, the filters run as expected on arriving mail, but not after-the-fact.

Magnus, is that what you see (per comment 20) or are you seeing a problem with arriving mail?

Yes, that's the same thing I saw - arriving mail seems ok.

*** Bug 372828 has been marked as a duplicate of this bug. ***

*** Bug 374258 has been marked as a duplicate of this bug. ***

*** Bug 377505 has been marked as a duplicate of this bug. ***

*** Bug 377926 has been marked as a duplicate of this bug. ***

can this be the reason why "Trust junk mail header set by SpamAssassin" doesn't work? same story: it used to work on 1.5.0.x, stopped working after upgrade.

*** Bug 381051 has been marked as a duplicate of this bug. ***

Any chance we could fix this asap, the dupes are really filling up on it, not to mention suspected dupes.

The number of dup reports on this would seem to indicate that it is something that should be addressed.

*** Bug 382684 has been marked as a duplicate of this bug. ***

I also have a problem with this exact same behaviour. It is a problem because I want to sort mail already identified as spam that has been used to learn bayes tokens from spam that hasn't. I run my filters on the spam folder where spamassassin puts it. Worked in 1.5, I am now using 2.0.0.0, Linux.

As noted in BZ 205501, it appears that all we need to do to fix this is to make the header caching code cache user-added custom headers (no need to cache all of them) when the messages contain those headers. If someone who knows this code well will point me in the right direction, I will write this patch myself.

These type of filters worked on existing message on version 1.5.x.x but now under version 2.0 they don't work. I wish that Thunderbird had more filter items that were standard instead of custom. -- esp. List-ID as all list providers use this one

Linkify bug pointed by Comment #34. Bug 205501.

*** Bug 387417 has been marked as a duplicate of this bug. ***

I just discovered this because "Sender" wouldn't work for me, TB3/Ubuntu; mail was already in a different folder. To force the filter to run, I marked this mail as unread, then dragged it to Inbox - and seemingly, it got filtered correctly (although that rule looks either in Sender or in Reply-To, so I cannot tell what piece actually triggered).. The problem here, also (seemingly) was that the same message according to another filter I have for "To:" should have filtered in to the original folder..

I guess one thing missing from the Filter Log, is actually a note about which header was it that matched a particular rule; currently it just says:

> Applied filter "My filter ONE" to message from Some Person
> <email address hidden> - My Subject at 2011-04-12 15:34:53 moved
> message id = <email address hidden> to
> imap://myaccount%<email address hidden>/INBOX/folderONE
>
> Applied filter "launchpad-bugs" to message from Some Person
> <email address hidden> - My Subject at 2011-04-12 15:34:53 moved
> message id = <email address hidden> to
> imap://myaccount%<email address hidden>/INBOX/folderTWO
>

... and there isn't any information which of the three rules I have in my "launchpad-bugs" rule actually triggered its run... And so, in the end, turns out it works for me (so I'm not really reporting a bug) - but it was just a bit difficult, to determine how to test the rule and see if it works.

I used to get this problem a lot but not recently.
Do you still see it in the latest 8.0?

*** Bug 564189 has been marked as a duplicate of this bug. ***

Still very much broken on 10.02 in IMAP as it has been for many years.

I created a custom header rule:

version="9"
logging="yes"
name="TEST"
enabled="yes"
type="17"
action="Copy to folder"
actionValue="mailbox://nobody@Local%20Folders/TEST"
condition="OR (\"mime-version\",contains,1.0)"

This should have copied at least one email!
But none appeared and the Filter log remained totally blank.

Come December this bug will be 10 years old!
Amazing.

Update:

It did actually work when receiving new mail!

But not when manually run.

The bug is there in TB10.0.2. The filters did work for my IMAP account till today, but when I tried to read filter log (which was 850M at the moment) TB hung, I killed it and restarted, and custom header filters stopped working. But Body filters do work.

Here in my 10.0.2 they only work with new incoming IMAP mail. The manually run filter of custom headers simply doesn't work. So the bug is in there.

And for me (on 10.0.2 too) custom header filter is not working on new incoming mail, but works if run manually.

Are you sure Constantin?

For me it's the opposite and this bug report is for the custom filters not working when run manually.

Yes, I am quite sure. I have List-ID filter. Al message that match that filter drop into my INBOX, but running filters from menu (Tools -> Run filters on folder) in INBOX move them to desired destination.
Moreover, everything was working fine till I tried to see Filter Log, which grew up to 850M to that moment. TB hung, I killed it, manually deleted filterlog.html, and filters did broke.

Maybe I should open separate bug?

This behavior is still present in the most current versions (note message date) of Thunderbird on Ubuntu and SeaMonkey on Windows 7. It does appear to be an issue when applying after the fact to IMAP and POP mail. Someone commented about a search function for AOL. Every webmail account I have, including AOL, has a search function. Many of these *custom* headers are standard headers. The best workaround, in my opinion, is to simply add common headers to future builds. This can also be an issue because occasionally the mail filters cannot find the folder the message is to be moved (another bug?). In my mind, X should imply custom headers since that is the recommended standard. All other common headers should be considered standard.

A suggest list, including those already listed, should contain Authentication-Results, BCC, CC, Content-Type, Content-Transfer-Encoding, Date, Delivered-To, DKIM-Signature, DomainKey-Signature, From, List-Id, List-Subscribe, List-Unsubscribe, Mailing-List, Message-ID, MIME-Version, Precedence, Received, References, Reply-To, Return-Path, Sender, Subject, To.

I'm sure this is not a comprehensive list. Also, some of these may be meaningless and some starting with X might be added. However, the person working on this bug will make the final decision when editing the code. Please offer further suggestions so that we may come to an agreement. I don't think this work around will add bulk or slow down mail.

*** Bug 531483 has been marked as a duplicate of this bug. ***

The bug is still in TB 15.

TB 16.0.1, still broken.

Any link with Bug 678322 ?

Please fix it.

TB 17.0.3/Mac, still broken. And so annoying...

Well I don't know the internal policies of Mozilla, but this bug has been around since 2002 and yet remains unfixed.

So either we are wrong complaining about this or something is a miss with Mozilla and the way they fix bugs?

Any way we can raise the profile of this sticky bug? :)

Thanks for the ping, Stuart.

I have not thought about this for awhile, but jumping from comment 101 to the referenced bug 363238, that bug is only implemented for Thunderbird 19 and later, so the fix would not be in the current shipping product, but would be available in the current beta.

Yet that fix is only automating the workarounds mentioned in bug 101. That is, when you define a new custom header to use in filters, that custom header will only work for future messages unless you rebuild the folder.

But the current situation (that is, what works in the current Thunderbird beta) is that when you define these custom headers, then they will work for all messages received after that point. They can work for past messages if you rebuild the folder. Any other functionality is very difficult to implement given the current design of the backend, and is unlikely to get any additional work for the foreseeable future.

(In reply to Kent James (:rkent) from comment #130)
> Any other functionality is very difficult to implement given the current design of the backend, (snip)

Phenomenon itself is pretty simple;
- If filtering upon fetch of new mail download, custom headers are fetched
  because used custom headers are placed in mailnews.customHeaders by
  message filter definition, and Tb's IMAP code includes the custom headers in
  uid xx fech BODY[HEADER.FIELDS (DATE FROM ... custom_headers ...)],
  then filter can process the custom headers.
- If after the fact filtering, and if custom header is not held in offline-store
  file, Tb doesn't request re-fetch of custom header and try to use locally held
  data(.msf, offline-store file), then filter can't obtain custom header data.

So, any of following can be a solution, although some of them have issue of "Repair Folder is needed".
(note:)
(Offline-Use=On here == auto-sync is enabled && selected for offline use )
(Offline-Use=Off here == auto-sync is disabled || not selected for offline use)
a) Force saving custom header data in .msf by setting in mailnews.customDBHeaders
   in addition to current mailnews.customHeaders upon filter definition.
b) If custom header is used, save custom header data in offline-store file upon
   fetch even when Offlne-use=Off folder, in conjunction with change to "fetch
   body[HEADERS]" from "fetch body[HEADER.FIELDS]", and use the header data in
   offlie-store file even when Offline-use=Off folder.
c) If custom header is used by filter, and if the custom header is not held in
   offline-store file, and if it's Work Online mode,
   issue uid xx fech BODY[HEADER.FIELDS (custom_headers)] according to
   mailnews.customHeaders
d) If custom header is requested, and if IMAP Offline-Use=Off folder, and if
   filter execution request is not "when Checking new mail" only,
   ask user to change the folder to Offline-use=On or not,
   and if answer is no, reject the rule with message of considerate apology.
   In addition to it, even when Offline-Use=On folder, if retention setting for
   offline-store file size limitation is used, ask user,
      Can you promise that you will never complaint about bug 184490?
   and, if answer is no, don't accept the rule.
e) When custom header is fetched upon initial fetch by mailnews.customHeaders,
   if all fetched headers is saved in Disk/Memory cache when Offline-use=Off,
   and if Disk/Memory cache is accessed when custom header is missing,
   problem of this bug may be resolved, but I'm not sure.

I perefer d) if short-term or mid-term solution, and I think d) is kinder to user than current bypass of "hiding Body at filter definition panel" and d) is applcable to "Body filter upon IMAP fetch" case too(see bug 806308 for Body only and bug 199689 for customized header and Body, please).

From perspective of implementation workload only, a) is simplest and easiest. But I don't think it's safe, because it's easily increases memory requirement and may cause performace problem. "setting in mailnews.customDBHeaders", which is known as a simpler workaround of bug 402594 than "filter rule of custom header of Received", should be done by user as a...

Adding some words in bug summary for ease of search.

By the way, Wayne, regression over what?
> <email address hidden> 2007-07-23 Keywords regression
If I understand problem correctly, I think this bug is issue since initial of "message filter on custom header".

(In reply to WADA from comment #132)
> By the way, Wayne, regression over what?

do you mean bug#? no idea.
And I did not confirm via testing.

As written in comment 44, I added regression keyword based on prior comments that it worked in version 1.5

WADA: (re your comment 131): I'm pretty sure that your a) and b) are both implemented by TB 19. It is the alternatives that are quite difficult, for example your c) would require filters to handle async searches, which are quite difficult given the current architecture.

(In reply to Kent James (:rkent) from comment #134)
> I'm pretty sure that your a) and b) are both implemented by TB 19.

Very good news for many bugs!
But one bad... No performane impact by (a)?
(Huge Mbox like [Gmail]/All Mail is very popular, and I saw a bug report by user on slowness with 136MB Junk.msf in the past...)

(In reply to Wayne Mery (:wsmwk) from comment #133)
> As written in comment 44, (snip)

Wayne, thanks for pointing the comment. By you, I could reach comment #45 by David with no effort :-)

(In reply to WADA from comment #135)

> Very good news for many bugs!
> But one bad... No performane impact by (a)?
> (Huge Mbox like [Gmail]/All Mail is very popular, and I saw a bug report by
> user on slowness with 136MB Junk.msf in the past...)

The changes that I am thinking of only expand the size of the mbox for custom headers added by the user. Although there is a performance impact in that case, it applies only to a small minority of users who add a particular custom header. By adding the header, they are saying that it is important to them, so adding it to the database is necessary for them to be able to use it effectively.

(In reply to Kent James (:rkent) from comment #137)
I see, I stop worryng about edge cases by edge users. As Fx 19 is already released, I hope Tb 19 will be available soon. Thanks for your great effort for resolving problems.

> I stop worryng about edge cases by edge users

Are those not the users who help nail down the bug per se?

 #c45 stop imap search

Wouldn't it be better to indicate IMAP server deficiencies to the user and disable features for problematic users and not punish everyone else with dimished functionality? [ditto #46]

> There were other reasons to switch it - e.g., filters that check if a sender is in an address book only work locally

How is that a reason?

Local filters should run as local filters and not attempt to run as server filters.

> or we could figure out how to do hybrid searches, partly locally, partly on the server

yes, please

#48
> Since I am the admin of my IMAP server

That could allow for extensions to manipulate server's EXIM rules [for future messages]

> other workaround is to configure your IMAP folders for offline use

That's not helpful in an ideal way.

#39
> [2007] Isn't IMAP trendy enough for the programmers to make it work?

It should be by now as many smart phones default to IMAP -- as android when microsoft ActiceSync is not available.

Else the option of Z-Push or ZCP or whatever it's been forked in to N months from now.

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Ok, I have just done a complete re-test and confirm that it seems to be working for me now. To be clear, my (previously failing) test was:

1, Receive an email that may have custom headers in it. (eg, X-customheader-set)
2, set up a message filter to look for and action on that customer header (X-customheader-set)
3, Run Manually the new filter

The result was that the filter didnt detect the headers.

HOWEVER.... after reading this I see that Thunderbird needs to have the CUSTOM HEADER already defined in the Message Filter Setup (and therefore begins to record the occurrence) of the header prior to the messages being received. It is because the customer header is being set up AFTER the message arrives that it is not being found.

SO THE SOLUTION IS:

After you set up a new CUSTOMER HEADER (in Message Filter setup) to search for in the Filters List, you need to do a 'Folder Repair' (which rebuilds the MSF file for the folder) on the folders concerned. This then forces Thunderbird to record the existence of the custom header in the MSF file and you will then see that the messages are actioned accordingly to the new filter. Also, any NEW messages received AFTER THE CREATION of your customer header will be actioned on automatically (according to your filter definition)

So, to set up a NEW filter for Custom Headers on existing messages, the process should be:

1, Receive an email that may have custom headers in it. (eg, X-customheader-set)
2, set up a message filter to look for and action on that customer header (X-customheader-set)
3, do FOLDER REBUILD on the folders that you will be running this filter against.
4, (Manually Run the new filter if you need to - depending on your Filter definition)
5, Any NEW messages with X-customheader-set will be actioned accordingly as per your Filter

For me there is now no problem regarding this 'issue'.

Using TB 31.6

Eh?

Surely you can't expect mere mortal humans to do this?

What if they've never read your posting?

No, the solution is for the *program* to prompt the user to perform a Folder rebuild and then do it.

Yes, I saw your comment (Comment 58)

> "If the developers don't want to pre-index all the headers of every email, then I can only suggest that when a user adds a custom header to search with, that this action prompts a request to automatically reindex all the email in the persons account. Alternatively just get the program to re-download the headers of each email as it searches"

This just doesnt seem feasible to me. Imagine an account that has a few hundred thousand emails spread throughout 50 or more folders. And then a user that just added a new custom header with a view of doing a search on the folder (example) "DEALT_2014" is now prompted (or worse, automatically FORCED to wait for) his thunderbird client to cycle through all 50+ folders, rebuilding them and re-downloading those hundreds of thousands emails again. It could take hours (being server, network and machine environment dependant). And after all he just wants to find that ONE or TWO emails he knows exists in 'DEALT_2014' (that contains only 200 emails). It would have been better for him to simply manually choose to Rebuild the folder 'DEALT_2104' - taking a few seconds and getting his search results he is looking for as quick as possible.

Is it really that hard to do 'right-click - properties - Repair Folder'? Im sure that anyone that has the knowledge, or is learning the knowledge, of creating 'custom header-based filters' is also very well adapted to learning the extra part of the procedure (namely "after creating your custom header and the filter, rebuild the desired folder you choose to run the filter on").

(In reply to jimimaseye from comment #143)

> Is it really that hard to do 'right-click - properties - Repair Folder'? Im
> sure that anyone that has the knowledge, or is learning the knowledge, of
> creating 'custom header-based filters' is also very well adapted to learning
> the extra part of the procedure (namely "after creating your custom header
> and the filter, rebuild the desired folder you choose to run the filter on").

Somewhere during the process of creating custom-header, it would be great to show the message above, i.e.,

 "after creating your custom header
  and the filter, rebuild the desired folder you choose to run the filter on"

then at least, someone like me, won't forget to do this (unless a phone call comes in
and I get distracted :-)

TB needs better messages in certain situations IMHO.

Yes, a fair point. Maybe a switchable toggle eg, "Do not show this again" stored in CONFIG.

Anyone interested in fixing this should vote for bug 543956 (Always download All Headers, even for folders *not* set to offline mode)...

*** Bug 1643904 has been marked as a duplicate of this bug. ***

*** Bug 1770440 has been marked as a duplicate of this bug. ***

*** Bug 1787394 has been marked as a duplicate of this bug. ***

*** Bug 1805909 has been marked as a duplicate of this bug. ***