KMail duplicates emails

Version: 4.7 (using Devel)
OS: Linux

I have a Jira account. I have set up 2 filters: one is used to move messages for JIRA folder and mark it as read. Filtered by sender (my own activieties).
The second one is used to move all other JIRA maessages to the same folder, but do not mark them as read.

As a result, I have a lot of duplicated messages in my JIRA folder. Some messages (during 2-3 weeks) became duplicated to 10 times.

Reproducible: Sometimes

Steps to Reproduce:
1. create some filters, as I mentioned.
2. try to get the mail.

Actual Results:
New messages arrive in the INBOX folder (that are already placed to the JIRA folder). After this, KMail starts to filter them and send the duplicates to JIRA folder again.

Expected Results:
Messages should be placed to the folder. No duplicating.

As I remember, one of e-mail clients, used by mine, has a function 'remove duplicates'. This should be very useful in this case.

Could be the same bug: https://bugs.launchpad.net/ubuntu/+source/kdepim/+bug/883459

*** Bug 286053 has been marked as a duplicate of this bug. ***

I have the same problem, nearly all messages in my inbox are duplicated. I use only IMAP so don't know if this affects pop3 accounts. I do have filtering enabled.

Running Gentoo stable, just pulled in kmail2. my pop account is showing multiple copies of messages that have been moved to another folder by a filter rule. Also, when I click on the Inbox messages that had been moved by filter suddenly appear in the inbox, then are moved back into the the folder specified by the filter rule (and thus the duplicate).
Also checking email will cause the filtered messages to duplicate.
This is not happening with my IMAP accounts (though none of my filters affect messages that typically come in through the accounts I access via IMAP).

Happy hunting.

I exported all my filters, and deleted them from KMail2. KMail2 is now running without filters. If I move an email from the inbox to another folder (ie, the "destination" folder), the email will appear again in the inbox (and remain in the destination folder as well). This occurs both when the destination folder is a sub-folder of the inbox, and when the destination folder is directly under the Local Folders.

I experience this problem in 4.7.90 (4.8 beta2).

If a mail just downloaded is moved by a filter to a different folder then there is a good chance that it will re-appear again in the original folder at the next update

It does not happen with all the accounts or all the filtered messages. However if a message is affected then it's 100% reproduceable: it will be duplicated at every folder update (forever?).

Same here... with 4.7.4 on Fedora16
Most mails I try to move from my Inbox will appear again as duplicate.
This happens also, when moved by filter rule.
Pressing <Del> to move them into trash surprisingly seems to work.

Same problem for me. I am using kmail 4.7.4-r1 from the Gentoo Stable amd64 repository. I have to POP3 accounts and some filters that examine the mail header (mostly the "from" attribute) and move the mails into folders and marks them as read. Sometimes new messages are duplicated. This also affects mails that are passed through the filter but not modified, because the filter condition does not apply.

The mails are not duplicated immediately but reemerge after a folder update, especially after starting kmail. Suddenly mails that are already some days old and have already been processed by the filter show up again.

I can observe the same behaviour as in comment #6. Not every mail that is processed by some filter is affected, but if some mail is affected once, the mail reappears over and over again. I have the feeling that a mail is more likely to be affected if the filter has to process a lot of new mails at once.

Hej all,

can you please check whether akonadi reports some errors regarding the target folders you try to move the mails to, about a missing remote id?
You can also use akonadiconsole and check the folder properties on the 'internal' tab to find out the remote id.

Found this while trying to reproduce and understand akonadiconsole logs:

AgentBase(akonadi_imap_resource_0): Select failed, server replied: A000068 NO Mailbox doesn't exist: &APw-lleimer

The local Trash folder is called "Lokale Ordner/Mülleimer". This is where the spam check wants to move spam. Putting it on the IMAP Junk folder as Thunderbird does is not possible/allowed.

(In reply to comment #9)
> Hej all,
>
> can you please check whether akonadi reports some errors regarding the target
> folders you try to move the mails to, about a missing remote id?
> You can also use akonadiconsole and check the folder properties on the
> 'internal' tab to find out the remote id.

Tobias,

Sorry this will sound lame but I'm not sure where to look for the akonadi error log. I did try this:

$ grep -R remote ~/.local/share/akonadi/
grep: /home/geoff/.local/share/akonadi/socket-dhcppc2: No such file or directory
grep: /home/geoff/.local/share/akonadi/socket-dhcppc1: No such file or directory
Binary file /home/geoff/.local/share/akonadi/db_data/ibdata1 matches
Binary file /home/geoff/.local/share/akonadi/db_data/akonadi/pimitemtable.frm matches
Binary file /home/geoff/.local/share/akonadi/db_data/akonadi/collectiontable.frm matches
Binary file /home/geoff/.local/share/akonadi/db_data/ib_logfile0 matches
grep: /home/geoff/.local/share/akonadi/akonadiserver.socket: No such device or address
grep: /home/geoff/.local/share/akonadi/socket-scotty/akonadiserver.socket: No such device or address
grep: /home/geoff/.local/share/akonadi/socket-scotty/mysql.socket: No such device or address

As the only matches are in binary files I didn't take this any further.

Using akonadiconsole there is nothing at all displayed in the browser->internals tab. I do notice that non of the remote IMAP folders are displayed in the panel on the left hand side. I see only:

Personal Contacts
Local Folders
    |
     - Outbox
Search

All I can see when moving a Mail is:

==================================
NotificationManager::notify ( Item (129274, 1329071216.R951.neptun:2,S) from collection 467 movedto collection 535 )
==================================

When rescanning the derectory, I just moved it out with F5:
========================================
NotificationManager::notify ( Item (129275, 1329071216.R951.neptun:2,S) in collection 467 added )
NotificationManager::notify ( Item (129275, 1329071216.R951.neptun:2,S) in collection 467 modified parts (PLD:RFC822) )
=======================================

There is a lot of debug traffic in the upper window, but (as I can see) nothin g suspecious.

Tobias, please give us more information on what exactly we should look at and post here.

rightclick on (some) of my target folders in akonadiconsole shows that they all have a "remote identifier" set.

I note though, that the situation has improved, I don't get as many duplicated emails anymore as I used to before. I'll see whehter I can reproduce it at all, maybe this is actually fixed in recent git?

*** Bug 282296 has been marked as a duplicate of this bug. ***

(In reply to comment #9)
> Hej all,
>
> can you please check whether akonadi reports some errors regarding the target
> folders you try to move the mails to, about a missing remote id?
> You can also use akonadiconsole and check the folder properties on the
> 'internal' tab to find out the remote id.

Tobias,

I have checked a number of times now in akonadiconsole, the debugger does not show any error messages when the messages are duplicated. Please note however that the duplicated messages are the ones that are /not/ moved because they do not meet any filter criteria. The messages that are moved by the filters are not duplicated.

Opposed to comment #16 the mails that are moved are duplicated with me.

Even mails that are not moved by a filter but that are move manually (drag and drop) are duplicated. Mails that are "moved" to the bin folder by hitting "DEL" are duplicated, too. In all three cases the mail re-appears in the original folder after some period of time. The problem can be avoid by copying the mail to the destination folder and then purge the original mail from the source folder by hitting "Shift+DEL". This work-around also applies to the filter. If I create two filter actions - copy mail first, then purge the original - everything is fine.

And in all three cases there is no error report in the akonadiconsole.

reopening, as I also just noticed this again. The target folder has a proper remote identifier set, yet I saw that some mails where duplicated by the filter manager...

Tobias, any other ideas what I could look into?

This nasty bugs affect 4.8.1 too.

I believe the problem I have in 4.8.4 is the same problem: emails passing through filters are sometimes being duplicated.

I have a cron job on my servers which send me an "hourly test email" message which I receive on my desktop PC; these have been running for about two years without trouble. I have upgraded my desktop from Ubuntu 10.04 and kmail to Xubuntu 64-bit 11.10 and kmail v4.7.3 on my desktop and I am now seeing some of these hourly emails as identical duplicates (same ids in the headers, etc.) in their folders in kmail2.

For example, my PC was unattended for 11 hours last night and during that time 3 of the hourly emails appear as duplicates in their folder.

The emails are fetched by kmail2 using POP3 and filed into mbox folders by filters. When using kmail I had it set up to remove emails from servers; I set kmail2 to leave it on the servers (just in case I reverted back). On the server I can see there are no duplicates so they were not sent as duplicates.

The rule checks the subject of the email contains "Hourly Test Email" and the "Envelope-to" equals the email address I specified for this sending server. it then moves the email to the correct folder and processing rules is then stopped.

These email checking filters are applied before "Bogofilter check", "Spam Handling" "Classify as Spam" and "Classify as NOT spam" filters, so they are all probably innocent.

The email was downloaded and processed by "interval email checking".

To reproduce it:
1. Set up a filter to receiving incoming email and automatically file it in a folder.
2. Send lots of emails to it.
3. Check the folder.

I just did this by manually sending 18 emails; one of them in the middle was duplicated.

I set up a script to send an email every 5 seconds and sent 77 emails. 79 arrived; two had been duplicated.

The package I have installed is: kmail 4:4.7.4+git111222-0ubuntu0.1
"About KMail" says "KMail Version 4.7.3, Using KDE Development Platform 4.7.4 (4.7.4)"

Just for information, the bug is still seen in 4.8.2.

Affects heavily KDE/KMail 4.8.4 too...
The problem is definitely in moving messages by an applied filter. Manual filtering doesn't have it.
The affected messages are marked for deletion on the remote server (IMAP in my case), deleted from the KMail folder and then reappearing back in the same Kmail folder and in the remote folder too. So at some point in time, the remote folder has the marked for deletion messages + one fresh new and duplicate of all marked for deletion. The strange thing is that some of the duplicates have only headers and not the full messages.
So the problem seems to be in the Move action function of the filtering rules.

Git commit d64d61b470c56b92294f6adee6d74305a217628d by Andras Mantia.
Committed on 04/07/2012 at 17:35.
Pushed by amantia into branch 'master'.

A biggish refactoring of the filter functionality:

1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only).
This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list)
Might fix 293768.

2) Download only what is really needed for the filter and add safety checks that
the mail is not modified if the requested part is missing. So far the distinction
was only betwee the body and rest. This should speed up filtering.
Also related to 287752, but should fix also 292283, 288109.

3) Rename qDebug->kDebug + fix some includes

4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates
with empty content), eg. 275233

All bug reporters: in case you still see the bug, reopen the corresponding one.
For the reports that were CC'd, but not closed, please try to reproduce and tell
the result.

So far the bug is fixed on master, but might be backported later to 4.9.
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364

M +7 -11 kmail/kmcommands.cpp
M +10 -7 kmail/kmcommands.h
M +1 -1 kmail/kmmainwidget.cpp
M +1 -5 mailcommon/filter/filteraction.cpp
M +6 -5 mailcommon/filter/filteraction.h
M +6 -0 mailcommon/filter/filteractionaddheader.cpp
M +2 -0 mailcommon/filter/filteractionaddheader.h
M +2 -2 mailcommon/filter/filteractionaddtag.cpp
M +1 -1 mailcommon/filter/filteractionaddtag.h
M +9 -3 mailcommon/filter/filteractionaddtoaddressbook.cpp
M +2 -0 mailcommon/filter/filteractionaddtoaddressbook.h
M +7 -0 mailcommon/filter/filteractionbeep.cpp
M +1 -0 mailcommon/filter/filteractionbeep.h
M +18 -1 mailcommon/filter/filteractioncopy.cpp
M +7 -0 mailcommon/filter/filteractioncopy.h
M +6 -0 mailcommon/filter/filteractiondelete.cpp
M +1 -0 mailcommon/filter/filteractiondelete.h
M +5 -0 mailcommon/filter/filteractionexec.cpp
M +1 -0 mailcommon/filter/filteractionexec.h
M +5 -0 mailcommon/filter/filteractionforward.cpp
M +1 -0 mailcommon/filter/filteractionforward.h
M +6 -0 mailcommon/filter/filteractionmove.cpp
M +2 -0 mailcommon/filter/filteractionmove.h
M +6 -0 mailcommon/filter/filteractionpipethrough.cpp
M +1 -0 mailcommon/filter/filteractionpipethrough.h
M +6 -0 mailcommon/filter/filteractionplaysound.cpp
M +2 -0 mailcommon/filter/filteractionplaysound.h
M +5 -0 mailcommon/filter/filteractionredirect.cpp
M +1 -0 mailcommon/filter/filteractionredirect.h
M +6 -1 mailcommon/filter/filteractionremoveheader.cpp
M +1 -0 mailcommon/filter/filteractionremoveheader.h
M +6 -0 mailcommon/filter/filteractionreplyto.cpp
M +1 -0 mailcommon/filter/filteractionreplyto.h
M +6 -0 mailcommon/filter/filteractionrewriteheader.cpp
M +1 ...

I really hope this does get backported. The duplicate emails is a deal-killer for me and I'd LOVE to dump Thunderbird.

Git commit e3a03f1d704838a9ced4ff41358785392774ae69 by Andras Mantia.
Committed on 04/07/2012 at 17:35.
Pushed by amantia into branch 'KDE/4.9'.

(Backport filter fix from master)

A biggish refactoring of the filter functionality:

1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only).
This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list)
Might fix 293768.

2) Download only what is really needed for the filter and add safety checks that
the mail is not modified if the requested part is missing. So far the distinction
was only betwee the body and rest. This should speed up filtering.
Also related to 287752, but should fix also 292283, 288109.

3) Rename qDebug->kDebug + fix some includes

4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates
with empty content), eg. 275233

All bug reporters: in case you still see the bug, reopen the corresponding one.
For the reports that were CC'd, but not closed, please try to reproduce and tell
the result.

So far the bug is fixed on master, but might be backported later to 4.9.
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364
(cherry picked from commit d64d61b470c56b92294f6adee6d74305a217628d)

FIXED-IN: 4.9

M +7 -11 kmail/kmcommands.cpp
M +10 -7 kmail/kmcommands.h
M +1 -1 kmail/kmmainwidget.cpp
M +1 -5 mailcommon/filter/filteraction.cpp
M +6 -5 mailcommon/filter/filteraction.h
M +6 -0 mailcommon/filter/filteractionaddheader.cpp
M +2 -0 mailcommon/filter/filteractionaddheader.h
M +2 -2 mailcommon/filter/filteractionaddtag.cpp
M +1 -1 mailcommon/filter/filteractionaddtag.h
M +9 -3 mailcommon/filter/filteractionaddtoaddressbook.cpp
M +2 -0 mailcommon/filter/filteractionaddtoaddressbook.h
M +7 -0 mailcommon/filter/filteractionbeep.cpp
M +1 -0 mailcommon/filter/filteractionbeep.h
M +18 -1 mailcommon/filter/filteractioncopy.cpp
M +7 -0 mailcommon/filter/filteractioncopy.h
M +6 -0 mailcommon/filter/filteractiondelete.cpp
M +1 -0 mailcommon/filter/filteractiondelete.h
M +5 -0 mailcommon/filter/filteractionexec.cpp
M +1 -0 mailcommon/filter/filteractionexec.h
M +5 -0 mailcommon/filter/filteractionforward.cpp
M +1 -0 mailcommon/filter/filteractionforward.h
M +6 -0 mailcommon/filter/filteractionmove.cpp
M +2 -0 mailcommon/filter/filteractionmove.h
M +6 -0 mailcommon/filter/filteractionpipethrough.cpp
M +1 -0 mailcommon/filter/filteractionpipethrough.h
M +6 -0 mailcommon/filter/filteractionplaysound.cpp
M +2 -0 mailcommon/filter/filteractionplaysound.h
M +5 -0 mailcommon/filter/filteractionredirect.cpp
M +1 -0 mailcommon/filter/filteractionredirect.h
M +6 -1 mailcommon/filter/filteractionremoveheader.cpp
M +1 -0 mailcommon/filter/filteractionremoveheader.h
M +6 -0 mailcommon/filter/filteractionreplyto.cpp
M +1 ...

Nice job !
/me is eager to try 4.9

Git commit 8846d44b217108351c35f8afd883f72ee88388ec by Andras Mantia.
Committed on 06/07/2012 at 11:12.
Pushed by amantia into branch 'KDE/4.8'.

Backport part of the filter fix: this is the actual fix, the rest were optimizations and safety checks, but it is difficult to backport.
An unwanted side-effect might be that more data is downloaded than before,
based on the type of filters used.

FIXED-IN: 4.8.5
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364

M +0 -1 mailfilteragent/mailfilteragent.cpp

http://commits.kde.org/kdepim/8846d44b217108351c35f8afd883f72ee88388ec

Just hit the same. I noted that the duplicated messages, were messages never fully moved in the first place. They ghosted in the original folder, and the ghosts were copied.

The folders this happens to also does not have working folder properties. Nothing you change in folder properties have ANY effect and are immediately lost

With upgrading to version 5.5.3 on Kmail, the duplication problem got worse in my case.
I have a rather large (~ 10000) collection of messages. It happens after filtering, it can be "fixed" by clearing akonadi cache with akonadiconsole, but it appears again.
Adding a filter on top of the filter list, that executes "/usr/bin/sleep 15" did not help. I could not find out if it happens with the folders that contain many messages, or in all of them (i think the latter).
What did change with the new version (compared to the one there was in kubuntu 14.04) is that now I get an error message when trying to access the folder with the duplicate messages, that kmail cannot fetch the message because of multiple candidates. Previously, i could erase one.

I can confirm this bug with both an IMAP and a POP account. Mails are sometimes (not always, filters don't always work well) transferred from the inbox to the correct folder and a little bit later the mails are also back in the inbox. This can go on forever.
When I delete them from the inbox it is over and they don't appear anymore.

I have also seen this (and my wife on her setup - similar to mine)
and I wonder if some of this is due to the huge number of options and the more complex environment.

I want to be able to read mails both on my computer and on my mobile.
I want the computer to be the primary storage of my mails.
POP/IMAP account - using IMAP for both

Computer
- copied new messages locally using filters into correct folder
- filters that mark a message, resulted in two copies (one shadow - not readable) the updated state could be seen from mobile too...
- had "download for offline processing" enabled (trying with it off now)
   noticed that it
- push on
- recheck 5 minutes
...

I wonder if my settings caused numerous rereads, store on disk with different generations of each mail...

-rw-r--r-- 1 roger users 4589 11 mar 00.20 /home/roger/.local/share/akonadi/file_db_data/99/822899_r0
-rw-r--r-- 1 roger users 4100 6 apr 18.29 /home/roger/.local/share/akonadi/file_db_data/99/827899_r61
-rw-r--r-- 1 roger users 6372 6 apr 18.29 /home/roger/.local/share/akonadi/file_db_data/99/828199_r1360
-rw-r--r-- 1 roger users 4348 6 apr 18.30 /home/roger/.local/share/akonadi/file_db_data/99/831799_r1359
-rw-r--r-- 1 roger users 4126 6 apr 18.30 /home/roger/.local/share/akonadi/file_db_data/99/833599_r61

Is this problem still reproducible in the latest version of KMail? There haven't been any comments on this bug in over a year; I'm wondering if that's because the problem has been fixed or because it has driven most of the user base away. I'd love to return to using KMail but can't as long as this message duplication problem persists. (I was getting dozens of duplicated messages every day, which was too much to put up with.)

(In reply to Tristan Miller from comment #202)
> Is this problem still reproducible in the latest version of KMail? There
> haven't been any comments on this bug in over a year; I'm wondering if
> that's because the problem has been fixed or because it has driven most of
> the user base away. I'd love to return to using KMail but can't as long as
> this message duplication problem persists. (I was getting dozens of
> duplicated messages every day, which was too much to put up with.)

It has been fixed I believe, since it disappeared from my kmail in 2015.

(In reply to Eric Depagne from comment #203)

> It has been fixed I believe, since it disappeared from my kmail in 2015.

And yet there are plenty of comments here from 2015 to 2018 claiming the problem still exists. I would prefer to hear from someone who was experiencing the problem more recently (say, 2018) who can confirm that it is no longer reproducible.

(In reply to Eric Depagne from comment #203)
> (In reply to Tristan Miller from comment #202)
> > Is this problem still reproducible in the latest version of KMail? There
> > haven't been any comments on this bug in over a year; I'm wondering if
> > that's because the problem has been fixed or because it has driven most of
> > the user base away. I'd love to return to using KMail but can't as long as
> > this message duplication problem persists. (I was getting dozens of
> > duplicated messages every day, which was too much to put up with.)
>
> It has been fixed I believe, since it disappeared from my kmail in 2015.

It certainly hasn't. I still get numerous duplicates daily on this Gentoo box.

I think the comments dried up because (a) there was nothing new to add; (b) users grew tired of banging on to no avail.

I did get numerous duplicates when I installed kmail 5.7.x (I think, x=3), that came with Ubuntu 18.04. Then I started using the workaround from comment #118.

(In reply to Árpi from comment #206)
> I did get numerous duplicates when I installed kmail 5.7.x (I think, x=3),
> that came with Ubuntu 18.04. Then I started using the workaround from
> comment #118.

Lucky you; that workaround did nothing for me.

Is there some sort of bug bounty program for KDE? If so I wouldn't mind chipping in €20 or so towards a fix for this issue.

I haven't seen the problem for a while, maybe two years. I use POP3 only, not IMAP. I run Kubuntu; currently I'm on 19.04.

Sometimes kmail crashes, and I wonder if the way I clean up is related to my not seeing duplicate filtered messages. When kmail crashes, I stop akonadi ("akonadictl stop") and look in the maildir structure like this:

cd .local/share/.local-mail.directory
ls */new/*S

The S indicates messages which have been Seen, and according to the Maildir specification the files should not be in a"new" subdirectory; they should have been moved to a "cur" subdirectory. I therefore move them while akonadi is stopped. E.g.,

mv -i wastebin/new/*S wastebin/cur

I then run "akonadictl fsck" and restart akonadi ("akonadictl start").

As I say, it's pure speculation that this is why I don't see duplicate messages now.

Apologies.

akonadictl start
akonadictl fsck

is the correct order.

On my end, I'm currently using Kmail 5.9.3 (Debian buster), and am seeing no duplicates using imap only.
Some very rare times I can see a few duplicates, but they disappear by themselves pretty quicly.

(In reply to Tristan Miller from comment #202)
> [...] because the problem has been fixed or because it has driven most of
> the user base away. I'd love to return to using KMail [...]

I didn't stop using KMail. I stopped using filters.

If I find time to explore alternatives and find something suitable I may switch. Also possible I will use two mail clients in parallel.

Coincidentally, I am well positioned to test alternatives as I moved away from using folders under "Local Folders" to using a local imap server a while ago (because, at that time, KMail stored messages in a database instead of a maildir). This should make it easy to use multiple clients in parallel.

(In reply to Eric Depagne from comment #203)
> (In reply to Tristan Miller from comment #202)
> > Is this problem still reproducible in the latest version of KMail? There
> > haven't been any comments on this bug in over a year; I'm wondering if
> > that's because the problem has been fixed or because it has driven most of
> > the user base away. I'd love to return to using KMail but can't as long as
> > this message duplication problem persists. (I was getting dozens of
> > duplicated messages every day, which was too much to put up with.)
>
> It has been fixed I believe, since it disappeared from my kmail in 2015.

Most definitely not fixed. Still run into it multiple times per day with some messages showing up 5 or more times. Sometimes I just delete the entire folder and filter and just set everything up again. Sometimes seems to do the trick. I have been unable to find any logic behind why some messages get duplicated where others don't (in the same folder).

(In reply to Tristan Miller from comment #207)
> (In reply to Árpi from comment #206)
> > I did get numerous duplicates when I installed kmail 5.7.x (I think, x=3),
> > that came with Ubuntu 18.04. Then I started using the workaround from
> > comment #118.
>
> Lucky you; that workaround did nothing for me.
>
> Is there some sort of bug bounty program for KDE? If so I wouldn't mind
> chipping in €20 or so towards a fix for this issue.

I checked my filters; what I am doing is not exactly as described in comment #118, but instead, I added an extra action, to execute
/bin/sleep 1
to each of my filters. Since then, no duplicates.

I've been using IMAP for some years now, and I don't get duplicates anymore when filtering. In the case my filters ever do filter, which is roughly the case in 80 % of the mails they should work on (otherwise, nothing happens).

After almost eight years and more than 200 comments, I think one should rather wonder if this even can be fixed inside this Akonadi monster I personally wouldn't even dare to look at the code, or if one should reconsider the whole thing.

Only thing I can't understand is that the KMail guys are not annoyed themselves by this ... I mean, I don't think they use Trojitá, do they?! But no harm meant,
remarkably enough I still stick to KMail ;-)

(In reply to Tristan Miller from comment #202)
> Is this problem still reproducible in the latest version of KMail? There
> haven't been any comments on this bug in over a year; I'm wondering if
> that's because the problem has been fixed or because it has driven most of
> the user base away. I'd love to return to using KMail but can't as long as
> this message duplication problem persists. (I was getting dozens of
> duplicated messages every day, which was too much to put up with.)

The way i reduced the duplicates to a point where it hardly happens (maybe once a week) is to stop filtering on incoming messages and only apply the filters manually once all emails are in. I'm using IMAP but I think the POP3 still has a lot of issues.

(In reply to Tristan Miller from comment #202)
> Is this problem still reproducible in the latest version of KMail? There
> haven't been any comments on this bug in over a year; I'm wondering if
> that's because the problem has been fixed or because it has driven most of
> the user base away.

Driven away.

Was a KMail for what seems like forever (~2000?) (I've been a KDE user since it first came out in '96), but after dealing with workarounds and hacks for two+ years I eventually gave in and abandoned KMail back in mid-2017.

Took me multiple weeks to get all of my mail (~12GB) imported into Thunderbird, get my 200+ mail filters set up again, and to remove all of the duplicates and junk that various KMail bugs had done to my mail. Was the most painful thing I'd done in years, and I knew the switch was going to suck bad so I held out as long as I could.

KMail eventually just got unusable, period. I'd have to do workarounds and restart akonadi every single time I picked up my POP mail, which just made KMail effectively useless. After realizing I was using Alpine to read all of my new mail and delete junk _before_ fetching it into KMail, I finally gave in and switched.

LOVED KMail, but THIS bug led me to finally throw in the towel and give it up.

(In reply to graham from comment #216)
> Was a KMail for what seems like forever (~2000?) (I've been a KDE user since
> it first came out in '96), but after dealing with workarounds and hacks for
> two+ years I eventually gave in and abandoned KMail back in mid-2017.

There was a commit in 2017-05-04
https://phabricator.kde.org/R206:43f2cde61f98317eb13d98222a57bc6df323a308
that was about this bug.
And indeed in openSUSE Leap-15 (kdepim-runtime 17.12.3),
there are much fewer duplicates than before.

There is now a related bug about unreachable duplicates
that has already been reported as #388869.

(In reply to Tobias Leupold from comment #214)
> I think one should rather wonder if this even can be fixed
> inside this Akonadi monster I personally wouldn't even dare to look at
> the code, or if one should reconsider the whole thing.

kmail/akonadi handle my > 15GB maildir gracefully,
which is quite a performance. Congratulations !

So it is not perfect, but to me it is an amazing piece of software.
Please consider that it is very hard for maintainers
to fix a bug that is not reproducible on their setup.

With a working link: bug #388869

> Please consider that it is very hard for maintainers
> to fix a bug that is not reproducible on their setup.

True, but there's nothing to stop them from enlisting help from knowledgeable users (at least a few have offered, including me). They won't do it though; I can only speculate as to why not.

Version KMail 5.13.3 (19.12.3)
OS Gentoo Linux

I have been plagued by this bug for a very long time. Today I found out, that for ages there has been an error message in my mysql/mariadb log-file:

[ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set ...

This is the first time I saw this error, as there is no other application than akonadi using mariadb on my pc and akonadi was never complaining. This mariadb-error could be fixed by running mysql_upgrade (google it to find an explanation, this is due to an incomplete upgrade of the mariadb-package)

After fixing the database, kmail has not been showing duplicated mails any longer. Also filters have been working flawlessly.

(In reply to linuxfan from comment #220)
> for ages there has been an error message in my mysql/mariadb log-file:
>
> [ERROR] Incorrect definition of table mysql.event: expected column
> 'sql_mode' at position 14 to have type set ...

Thanks for sharing.
In which log file exactly did you find this error ?

/var/log/mysql/ is empty here, so

pgrep -af mysqld
/usr/sbin/mysqld --defaults-file=/home/ederag/.local/share/akonadi/mysql.conf --datadir=/home/ederag/.local/share/akonadi/db_data/ --socket=/tmp/akonadi-ederag.Yqlbg1/mysql.socket --pid-file=/tmp/akonadi-ederag.Yqlbg1/mysql.pid

grep -1 "error log" /home/ederag/.local/share/akonadi/mysql.conf

# # error log file name, relative to datadir (default:hostname.err)
log_error=mysql.err

grep -i error /home/ederag/.local/share/akonadi/db_data/mysql.err
<nothing>

That was with the "1 s" delay filter active,
just deactivated; we'll see.

(In reply to ederag from comment #221)

> grep -i error /home/ederag/.local/share/akonadi/db_data/mysql.err
> <nothing>

Here, I see:

$ grep -i error .local/share/akonadi/db_data/mysql.err
2020-05-08 13:12:10 0 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
2020-05-08 13:12:10 9 [ERROR] InnoDB: Table `mysql`.`innodb_table_stats` not found.

I don't know how long that's been happening, because I haven't had any duplicates for quite a long time. I remember, though, that when I was getting them, that error log had numerous errors of various types.

(In reply to Peter Humphrey from comment #222)
> (In reply to ederag from comment #221)
>
> > grep -i error /home/ederag/.local/share/akonadi/db_data/mysql.err
> > <nothing>
>
> Here, I see:
>
> $ grep -i error .local/share/akonadi/db_data/mysql.err
> 2020-05-08 13:12:10 0 [ERROR] Can't open and lock privilege tables: Table
> 'mysql.servers' doesn't exist
> 2020-05-08 13:12:10 9 [ERROR] InnoDB: Table `mysql`.`innodb_table_stats` not
> found.
>
> I don't know how long that's been happening, because I haven't had any
> duplicates for quite a long time. I remember, though, that when I was
> getting them, that error log had numerous errors of various types.

I've been using postgresql and i've hardly seen any since but as of today i got duplicates by the bucket load.

opensuse:tumbleweed:20200506
Qt: 5.14.1 KDE Frameworks: 5.69.0 - KDE Plasma: 5.18.4 - kwin 5.18.4
kmail2 5.14.0 (20.04.0) - akonadiserver 5.14.0 (20.04.0) -

the log file is /var/log/mysql/mysqld.err

However, digging deeper, the problem might be related to two instances of mariadb being present.

pgrep -af mysqld yields

2402 /usr/sbin/mysqld --defaults-file=/etc/mysql/my.cnf
2881 /usr/sbin/mysqld --defaults-file=/home/linuxfan/.local/share/akonadi/mysql.conf --datadir=/home/linuxfan/.local/share/akonadi/db_data/ --socket=/run/user/10001/akonadi/mysql.socket --pid-file=/run/user/1000/akonadi/mysql.pid

While /var/log/mysql/mysqld.err is now clear, in

/home/linuxfan/.local/share/akonadi/db_data/mysql.err

I can see two further errors and some warnings:

[ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
 [Warning] Can't open and lock time zone table: Table 'mysql.time_zone_leap_second' doesn't exist trying to live without them
 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1017: Can't find file: './mysql/' (errno: 2 "No such file or directory")
 [Note] Reading of all Master_info entries succeeded
 [Note] Added new Master_info '' to hash table
 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.4.12-MariaDB' socket: '/run/user/1000/akonadi/mysql.socket' port: 0 Source distribution
[ERROR] InnoDB: Table `mysql`.`innodb_table_stats` not found.

what is more important, I believe, is that akonadi seems to start the second mysql instance but some of the data is actually going to my main mysql server instead and its data directory /home/mysql_data/ while other parts are ending up in the akonadi-instance of mysql.

After disabling the stand-alone server, I found that ~/.config/akonadi/akonadiserverrc

still contained the line

Options="UNIX_SOCKET=/var/run/mysql/mysqld.socket"

pointing to the main server socket. Clearing this up, everything seems to end up in the akonadi-version of mysql (with the above warnings/errors, not sure what that means). I am unsure, why some of the data still went to the akonadi-version of mysql, though.

From my point of view, I seem to have had some mess in the config files as I have been dragging/upgrading my /home/ directory with me since 1997 .... (and yes, I have been using KDE ever since the original beta1)

Well, only half a day before a duplicate is back.
(again, from lists.gnu.org)
Same status as in comment #221.
And nothing suspicious in mysql.err (no ERROR).

(In reply to Alexandre Bonneau from comment #210)
> On my end, I'm currently using Kmail 5.9.3 (Debian buster), and am seeing no
> duplicates using imap only.
> Some very rare times I can see a few duplicates, but they disappear by
> themselves pretty quicly.

For info (I posted that info in another related issue already), I found out that I only get duplicates when I have 2 computers running akonadi-server.

For instance it happens when I'm on my main rig, then casually open my laptop. Then mails are duplicated a lot...until I close one of the akonadi server (usually the one from the laptop).

(In reply to Alexandre Bonneau from comment #226)
> For info (I posted that info in another related issue already), I found out
> that I only get duplicates when I have 2 computers running akonadi-server.

Here also, two computers once concurrently ran akonadi
with the same configuration (synchronized with unison).
But duplicates do appear, even when the other akonadi is stopped.
(pgrep akonadi yields nothing on the other computer)
[akonadictl 5.16.1 (20.12.0)]

The problem is still present in kmail for a long time. I have filters for using bogofilter in kmail.
Is there any workaround or something? Or a filter for kmail which removes all duplicates?
And is there any explanation why this is happenning?

(In reply to r.kunschke from comment #228)
> Is there any workaround or something?

Adding this filter helped a lot here:
- match all messages
- Executed Command: /usr/bin/sleep 1,

Created attachment 148406
signature.asc

Am Mittwoch, 27. April 2022, 10:57:26 CEST schrieb ederag:
> https://bugs.kde.org/show_bug.cgi?id=283682
>
> --- Comment #229 from ederag <email address hidden> ---
> (In reply to r.kunschke from comment #228)
> > Is there any workaround or something?
>
> Adding this filter helped a lot here:
> - match all messages
> - Executed Command: /usr/bin/sleep 1,

Depending on how quick the IMAP server is, this had
little to no effect for me when I tried, so YMMV.

(bye)^2

(In reply to r.kunschke from comment #228)
> The problem is still present in kmail for a long time. I have filters for
> using bogofilter in kmail.
> Is there any workaround or something? Or a filter for kmail which removes
> all duplicates?
> And is there any explanation why this is happenning?

There is an option on the "Folder" menu to remove duplicates that works reasonably well.
I used to have this issue but its now pretty much gone since i changed from 1. POP3 to Imap and 2. Mysql to Postgresql for the backend, its very very rare for me to get a duplicate now

Still valid in akonadi 21.12.3 on Kubuntu 22.04.

I think akonadi should automatically handle this problem.

New symptom.

I have a local server that fetches POP3 mail from my ISP and serves it as IMAP4. I've had only rare duplicates since setting this arrangement up a few years ago.

I discovered yesterday that creating a KMail POP3 account led immediately to numerous duplicates, as half-expected, but the key point here is that duplicates kept occurring, even after I'd deleted the POP3 account and rebooted.

This is KMail version 5.20.3 (22.04.3) on Gentoo stable, Plasma version 5.24.6-r1. I tried postgres several years ago but it didn't improve anything, so I'm sticking with mariadb.

Here on a current openSuse Tumbleweed it happens too, akonadi 5.22.1 (22.12.1).
Or at least the old duplicates are still here, not sure whether new ones have appeared this week already.

Comment on attachment 74538
filter log, a duplicated message is fetched

><html>
><body>
><p>------------------------------</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;Subject&quot; &lt;contains&gt; &quot;Dit kreditkort blev brugt pￜ bￜrnepornografi site&quot;</p>
><p>[11:48:19] <font color=#FF0000>0 = </font>&quot;Subject&quot; &lt;contains&gt; &quot;Dit kreditkort blev brugt pï¿œ bï¿œrnepornografi site&quot; (<i>Donación de la Hermana Alice Jahab</i>)</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;List-Id&quot; &lt;contains&gt; &quot;&lt;nepomuk.kde.org&gt;&quot;</p>
><p>[11:48:19] <font color=#FF0000>0 = </font>&quot;List-Id&quot; &lt;contains&gt; &quot;&lt;nepomuk.kde.org&gt;&quot; (<i></i>)</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;Subject&quot; &lt;contains&gt; &quot;Beskyt dit Visa kort ved internethandel&quot;</p>
><p>[11:48:19] <font color=#FF0000>0 = </font>&quot;Subject&quot; &lt;contains&gt; &quot;Beskyt dit Visa kort ved internethandel&quot; (<i>Donación de la Hermana Alice Jahab</i>)</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;From&quot; &lt;contains&gt; &quot;<email address hidden>&quot;</p>
><p>[11:48:19] <font color=#FF0000>0 = </font>&quot;From&quot; &lt;contains&gt; &quot;<email address hidden>&quot; (<i>alice jahab &lt;<email address hidden>&gt;</i>)</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;Subject&quot; &lt;contains&gt; &quot;aktivering af nye sikkerheds&quot;</p>
><p>[11:48:19] <font color=#FF0000>0 = </font>&quot;Subject&quot; &lt;contains&gt; &quot;aktivering af nye sikkerheds&quot; (<i>Donación de la Hermana Alice Jahab</i>)</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;Subject&quot; &lt;contains&gt; &quot;venligst din opdatering&quot;</p>
><p>[11:48:19] <font color=#FF0000>0 = </font>&quot;Subject&quot; &lt;contains&gt; &quot;venligst din opdatering&quot; (<i>Donación de la Hermana Alice Jahab</i>)</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;List-Id&quot; &lt;contains&gt; &quot;&lt;openseamap-maps.lists.sourceforge.net&gt;&quot;</p>
><p>[11:48:19] <font color=#FF0000>0 = </font>&quot;List-Id&quot; &lt;contains&gt; &quot;&lt;openseamap-maps.lists.sourceforge.net&gt;&quot; (<i></i>)</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;List-Id&quot; &lt;contains&gt; &quot;&lt;marble-devel.kde.org&gt;&quot;</p>
><p>[11:48:19] <font color=#FF0000>0 = </font>&quot;List-Id&quot; &lt;contains&gt; &quot;&lt;marble-devel.kde.org&gt;&quot; (<i></i>)</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;List-Id&quot; &lt;contains&gt; &quot;&lt;kde-core-devel.kde.org&gt;&quot;</p>
><p>[11:48:19] <font color=#FF0000>0 = </font>&quot;List-Id&quot; &lt;contains&gt; &quot;&lt;kde-core-devel.kde.org&gt;&quot; (<i></i>)</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;List-Id&quot; &lt;contains&gt; &quot;&lt;digikam-users.kd...