NNTP gatewaying trashes Message-IDs

Finally! I thought I was the only one who noticed this.
See http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=346413

Why would the mail message-id ever cause a conflict with
the NNTP server? message-ids are supposed to be globally
unique and there is a convention (in RFC 2822?) that ensures
it (put in the FQDN and something unique to the generating
software). So I think 1) is not ugly but correct :)

2) wouldn't completely solve the problem.
I can post one copy of a message to a mailman list and
another directly to another recipient, including myself.
(I actually Cc myself on all mail I send). If mailman
does 2), the extra recipient will still be confused.

Ian

the conflicts occur when a user cross-posts a message to
two lists. the newsrunner won't turn it into a cross-post,
instead it tries to post it separately to both groups, and
the news server rejects the second one for a duplicate
message-id (tested with innd).

of course, you could say this behaviour is a bug too :) i'm
not sure what the best way to group cross-posted messages
is, since they won't be received by the same mailman
instance. (cancelling the first and re-posting to both
groups might work, but i don't know if inn will still
record the message-id and not allow it to be reused.)

Originator: NO

nobrowser: the problem is, that newsserver drop messages in crossposts
when one is a mailinglist and the other is a newsgroup.

i just had an idea. we you use a small sqlite table:

| Original-ID | Translated-ID | Last-Used |

When a message is pulled von news, we replace in the reference header alle
ocurrences of translated-id's with their original id's.
When a message is posted to news, we replace all original-id's with
translated-id's.
A new message still generates a new message id, the original is inserted
into the table along with it's original id.
On all actions, we update last-used.

The daily cron job, deletes all last-used older than x days (60 for
example) to keep the table small. This way, no thread trashing would occure
on both sides, while keeping crossposting possible and overhead isn't too
much i think.

We have patched this issue. (Attached)

Newsrunner.py:
Do not replace Message-ID, just add X-Mailman-ID
Skip message if X-Mailman-ID is set
That skips the rewriting of Headers and keeps threading for Mailinglists as well as Newsgroups intact.

a minor fix for stupid MUAs in gate_news:
del 'Newsgroups' header, because MUAs like Evolution take the first useful occurance of a valid Header line to reply. For messages coming from NNTP that is the 'Newsgroups' header. But having that header in a Message received via Mailinglist, leads to a reply towards the newsgroup, even if there a valid To: and/or Reply-To: headers set. Replying to a Newsgroup from a Mailaccount breaks usabilty (user might not even have a nntp account).

hth

Hi!

Best way to handle the crossposting issue would be the following:

Instead of speaking directly with the news server the newsrunner should write the outgoing messages into a queue. This queue is handled by another script that does the queue to news server stuff (eg gate_news?)

When another newsrunner (another mailinglist on the same host) is posting a message it should take a look if a message with the same message-id is already there. (maybe the queue name could be a hash of the message-id). Then it could modify the message with adding a second newsgroup name.

This should handle most problems.

Here is my proposal for what to do:

1. Find all known mailing lists in the to/cc headers, and simultaneously crosspost to those newsgroups
2. Only rewrite a message ID if the news server has one on the server that does not represent this message.

This means that all of the problems with message IDs should be limited to people who put the mailing lists in BCCs, or people who are sending to mailing lists on multiple servers that inject into NNTP in different places.

I do not yet have a patch for either yet.

I solved this problem in another way. Someone wrote for me the SynFU-script: https://github.com/Shirk/SynFU

It is a very good script for synchronizing mails between mailing lists and a newsserver. I'm using it on a system with more than 200 mailing lists and up to 1000 postings a day.

BTW, thus should be pretty easy now in current mm3 bzr trunk.

Is this related to https://bugs.launchpad.net/mailman/+bug/496233 and https://bugs.launchpad.net/mailman/+bug/557955 ?

I have marked Bug #496233 as a duplicate of this bug.

Bug #557955 is related to this but is the converse. I.e., instead of not munging message IDs from Usnet, it wants to mung all message IDs.

This bug has been moved to the new gitlab repo here: https://gitlab.com/mailman/mailman/issues/24