workaround for 8 bit headers containing invalid UTF-8
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
dkimpy-milter |
Confirmed
|
Medium
|
Unassigned |
Bug Description
Pasting from an email conversation with Scott, so he won't forget this bug:
Scott: I think the correct answer is to use bytes, but the work-around you propose is reasonable and I am pretty sure I know how to implement it. Please file a bug on launchpad asking for the work-around so I don't forget.
Florian and Patrick:
In a nutshell, the problem is that python3 strings are utf-8, while email (technically) is 8 bit anything. Of course in headers not RFC compliant, but those mails are out there and they get delivered usually.
So the problem occurs, if there is an 8 bit header which is NOT valid UTF-8. In Germany for example ISO-85591 "Umlauts". Usually in the To or Subject header. See for example comment by " scandox commented on Jul 25, 2019" in issue https:/
For our milters we uses the fix proposed by "william6502 commented on Jun 27, 2019" which changes the type to bytes in the lib, which of course implies you have to change the python code also to these types, but that was pretty straight forward and worked for us. We could change dkimpy-milter the same way I guess, but then it only works with a patched libmilter and therefore is nothing for general use.
The hotfix alternative, and thats what Patrick suggested I think, would be to catch the exception and just not process the mail, but not return an error. Then invalid mails would not be signed but still transported.
"sdgathman commented on Jun 19, 2020" suggests an other approach, but still seems to be uncertain if to proceed this way. As "our" problem was solved we did not reall look/push deeper there so far.
> I'd like to do some experimentation
Simply create a mail with an non-utf-8 8bit subject, feed it through your milter with swaks or so, and you should get the effect. For example this Subject header (base64 here...
With pymilter 1.0.5 this is much more tractable. Things that were crashing in the milter bindings now don't. With a few changes in the logging, we can get to "don't crash". Figuring out making signing/validation work is more complicated.