validation by yahoo/hotmail/dkimpy fails because of whitespace in h= or header folding issues

Bug #939128 reported by Martin Pool
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
dkimpy
Fix Released
Medium
Stuart Gathman

Bug Description

Hello. I am new to DKIM, and might be doing something silly.

I build a message using email.mime, call as_string() on it, construct a DKIM object with the msg, and call sign. I prepend that signature onto my message, and send it. An email to a google apps account gives me "dkim=pass. An email to a yahoo account gives "dkim=permerror (bad sig)". This is using the same code, same from email and domain, etc.. I get the same results using pydkim-3 and with the newer dkimpy-5.1.

While I was still using pydkim-3, I tried different things, and managed to get both yahoo and gmail to give me a pass. I changed the sign function to omit the spaces around the colon in the "h=" list, and removed the call to the fold utility in the sign function so that the signature header was all on one long line. With those changes, yahoo and google were both happy.

Any suggestions?

Thanks,
Rob

Revision history for this message
Martin Pool (mbp) wrote :

Hi, Rob, thanks for reporting this.

In <http://tools.ietf.org/html/rfc4871#section-3.5> describing "h=":

  Folding whitespace (FWS) MAY be included on either side of the colon separator.

It looks like Yahoo doesn't handle that properly. Perhaps it is worth trying to report an issue to them.

I think a patch to pydkim that makes it not put whitespace in the h= field would be fine. You should add a comment explaining the issue with Yahoo. If there is no FWS I think it won't fold; otherwise we could use strict signing.

Changed in dkimpy:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Stuart Gathman (stuart-gathman) wrote :

Possible patch for compatibility with yahoo braindamage. Passes test suite.

--- dkim/__init__.py 2012-02-03 22:10:49 +0000
+++ dkim/__init__.py 2012-02-23 03:06:30 +0000
@@ -238,7 +238,7 @@
             j = i + 1
         pre += header[:j] + b"\r\n "
         header = header[j:]
- return pre + header
+ return (pre + header).replace(' : ',':')

A better way to do it would be to have the fold method break on either ':' or ' '.

Revision history for this message
Rob Nichols (0x2jjt-rob-0tlbog) wrote :

See related question for a little discussion. It's the folding that seems to bother yahoo, and not the space. Here's a patch of what I mean. I've asked the yahoo feedback loop for input, and will report back if/when they reply.

--- dkim/__init__.py 2012-02-22 21:56:26.000000000 -0600
+++ dkim/__init__.py 2012-02-22 21:19:23.000000000 -0600
@@ -441,7 +441,8 @@
     # record what verify should extract
     self.include_headers = tuple(include_headers)

- sig_value = fold(b"; ".join(b"=".join(x) for x in sigfields))
+ sig_value = b"; ".join(b"=".join(x) for x in sigfields)
     dkim_header = (b'DKIM-Signature', b' ' + sig_value)
     h = hashlib.sha256()
     sig = dict(sigfields)

Revision history for this message
Martin Pool (mbp) wrote : Re: [Bug 939128] Re: validation by yahoo fails because of whitespace in h=

It would be good to know if folding is a problem only in strict mode
or only in relaxed mode or both.

Revision history for this message
Rob Nichols (0x2jjt-rob-0tlbog) wrote : Re: validation by yahoo fails because of whitespace in h=

I'll check tomorrow and let you know.

Revision history for this message
Rob Nichols (0x2jjt-rob-0tlbog) wrote :

Previous tests done in the default, "simple" mode for both headers and body.

Here are the results using dkimpy-5.1 as-is, meaning is still has the space in "h=" and the header is folded.

(simple, simple) -> google pass / yahoo dkim=permerror (bad sig)
(relaxed, relaxed) -> google pass / yahoo pass
(relaxed, simple) -> google pass / yahoo pass
(simple, relaxed) -> google pass / yahoo dkim=permerror (bad sig)

No response yet from yahoo. For my own usage, I'm going with stock dkimpy-5.1, and (relaxed, simple) conanicalization.

Revision history for this message
Rob Nichols (0x2jjt-rob-0tlbog) wrote :

Okay, one more piece of information.

MSN/live/hotmail respond like yahoo. It fails with simple, but passes with relaxed. I have no idea why...?

Can you change the title of this bug? Or should I move the additional details to the question and open a new bug?

Revision history for this message
Stuart Gathman (stuart-gathman) wrote :

In simple mode, you are supposed to hash header fields without unfolding them, including the DKIM-Signature header field. BUT, you MUST treat the b= param of the current signature as if it were the empty string. That leaves an ambiguity in the spec. What if the b= value, which is long, is then folded? When replacing the b= value with empty string, should you also remove any FWS inside the value? Adjacent to the value? I can see different interpretations. dkimpy puts b= as the last sig param, and does not fold again after signing (thus the b= line can be longer than 72 chars). However, perhaps your mail relay is folding the headers? And google is actually being more tolerant than it should? Or the difference is folding in the b= param.

summary: - validation by yahoo fails because of whitespace in h=
+ validation by yahoo/hotmail fails because of whitespace in h= or header
+ folding issues
Revision history for this message
Scott Kitterman (kitterman) wrote : Re: validation by yahoo/hotmail fails because of whitespace in h= or header folding issues

I asked one of the original DKIM developers about this (who also maintains their own DKIM library) and he said:

Can you capture a copy of the canonicalized header and body of a message
that goes through that software? Seeing the original message and the raw
data that went through the hash will help.

I seem to remember RFC4871 had an erratum about this that I think we fixed
in RFC6376, but I'll have to go back and double-check later this evening.

So if someone could capture that, it would be helpful.

Revision history for this message
Scott Kitterman (kitterman) wrote :

This erratum for RFC 4871 looks like it covers this exact issue:

http://www.rfc-editor.org/errata_search.php?eid=1596

Revision history for this message
Stuart Gathman (stuart-gathman) wrote :

Yep. That certainly sounds like this. And you can work around it by ensuring there is no whitespace or folding before or after the b= value. And dkimpy already does this by ensuring the b= is last, and folding before computing sig and appending value. I suspect that the header is getting refolded later on by some other software (which is calling dkimpy) - which could trigger this ambiguity.

Revision history for this message
Rob Nichols (0x2jjt-rob-0tlbog) wrote :

Yes, I can see that this might be related, but I don't think this is quite everything. When I removed the call to fold() in the DKIM.sign(), which made the signature one long line, yahoo accepted the signature. (However, MSN still did not.) What I've decided to do with my current project is use (relaxed, simple) canonicalization. That made google, yahoo, and MSN happy. It avoids any related confusion about this errata.

What about changing the default c14n to (relaxed, simple)? I think it would move closer to "just working" for new users of the module.

Revision history for this message
Scott Kitterman (kitterman) wrote : Re: [Bug 939128] Re: validation by yahoo/hotmail fails because of whitespace in h= or header folding issues

I'd prefer a proper resolution of the issue, but we may have to do that. I'm in communication with the author of the DKIM library used by Yahoo.

Revision history for this message
Scott Kitterman (kitterman) wrote : Re: validation by yahoo/hotmail fails because of whitespace in h= or header folding issues

Would you please run your tests with the different canonicalization methods again sending to <email address hidden> and report your results. It runs the same DKIM library as Yahoo! and will return the body of your messages as an attachment so you can see if it's modified in transit.

Revision history for this message
Rob Nichols (0x2jjt-rob-0tlbog) wrote :

Certainly, happy to. (Looking at my schedule, it will probably be a few days...)

Revision history for this message
Scott Kitterman (kitterman) wrote :

ping about getting signed mails that fail at yahoo/hotmail so I can get them to look into it.

Revision history for this message
Rob Nichols (0x2jjt-rob-0tlbog) wrote : Re: [Bug 939128] Re: validation by yahoo/hotmail fails because of whitespace in =?utf-8?Q?h=3D_?=or header folding issues

Thanks for the reminder, and my apologies for the the delay. I will get to this as soon as I can, hopefully in the next couple of days.

On Thursday, April 5, 2012 at 8:12 PM, Scott Kitterman wrote:

> ping about getting signed mails that fail at yahoo/hotmail so I can get
> them to look into it.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/939128
>
> Title:
> validation by yahoo/hotmail fails because of whitespace in h= or
> header folding issues
>
> Status in Module for DKIM signing and verification in Python:
> Confirmed
>
> Bug description:
> Hello. I am new to DKIM, and might be doing something silly.
>
> I build a message using email.mime, call as_string() on it, construct
> a DKIM object with the msg, and call sign. I prepend that signature
> onto my message, and send it. An email to a google apps account gives
> me "dkim=pass. An email to a yahoo account gives "dkim=permerror (bad
> sig)". This is using the same code, same from email and domain, etc..
> I get the same results using pydkim-3 and with the newer dkimpy-5.1.
>
> While I was still using pydkim-3, I tried different things, and
> managed to get both yahoo and gmail to give me a pass. I changed the
> sign function to omit the spaces around the colon in the "h=" list,
> and removed the call to the fold utility in the sign function so that
> the signature header was all on one long line. With those changes,
> yahoo and google were both happy.
>
> Any suggestions?
>
> Thanks,
> Rob
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/dkimpy/+bug/939128/+subscriptions
>
>

Revision history for this message
Scott Kitterman (kitterman) wrote : Re: validation by yahoo/hotmail fails because of whitespace in h= or header folding issues

I think for Ubuntu precise (12.04) I'm going for the relaxed/simple option as a workaround since we're very close to release.

Revision history for this message
Scott Kitterman (kitterman) wrote :

dkimpy (0.5.1-1ubuntu1) precise; urgency=low

  * Change default header signing canonicalization from 'simple' to
    'relaxed' to avoid interoperability problems like those described in
    LP 939128
    - This is a work-around, not a fix

 -- Scott Kitterman <email address hidden> Wed, 18 Apr 2012 01:00:18 -0400

Revision history for this message
Stuart Gathman (stuart-gathman) wrote :

Changed defaults in revision 86

Revision history for this message
Rob Nichols (0x2jjt-rob-0tlbog) wrote :

Sorry for the delay. I no longer have access to some of the test accounts. I created new accounts, and verified that the results were the same as previously reported. I wrote each messages to a file. Looks like I can only attach one, so I'll go with msn (actually live.com). Can attach the others to additional posts if helpful. This is using dkimpy-5.1, and setting c14n to simple/simple. Results were:

google pass
yahoo dkim=permerror (bad sig)
live X-DKIM-Result: Fail(t)

I sent a copy to <email address hidden>, and will post the response when I get it.

Revision history for this message
Scott Kitterman (kitterman) wrote : Fwd: Re: dkimpy interop issue with Microsoft
  • gmail.txt Edit (1021 bytes, application/octet-stream; name="Attachment 1")
  • live.txt Edit (1016 bytes, application/octet-stream; name="Attachment 2")
  • yahoo.txt Edit (1019 bytes, application/octet-stream; name="Attachment 3")

Additional test data.

Forwarded Message ----------

Subject: Re: dkimpy interop issue with Microsoft
Date: Monday, April 23, 2012, 01:52:42 PM
From: Rob Nichols <email address hidden>
To: Scott Kitterman <email address hidden>

Very sorry for the slow response time. Here are test messages as sent to
yahoo, msn an gmail. Gmail passed, both yahoo and msn failed. Sign generated
with 5.1 using c14n simple/simple. Guess you can see simple/simple in the
header...

On Monday, March 12, 2012 at 5:10 AM, Rob Nichols wrote:

> Great! It will be a few more days before I'm back and can generate those
emails, but I'll get to it as soon as I can.
>
> On Wednesday, March 7, 2012 at 2:02 PM, Scott Kitterman wrote:
>
> > I'm writing you directly because I've got a contact at Microsoft that will
> > work with us on solving your issue. He's asked for signed messages that
fail.
> > If you can generate them and then attach them in an email to me I can pass
> > them on and they'll have an engineer look into it.
> >
> > Scott K
>

-----------------------------------------

Revision history for this message
Stuart Gathman (stuart-gathman) wrote : Re: validation by yahoo/hotmail fails because of whitespace in h= or header folding issues

Aha! Notice that for all 3 messages (attached to comment #22) the DKIM-Signature was folded on the email services, but *not* folded in the original as sent out (attached to comment #21). So as I suspected in comment #11, something else is refolding the DKIM-Signature header (probably an MTA, probably Rob Nichols MTA). According to the spec, this should not affect the signature, the DKIM-Signature being verified is treated specially, and folding whitespace is specifically allowed.

For what it's worth, dkimpy also fails to validate the message when b= is folded. :-( . I suggest we

1) double check our interpretation of the spec.

2) fix dkimpy to accept folding whitespace within the b= parm of the DKIM-Signature being validated.

3) *Then* loudly tell M$ and Yahoo they are doing it wrong. :-)

Revision history for this message
Stuart Gathman (stuart-gathman) wrote :

From the RFC:

   b= The signature data (base64; REQUIRED). Whitespace is ignored in
       this value and MUST be ignored when reassembling the original
       signature. In particular, the signing process can safely insert
       FWS in this value in arbitrary places to conform to line-length
       limits. See Signer Actions (Section 5) for how the signature is
       computed.

gmail gets it right. dkimpy, live.com, yahoo get it wrong.

Revision history for this message
Rob Nichols (0x2jjt-rob-0tlbog) wrote :

The file attached to #21 and #22 should be identical. I attached #21, and Scott attached #22 from files I emailed him. The files on #21 and #22 were captured before being sent. Neither is post mailing. I sent these with twisted, which I think does *not* fold lines... but I could be wrong.

I looked at my live.com test address. It has a "view message source" option, that opens a new window and shows the message with minimal formatting. I've never much trusted such pages to show the "real" source... Anyway, when I compared the DKIM header in what I sent to what live.com showed, they match. They have the same folding. I hesitate to draw too much from that, but it would seem to mean that no additional folding happened between signing and delivery.

Revision history for this message
Stuart Gathman (stuart-gathman) wrote :

Added test case with revision 87.

summary: - validation by yahoo/hotmail fails because of whitespace in h= or header
- folding issues
+ validation by yahoo/hotmail/dkimpy fails because of whitespace in h= or
+ header folding issues
Changed in dkimpy:
assignee: nobody → Stuart Gathman (stuart-gathman)
milestone: none → 0.5.2
Revision history for this message
Stuart Gathman (stuart-gathman) wrote :

Fix for dkim.verify commited with revision 88. This does not, of course, fix yahoo and live. So the default will still be relaxed header c14n.

Changed in dkimpy:
status: Confirmed → Fix Committed
Revision history for this message
Rob Nichols (0x2jjt-rob-0tlbog) wrote :

Just for completeness, thought I would attach what live.com returns via "view message source." This is corresponds to the file attached to #21 above. I realize this issue is closed, but thought having everything together might be useful to msn/yahoo...?

Thanks all!

Revision history for this message
Scott Kitterman (kitterman) wrote : Re: [Bug 939128] Re: validation by yahoo/hotmail/dkimpy fails because of whitespace in h= or header folding issues

I just cherrypicked this for the Ubuntu release this Thursday, so it should be
good there.

Changed in dkimpy:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.