Puppet Agent Master sync issue

Puppet agents on the controller or the compute nodes have the following issue while communicating to the pupet master on the build node.

SYSLOG output

(/File[/var/lib/puppet/lib]) Failed to generate additional resources using 'eval_generate: The certificate retrieved from the master does not match the agent's private key.#012Certificate fingerprint: 04:F2:9F:D8:C9:CA:8B:C9:F7:F9:0E:1F:34:28:58:C4#012To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certficate.#012On the master:#012 puppet cert clean p4-control01.ci.ctocllab.cisco.com#012On the agent:#012 rm -f /var/lib/puppet/ssl/certs/p4-control01.ci.ctocllab.cisco.com.pem#012 puppet agent -t
Dec 10 15:20:30 p4-control01 puppet-agent[1170]: (/File[/var/lib/puppet/lib]) Could not evaluate: SSL_CTX_use_PrivateKey:: key values mismatch Could not retrieve file metadata for puppet://pod4node1.ci.ctocllab.cisco.com/plugins: SSL_CTX_use_PrivateKey:: key values mismatch
Dec 10 15:20:31 p4-control01 puppet-agent[1170]: Could not retrieve catalog from remote server: SSL_CTX_use_PrivateKey:: key values mismatch
Dec 10 15:20:31 p4-control01 puppet-agent[1170]: Using cached catalog
Dec 10 15:20:31 p4-control01 puppet-agent[1170]: Could not retrieve catalog; skipping run
Dec 10 15:20:31 p4-control01 puppet-agent[1170]: Could not send report: SSL_CTX_use_PrivateKey:: key values mismatch
Dec 10 15:25:21 p4-control01 puppet-agent[1734]: Creating a new SSL certificate request for p4-control01.ci.ctocllab.cisco.com
Dec 10 15:25:21 p4-control01 puppet-agent[1734]: Certificate Request fingerprint (md5): 6C:D0:C0:F0:80:6B:1A:9A:88:64:86:23:B6:77:1E:29
Dec 10 15:25:21 p4-control01 puppet-agent[1734]: Caching certificate for p4-control01.ci.ctocllab.cisco.com
Dec 10 15:25:21 p4-control01 puppet-agent[1734]: Retrieving plugin
Dec 10 15:25:21 p4-control01 puppet-agent[1734]: (/File[/var/lib/puppet/lib]) Failed to generate additional resources using 'eval_generate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed. This is often because the time is out of sync on the server or client
Dec 10 15:25:21 p4-control01 puppet-agent[1734]: (/File[/var/lib/puppet/lib]) Could not evaluate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed. This is often because the time is out of sync on the server or client Could not retrieve file metadata for puppet://pod4node1.ci.ctocllab.cisco.com/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed. This is often because the time is out of sync on the server or client
Dec 10 15:25:23 p4-control01 puppet-agent[1734]: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed. This is often because the time is out of sync on the server or client
Dec 10 15:25:23 p4-control01 puppet-agent[1734]: Not using cache on failed catalo

Workaround
On the controller and the compute nodes
stop the puppet agent
rm -f /var/lib/puppet/ssl/
Its not sufficient to just delete the specific certificate(*.pem file ), the entire ssl directory needs to be deleted.
start the puppet agent.

Related bug
http://projects.puppetlabs.com/issues/15290