ERROR juju.worker runner.go:219 exited "rsyslog": x509: certificate signed by unknown authority

This may be a duplicate or related issue to bug 1387388.

Here is the record of a manual attempt to connect to the machine 0 rsyslogd:

root@juju-machine-0-lxc-8:/var/log/juju# openssl s_client -connect 10.49.4.0:6514 -CAfile ca-cert.pem
CONNECTED(00000003)
depth=0 O = juju, CN = *
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = juju, CN = *
verify error:num=27:certificate not trusted
verify return:1
depth=0 O = juju, CN = *
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/O=juju/CN=*
   i:/O=juju/CN=juju-generated CA for environment "rsyslog"
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDSDCCArOgAwIBAgIBADALBgkqhkiG9w0BAQUwRTENMAsGA1UEChMEanVqdTE0
MDIGA1UEAwwranVqdS1nZW5lcmF0ZWQgQ0EgZm9yIGVudmlyb25tZW50ICJyc3lz
bG9nIjAeFw0xNTAxMjgwNTI0MDBaFw0yNTAyMDQwNTIzNTlaMBsxDTALBgNVBAoT
BGp1anUxCjAIBgNVBAMTASowgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAKpH
52wcKtCLnVY8goqRAvBKierpAUHACcakSG34LysAEnVWb+GcKvMRNKOWs38DBkK4
KEdOzNRSgsaShFKg4omlfHDwUyVXOV0NsNM6/jSUcLyhM/KLcvVFEM9a5QNxU+53
H5wPSJEZklm228jRWKeRMnf2IZkfQJndKrLDw1NTAgMBAAGjggF0MIIBcDAOBgNV
HQ8BAf8EBAMCAKgwEwYDVR0lBAwwCgYIKwYBBQUHAwEwHQYDVR0OBBYEFOX8AXWM
mL1e+9dnPJBGKYHt4HhaMB8GA1UdIwQYMBaAFDJS+w8VL6hM7MZmsEPK7P1MvvHT
MIIBBwYDVR0RBIH/MIH8ghxyYXNhbGhhZ3VlLmxjeTAzLmNhbm9uaXN0YWNrghhn
aWVuYWgubGN5MDMuY2Fub25pc3RhY2uCF3Jpc2hhLmxjeTAzLmNhbm9uaXN0YWNr
ggEqhwQKMQQBhxD+gAAAAAAAAJ6Omf/+/PeYhxD+gAAAAAAAAJ6Omf/+/PeYhxD+
gAAAAAAAAJ6Omf/+/PeYhxD+gAAAAAAAAJ6Omf/+/PeYhxD+gAAAAAAAAJi+3f/+
ZN2QhxD+gAAAAAAAAEjPj//+BuqVhxD+gAAAAAAAAJi+3f/+ZN2QhxD+gAAAAAAA
APTGKf/+6bHAhxD+gAAAAAAAABAuT//+1od3MAsGCSqGSIb3DQEBBQOBgQBZ4nmW
ZGaj7j0rFSUBzz7njweBH7LpPkcfvetfVE0WMbhBKND+dYH83zAAejBe9QWxdlY+
TiHkf0pEXGLR+R9fKipDcNs3vMaCZYimLgqmPq/hS9YzUf7v0gvkLeqBICFXV/RQ
RGrddPFwJG7rKnxX7tbQ93Nxw9S4Yr80OevbsQ==
-----END CERTIFICATE-----
subject=/O=juju/CN=*
issuer=/O=juju/CN=juju-generated CA for environment "rsyslog"
---
Acceptable client certificate CA names
/O=juju/CN=juju-generated CA for environment "rsyslog"
/O=juju/CN=juju-generated CA for environment "rsyslog"
---
SSL handshake has read 1295 bytes and written 547 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA256
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol : TLSv1.2
    Cipher : AES256-SHA256
    Session-ID: D057E1CB11EA37C70F8C9539E95EE675DB2B3F58A5834CDCEEE48F95464B00F5
    Session-ID-ctx:
    Master-Key: E5F761603E97CE6401F74A4EBF232FF5474DE033D4E0528026B5D1790251758545701294A249DE807D8B7ACC71B05678
    Key-Arg : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1423028794
    Timeout : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---
^C
root@juju-machine-0-lxc-8:/var/log/juju#

Here's the openssl dump of ca-cert.pem:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: O=juju, CN=juju-generated CA for environment "rsyslog"
        Validity
            Not Before: Jan 28 05:24:43 2015 GMT
            Not After : Feb 4 05:24:43 2025 GMT
        Subject: O=juju, CN=juju-generated CA for environment "rsyslog"
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (1024 bit)
                Modulus:
                    00:c4:9d:e0:4d:cb:66:f8:cf:b3:a4:7c:d9:51:fe:
                    cb:6b:fe:71:cc:16:6e:64:38:c0:dc:ae:57:ed:82:
                    41:eb:95:80:e7:ab:29:b9:bd:82:0b:71:82:1e:62:
                    91:5e:2e:f9:c5:a8:8f:8b:c0:e1:97:28:47:65:e2:
                    69:20:d8:86:17:ca:18:96:63:78:9a:d3:6b:1b:45:
                    90:ab:c4:d1:60:a0:09:82:84:66:05:93:82:e2:9d:
                    36:96:f8:e3:48:b9:59:ce:fd:54:d9:b9:41:bb:39:
                    61:ad:ab:92:01:eb:50:eb:b8:ef:2a:31:7f:7e:d9:
                    bc:6a:1a:31:f1:80:20:0b:cd
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier:
                8B:AE:D9:4E:D2:DA:89:E1:F6:3F:4F:F9:F1:25:48:D2:E6:BF:81:2F
            X509v3 Authority Key Identifier:
                keyid:8B:AE:D9:4E:D2:DA:89:E1:F6:3F:4F:F9:F1:25:48:D2:E6:BF:81:2F

    Signature Algorithm: sha1WithRSAEncryption
         32:78:0e:53:05:7d:60:db:7b:33:63:4c:8c:c5:c5:64:10:fa:
         9b:29:26:24:85:ab:35:b1:01:24:37:48:79:59:5c:a8:36:ea:
         0d:1a:83:71:6e:4e:12:1c:7a:93:7f:72:b9:e0:be:c9:a8:49:
         68:61:1f:a8:fb:b1:e3:83:6f:62:5d:2e:90:91:e7:94:6c:9c:
         f9:65:4a:c4:d8:c7:d0:81:4e:74:14:cd:f5:ae:c9:99:06:2e:
         34:e0:70:03:71:4b:e6:3e:9c:11:11:62:a0:ad:31:5e:4a:13:
         8d:29:f9:ce:a1:d9:4f:d1:e7:cd:46:46:9a:f0:61:de:4c:2e:
         08:9e

Here's the corresponding dump of rsyslog-cert.pem:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: O=juju, CN=juju-generated CA for environment "rsyslog"
        Validity
            Not Before: Jan 28 05:01:18 2015 GMT
            Not After : Feb 4 05:01:17 2025 GMT
        Subject: O=juju, CN=*
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (1024 bit)
                Modulus:
                    00:ae:5f:ea:b1:34:28:32:bc:bd:4e:dd:97:46:5f:
                    4c:c6:57:9d:1a:fb:d3:3f:01:07:22:b5:45:9b:ba:
                    b9:79:d2:5e:0a:cb:5a:c5:b0:53:bf:9c:e7:0d:2a:
                    72:1b:c3:e2:2d:90:b4:7c:b7:6d:5e:df:5d:ac:50:
                    67:bc:63:18:f1:35:7e:42:8b:41:48:28:a5:6d:36:
                    c0:b7:35:ab:a4:02:28:14:e1:a6:be:69:fd:fa:6a:
                    d7:6c:d3:7c:96:55:97:1d:f1:f8:a7:86:a0:d2:61:
                    27:7d:55:86:98:ab:20:97:7...

I neglected to mention earlier that these problems were showing up on juju-core 1.21.1-0ubuntu1~14.04.1~juju1.

I tried again with the same environment on juju-core 1.20.14-0ubuntu1~14.04.1~juju1 (which no longer seems to be present in the repos - why?) and the problem does not occur. Here is a log of a manual connection attempt with the rsyslog configuration deployed by 1.20.14:

root@rasalhague:/var/log/juju# openssl s_client -CAfile ca-cert.pem -connect 10.49.4.0:6514
CONNECTED(00000003)
depth=1 O = juju, CN = juju-generated CA for environment \"rsyslog\"
verify return:1
depth=0 O = juju, CN = *
verify return:1
---
Certificate chain
 0 s:/O=juju/CN=*
   i:/O=juju/CN=juju-generated CA for environment "rsyslog"
---
Server certificate
-----BEGIN CERTIFICATE-----
MIICOTCCAaSgAwIBAgIBADALBgkqhkiG9w0BAQUwRTENMAsGA1UEChMEanVqdTE0
MDIGA1UEAwwranVqdS1nZW5lcmF0ZWQgQ0EgZm9yIGVudmlyb25tZW50ICJyc3lz
bG9nIjAeFw0xNTAyMDQwNjI3MjZaFw0yNTAyMDQwNjMyMjVaMBsxDTALBgNVBAoT
BGp1anUxCjAIBgNVBAMTASowgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBALek
sdX50hAu67zJXx3pgIQgZRS9VxU9+hFJCNBdP4f4AAr66kIiD3mDqFgK/m6Hikjz
Ygl5I1mR4IPvU8fdLDO73N09Z79dAo/6BmJe4GMG8q5ZuzKvGuBKQEwyH5Au4hzp
Hh+CK8uzJ4z9D+wL90nCuBlrNIonWqnCD9uRFBkVAgMBAAGjZzBlMA4GA1UdDwEB
/wQEAwIAqDATBgNVHSUEDDAKBggrBgEFBQcDATAdBgNVHQ4EFgQUYchdTv6AxFgO
Vhl0CCdbD5tcDW8wHwYDVR0jBBgwFoAUi89ZzFuSarbzgsLFeQc7llHZsNcwCwYJ
KoZIhvcNAQEFA4GBAFD69qxiD/ZCdRz8z1+m/up11gsq78GgE0/pLXV+EB7tZfAz
3mehJEzGNYZV365U6//6+Tw07nUY/S8HHliPsVpmzdS8JdzmInlQnbNPHokwNySP
LbQJlmkGKNy5RIG/zuT5aMpGEdd6wkqfMLSIbvY89dF3+YL2rnPe+TBll2Ig
-----END CERTIFICATE-----
subject=/O=juju/CN=*
issuer=/O=juju/CN=juju-generated CA for environment "rsyslog"
---
Acceptable client certificate CA names
/O=juju/CN=juju-generated CA for environment "rsyslog"
/O=juju/CN=juju-generated CA for environment "rsyslog"
---
SSL handshake has read 1120 bytes and written 547 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA256
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol : TLSv1.2
    Cipher : AES256-SHA256
    Session-ID: CA6FCBCD53F11C86AE130265CC4CA7568876B202ABD9EDC984FF85FE9CD8BA3D
    Session-ID-ctx:
    Master-Key: AA3880BEB10E8EE5B7AF58ED3477823079E3523F98756D42366B521D6361E5E111B82165EEED67D21371EF4A643A6888
    Key-Arg : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1423031626
    Timeout : 300 (sec)
    Verify return code: 0 (ok)
---
^C
root@rasalhague:/var/log/juju#

Hey Paul,

Can you give me the bootstrap and deploy commands you've used for this environment? I'm having issues replicating the behavior.

FROM IRC: deploy command is 'juju bootstrap --constraints "tags=bootstrap" --upload-tools --show-log --debug'

Why is this targeted to 1.21, but the milestone is 1.22-beta3? It's also targeted to 1.22 and the milestone there is also 1.22-beta3. I've changed the 1.21 milestone to 1.21.3 not to block the already ready to release 1.21.2.

Despite my best efforts I am unable to reproduce this error.

I'm using MAAS 1.7.0 with any of the Juju branches listed as also being affected.

 I've asked another Juju dev with a mass setup to attempt to reproduce the error.

Would it be possible to get access to the environment that the error is happening on?

I can't reproduce either. (Latest MAAS, juju trunk.)

Ok, good to know it isn't just me, thanks Michael.

On Wed, Feb 11, 2015 at 6:05 AM, Michael Foord <email address hidden>
wrote:

> I can't reproduce either. (Latest MAAS, juju trunk.)
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1417875
>
> Title:
> ERROR juju.worker runner.go:219 exited "rsyslog": x509: certificate
> signed by unknown authority
>
> Status in juju-core:
> Triaged
> Status in juju-core 1.21 series:
> Triaged
> Status in juju-core 1.22 series:
> Triaged
>
> Bug description:
> In a new deploy with MAAS as the provider, juju's rsyslog
> configuration is unworkable, with machine 0's units unable to connect
> to its own rsyslogd, due to certificate validation errors:
>
> Here's an excerpt of tail -F /var/log/juju/unit-*.log:
>
> ==> /var/log/juju/unit-neutron-gateway-0.log <==
> 2015-02-04 05:28:11 INFO juju.worker runner.go:261 start "rsyslog"
> 2015-02-04 05:28:11 DEBUG juju.worker.rsyslog worker.go:86 starting
> rsyslog worker mode 1 for "unit-neutron-gateway-0" ""
> 2015-02-04 05:28:11 DEBUG juju.worker.rsyslog worker.go:183 making
> syslog connection for "juju-unit-neutron-gateway-0" to 10.49.4.0:6514
> 2015-02-04 05:28:11 ERROR juju.worker runner.go:219 exited "rsyslog":
> x509: certificate signed by unknown authority (possibly because of
> "crypto/rsa: verification error" while trying to verify candidate authority
> certificate "juju-generated CA for environment \"rsyslog\"")
> 2015-02-04 05:28:11 INFO juju.worker runner.go:253 restarting "rsyslog"
> in 3s
>
> ==> /var/log/juju/unit-openstack-ha-0.log <==
> 2015-02-04 05:28:11 INFO juju.worker runner.go:261 start "rsyslog"
> 2015-02-04 05:28:11 DEBUG juju.worker.rsyslog worker.go:86 starting
> rsyslog worker mode 1 for "unit-openstack-ha-0" ""
> 2015-02-04 05:28:11 DEBUG juju.worker.rsyslog worker.go:183 making
> syslog connection for "juju-unit-openstack-ha-0" to 10.49.4.0:6514
> 2015-02-04 05:28:11 ERROR juju.worker runner.go:219 exited "rsyslog":
> x509: certificate signed by unknown authority (possibly because of
> "crypto/rsa: verification error" while trying to verify candidate authority
> certificate "juju-generated CA for environment \"rsyslog\"")
> 2015-02-04 05:28:11 INFO juju.worker runner.go:253 restarting "rsyslog"
> in 3s
>
> ==> /var/log/juju/unit-neutron-gateway-0.log <==
> 2015-02-04 05:28:14 INFO juju.worker runner.go:261 start "rsyslog"
> 2015-02-04 05:28:14 DEBUG juju.worker.rsyslog worker.go:86 starting
> rsyslog worker mode 1 for "unit-neutron-gateway-0" ""
> 2015-02-04 05:28:14 DEBUG juju.worker.rsyslog worker.go:183 making
> syslog connection for "juju-unit-neutron-gateway-0" to 10.49.4.0:6514
> 2015-02-04 05:28:14 ERROR juju.worker runner.go:219 exited "rsyslog":
> x509: certificate signed by unknown authority (possibly because of
> "crypto/rsa: verification error" while trying to verify candidate authority
> certificate "juju-generated CA for environment \"rsyslog\"")
> 2015-02-04 05:28:14 INFO juju.worker runner.go:253 restarting "rsyslog"
> in 3s
>
> ==> /var/log/juju/unit-openstack-ha-0.log <==
> 2015-02-04 05:28:14 IN...

Was able to reproduce the issue. The certificates are only a problem when you use ensure-availability. Since each state machine generates its own certificate, the result is the client can only connect to the state server that gave the node its client key.

Solution is to ensure that all bootstrap nodes share the master nodes rsyslog-cert.pem and rsyslog-key.pem files.

After further investigation it looks like my previous statement is incorrect. We call composeTLS during the connect to the remote rsyslog, passing in the ca-cert.pem to generate the client cert for the connection. We do this on the fly.

In replaceRemoteLogger we have a range that loops over all of the StateServerAddresses, it attempts to connect to each state server using this client cert, generated using composeTLS, but each of the state machines have different ca-certs. This is why we see the error repeated for each of the other state machines.

A manual work around:

After running ensure-availability one of the state machines will have a set of certificates that works for connecting.

Using the juju ssh command and the openssl command we can determine which state machine has the "good" certificates:

    juju ssh 0
    openssl s_client -connect 01-current-maas-node:6514 -CAfile /var/log/juju/ca-cert.pem

Continue with the other state servers until you find the one that returns the OK response for the openssl command. Copy the rsyslog-*.pem certs from that state server to the /var/log/juju folder of the other state servers and restart their respective rsyslog services.

There is a fix ready to merge here: https://github.com/juju/juju/pull/1602

It is currently blocked by CI waiting on fixes for lp:1421687 and lp:1421606

Getting something similar to this bug while trying to install openstack-base bundle on clean MAAS environment.

JUJU version 1.24.6-trusty-amd64

status on this units report hook failed

    units:
      nova-cloud-controller/0:
        workload-status:
          current: error
          message: 'hook failed: "cloud-compute-relation-changed" for nova-compute:cloud-compute'
          since: 22 Oct 2015 15:54:39-06:00
        agent-status:
          current: idle
          since: 22 Oct 2015 15:54:39-06:00
          version: 1.24.7
        agent-state: error
        agent-state-info: 'hook failed: "cloud-compute-relation-changed" for nova-compute:cloud-compute'
        agent-version: 1.24.7
        machine: 3/lxc/1
        open-ports:
        - 3333/tcp
        - 8773/tcp
        - 8774/tcp
        - 9696/tcp
        public-address: 172.16.1.144

and logs full of this:

2015-10-22 21:37:34 ERROR juju.worker runner.go:223 exited "rsyslog": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "juju-generated CA for environment \"rsyslog\"")
2015-10-22 21:37:38 ERROR juju.worker runner.go:223 exited "rsyslog": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "juju-generated CA for environment \"rsyslog\"")
2015-10-22 21:37:41 ERROR juju.worker runner.go:223 exited "rsyslog": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "juju-generated CA for environment \"rsyslog\"")
2015-10-22 21:37:45 ERROR juju.worker runner.go:223 exited "rsyslog": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "juju-generated CA for environment \"rsyslog\"")
2015-10-22 21:37:48 ERROR juju.worker runner.go:223 exited "rsyslog": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "juju-generated CA for environment \"rsyslog\"")
2015-10-22 21:37:51 ERROR juju.worker runner.go:223 exited "rsyslog": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "juju-generated CA for environment \"rsyslog\"")
2015-10-22 21:37:56 ERROR juju.worker runner.go:223 exited "rsyslog": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "juju-generated CA for environment \"rsyslog\"")
2015-10-22 21:37:59 ERROR juju.worker runner.go:223 exited "rsyslog": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "juju-generated CA for environment \"rsyslog\"")
2015-10-22 21:38:03 ERROR juju.worker runner.go:223 exited "rsyslog": x509: certificate signed by unknown authority (possibl...

speransky - do you have DEBUG logs for this? There's a new bug, bug #1491688, that's opened for this problem on 1.24.5+ and we didn't get enough data to debug the issue. If you have DEBUG logs, can you add them to bug #1491688?

Can't reproduce on 1.24.7-0ubuntu1~14.04.1~juju1