Canonical Juju

juju bootstrap with ca-cert-path and ca-private-key-path not using assigned cert/key

Bug #2038974 reported by Jeff Hillman on 2023-10-11

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Canonical Juju	In Progress	High	Thomas Miller	Canonical Juju 3.1.7
3.2	In Progress	High	Thomas Miller	Canonical Juju 3.2.4
3.3	In Progress	High	Thomas Miller	Canonical Juju 3.3.0

Bug Description

juju 3.1.6
Focal 20.04.6

Whenrunning a local bootstrap pointing to use a local set of CA certs/keys with a decrypted key, juju doesn't utilize that CA/key to generate a certificate against the pair.

A juju-ca issuer is still utilized.

juju bootstrap debug output:

```
$ juju bootstrap maas maas --bootstrap-series focal --config ca-cert-path=./controller_ca.pem --config ca-private-key-path=./decrypt.pem --debug
02:09:36 INFO juju.cmd supercommand.go:56 running juju [3.1.6 f6a66aa91eec620f5ac04a19d8c06bef03ae6228 gc go1.20.8]
02:09:36 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/24626/bin/juju", "bootstrap", "maas", "maas", "--bootstrap-series", "focal", "--config", "ca-cert-path=./controller_ca.pem", "--config", "ca-private-key-path=./decrypt.pem", "--debug"}
02:09:36 DEBUG juju.cmd.juju.commands bootstrap.go:1392 authenticating with region "" and credential "maas" ()
02:09:36 DEBUG juju.cmd.juju.commands bootstrap.go:1551 provider attrs: map[]
02:09:36 INFO cmd authkeys.go:113 Adding contents of "/home/ubuntu/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
02:09:36 DEBUG juju.cmd.juju.commands bootstrap.go:1621 preparing controller with config: map[agent-metadata-url: agent-stream:released apt-ftp-proxy: apt-http-proxy: apt-https-proxy: apt-mirror: apt-no-proxy: authorized-keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDTsBz0GmLHKL5fEZsJ87axUqxETiAnb6Pig3lG9WEkoxzop6KzMQ3RUSGWhGmfRbsoK/+ShiHvyILS2LD5yuJsKvc+aqP4h6pZ+HuPtwiACzCHxpKqSptwGkW/MilInmmoyhy3Ei4A4Oqe61ixcFxv3Sb7CIxrYz0XujVg1R4uYOqLKKhEZbHHISi8W0oEA91MUKk3NNJ/UjQtiyWYhsl2Rz08CkFJu9jR6PhsS4qz/H1uuUszDAN8fz+ANwHJVQKca0aOjpD0uuIINcdl9RNH3ZTTikC9YuK3xHkW7ivnl2WPLlgiOEIvuny5X/wtVkNaNR76eOf6L0ODZYnYxwtacnjNQo5DYnhhIbW8W+HePc2NbenGAIEQXz/B9qxxWkDJW4BI/A6HY3ML8wvWL//F8sp4+7fIv3KOlQVKQiCncep8R3tAGj1w9kYFZw7vX9+nM33yosW6ByAxduevITl+EvF8cHqQ6aKw/Zy7IvuHT7YIVNhlGYmN0zRsnsg/6UoVsqbCJfxDGML9TTpAWC8KBvGqOQXXp2kE4T1sX5/LpLJYRu2eQfw/ICFHZh3CtgBsZJ/aVuwKJtXJRLHxU7BIJOEaAMpnR0EdNLS4T8kqqjfI7mAHBpZGABRkk0hMgDKfPFSnTzAuSZ38G9mNREbXCDXlwhivYDahu4C+6uTVNQ== juju-client-key
automatically-retry-hooks:true backup-dir: ca-cert-path:./controller_ca.pem ca-private-key-path:./decrypt.pem charmhub-url:https://api.charmhub.io cloudinit-userdata: container-image-metadata-url: container-image-stream:released container-inherit-properties: container-networking-method: default-base: default-space: development:false disable-network-management:false disable-telemetry:false egress-subnets: enable-os-refresh-update:true enable-os-upgrade:true fan-config: firewall-mode:instance ftp-proxy: http-proxy: https-proxy: ignore-machine-addresses:false image-metadata-url: image-stream:released juju-ftp-proxy: juju-http-proxy: juju-https-proxy: juju-no-proxy:127.0.0.1,localhost,::1 logforward-enabled:false logging-config: logging-output: lxd-snap-channel:5.0/stable max-action-results-age:336h max-action-results-size:5G max-status-history-age:336h max-status-history-size:5G mode: name:controller net-bond-reconfigure-delay:17 no-proxy:127.0.0.1,localhost,::1 num-container-provision-workers:4 num-provision-workers:16 provisioner-harvest-mode:destroyed proxy-ssh:false resource-tags: secret-backend:auto snap-http-proxy: snap-https-proxy: snap-store-assertions: snap-store-proxy: snap-store-proxy-url: ssl-hostname-verification:true test-mode:false transmit-vendor-metrics:true type:maas update-status-hook-interval:5m uuid:3eea9ec8-8c3b-4e3c-8868-97921da77ca1]
02:09:36 DEBUG juju.provider.maas environprovider.go:63 opening model "controller".
02:09:36 WARN juju.environs.config config.go:1908 unknown config field "ca-private-key-path"
02:09:36 WARN juju.environs.config config.go:1908 unknown config field "ca-cert-path"
02:09:36 WARN juju.environs.config config.go:1908 unknown config field "ca-private-key-path"
02:09:36 WARN juju.environs.config config.go:1908 unknown config field "ca-cert-path"
02:09:36 INFO cmd bootstrap.go:939 Creating Juju controller "maas" on maas/default
02:09:36 INFO juju.cmd.juju.commands bootstrap.go:1010 combined bootstrap constraints:
02:09:36 DEBUG juju.environs.bootstrap bootstrap.go:319 model "controller" supports application/machine networks: true
02:09:36 DEBUG juju.environs.bootstrap bootstrap.go:321 network management by juju enabled: true
02:09:36 INFO cmd bootstrap.go:406 Loading image metadata
02:09:36 INFO cmd bootstrap.go:479 Looking for packaged Juju agent version 3.1.6 for amd64
02:09:36 INFO juju.environs.bootstrap tools.go:78 looking for bootstrap agent binaries: version=3.1.6
02:09:36 DEBUG juju.environs.tools tools.go:87 finding agent binaries in stream: "released"
02:09:36 DEBUG juju.environs.tools tools.go:89 reading agent binaries with major.minor version 3.1
02:09:36 DEBUG juju.environs.tools tools.go:98 filtering agent binaries by version: 3.1.6
02:09:36 DEBUG juju.environs.tools tools.go:101 filtering agent binaries by os type: ubuntu
02:09:36 DEBUG juju.environs.tools tools.go:104 filtering agent binaries by architecture: amd64
02:09:36 DEBUG juju.environs.tools urls.go:133 trying datasource "keystone catalog"
02:09:36 DEBUG juju.environs.simplestreams simplestreams.go:417 searching for signed metadata in datasource "default simplestreams"
02:09:36 DEBUG juju.environs.simplestreams simplestreams.go:451 looking for data index using path streams/v1/index2.sjson
02:09:38 DEBUG juju.environs.simplestreams simplestreams.go:747 using default candidate for content id "com.ubuntu.juju:released:agents" are {20210329 mirrors:1.0 content-download streams/v1/cpc-mirrors-agents.sjson []}
02:09:38 DEBUG juju.environs.simplestreams simplestreams.go:463 looking for data index using URL https://streams.canonical.com/juju/tools/streams/v1/index2.sjson
02:09:38 DEBUG juju.environs.simplestreams simplestreams.go:486 read metadata index at "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson"
02:09:38 DEBUG juju.environs.simplestreams simplestreams.go:1018 finding products at path "streams/v1/com.ubuntu.juju-released-agents.sjson"
02:09:38 INFO juju.environs.bootstrap tools.go:80 found 1 packaged agent binaries
02:09:38 INFO cmd bootstrap.go:492 Located Juju agent version 3.1.6-ubuntu-amd64 at https://streams.canonical.com/juju/tools/agent/3.1.6/juju-3.1.6-linux-amd64.tgz
02:09:38 WARN juju.environs.config config.go:1908 unknown config field "ca-private-key-path"
02:09:38 WARN juju.environs.config config.go:1908 unknown config field "ca-cert-path"
02:09:38 WARN juju.environs.config config.go:1908 unknown config field "ca-private-key-path"
02:09:38 WARN juju.environs.config config.go:1908 unknown config field "ca-cert-path"
02:09:38 INFO cmd bootstrap.go:590 Starting new instance for initial controller
02:09:39 INFO cmd bootstrap.go:184 Launching controller instance(s) on maas/default...
02:09:39 DEBUG juju.provider.maas environ.go:685 attempting to acquire node in zone "default"
02:09:39 DEBUG juju.cloudconfig.instancecfg instancecfg.go:945 Setting numa ctl preference to false
02:09:39 DEBUG juju.provider.maas environ.go:769 maas user data; 3720 bytes
02:09:43 DEBUG juju.provider.maas environ.go:795 started instance "eghprm"
02:09:43 INFO cmd bootstrap.go:322 - eghprm (arch=amd64 mem=4G cores=2)
02:09:43 INFO juju.environs.bootstrap bootstrap.go:1010 newest version: 3.1.6
02:09:43 INFO juju.environs.bootstrap bootstrap.go:1025 picked bootstrap agent binary version: 3.1.6
02:09:43 INFO cmd bootstrap.go:633 Installing Juju agent on bootstrap instance
02:12:20 DEBUG juju.cloudconfig.instancecfg instancecfg.go:945 Setting numa ctl preference to false
Waiting for address
02:12:20 DEBUG juju.provider.maas instance.go:88 "juju" has addresses ["local-cloud:192.168.122.13@undefined(id:-1)"]
Attempting to connect to 192.168.122.13:22
02:12:21 DEBUG juju.provider.common bootstrap.go:668 connection attempt for 192.168.122.13 failed: /var/lib/juju/nonce.txt does not exist
02:12:26 DEBUG juju.provider.common bootstrap.go:668 connection attempt for 192.168.122.13 failed: /var/lib/juju/nonce.txt does not exist
02:12:30 DEBUG juju.provider.maas instance.go:88 "juju" has addresses ["local-cloud:192.168.122.13@undefined(id:-1)"]
02:12:32 DEBUG juju.provider.common bootstrap.go:668 connection attempt for 192.168.122.13 failed: /var/lib/juju/nonce.txt does not exist
02:12:37 INFO cmd bootstrap.go:436 Connected to 192.168.122.13
02:12:37 INFO juju.cloudconfig userdatacfg_unix.go:598 Fetching agent: curl -sSf --retry 10 -o $bin/tools.tar.gz <[https://streams.canonical.com/juju/tools/agent/3.1.6/juju-3.1.6-linux-amd64.tgz]>
02:12:37 INFO cmd bootstrap.go:506 Running machine configuration script...
02:15:35 INFO cmd bootstrap.go:744 Bootstrap agent now started
02:15:36 DEBUG juju.provider.maas instance.go:88 "juju" has addresses ["local-cloud:192.168.122.13@undefined(id:-1)"]
02:15:36 INFO juju.juju api.go:354 API endpoints changed from [] to [192.168.122.13:17070]
02:15:36 INFO cmd controller.go:88 Contacting Juju controller at 192.168.122.13 to verify accessibility...
02:15:36 INFO juju.juju api.go:86 connecting to API addresses: [192.168.122.13:17070]
02:15:38 DEBUG juju.api apiclient.go:1171 successfully dialed "wss://192.168.122.13:17070/model/3eea9ec8-8c3b-4e3c-8868-97921da77ca1/api"
02:15:38 INFO juju.api apiclient.go:706 connection established to "wss://192.168.122.13:17070/model/3eea9ec8-8c3b-4e3c-8868-97921da77ca1/api"
02:15:38 DEBUG juju.api monitor.go:35 RPC connection died
02:15:38 INFO cmd controller.go:108
Bootstrap complete, controller "maas" is now available
Controller machines are in the "controller" model
02:15:38 INFO cmd bootstrap.go:658
Now you can run
juju add-model <model-name>
to create a new model to deploy workloads.
02:15:38 INFO cmd supercommand.go:535 command finished
```

openssl output from newly generated cert:

```$ openssl s_client -showcerts -connect 192.168.122.13:17070
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 O = Juju, CN = juju-ca
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 O = Juju, CN = juju-ca
verify return:1
depth=0 O = Juju, CN = Juju server certificate - controllerip, serialNumber = 3d3bee3c-10b7-4e03-8696-ce3fca64b803
verify return:1
---
Certificate chain
0 s:O = Juju, CN = Juju server certificate - controllerip, serialNumber = 3d3bee3c-10b7-4e03-8696-ce3fca64b803
i:O = Juju, CN = juju-ca
-----BEGIN CERTIFICATE-----
MIIEUzCCArugAwIBAgIVAMSE9KN5mL18zFdcD4E4Z8y46TYQMA0GCSqGSIb3DQEB
CwUAMCExDTALBgNVBAoTBEp1anUxEDAOBgNVBAMTB2p1anUtY2EwHhcNMjMxMDEx
MDIxMDM2WhcNMzMxMDExMDIxNTM2WjBvMQ0wCwYDVQQKEwRKdWp1MS8wLQYDVQQD
EyZKdWp1IHNlcnZlciBjZXJ0aWZpY2F0ZSAtIGNvbnRyb2xsZXJpcDEtMCsGA1UE
BRMkM2QzYmVlM2MtMTBiNy00ZTAzLTg2OTYtY2UzZmNhNjRiODAzMIIBojANBgkq
hkiG9w0BAQEFAAOCAY8AMIIBigKCAYEA05qrGU2EOlktpy1hpwY6G94Bs0kBxohN
pabCiBuvlCO77S0fK/N58MvmaPrKrXEs4e0fukBQf+MQnf6j/05J2+oE+EG0nqbv
s8vwAXcj71yeTHC8AxOv7LbR0XySy2mDsiou8o4Kvz6NgyvrU2VTEbPnPGhhx5Mi
iIB+/giCt69am0HbMkF+P+1F21YlVeSVqTiaxU4XmmtwqRKZuaiBWbwoIGPE3yJH
fG5KGhfGamFbKKJj3iMwiyG/BZIu3kW8fakyxkE0++pyBlVaxKjK0VcvWDxCvPri
if0Q+PEW0x9VRTiiWg0ACS7SrDyWk+JodVq4uSd+v11KvpzqzuU7SQIPJebAuSFV
+pcIOndEmv41X5Tdtcvq1NPbN7F9/AM1ltK0ck94guY9lJKQEnKyOdf+uo3DzFKr
uKrASaq9iw1/FmLy7TYfsuKNH+s2YW35gBGXaczB0ftR3ZmpD/zw0EdIH3bQEmne
WG0nkKKFX16FZ0dXqMazt1uUb3oKDJ9xAgMBAAGjNDAyMB8GA1UdIwQYMBaAFEWl
r6Ab45m7IzD/5t57F+5pE2QpMA8GA1UdEQQIMAaHBMCoeg0wDQYJKoZIhvcNAQEL
BQADggGBAJZrTMkBbyRKeDRx4XeB9ouUIxQIudVkUesEHgg3aqPwgStyOLp6nq1e
wkvsotGYKAZhUNkPf+Rt4VuT4JO2691JXFaSfJc1VhdIcxE5BknOhwzxkfLaufSX
7NRngcqMoJWKP8PQ2dv3xqwpb77IMXyJuKzMmD1gKUXPbFeCKHRPh81OdWMcA25C
RsQrVwxelrFksneubu7GOkuK9C/KHWKCmB1mx2OK2bLRWBD+nV19yxKIi15a4cCA
t2eWSHb3BhodlgdRFet5EqyFlJB3SOhXGceNLj/GKZ5jSnxJqKDsyDZ3q9VVkfAQ
pY4SA8f7TaxzlVtb/MYu0KHkrbBM4sSpdn8pAt86iZYpibjUD6SPNjxbRdviY7zG
Y1swDCpdriHLTDkn/9QmuUqNCU9VcoIIkFahwwHcVH5ZlRXO6aNts74RBtPcMMGi
/Xbdmt16LRbiF9g150RmJI1/VA48ZEgbHwA8b3k1+IJaXxSzPPyNiaN0l0SC+9N4
R3RVYPFrvA==
-----END CERTIFICATE-----
1 s:O = Juju, CN = juju-ca
i:O = Juju, CN = juju-ca
-----BEGIN CERTIFICATE-----
MIIEEjCCAnqgAwIBAgIUVLlMrIoXMF5lx1/dVMANWRoWNJAwDQYJKoZIhvcNAQEL
BQAwITENMAsGA1UEChMESnVqdTEQMA4GA1UEAxMHanVqdS1jYTAeFw0yMzEwMTEw
MjA0MzZaFw0zMzEwMTEwMjA5MzZaMCExDTALBgNVBAoTBEp1anUxEDAOBgNVBAMT
B2p1anUtY2EwggGiMA0GCSqGSIb3DQEBAQUAA4IBjwAwggGKAoIBgQDK4BU2TsKo
L4M2Inv4liaHN14KkK4SXHJNOSTqtGxEPkprkNRlz0BV3ybY/Qo8xDeAqQYCCt2o
SNZW1KqJKCP+FJ5NU7o3ZRD9Weru69y+71C2LCLlAtK0zwqHLLa6J4IWqmjD9kea
efkFV4pfUQA9xtF7+SY9AdGfPOx8aFeCiKRMgP+HZ9/212eoFormA5rdOXEYA1KE
uq1k4Ibfr3lIiwFmzu3j2XCZ++H+VUtO/kFaOzSX9an88a3/s1V8jNz/KB1E4Jwx
FmLduX2+MiBWWl/Sf3LnyHtGEOQ6mfu+gF9deJN1NLwSOVnECMzUFu93wX2A1XMw
Svv4wI41ADE8FQqWy0ZM5QIzQgm2KVzo9OIfC+JEk+5gKnO+tf6w/kDru+bqnkUK
6njiDMxTQBPvyF03bS5fYlC5JrS2vmgO08Bg48tebo5vl5GeF2/1EdJjU/Lk7XzP
dMplx3NZnLBE1QgxdShTPM+VQfdbXtN1ITI4AgNGDIQVOcto4UNg490CAwEAAaNC
MEAwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFEWl
r6Ab45m7IzD/5t57F+5pE2QpMA0GCSqGSIb3DQEBCwUAA4IBgQCxV+DBYleAu1oz
VnCG66nc9gZigzz499K9cyk9lor3DDiOF9olC4eYuCscXZQA8b0UXD2oOmle3GlD
7hAXglCFjJWkVHLUhm/ZMgZh8G8KVHvz6WOkkXpke4pF9XOdqfmx09T8Buf0eT32
SHlw9bk/78nmU6PeeZV5leVUiJLW9EVTVd3yZz/RTkeHBnHwAgkiMhJ+wCTk3wHS
2z7nOMWmKV0TyzSANCMyANlK8MaQ0wi2Fuh/YEBoNz96r3tjlDV1/q3n4SiK3G3e
qElTpdrK37GENGo+uKsQMpbgL7FnAhJ5RdZa447BRlaEI3kFD+ziB8om6itxSaK/
AVqIqVylxrjLc98ARoDJ61tC3kAZ4lMvq6MIQHgFCaYMnxN2CQHjHx5z9yXcPuxB
QrN5FS17RMDQ1Vs8NMjfxYnsZGIphhKyukc6ruNtZX0qhdrr3xU+aCRIUNMRTwhP
quQKdcNyEY5rxlczGAjwlFcumjcgVikDY9e++hbP7vx3RJjDfC0=
-----END CERTIFICATE-----
---
Server certificate
subject=O = Juju, CN = Juju server certificate - controllerip, serialNumber = 3d3bee3c-10b7-4e03-8696-ce3fca64b803

issuer=O = Juju, CN = juju-ca

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2863 bytes and written 370 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 3072 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self signed certificate in certificate chain)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol : TLSv1.3
    Cipher : TLS_AES_128_GCM_SHA256
    Session-ID: B61B5D9C853774D3E0D87FD523F4B8582F4657C7443E521A19925DD746650AC8
    Session-ID-ctx:
    Resumption PSK: 91CB1A72A959F9F8B60733B0DC701602596ABA65235F541569DFCEB007E0CF64
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - 8a 5c 38 4a 68 0a c2 ac-47 03 96 3f c5 73 1b 34 .\8Jh...G..?.s.4
    0010 - 69 3f 35 36 56 53 40 e1-41 a8 d8 b2 42 5a 06 46 i?56VS@.A...BZ.F
    0020 - 74 bf 9b 2e cf 26 3a 7a-63 7c 6a f2 d0 94 b3 90 t....&:zc|j.....
    0030 - cb c5 55 de 01 86 44 77-0b e4 fd 24 cc cc 17 c7 ..U...Dw...$....
    0040 - 64 68 2b fd d3 2d b3 02-ec 28 59 65 48 9f ce 4d dh+..-...(YeH..M
    0050 - c2 2d 9c 03 aa 86 5a ad-34 e0 ee 28 50 2f 0f 87 .-....Z.4..(P/..
    0060 - 48 1a a5 8f 56 db e7 1a-17 1f 23 f5 21 04 38 d3 H...V.....#.!.8.
    0070 - 15 .

    Start Time: 1696990592
    Timeout : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
```

Revision history for this message

Jeff Hillman (jhillman) wrote on 2023-10-11:

Download full text (10.6 KiB)

Possibly unrelated, but when using just ca-cert and ca-private-key without the -path reads the files but fails to bootstrap:

```
~$ juju bootstrap maas maas --bootstrap-series focal --config ca-cert="$(cat ./ca-cert.pem)" --config ca-private-key="$(cat ca-private-key.pem)" --debug
15:42:33 INFO  juju.cmd supercommand.go:56 running juju [3.1.6 f6a66aa91eec620f5ac04a19d8c06bef03ae6228 gc go1.20.8]
15:42:33 DEBUG juju.cmd supercommand.go:57   args: []string{"/snap/juju/24626/bin/juju", "bootstrap", "maas", "maas", "--bootstrap-series", "focal", "--config", "ca-cert=-----BEGIN CERTIFICATE-----\nMIIDVzCCAj+gAwIBAgIUUKjZYxXOfxfk1pPA4Lu2PmecXK8wDQYJKoZIhvcNAQEL\nBQAwOzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkZMMQ0wCwYDVQQKDARVU0FGMRAw\nDgYDVQQDDAdhbXBob3JhMB4XDTIzMDkyNTE4NTUxMFoXDTMzMDkyMjE4NTUxMFow\nOzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkZMMQ0wCwYDVQQKDARVU0FGMRAwDgYD\nVQQDDAdhbXBob3JhMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArHZt\n2UID0WzK32qyJH7KFtRk7VLQ/FZSOqwQQmjPCk9ffsmVciMqsD02gAluCQyUwOBS\ndPCv52kz95rQfC2LSh1OQYG2GraiF/QFnAEbYF+77W4xAOmgYe77olOxEyWtRD6+\nK8uIov06EROzmnnELdHEua6eCZRIC/W8OIpj7rzThMs4G8vpQam82Xpk1gJMrJws\nFCDUEF8+2intZ6zqL/q5Q6SZ+h0nZSnhmlCw/UTCH12H1ZjI9+W6ZRQ1ZnOHxkvi\nFBjg/SMGnfmId9jrXhEl/FvuY/nhcfC40vW79Vp3ZLdMRyKz4vflDUE4MJm/DIJy\nZyzzv7fkJ+uCgYcB/wIDAQABo1MwUTAdBgNVHQ4EFgQU7OIMjsZCoj2yUBUi/3Fe\nmJg4ZgowHwYDVR0jBBgwFoAU7OIMjsZCoj2yUBUi/3FemJg4ZgowDwYDVR0TAQH/\nBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAOZNLVxaOO3ihc9Ve4yW/tDA9DT7D\nPGr+z/wZGp1bzXF5aN/lW0hNE4VcgrvYkrVJMGRoi9rosZU+M4lN2lK2VaIycw58\n6lGCwugx/HxmmnKji+fv4+OzFI9xuplEAFee+ZL8weJIL7jD0QCrL86K+RPEBXyw\nZtSwR5ZTfoQOxKFvk/I5IKa4piyBOQJzGW/5aorxVC27gwbOzGOhTnwU32B5nCp+\nzVJdhyCMpEtnGiP9M+C3lZD4yBqAeHqLZQ3SpkNs8yIT6ikyCDJshXzdeUQ+yiWB\nRCMRarEEGVdPJXwiJsW2E5XSFPbo4HGxRY8uy47AVgQz06rONS/LMEzgEw==\n-----END CERTIFICATE-----", "--config", "ca-private-key=-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCsdm3ZQgPRbMrf\narIkfsoW1GTtUtD8VlI6rBBCaM8KT19+yZVyIyqwPTaACW4JDJTA4FJ08K/naTP3\nmtB8LYtKHU5BgbYatqIX9AWcARtgX7vtbjEA6aBh7vuiU7ETJa1EPr4ry4ii/ToR\nE7OaecQt0cS5rp4JlEgL9bw4imPuvNOEyzgby+lBqbzZemTWAkysnCwUINQQXz7a\nKe1nrOov+rlDpJn6HSdlKeGaULD9RMIfXYfVmMj35bplFDVmc4fGS+IUGOD9Iwad\n+Yh32OteESX8W+5j+eFx8LjS9bv1Wndkt0xHIrPi9+UNQTgwmb8MgnJnLPO/t+Qn\n64KBhwH/AgMBAAECggEBAJVNcYvEOsnzBS6lj9NIcGuTNEzAhWvuzwE8NFdJpPOo\nA2Qf3+EP0MUCOUxe3YO4AI95o+jY9W/BA9w3ioX2Z7+h3z2WrbAyq+FQwApuauTj\n/C1wV7wShWZDHlVJXuLUm0uZhckkSwXPc5WVGMeC4vNHxqd5JBfm5vhzpJ7OL/Vc\n1tgvUGYv3wKE38Cf/vtFr4Z5/PgJnpq6QAG1rx8kjvrvnVwB+KTJMUFaUeKqR7+I\nyZkUn900Y62D9zTqTHb9NJFw9mCHi47u13uZU4W2y1KOKxPThx0+NzKFCaD1I34w\ntPqf2a+mQauaq32LIpV/zoBaezSeej4YkRn8pCrW5QECgYEA3oTFp31bRLws2eJO\nNI6sMH7OFGbuXE9yIsxSZlVO/XpIYazN+LGYZSOsPGPwApjosQQeoamqTfUz/dqV\n8nSj57ZzOOkJsJZSrB5S+/IYq8Ngki5NFYWdV9FDEEHOu76VqHCLebC2Jm3K+ZzA\nJeUbVNFwbeOI6Dq5LTZC+nYdVwUCgYEAxmmKpubr7TXEox8GXFae1TrBzdAb/EhA\nAxfZ6AGsaHPcrr9FI1iuiCoMNsTn7QMco6bggEwu36Pw9Qdhz5KdnWWWfu2PNOut\nGAh0YMb3ws+QraQxFwYwQzwzhgf9QKsN7TliKk468wa8MVvBLd7CQmEA//hVuDkb\nPCFjhbNbvDMCgYAmyN+mrOvelCoBJDfbY07R+Rg0aCh9wH81X4WxPodRGSJKnBMe\nmN3mKwyXThgEa+CLEhvAs9DyW53fTl06cGgtOBjnP38n31uzkmmGbpEsRxarBBT9\nfleJefkFeWLuSG7PeCZnZIyrMBHj5hV7xW45H3RIQxOW3r1uCvorq7C7fQKBgQDD\nBw0d35B6O0bM3HshGpfHmpjTmENZ6R7aq4AquDJi/bgnIKVyCAKOn14gG86pP0zQ\nZAD6u2e7tmbsQ1RWd5RsGWYj35jLePlMzUqvsgPpvqNEu0bxjci03iD7FzFNdpYa\n3zPpx7nUFoO3tKwfTX3/R2B3Ov/vNFrb8SnlSxbZ1wKBgFKb3pF7bdFMUyIEn3S9\ntUV1ltIBiHxCGGCSo/Luwbs47yzdFtUWbTmtbVrFwkEKJRbTFvV8rZK01eb9UmeV\nGNFpfrgXJp9ZSeRVndiDPCc6GDVuenXR3ckJyhCWYIBZgcKw9Lmj84Yg3nHy84Qg\nAvNqCg3JVQLabB+C8GtXRD/y\n-----END PRIVATE KEY-----", "--debug"}
15:42:33 DEBUG juju.cmd.juju.commands bootstrap.go:1392 authenticating with region "" and credential "maas" ()
15:42:33 DEBUG juju.cmd.juju.commands bootstrap.go:1551 provider attrs: map[]
15:42:33 INFO  cmd authkeys.go:113 Adding contents of "/home/ubuntu/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
15:42:33 DEBUG juju.cmd.juju.commands bootstrap.go:1621 preparing controller with config: map[agent-metadata-url: agent-stream:released apt-ftp-proxy: apt-http-proxy: apt-https-proxy: apt-mirror: apt-no-proxy: authorized-keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDTsBz0GmLHKL5fEZsJ87axUqxETiAnb6Pig3lG9WEkoxzop6KzMQ3RUSGWhGmfRbsoK/+ShiHvyILS2LD5yuJsKvc+aqP4h6pZ+HuPtwiACzCHxpKqSptwGkW/MilInmmoyhy3Ei4A4Oqe61ixcFxv3Sb7CIxrYz0XujVg1R4uYOqLKKhEZbHHISi8W0oEA91MUKk3NNJ/UjQtiyWYhsl2Rz08CkFJu9jR6PhsS4qz/H1uuUszDAN8fz+ANwHJVQKca0aOjpD0uuIINcdl9RNH3ZTTikC9YuK3xHkW7ivnl2WPLlgiOEIvuny5X/wtVkNaNR76eOf6L0ODZYnYxwtacnjNQo5DYnhhIbW8W+HePc2NbenGAIEQXz/B9qxxWkDJW4BI/A6HY3ML8wvWL//F8sp4+7fIv3KOlQVKQiCncep8R3tAGj1w9kYFZw7vX9+nM33yosW6ByAxduevITl+EvF8cHqQ6aKw/Zy7IvuHT7YIVNhlGYmN0zRsnsg/6UoVsqbCJfxDGML9TTpAWC8KBvGqOQXXp2kE4T1sX5/LpLJYRu2eQfw/ICFHZh3CtgBsZJ/aVuwKJtXJRLHxU7BIJOEaAMpnR0EdNLS4T8kqqjfI7mAHBpZGABRkk0hMgDKfPFSnTzAuSZ38G9mNREbXCDXlwhivYDahu4C+6uTVNQ== juju-client-key
 automatically-retry-hooks:true backup-dir: charmhub-url:https://api.charmhub.io cloudinit-userdata: container-image-metadata-url: container-image-stream:released container-inherit-properties: container-networking-method: default-base: default-space: development:false disable-network-management:false disable-telemetry:false egress-subnets: enable-os-refresh-update:true enable-os-upgrade:true fan-config: firewall-mode:instance ftp-proxy: http-proxy: https-proxy: ignore-machine-addresses:false image-metadata-url: image-stream:released juju-ftp-proxy: juju-http-proxy: juju-https-proxy: juju-no-proxy:127.0.0.1,localhost,::1 logforward-enabled:false logging-config: logging-output: lxd-snap-channel:5.0/stable max-action-results-age:336h max-action-results-size:5G max-status-history-age:336h max-status-history-size:5G mode: name:controller net-bond-reconfigure-delay:17 no-proxy:127.0.0.1,localhost,::1 num-container-provision-workers:4 num-provision-workers:16 provisioner-harvest-mode:destroyed proxy-ssh:false resource-tags: secret-backend:auto snap-http-proxy: snap-https-proxy: snap-store-assertions: snap-store-proxy: snap-store-proxy-url: ssl-hostname-verification:true test-mode:false transmit-vendor-metrics:true type:maas update-status-hook-interval:5m uuid:ab73669d-4228-4217-84b0-8992c0299973]
15:42:33 DEBUG juju.provider.maas environprovider.go:63 opening model "controller".
15:42:33 INFO  cmd bootstrap.go:939 Creating Juju controller "maas" on maas/default
15:42:33 INFO  juju.cmd.juju.commands bootstrap.go:1010 combined bootstrap constraints: 
15:42:33 DEBUG juju.environs.bootstrap bootstrap.go:319 model "controller" supports application/machine networks: true
15:42:33 DEBUG juju.environs.bootstrap bootstrap.go:321 network management by juju enabled: true
15:42:33 INFO  cmd bootstrap.go:406 Loading image metadata
15:42:33 INFO  cmd bootstrap.go:479 Looking for packaged Juju agent version 3.1.6 for amd64
15:42:33 INFO  juju.environs.bootstrap tools.go:78 looking for bootstrap agent binaries: version=3.1.6
15:42:33 DEBUG juju.environs.tools tools.go:87 finding agent binaries in stream: "released"
15:42:33 DEBUG juju.environs.tools tools.go:89 reading agent binaries with major.minor version 3.1
15:42:33 DEBUG juju.environs.tools tools.go:98 filtering agent binaries by version: 3.1.6
15:42:33 DEBUG juju.environs.tools tools.go:101 filtering agent binaries by os type: ubuntu
15:42:33 DEBUG juju.environs.tools tools.go:104 filtering agent binaries by architecture: amd64
15:42:33 DEBUG juju.environs.tools urls.go:133 trying datasource "keystone catalog"
15:42:33 DEBUG juju.environs.simplestreams simplestreams.go:417 searching for signed metadata in datasource "default simplestreams"
15:42:33 DEBUG juju.environs.simplestreams simplestreams.go:451 looking for data index using path streams/v1/index2.sjson
15:42:34 DEBUG juju.environs.simplestreams simplestreams.go:747 using default candidate for content id "com.ubuntu.juju:released:agents" are {20210329 mirrors:1.0 content-download streams/v1/cpc-mirrors-agents.sjson []}
15:42:34 DEBUG juju.environs.simplestreams simplestreams.go:463 looking for data index using URL https://streams.canonical.com/juju/tools/streams/v1/index2.sjson
15:42:34 DEBUG juju.environs.simplestreams simplestreams.go:486 read metadata index at "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson"
15:42:34 DEBUG juju.environs.simplestreams simplestreams.go:1018 finding products at path "streams/v1/com.ubuntu.juju-released-agents.sjson"
15:42:34 INFO  juju.environs.bootstrap tools.go:80 found 1 packaged agent binaries
15:42:34 INFO  cmd bootstrap.go:492 Located Juju agent version 3.1.6-ubuntu-amd64 at https://streams.canonical.com/juju/tools/agent/3.1.6/juju-3.1.6-linux-amd64.tgz
15:42:34 INFO  cmd bootstrap.go:590 Starting new instance for initial controller
15:42:34 INFO  cmd bootstrap.go:184 Launching controller instance(s) on maas/default...
15:42:34 DEBUG juju.provider.maas environ.go:685 attempting to acquire node in zone "default"
15:42:35 DEBUG juju.cloudconfig.instancecfg instancecfg.go:945 Setting numa ctl preference to false
15:42:35 DEBUG juju.provider.maas environ.go:769 maas user data; 3720 bytes
15:42:39 DEBUG juju.provider.maas environ.go:795 started instance "wfdwby"
15:42:39 INFO  cmd bootstrap.go:322  - wfdwby (arch=amd64 mem=4G cores=2)  
15:42:39 INFO  juju.environs.bootstrap bootstrap.go:1010 newest version: 3.1.6
15:42:39 INFO  juju.environs.bootstrap bootstrap.go:1025 picked bootstrap agent binary version: 3.1.6
15:42:39 INFO  cmd bootstrap.go:633 Installing Juju agent on bootstrap instance
15:42:39 ERROR juju.cmd.juju.commands bootstrap.go:968 failed to bootstrap model: finalizing bootstrap instance config: loading juju certificate authority: generating CA leaf: found zero certificates in pem bundle
15:42:39 DEBUG juju.cmd.juju.commands bootstrap.go:969 (error details: [{github.com/juju/juju/cmd/juju/commands.(*bootstrapCommand).Run:1067: failed to bootstrap model} {github.com/juju/juju/environs/bootstrap.Bootstrap:742: } {github.com/juju/juju/environs/bootstrap.bootstrapIAAS:670: finalizing bootstrap instance config} {github.com/juju/juju/environs/bootstrap.finalizeInstanceBootstrapConfig:773: loading juju certificate authority} {github.com/juju/juju/pki.NewDefaultAuthorityPem:224: generating CA leaf} {github.com/juju/juju/pki.NewDefaultLeafPem:198: found zero certificates in pem bundle}])
15:42:39 DEBUG juju.cmd.juju.commands bootstrap.go:1778 cleaning up after failed bootstrap
15:42:39 INFO  juju.provider.common destroy.go:21 destroying model "controller"
15:42:39 INFO  juju.provider.common destroy.go:32 destroying instances
15:42:40 INFO  juju.provider.maas environ.go:961 ignoring error while releasing nodes (Invalid transition: Releasing -> Deploying.); all nodes released OK
15:42:40 INFO  juju.provider.common destroy.go:56 destroying storage
15:42:40 INFO  cmd supercommand.go:535 command finished
```

Revision history for this message

John A Meinel (jameinel) wrote on 2023-10-11:

We worked out the issue is that bash strips the trailing newlines from "--config ca-cert=$(cat ./foo.pem)"
And then juju just concatenates the two files when parsing it, but you end up with:

```
-----BEGIN CERTIFICATE-----
MIIDVzCCAj+gAwIBAgIUUKjZ...
-----END CERTIFICATE----------BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
```

The workaround is to
a) Patch juju with:
diff --git a/pki/authority.go b/pki/authority.go
index 9d812540d5..b255dc9495 100644
--- a/pki/authority.go
+++ b/pki/authority.go
@@ -236,7 +236,10 @@ func NewDefaultAuthorityPem(pemBlock []byte) (*DefaultAuthority, error) {
// pem ca and key. Returns error if the supplied cert is not a ca or passing of
// the pem data fails.
func NewDefaultAuthorityPemCAKey(caPem, keyPem []byte) (*DefaultAuthority, error) {
- return NewDefaultAuthorityPem(append(caPem, keyPem...))
+ combined := append(caPem, []byte("\n\n")...)
+ combined = append(combined, keyPem...)
+ combined = append(combined, []byte("\n\n")...)
+ return NewDefaultAuthorityPem(combined)
}

or
b) insert a newline at the *start* of the certificate file (it seems bash strips trailing newlines, but not prefixed ones)

c) create a config.yaml with the fields filled out:
$ cat ./config.yaml
ca-cert: |
  -----BEGIN CERTIFICATE-----
  MIIFqTCCA5GgAwIBAgIUVDF76nUFr9DdDbv8+0CGB3Oxsq8wDQYJKoZIhvcNAQEL
...
  gjJGfWMpEPARB6VLJA==
  -----END CERTIFICATE-----
ca-private-key: |
  -----BEGIN PRIVATE KEY-----
  MIIJQAIBADANBgkqhkiG9w0BAQEFAASCCSowggkmAgEAAoICAQC46EMKXPP4xdys
...
  SvhLa0tEpRcPUvIk6UVTPiWJ290=
  -----END PRIVATE KEY-----

$ juju bootstrap lxd --debug --config ./config.yaml

$ openssl s_client 10.10.30.62:17070
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 C = US, ST = FL, L = Gainesville, O = Internet Widgits Pty Ltd, CN = meinel
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 C = US, ST = FL, L = Gainesville, O = Internet Widgits Pty Ltd, CN = meinel
...

We worked out the issue is that bash strips the trailing newlines from "--config ca-cert=$(cat ./foo.pem)"
And then juju just concatenates the two files when parsing it, but you end up with:

```
-----BEGIN CERTIFICATE-----
MIIDVzCCAj+gAwIBAgIUUKjZ...
-----END CERTIFICATE----------BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
```

The workaround is to
a) Patch juju with:
diff --git a/pki/authority.go b/pki/authority.go
index 9d812540d5..b255dc9495 100644
--- a/pki/authority.go
+++ b/pki/authority.go
@@ -236,7 +236,10 @@ func NewDefaultAuthorityPem(pemBlock []byte) (*DefaultAuthority, error) {
 // pem ca and key. Returns error if the supplied cert is not a ca or passing of
 // the pem data fails.
 func NewDefaultAuthorityPemCAKey(caPem, keyPem []byte) (*DefaultAuthority, error) {
-       return NewDefaultAuthorityPem(append(caPem, keyPem...))
+       combined := append(caPem, []byte("\n\n")...)
+       combined = append(combined, keyPem...)
+       combined = append(combined, []byte("\n\n")...)
+       return NewDefaultAuthorityPem(combined)
 }

or
b) insert a newline at the *start* of the certificate file (it seems bash strips trailing newlines, but not prefixed ones)

c) create a config.yaml with the fields filled out: 
$ cat ./config.yaml
ca-cert: |
  -----BEGIN CERTIFICATE-----
  MIIFqTCCA5GgAwIBAgIUVDF76nUFr9DdDbv8+0CGB3Oxsq8wDQYJKoZIhvcNAQEL
...
  gjJGfWMpEPARB6VLJA==
  -----END CERTIFICATE-----
ca-private-key: |
  -----BEGIN PRIVATE KEY-----
  MIIJQAIBADANBgkqhkiG9w0BAQEFAASCCSowggkmAgEAAoICAQC46EMKXPP4xdys
...
  SvhLa0tEpRcPUvIk6UVTPiWJ290=
  -----END PRIVATE KEY-----
 
$ juju bootstrap lxd --debug --config ./config.yaml

Revision history for this message

Jeff Hillman (jhillman) wrote on 2023-10-11: Re: [Bug 2038974] Re: juju bootstrap with ca-cert-path and ca-private-key-path not using assigned cert/key

Download full text (18.7 KiB)

The newline goes above the key, not the cert.

On Wed, Oct 11, 2023, 12:56 PM John A Meinel <email address hidden>
wrote:

The newline goes above the key, not the cert.

On Wed, Oct 11, 2023, 12:56 PM John A Meinel <2038974@bugs.launchpad.net>
wrote:

> We worked out the issue is that bash strips the trailing newlines from
> "--config ca-cert=$(cat ./foo.pem)"
> And then juju just concatenates the two files when parsing it, but you end
> up with:
>
> ```
> -----BEGIN CERTIFICATE-----
> MIIDVzCCAj+gAwIBAgIUUKjZ...
> -----END CERTIFICATE----------BEGIN PRIVATE KEY-----
> ...
> -----END PRIVATE KEY-----
> ```
>
> The workaround is to
> a) Patch juju with:
> diff --git a/pki/authority.go b/pki/authority.go
> index 9d812540d5..b255dc9495 100644
> --- a/pki/authority.go
> +++ b/pki/authority.go
> @@ -236,7 +236,10 @@ func NewDefaultAuthorityPem(pemBlock []byte)
> (*DefaultAuthority, error) {
>  // pem ca and key. Returns error if the supplied cert is not a ca or
> passing of
>  // the pem data fails.
>  func NewDefaultAuthorityPemCAKey(caPem, keyPem []byte)
> (*DefaultAuthority, error) {
> -       return NewDefaultAuthorityPem(append(caPem, keyPem...))
> +       combined := append(caPem, []byte("\n\n")...)
> +       combined = append(combined, keyPem...)
> +       combined = append(combined, []byte("\n\n")...)
> +       return NewDefaultAuthorityPem(combined)
>  }
>
>
> or
> b) insert a newline at the *start* of the certificate file (it seems bash
> strips trailing newlines, but not prefixed ones)
>
> c) create a config.yaml with the fields filled out:
> $ cat ./config.yaml
> ca-cert: |
>   -----BEGIN CERTIFICATE-----
>   MIIFqTCCA5GgAwIBAgIUVDF76nUFr9DdDbv8+0CGB3Oxsq8wDQYJKoZIhvcNAQEL
> ...
>   gjJGfWMpEPARB6VLJA==
>   -----END CERTIFICATE-----
> ca-private-key: |
>   -----BEGIN PRIVATE KEY-----
>   MIIJQAIBADANBgkqhkiG9w0BAQEFAASCCSowggkmAgEAAoICAQC46EMKXPP4xdys
> ...
>   SvhLa0tEpRcPUvIk6UVTPiWJ290=
>   -----END PRIVATE KEY-----
>
> $ juju bootstrap lxd --debug --config ./config.yaml
>
> $ openssl s_client 10.10.30.62:17070
> CONNECTED(00000003)
> Can't use SSL_get_servername
> depth=1 C = US, ST = FL, L = Gainesville, O = Internet Widgits Pty Ltd, CN
> = meinel
> verify error:num=19:self-signed certificate in certificate chain
> verify return:1
> depth=1 C = US, ST = FL, L = Gainesville, O = Internet Widgits Pty Ltd, CN
> = meinel
> ...
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2038974
>
> Title:
>   juju bootstrap with ca-cert-path and ca-private-key-path not using
>   assigned cert/key
>
> Status in Canonical Juju:
>   New
>
> Bug description:
>   juju 3.1.6
>   Focal 20.04.6
>
>   Whenrunning a local bootstrap pointing to use a local set of CA
>   certs/keys with a decrypted key, juju doesn't utilize that CA/key to
>   generate a certificate against the pair.
>
>   A juju-ca issuer is still utilized.
>
>   juju bootstrap debug output:
>
>   ```
>   $ juju bootstrap maas maas --bootstrap-series focal --config
> ca-cert-path=./controller_ca.pem --config
> ca-private-key-path=./decrypt.pem   --debug
>   02:09:36 INFO  juju.cmd supercommand.go:56 running juju [3.1.6
> f6a66aa91eec620f5ac04a19d8c06bef03ae6228 gc go1.20.8]
>   02:09:36 DEBUG juju.cmd supercommand.go:57   args:
> []string{"/snap/juju/24626/bin/juju", "bootstrap", "maas", "maas",
> "--bootstrap-series", "focal", "--config",
> "ca-cert-path=./controller_ca.pem", "--config",
> "ca-private-key-path=./decrypt.pem", "--debug"}
>   02:09:36 DEBUG juju.cmd.juju.commands bootstrap.go:1392 authenticating
> with region "" and credential "maas" ()
>   02:09:36 DEBUG juju.cmd.juju.commands bootstrap.go:1551 provider attrs:
> map[]
>   02:09:36 INFO  cmd authkeys.go:113 Adding contents of
> "/home/ubuntu/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
>   02:09:36 DEBUG juju.cmd.juju.commands bootstrap.go:1621 preparing
> controller with config: map[agent-metadata-url: agent-stream:released
> apt-ftp-proxy: apt-http-proxy: apt-https-proxy: apt-mirror: apt-no-proxy:
> authorized-keys:ssh-rsa
> AAAAB3NzaC1yc2EAAAADAQABAAACAQDTsBz0GmLHKL5fEZsJ87axUqxETiAnb6Pig3lG9WEkoxzop6KzMQ3RUSGWhGmfRbsoK/+ShiHvyILS2LD5yuJsKvc+aqP4h6pZ+HuPtwiACzCHxpKqSptwGkW/MilInmmoyhy3Ei4A4Oqe61ixcFxv3Sb7CIxrYz0XujVg1R4uYOqLKKhEZbHHISi8W0oEA91MUKk3NNJ/UjQtiyWYhsl2Rz08CkFJu9jR6PhsS4qz/H1uuUszDAN8fz+ANwHJVQKca0aOjpD0uuIINcdl9RNH3ZTTikC9YuK3xHkW7ivnl2WPLlgiOEIvuny5X/wtVkNaNR76eOf6L0ODZYnYxwtacnjNQo5DYnhhIbW8W+HePc2NbenGAIEQXz/B9qxxWkDJW4BI/A6HY3ML8wvWL//F8sp4+7fIv3KOlQVKQiCncep8R3tAGj1w9kYFZw7vX9+nM33yosW6ByAxduevITl+EvF8cHqQ6aKw/Zy7IvuHT7YIVNhlGYmN0zRsnsg/6UoVsqbCJfxDGML9TTpAWC8KBvGqOQXXp2kE4T1sX5/LpLJYRu2eQfw/ICFHZh3CtgBsZJ/aVuwKJtXJRLHxU7BIJOEaAMpnR0EdNLS4T8kqqjfI7mAHBpZGABRkk0hMgDKfPFSnTzAuSZ38G9mNREbXCDXlwhivYDahu4C+6uTVNQ==
> juju-client-key
>    automatically-retry-hooks:true backup-dir:
> ca-cert-path:./controller_ca.pem ca-private-key-path:./decrypt.pem
> charmhub-url:https://api.charmhub.io cloudinit-userdata:
> container-image-metadata-url: container-image-stream:released
> container-inherit-properties: container-networking-method: default-base:
> default-space: development:false disable-network-management:false
> disable-telemetry:false egress-subnets: enable-os-refresh-update:true
> enable-os-upgrade:true fan-config: firewall-mode:instance ftp-proxy:
> http-proxy: https-proxy: ignore-machine-addresses:false image-metadata-url:
> image-stream:released juju-ftp-proxy: juju-http-proxy: juju-https-proxy:
> juju-no-proxy:127.0.0.1,localhost,::1 logforward-enabled:false
> logging-config: logging-output: lxd-snap-channel:5.0/stable
> max-action-results-age:336h max-action-results-size:5G
> max-status-history-age:336h max-status-history-size:5G mode:
> name:controller net-bond-reconfigure-delay:17
> no-proxy:127.0.0.1,localhost,::1 num-container-provision-workers:4
> num-provision-workers:16 provisioner-harvest-mode:destroyed proxy-ssh:false
> resource-tags: secret-backend:auto snap-http-proxy: snap-https-proxy:
> snap-store-assertions: snap-store-proxy: snap-store-proxy-url:
> ssl-hostname-verification:true test-mode:false transmit-vendor-metrics:true
> type:maas update-status-hook-interval:5m
> uuid:3eea9ec8-8c3b-4e3c-8868-97921da77ca1]
>   02:09:36 DEBUG juju.provider.maas environprovider.go:63 opening model
> "controller".
>   02:09:36 WARN  juju.environs.config config.go:1908 unknown config field
> "ca-private-key-path"
>   02:09:36 WARN  juju.environs.config config.go:1908 unknown config field
> "ca-cert-path"
>   02:09:36 WARN  juju.environs.config config.go:1908 unknown config field
> "ca-private-key-path"
>   02:09:36 WARN  juju.environs.config config.go:1908 unknown config field
> "ca-cert-path"
>   02:09:36 INFO  cmd bootstrap.go:939 Creating Juju controller "maas" on
> maas/default
>   02:09:36 INFO  juju.cmd.juju.commands bootstrap.go:1010 combined
> bootstrap constraints:
>   02:09:36 DEBUG juju.environs.bootstrap bootstrap.go:319 model
> "controller" supports application/machine networks: true
>   02:09:36 DEBUG juju.environs.bootstrap bootstrap.go:321 network
> management by juju enabled: true
>   02:09:36 INFO  cmd bootstrap.go:406 Loading image metadata
>   02:09:36 INFO  cmd bootstrap.go:479 Looking for packaged Juju agent
> version 3.1.6 for amd64
>   02:09:36 INFO  juju.environs.bootstrap tools.go:78 looking for bootstrap
> agent binaries: version=3.1.6
>   02:09:36 DEBUG juju.environs.tools tools.go:87 finding agent binaries in
> stream: "released"
>   02:09:36 DEBUG juju.environs.tools tools.go:89 reading agent binaries
> with major.minor version 3.1
>   02:09:36 DEBUG juju.environs.tools tools.go:98 filtering agent binaries
> by version: 3.1.6
>   02:09:36 DEBUG juju.environs.tools tools.go:101 filtering agent binaries
> by os type: ubuntu
>   02:09:36 DEBUG juju.environs.tools tools.go:104 filtering agent binaries
> by architecture: amd64
>   02:09:36 DEBUG juju.environs.tools urls.go:133 trying datasource
> "keystone catalog"
>   02:09:36 DEBUG juju.environs.simplestreams simplestreams.go:417
> searching for signed metadata in datasource "default simplestreams"
>   02:09:36 DEBUG juju.environs.simplestreams simplestreams.go:451 looking
> for data index using path streams/v1/index2.sjson
>   02:09:38 DEBUG juju.environs.simplestreams simplestreams.go:747 using
> default candidate for content id "com.ubuntu.juju:released:agents" are
> {20210329 mirrors:1.0 content-download streams/v1/cpc-mirrors-agents.sjson
> []}
>   02:09:38 DEBUG juju.environs.simplestreams simplestreams.go:463 looking
> for data index using URL
> https://streams.canonical.com/juju/tools/streams/v1/index2.sjson
>   02:09:38 DEBUG juju.environs.simplestreams simplestreams.go:486 read
> metadata index at "
> https://streams.canonical.com/juju/tools/streams/v1/index2.sjson"
>   02:09:38 DEBUG juju.environs.simplestreams simplestreams.go:1018 finding
> products at path "streams/v1/com.ubuntu.juju-released-agents.sjson"
>   02:09:38 INFO  juju.environs.bootstrap tools.go:80 found 1 packaged
> agent binaries
>   02:09:38 INFO  cmd bootstrap.go:492 Located Juju agent version
> 3.1.6-ubuntu-amd64 at
> https://streams.canonical.com/juju/tools/agent/3.1.6/juju-3.1.6-linux-amd64.tgz
>   02:09:38 WARN  juju.environs.config config.go:1908 unknown config field
> "ca-private-key-path"
>   02:09:38 WARN  juju.environs.config config.go:1908 unknown config field
> "ca-cert-path"
>   02:09:38 WARN  juju.environs.config config.go:1908 unknown config field
> "ca-private-key-path"
>   02:09:38 WARN  juju.environs.config config.go:1908 unknown config field
> "ca-cert-path"
>   02:09:38 INFO  cmd bootstrap.go:590 Starting new instance for initial
> controller
>   02:09:39 INFO  cmd bootstrap.go:184 Launching controller instance(s) on
> maas/default...
>   02:09:39 DEBUG juju.provider.maas environ.go:685 attempting to acquire
> node in zone "default"
>   02:09:39 DEBUG juju.cloudconfig.instancecfg instancecfg.go:945 Setting
> numa ctl preference to false
>   02:09:39 DEBUG juju.provider.maas environ.go:769 maas user data; 3720
> bytes
>   02:09:43 DEBUG juju.provider.maas environ.go:795 started instance
> "eghprm"
>   02:09:43 INFO  cmd bootstrap.go:322  - eghprm (arch=amd64 mem=4G
> cores=2)
>   02:09:43 INFO  juju.environs.bootstrap bootstrap.go:1010 newest version:
> 3.1.6
>   02:09:43 INFO  juju.environs.bootstrap bootstrap.go:1025 picked
> bootstrap agent binary version: 3.1.6
>   02:09:43 INFO  cmd bootstrap.go:633 Installing Juju agent on bootstrap
> instance
>   02:12:20 DEBUG juju.cloudconfig.instancecfg instancecfg.go:945 Setting
> numa ctl preference to false
>   Waiting for address
>   02:12:20 DEBUG juju.provider.maas instance.go:88 "juju" has addresses
> ["local-cloud:192.168.122.13@undefined(id:-1)"]
>   Attempting to connect to 192.168.122.13:22
>   02:12:21 DEBUG juju.provider.common bootstrap.go:668 connection attempt
> for 192.168.122.13 failed: /var/lib/juju/nonce.txt does not exist
>   02:12:26 DEBUG juju.provider.common bootstrap.go:668 connection attempt
> for 192.168.122.13 failed: /var/lib/juju/nonce.txt does not exist
>   02:12:30 DEBUG juju.provider.maas instance.go:88 "juju" has addresses
> ["local-cloud:192.168.122.13@undefined(id:-1)"]
>   02:12:32 DEBUG juju.provider.common bootstrap.go:668 connection attempt
> for 192.168.122.13 failed: /var/lib/juju/nonce.txt does not exist
>   02:12:37 INFO  cmd bootstrap.go:436 Connected to 192.168.122.13
>   02:12:37 INFO  juju.cloudconfig userdatacfg_unix.go:598 Fetching agent:
> curl -sSf --retry 10 -o $bin/tools.tar.gz <[
> https://streams.canonical.com/juju/tools/agent/3.1.6/juju-3.1.6-linux-amd64.tgz
> ]>
>   02:12:37 INFO  cmd bootstrap.go:506 Running machine configuration
> script...
>   02:15:35 INFO  cmd bootstrap.go:744 Bootstrap agent now started
>   02:15:36 DEBUG juju.provider.maas instance.go:88 "juju" has addresses
> ["local-cloud:192.168.122.13@undefined(id:-1)"]
>   02:15:36 INFO  juju.juju api.go:354 API endpoints changed from [] to [
> 192.168.122.13:17070]
>   02:15:36 INFO  cmd controller.go:88 Contacting Juju controller at
> 192.168.122.13 to verify accessibility...
>   02:15:36 INFO  juju.juju api.go:86 connecting to API addresses: [
> 192.168.122.13:17070]
>   02:15:38 DEBUG juju.api apiclient.go:1171 successfully dialed "wss://
> 192.168.122.13:17070/model/3eea9ec8-8c3b-4e3c-8868-97921da77ca1/api"
>   02:15:38 INFO  juju.api apiclient.go:706 connection established to
> "wss://192.168.122.13:17070/model/3eea9ec8-8c3b-4e3c-8868-97921da77ca1/api
> "
>   02:15:38 DEBUG juju.api monitor.go:35 RPC connection died
>   02:15:38 INFO  cmd controller.go:108
>   Bootstrap complete, controller "maas" is now available
>   Controller machines are in the "controller" model
>   02:15:38 INFO  cmd bootstrap.go:658
>   Now you can run
>         juju add-model <model-name>
>   to create a new model to deploy workloads.
>   02:15:38 INFO  cmd supercommand.go:535 command finished
>   ```
>
>   openssl output from newly generated cert:
>
>   ```$ openssl s_client -showcerts -connect 192.168.122.13:17070
>   CONNECTED(00000003)
>   Can't use SSL_get_servername
>   depth=1 O = Juju, CN = juju-ca
>   verify error:num=19:self signed certificate in certificate chain
>   verify return:1
>   depth=1 O = Juju, CN = juju-ca
>   verify return:1
>   depth=0 O = Juju, CN = Juju server certificate - controllerip,
> serialNumber = 3d3bee3c-10b7-4e03-8696-ce3fca64b803
>   verify return:1
>   ---
>   Certificate chain
>    0 s:O = Juju, CN = Juju server certificate - controllerip, serialNumber
> = 3d3bee3c-10b7-4e03-8696-ce3fca64b803
>      i:O = Juju, CN = juju-ca
>   -----BEGIN CERTIFICATE-----
>   MIIEUzCCArugAwIBAgIVAMSE9KN5mL18zFdcD4E4Z8y46TYQMA0GCSqGSIb3DQEB
>   CwUAMCExDTALBgNVBAoTBEp1anUxEDAOBgNVBAMTB2p1anUtY2EwHhcNMjMxMDEx
>   MDIxMDM2WhcNMzMxMDExMDIxNTM2WjBvMQ0wCwYDVQQKEwRKdWp1MS8wLQYDVQQD
>   EyZKdWp1IHNlcnZlciBjZXJ0aWZpY2F0ZSAtIGNvbnRyb2xsZXJpcDEtMCsGA1UE
>   BRMkM2QzYmVlM2MtMTBiNy00ZTAzLTg2OTYtY2UzZmNhNjRiODAzMIIBojANBgkq
>   hkiG9w0BAQEFAAOCAY8AMIIBigKCAYEA05qrGU2EOlktpy1hpwY6G94Bs0kBxohN
>   pabCiBuvlCO77S0fK/N58MvmaPrKrXEs4e0fukBQf+MQnf6j/05J2+oE+EG0nqbv
>   s8vwAXcj71yeTHC8AxOv7LbR0XySy2mDsiou8o4Kvz6NgyvrU2VTEbPnPGhhx5Mi
>   iIB+/giCt69am0HbMkF+P+1F21YlVeSVqTiaxU4XmmtwqRKZuaiBWbwoIGPE3yJH
>   fG5KGhfGamFbKKJj3iMwiyG/BZIu3kW8fakyxkE0++pyBlVaxKjK0VcvWDxCvPri
>   if0Q+PEW0x9VRTiiWg0ACS7SrDyWk+JodVq4uSd+v11KvpzqzuU7SQIPJebAuSFV
>   +pcIOndEmv41X5Tdtcvq1NPbN7F9/AM1ltK0ck94guY9lJKQEnKyOdf+uo3DzFKr
>   uKrASaq9iw1/FmLy7TYfsuKNH+s2YW35gBGXaczB0ftR3ZmpD/zw0EdIH3bQEmne
>   WG0nkKKFX16FZ0dXqMazt1uUb3oKDJ9xAgMBAAGjNDAyMB8GA1UdIwQYMBaAFEWl
>   r6Ab45m7IzD/5t57F+5pE2QpMA8GA1UdEQQIMAaHBMCoeg0wDQYJKoZIhvcNAQEL
>   BQADggGBAJZrTMkBbyRKeDRx4XeB9ouUIxQIudVkUesEHgg3aqPwgStyOLp6nq1e
>   wkvsotGYKAZhUNkPf+Rt4VuT4JO2691JXFaSfJc1VhdIcxE5BknOhwzxkfLaufSX
>   7NRngcqMoJWKP8PQ2dv3xqwpb77IMXyJuKzMmD1gKUXPbFeCKHRPh81OdWMcA25C
>   RsQrVwxelrFksneubu7GOkuK9C/KHWKCmB1mx2OK2bLRWBD+nV19yxKIi15a4cCA
>   t2eWSHb3BhodlgdRFet5EqyFlJB3SOhXGceNLj/GKZ5jSnxJqKDsyDZ3q9VVkfAQ
>   pY4SA8f7TaxzlVtb/MYu0KHkrbBM4sSpdn8pAt86iZYpibjUD6SPNjxbRdviY7zG
>   Y1swDCpdriHLTDkn/9QmuUqNCU9VcoIIkFahwwHcVH5ZlRXO6aNts74RBtPcMMGi
>   /Xbdmt16LRbiF9g150RmJI1/VA48ZEgbHwA8b3k1+IJaXxSzPPyNiaN0l0SC+9N4
>   R3RVYPFrvA==
>   -----END CERTIFICATE-----
>    1 s:O = Juju, CN = juju-ca
>      i:O = Juju, CN = juju-ca
>   -----BEGIN CERTIFICATE-----
>   MIIEEjCCAnqgAwIBAgIUVLlMrIoXMF5lx1/dVMANWRoWNJAwDQYJKoZIhvcNAQEL
>   BQAwITENMAsGA1UEChMESnVqdTEQMA4GA1UEAxMHanVqdS1jYTAeFw0yMzEwMTEw
>   MjA0MzZaFw0zMzEwMTEwMjA5MzZaMCExDTALBgNVBAoTBEp1anUxEDAOBgNVBAMT
>   B2p1anUtY2EwggGiMA0GCSqGSIb3DQEBAQUAA4IBjwAwggGKAoIBgQDK4BU2TsKo
>   L4M2Inv4liaHN14KkK4SXHJNOSTqtGxEPkprkNRlz0BV3ybY/Qo8xDeAqQYCCt2o
>   SNZW1KqJKCP+FJ5NU7o3ZRD9Weru69y+71C2LCLlAtK0zwqHLLa6J4IWqmjD9kea
>   efkFV4pfUQA9xtF7+SY9AdGfPOx8aFeCiKRMgP+HZ9/212eoFormA5rdOXEYA1KE
>   uq1k4Ibfr3lIiwFmzu3j2XCZ++H+VUtO/kFaOzSX9an88a3/s1V8jNz/KB1E4Jwx
>   FmLduX2+MiBWWl/Sf3LnyHtGEOQ6mfu+gF9deJN1NLwSOVnECMzUFu93wX2A1XMw
>   Svv4wI41ADE8FQqWy0ZM5QIzQgm2KVzo9OIfC+JEk+5gKnO+tf6w/kDru+bqnkUK
>   6njiDMxTQBPvyF03bS5fYlC5JrS2vmgO08Bg48tebo5vl5GeF2/1EdJjU/Lk7XzP
>   dMplx3NZnLBE1QgxdShTPM+VQfdbXtN1ITI4AgNGDIQVOcto4UNg490CAwEAAaNC
>   MEAwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFEWl
>   r6Ab45m7IzD/5t57F+5pE2QpMA0GCSqGSIb3DQEBCwUAA4IBgQCxV+DBYleAu1oz
>   VnCG66nc9gZigzz499K9cyk9lor3DDiOF9olC4eYuCscXZQA8b0UXD2oOmle3GlD
>   7hAXglCFjJWkVHLUhm/ZMgZh8G8KVHvz6WOkkXpke4pF9XOdqfmx09T8Buf0eT32
>   SHlw9bk/78nmU6PeeZV5leVUiJLW9EVTVd3yZz/RTkeHBnHwAgkiMhJ+wCTk3wHS
>   2z7nOMWmKV0TyzSANCMyANlK8MaQ0wi2Fuh/YEBoNz96r3tjlDV1/q3n4SiK3G3e
>   qElTpdrK37GENGo+uKsQMpbgL7FnAhJ5RdZa447BRlaEI3kFD+ziB8om6itxSaK/
>   AVqIqVylxrjLc98ARoDJ61tC3kAZ4lMvq6MIQHgFCaYMnxN2CQHjHx5z9yXcPuxB
>   QrN5FS17RMDQ1Vs8NMjfxYnsZGIphhKyukc6ruNtZX0qhdrr3xU+aCRIUNMRTwhP
>   quQKdcNyEY5rxlczGAjwlFcumjcgVikDY9e++hbP7vx3RJjDfC0=
>   -----END CERTIFICATE-----
>   ---
>   Server certificate
>   subject=O = Juju, CN = Juju server certificate - controllerip,
> serialNumber = 3d3bee3c-10b7-4e03-8696-ce3fca64b803
>
>   issuer=O = Juju, CN = juju-ca
>
>   ---
>   No client certificate CA names sent
>   Peer signing digest: SHA256
>   Peer signature type: RSA-PSS
>   Server Temp Key: ECDH, P-256, 256 bits
>   ---
>   SSL handshake has read 2863 bytes and written 370 bytes
>   Verification error: self signed certificate in certificate chain
>   ---
>   New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
>   Server public key is 3072 bit
>   Secure Renegotiation IS NOT supported
>   Compression: NONE
>   Expansion: NONE
>   No ALPN negotiated
>   Early data was not sent
>   Verify return code: 19 (self signed certificate in certificate chain)
>   ---
>   ---
>   Post-Handshake New Session Ticket arrived:
>   SSL-Session:
>       Protocol  : TLSv1.3
>       Cipher    : TLS_AES_128_GCM_SHA256
>       Session-ID:
> B61B5D9C853774D3E0D87FD523F4B8582F4657C7443E521A19925DD746650AC8
>       Session-ID-ctx:
>       Resumption PSK:
> 91CB1A72A959F9F8B60733B0DC701602596ABA65235F541569DFCEB007E0CF64
>       PSK identity: None
>       PSK identity hint: None
>       SRP username: None
>       TLS session ticket lifetime hint: 604800 (seconds)
>       TLS session ticket:
>       0000 - 8a 5c 38 4a 68 0a c2 ac-47 03 96 3f c5 73 1b 34
>  .\8Jh...G..?.s.4
>       0010 - 69 3f 35 36 56 53 40 e1-41 a8 d8 b2 42 5a 06 46   i?56VS@
> .A...BZ.F
>       0020 - 74 bf 9b 2e cf 26 3a 7a-63 7c 6a f2 d0 94 b3 90
>  t....&:zc|j.....
>       0030 - cb c5 55 de 01 86 44 77-0b e4 fd 24 cc cc 17 c7
>  ..U...Dw...$....
>       0040 - 64 68 2b fd d3 2d b3 02-ec 28 59 65 48 9f ce 4d
>  dh+..-...(YeH..M
>       0050 - c2 2d 9c 03 aa 86 5a ad-34 e0 ee 28 50 2f 0f 87
>  .-....Z.4..(P/..
>       0060 - 48 1a a5 8f 56 db e7 1a-17 1f 23 f5 21 04 38 d3
>  H...V.....#.!.8.
>       0070 - 15                                                .
>
>       Start Time: 1696990592
>       Timeout   : 7200 (sec)
>       Verify return code: 19 (self signed certificate in certificate chain)
>       Extended master secret: no
>       Max Early Data: 0
>   ---
>   read R BLOCK
>   ```
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/2038974/+subscriptions
>
>

Revision history for this message

John A Meinel (jameinel) wrote on 2023-10-11:

so Thomas said he would investigate
I think some easy fixes are to ensure that there is a '\n' in the cert and key, or just ensuring that when concatenating them we insert a newline.
We also could make sure the '-path' variants are handled correctly (as that makes it easier to use and doesn't depend on how bash interprets $(<file) content.)

Changed in juju:
assignee:	nobody → Thomas Miller (tlmiller)
importance:	Undecided → High
status:	New → In Progress
milestone:	none → 3.1.7

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.