If puppet host uid/gid is changed we cannot puppet apply anymore.

Hi,

this happened during a queens upgrade from pike but I believe it's still the case.

The client for some reason, changed the puppet uid/gid on the host. Then during upgrade he/she got:

 "Error running [\'docker\', \'run\', \'--name\', \'neutron_ovs_bridge\', \'--label\', \'config_id=tripleo_step3\', \'--label\', \'container_name=neutron_ovs_bridge\', \'--label\', \'managed_by=paunch\', \'--label\', \'config_data={\\"image\\": \\"XXXXXXXXXX/rhosp13/openstack-neutron-server:13.0-63\\", \\"pid\\": \\"host\\", \\"environment\\": [\\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\\", \\"TRIPLEO_CONFIG_HASH=97bd63f3e60339f5a0467f97a337b84b\\"], \\"command\\": [\\"puppet\\", \\"apply\\", \\"--modulepath\\", \\"/etc/puppet/modules:/usr/share/openstack-puppet/modules\\", \\"--tags\\", \\"file,file_line,concat,augeas,neutron::plugins::ovs::bridge,vs_config\\", \\"-v\\", \\"-e\\", \\"include neutron::agents::ml2::ovs\\"], \\"user\\": \\"root\\", \\"volumes\\": [\\"/etc/hosts:/etc/hosts:ro\\", \\"/etc/localtime:/etc/localtime:ro\\", \\"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\\", \\"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\\", \\"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\\", \\"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\\", \\"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\\", \\"/dev/log:/dev/log\\", \\"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\\", \\"/etc/puppet:/etc/puppet:ro\\", \\"/var/lib/kolla/config_files/neutron_ovs_agent.json:/var/lib/kolla/config_files/config.json:ro\\", \\"/var/lib/config-data/puppet-generated/neutron/:/var/lib/kolla/config_files/src:ro\\", \\"/lib/modules:/lib/modules:ro\\", \\"/run/openvswitch:/run/openvswitch\\", \\"/etc/puppet:/etc/puppet:ro\\", \\"/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro\\", \\"/var/run/openvswitch/:/var/run/openvswitch/\\"], \\"net\\": \\"host\\", \\"detach\\": false, \\"privileged\\": true}\', \'--env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\', \'--env=TRIPLEO_CONFIG_HASH=97bd63f3e60339f5a0467f97a337b84b\', \'--net=host\', \'--pid=host\', \'--privileged=true\', \'--user=root\', \'--volume=/etc/hosts:/etc/hosts:ro\', \'--volume=/etc/localtime:/etc/localtime:ro\', \'--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\', \'--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\', \'--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\', \'--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\', \'--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\', \'--volume=/dev/log:/dev/log\', \'--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\', \'--volume=/etc/puppet:/etc/puppet:ro\', \'--volume=/var/lib/kolla/config_files/neutron_ovs_agent.json:/var/lib/kolla/config_files/config.json:ro\', \'--volume=/var/lib/config-data/puppet-generated/neutron/:/var/lib/kolla/config_files/src:ro\', \'--volume=/lib/modules:/lib/modules:ro\', \'--volume=/run/openvswitch:/run/openvswitch\', \'--volume=/etc/puppet:/etc/puppet:ro\', \'--volume=/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro\', \'--volume=/var/run/openvswitch/:/var/run/openvswitch/\', \'XXXXXXXXXX/:5000/rhosp13/openstack-neutron-server:13.0-63\', \'puppet\', \'apply\', \'--modulepath\', \'/etc/puppet/modules:/usr/share/openstack-puppet/modules\', \'--tags\', \'file,file_line,concat,augeas,neutron::plugins::ovs::bridge,vs_config\', \'-v\', \'-e\', \'include neutron::agents::ml2::ovs\']. [1]", ',
 u' "stdout: \\u001b[mNotice: /File[/etc/puppet/ssl/certs]: Dependency File[/etc/puppet/ssl] has failures: true\\u001b[0m", ',
 u' "\\u001b[mNotice: /File[/etc/puppet/ssl/public_keys]: Dependency File[/etc/puppet/ssl] has failures: true\\u001b[0m", ',
 u' "\\u001b[mNotice: /File[/etc/puppet/ssl/certificate_requests]: Dependency File[/etc/puppet/ssl] has failures: true\\u001b[0m", ',
 u' "\\u001b[mNotice: /File[/etc/puppet/ssl/private_keys]: Dependency File[/etc/puppet/ssl] has failures: true\\u001b[0m", ',
 u' "\\u001b[mNotice: /File[/etc/puppet/ssl/private]: Dependency File[/etc/puppet/ssl] has failures: true\\u001b[0m", ',
 u' "stderr: \\u001b[1;31mError: Failed to set owner to \'52\': Read-only file system - /etc/puppet/ssl\\u001b[0m", ',
 u' "\\u001b[1;31mError: /File[/etc/puppet/ssl]/owner: change from 31076 to puppet failed: Failed to set owner to \'52\': Read-only file system - /etc/puppet/ssl\\u001b[0m", ',
 u' "\\u001b[1;31mError: Failed to set group to \'52\': Read-only file system - /etc/puppet/ssl\\u001b[0m", ',
 u' "\\u001b[1;31mError: /File[/etc/puppet/ssl]/group: change from 10609 to puppet failed: Failed to set group to \'52\': Read-only file system - /etc/puppet/ssl\\u001b[0m", ',
 u' "\\u001b[1;33mWarning: /File[/etc/puppet/ssl/certs]: Skipping because of failed dependencies\\u001b[0m", ',
 u' "\\u001b[1;33mWarning: /File[/etc/puppet/ssl/public_keys]: Skipping because of failed dependencies\\u001b[0m", ',
 u' "\\u001b[1;33mWarning: /File[/etc/puppet/ssl/certificate_requests]: Skipping because of failed dependencies\\u001b[0m", ',
 u' "\\u001b[1;33mWarning: /File[/etc/puppet/ssl/private_keys]: Skipping because of failed dependencies\\u001b[0m", ',
 u' "\\u001b[1;33mWarning: /File[/etc/puppet/ssl/private]: Skipping because of failed dependencies\\u001b[0m", ',
 u' "\\u001b[1;31mError: Could not prepare for execution: Got 1 failure(s) while initializing: File[/etc/puppet/ssl]: change from 31076 to puppet failed: Failed to set owner to \'52\': Read-only file system - /etc/puppet/ssl; File[/etc/puppet/ssl]: change from 10609 to puppet failed: Failed to set group to \'52\': Read-only file system - /etc/puppet/ssl\\u001b[0m", ',
 u' "stderr: Option \\"logdir\\" from group \\"DEFAULT\\" is deprecated. Use option \\"log-dir\\" from group \\"DEFAULT\\".", ',
 u' "stdout: Upgraded database to: queens_expand01, current revision(s): pike_contract01, queens_expand01", ',
[...]

The problem is that /etc/puppet is mounted read-only and that puppet apply still want to make sure that the /etc/puppet/ssl directory is correct. The mounted directory has uid/gid 31076 while inside the container puppet is 52/52. It tries to fix them and fails.

This is a requirements that I don't think should be there as the /etc/puppet/ssl in this use case is not used (puppet apply vs puppet master).