systemd wrappers (sidecars) locking doesn't really work
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| tripleo |
Incomplete
|
High
|
Unassigned | ||
Bug Description
IIUC The wrapper that populates the processes file and the sync executed by the related systemd service on the host are supposed to share a file lock to prevent races on the processes file. This is because the wrapper adds to the file and sync truncates the file after it runs. However, the lock used in the wrapper is under /var/lock in the container which is not shared with the host so the sync script never waits for the wrapper to be done. Moving the lock file to a path on a shared mount in the container seems to solve that particular race.
... in addition ...
it appears that the triggering of the systemd process that runs the sync command is also racy. It appears that if the processes file has entry added after the shared lock is released, but the sync process isn't completed, sync doesnt happen again.
This was reproduced by restarting the neutron dhcp agent container when 3 subnets were configured. The first problem resulted in only one sidecar being created, the second issue would occasionally result in one or more side car containers being missed in the sync. The processes file would have remaining entries and restarting the dhcp_dnsmasq service on the host would cause the remaining side cars to get created.
| Changed in tripleo: | |
| status: | New → Triaged |
| milestone: | none → ussuri-rc3 |
| importance: | Undecided → Critical |
| description: | updated |
| tags: | added: train-backport-potential |
| Bogdan Dobrelya (bogdando) wrote | #1 |
| Bogdan Dobrelya (bogdando) wrote | #2 |
| Bogdan Dobrelya (bogdando) wrote | #3 |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to tripleo-ansible (master) | #4 |
| Changed in tripleo: | |
| assignee: | nobody → Brent Eagles (beagles) |
| status: | Triaged → In Progress |
| OpenStack Infra (hudson-openstack) wrote | #5 |
| Changed in tripleo: | |
| assignee: | Brent Eagles (beagles) → Bogdan Dobrelya (bogdando) |
| Bogdan Dobrelya (bogdando) wrote | #6 |
| Bogdan Dobrelya (bogdando) wrote | #7 |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to tripleo-heat-templates (master) | #8 |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to tripleo-ansible (master) | #9 |
| OpenStack Infra (hudson-openstack) wrote Fix merged to tripleo-heat-templates (master) | #10 |
| OpenStack Infra (hudson-openstack) wrote Fix merged to tripleo-ansible (master) | #11 |
| OpenStack Infra (hudson-openstack) wrote Change abandoned on tripleo-ansible (master) | #12 |
| OpenStack Infra (hudson-openstack) wrote | #13 |
| Changed in tripleo: | |
| importance: | Critical → High |
| yatin (yatinkarel) wrote | #14 |
| Changed in tripleo: | |
| status: | In Progress → Triaged |
| assignee: | Bogdan Dobrelya (bogdando) → nobody |
| OpenStack Infra (hudson-openstack) wrote Related fix proposed to tripleo-heat-templates (master) | #15 |
| OpenStack Infra (hudson-openstack) wrote Change abandoned on tripleo-heat-templates (master) | #16 |
| Changed in tripleo: | |
| milestone: | ussuri-rc3 → victoria-1 |
| Changed in tripleo: | |
| milestone: | victoria-1 → victoria-3 |
| Changed in tripleo: | |
| milestone: | victoria-3 → wallaby-1 |
| Changed in tripleo: | |
| milestone: | wallaby-1 → wallaby-2 |
| Changed in tripleo: | |
| milestone: | wallaby-2 → wallaby-3 |
| Changed in tripleo: | |
| milestone: | wallaby-3 → wallaby-rc1 |
| Changed in tripleo: | |
| milestone: | wallaby-rc1 → xena-1 |
| Marios Andreou (marios-b) wrote | #17 |
| Changed in tripleo: | |
| milestone: | xena-1 → none |
| status: | Triaged → Incomplete |
> It appears that if the processes file has entry added after the shared lock is released, but the sync process isn't completed, sync doesnt happen again.
that particular part of the issue really sounds like a race in systemd watchers?..