neutron failed to deploy in ubuntu binary

Bug #1546798 reported by Lingfeng Xiong
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
kolla
Fix Released
Medium
Unassigned

Bug Description

neutron in master failed to deploy in a multi-node deployment (ubuntu binary)

Log:
TASK: [neutron | Starting neutron-dhcp-agent container] ***********************
skipping: [compute-node]
failed: [controller] => {"changed": true, "failed": true}
msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/neutron_dhcp_agent/start',),)

in controller:
ae2838c514e2 kolla:4000/kollaglue/ubuntu-binary-neutron-dhcp-agent:2.0.0 "kolla_start" 28 seconds ago Created neutron_dhcp_agent

docker start ae2838c514e2:
Error response from daemon: Cannot start container ae2838c514e24836b45a8ff4528ddc6bfb935fca60f3a32691354b7c5a044670: Path /run/netns is mounted on /run but it is not a shared mount.
Error: failed to start containers: ae2838c514e2

Revision history for this message
Steven Dake (sdake) wrote :

I believe this is fixed with using the shared flag in systemd. I believe Sam has a patch to modify the docs to resolve this problem.

Changed in kolla:
status: New → Triaged
importance: Undecided → Wishlist
milestone: none → mitaka-3
assignee: nobody → Sam Yaple (s8m)
importance: Wishlist → Low
Revision history for this message
Lingfeng Xiong (xionglingfeng) wrote :

Yes, I saw that patchset today
https://review.openstack.org/#/c/281279/

Thank you for your reply.

Revision history for this message
Liyi Meng (liyi-meng) wrote :

Hi Steven, how this fix problem in ubuntu where there is no systemd? I cherry-pick Sam's change. it make no difference for me. :(

Revision history for this message
Liyi Meng (liyi-meng) wrote :

commit d077fa279c69c9a56d2bae138e5b8cfa62991d78
Merge: 8812271 94a79b1
Author: Jenkins <email address hidden>
Date: Wed Feb 17 15:06:25 2016 +0000

    Merge "Trove ubuntu binary container"

Deploy with ubuntu binary

Here is my error:

<pre>
TASK: [neutron | Starting neutron-dhcp-agent container] ***********************
skipping: [k-blade-4]
skipping: [k-blade-5]
failed: [k-blade-1] => {"changed": true, "failed": true}
msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/neutron_dhcp_agent/start',),)
failed: [k-blade-3] => {"changed": true, "failed": true}
msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/neutron_dhcp_agent/start',),)
failed: [k-blade-2] => {"changed": true, "failed": true}
msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/neutron_dhcp_agent/start',),)
</pre>

On the controller node
<pre>
time="2016-02-18T14:45:46.198232044Z" level=warning msg="Auto-creating non-existent volume host path /run/netns, this is deprecated and will be removed soon"
time="2016-02-18T14:45:46.264529981Z" level=error msg="error locating sandbox id 712989fd759ec8902903cad4b7c562c12805c363c2b4c8d7c284b6cb5bd87ed9: sandbox 712989fd759ec8902903cad4b7c562c12805c363c2b4c8d7c284b6cb5bd87ed9 not found"
time="2016-02-18T14:45:46.265495865Z" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/lib/docker/containers/73f4ec8885377419c1694ab77dea494c1f23a88cf4bc300a87aa0304ff94b135/shm: invalid argument"
time="2016-02-18T14:45:46.265640402Z" level=error msg="Error unmounting container 73f4ec8885377419c1694ab77dea494c1f23a88cf4bc300a87aa0304ff94b135: not mounted"
time="2016-02-18T14:45:46.266043143Z" level=error msg="Handler for POST /v1.22/containers/neutron_dhcp_agent/start returned error: Cannot start container 73f4ec8885377419c1694ab77dea494c1f23a88cf4bc300a87aa0304ff94b135: Path /run/netns is mounted on /run but it is not a shared mount."
</pre>

Try to run the container manually
<pre>
docker start neutron_dhcp_agent
Error response from daemon: Cannot start container 73f4ec8885377419c1694ab77dea494c1f23a88cf4bc300a87aa0304ff94b135: Path /run/netns is mounted on /run but it is not a shared mount.
Error: failed to start containers: neutron_dhcp_agent
</pre>

sudo grep shared /proc/self/mountinfo
342 335 0:17 / /run rw,relatime shared:1 - tmpfs none rw
163 342 0:4 / /run/docker/netns/default rw shared:2 - nsfs nsfs rw

Revision history for this message
Liyi Meng (liyi-meng) wrote :

OK, I found a workaround for my problem. Before running deployment. I should run this command first.

 sudo mkdir -p /run/netns && sudo mount -o rw,relatime -t tmpfs tmpfs /run/netns && sudo mount --make-rshared /run

@Sam Yaple, this seems more reliable than simply run "mount --make-shared /run". do you think it is a good idea to incorporate this into the ubuntu playbook? Do you want me to upate the document?

Revision history for this message
Liyi Meng (liyi-meng) wrote :

OK, my apology. My way of doing thing is not stable either. Sometimes work, sometimes not.

Steven Dake (sdake)
Changed in kolla:
importance: Low → Medium
milestone: mitaka-3 → mitaka-rc1
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

One small comment: I'm using
- Ubuntu source
- All-in-one
and I also have this problem.

DOCKER LOGS:
DEBU[0156] copying image data from ba7b8918b37a0274ef74d6e2550bfc76312a8d665844c44aeafe4a25726de348:/var/lib/neutron/kolla, to neutron_metadata_socket
DEBU[0156] Calling GET /v1.22/containers/json
DEBU[0156] GET /v1.22/containers/json?all=1&limit=-1&trunc_cmd=0&size=0
DEBU[0156] Calling POST /v1.22/containers/neutron_dhcp_agent/start
DEBU[0156] POST /v1.22/containers/neutron_dhcp_agent/start
DEBU[0156] form data: {}
DEBU[0156] container mounted via layerStore: /var/lib/docker/aufs/mnt/8ec90df91cdb2a4590f6bba4d31aa4c4158010650c5fe93224b028df0000880a
WARN[0156] Auto-creating non-existent volume host path /run/netns, this is deprecated and will be removed soon
ERRO[0157] error locating sandbox id 7fbffe1a9572be17d6202e6af0dfc6c93cf8c4cfc1cbef3498a71b6595aeccc3: sandbox 7fbffe1a9572be17d6202e6af0dfc6c93cf8c4cfc1cbef3498a71b6595aeccc3 not found
WARN[0157] failed to cleanup ipc mounts:
failed to umount /var/lib/docker/containers/ba7b8918b37a0274ef74d6e2550bfc76312a8d665844c44aeafe4a25726de348/shm: invalid argument
ERRO[0157] Error unmounting container ba7b8918b37a0274ef74d6e2550bfc76312a8d665844c44aeafe4a25726de348: not mounted
ERRO[0157] Handler for POST /v1.22/containers/neutron_dhcp_agent/start returned error: Cannot start container ba7b8918b37a0274ef74d6e2550bfc76312a8d665844c44aeafe4a25726de348: Path /run/netns is mounted on /run but it is not a shared mount.

Revision history for this message
Mauricio Lima (mliima) (mauricio-lima) wrote :
Changed in kolla:
assignee: Sam Yaple (s8m) → nobody
assignee: nobody → Mauricio José de Lima (mlima) (limamauricio)
assignee: Mauricio José de Lima (mlima) (limamauricio) → nobody
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.