[R2.20-Build 17]:Docker container instances informations could not be recovered after agent crash

Bug #1454655 reported by alok kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
Fix Committed
High
Ignatious Johnson Christopher
Trunk
Fix Committed
High
Ignatious Johnson Christopher
nova-docker
Fix Committed
Undecided
Ignatious Johnson Christopher

Bug Description

if agent crashed or restarted because of any reason, then once agent is up and running metadata IP of docker containers are not reachable.

For qemu based computes, metadata IP is reachable once agent is UP.

alok kumar (kalok)
summary: - [R2.20-Build 17]:Docker container metadata service not resumed after
- agent crash
+ [R2.20-Build 17]:Docker container instances informations could not be
+ recovered after agent crash
Revision history for this message
alok kumar (kalok) wrote :

once agent is up, it doesn't have any instance information.
looks like instances info could not be recovered after agent crash/restart.

after this instances are not able to send traffic to each other, verified it through docker cli.

from analytics:
curl http://nodec64.englab.juniper.net:8081/analytics/uves/virtual-machines
[]

root@nodec62:~# nova list
+--------------------------------------+-------+--------+------------+-------------+-------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+-------------+
| 17dda6d0-89e1-4c72-845c-bdf124564a2d | vm10 | ACTIVE | - | Running | vn4=4.4.4.6 |
| b0716346-ef97-489e-b36e-3c3d9a92c4d6 | vm4-1 | ACTIVE | - | Running | vn4=4.4.4.3 |
| 556f84ce-2240-4fec-b0ec-69d17d5ba137 | vm4-2 | ACTIVE | - | Running | vn4=4.4.4.4 |
| 05c436ed-baca-4c10-95ca-5be0fff8630f | vm9 | ACTIVE | - | Running | vn4=4.4.4.5 |
+--------------------------------------+-------+--------+------------+-------------+-------------+

root@nodec64:/opt/contrail/utils# curl -u admin:contrail123 http://127.0.0.1:8095/virtual-machines | python -mjson.tool
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 778 100 778 0 0 176k 0 --:--:-- --:--:-- --:--:-- 189k
{
    "virtual-machines": [
        {
            "fq_name": [
                "05c436ed-baca-4c10-95ca-5be0fff8630f"
            ],
            "href": "http://127.0.0.1:8095/virtual-machine/05c436ed-baca-4c10-95ca-5be0fff8630f",
            "uuid": "05c436ed-baca-4c10-95ca-5be0fff8630f"
        },
        {
            "fq_name": [
                "17dda6d0-89e1-4c72-845c-bdf124564a2d"
            ],
            "href": "http://127.0.0.1:8095/virtual-machine/17dda6d0-89e1-4c72-845c-bdf124564a2d",
            "uuid": "17dda6d0-89e1-4c72-845c-bdf124564a2d"
        },
        {
            "fq_name": [
                "556f84ce-2240-4fec-b0ec-69d17d5ba137"
            ],
            "href": "http://127.0.0.1:8095/virtual-machine/556f84ce-2240-4fec-b0ec-69d17d5ba137",
            "uuid": "556f84ce-2240-4fec-b0ec-69d17d5ba137"
        },
        {
            "fq_name": [
                "b0716346-ef97-489e-b36e-3c3d9a92c4d6"
            ],
            "href": "http://127.0.0.1:8095/virtual-machine/b0716346-ef97-489e-b36e-3c3d9a92c4d6",
            "uuid": "b0716346-ef97-489e-b36e-3c3d9a92c4d6"
        }
    ]
}

information type: Proprietary → Public
tags: added: blocker
Changed in nova-docker:
assignee: nobody → Ignatious Johnson Christopher (ijohnson-x)
Changed in nova-docker:
status: New → In Progress
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/10731
Committed: http://github.org/Juniper/contrail-packaging/commit/22ac9793701d0844dea10bbcffdd3eabe4a2f821
Submitter: Zuul
Branch: R2.20

commit 22ac9793701d0844dea10bbcffdd3eabe4a2f821
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri May 22 16:33:26 2015 -0700

Removing veth interfaces if the contrainer launch failed
Closes-Bug: 1452151
Closes-Bug: 1454976

Using inet6 argument with ip command to configure ipv6 address to container
Closes-Bug: 1455616

Configure network using statically instead of using dhclient to avoid
it writting to the compute hosts resolv.conf
Closes-Bug: 1455168

Periodically check the vrouter agent conncetion and publish the port informaton to
the vrouter agent.
Closes-Bug: 1454655

Change-Id: I3f59055da0748df2a8dddfd82cf2eb062377dfab

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/10730
Committed: http://github.org/Juniper/contrail-packaging/commit/70bbb08f57773280a40c6f1565f853918c0bb120
Submitter: Zuul
Branch: master

commit 70bbb08f57773280a40c6f1565f853918c0bb120
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri May 22 16:09:24 2015 -0700

Removing veth interfaces if the contrainer launch failed
Closes-Bug: 1452151
Closes-Bug: 1454976

Using inet6 argument with ip command to configure ipv6 address to container
Closes-Bug: 1455616

Configure network using statically instead of using dhclient to avoid
it writting to the compute hosts resolv.conf
Closes-Bug: 1455168

Periodically check the vrouter agent conncetion and publish the port informaton to
the vrouter agent.
Closes-Bug: 1454655

Change-Id: I79360edd27ca59743f5ee6afee928f77c6a31435

Revision history for this message
Ignatious Johnson Christopher (ijohnson-x) wrote :

Reviewed: https://review.opencontrail.org/10730
Committed: http://github.org/Juniper/contrail-packaging/commit/70bbb08f57773280a40c6f1565f853918c0bb120
Submitter: Zuul
Branch: master

commit 70bbb08f57773280a40c6f1565f853918c0bb120
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri May 22 16:09:24 2015 -0700

Removing veth interfaces if the contrainer launch failed
Closes-Bug: 1452151
Closes-Bug: 1454976

Using inet6 argument with ip command to configure ipv6 address to container
Closes-Bug: 1455616

Configure network using statically instead of using dhclient to avoid
it writting to the compute hosts resolv.conf
Closes-Bug: 1455168

Periodically check the vrouter agent conncetion and publish the port informaton to
the vrouter agent.
Closes-Bug: 1454655

Change-Id: I79360edd27ca59743f5ee6afee928f77c6a31435

Revision history for this message
Ignatious Johnson Christopher (ijohnson-x) wrote :

Reviewed: https://review.opencontrail.org/10731
Committed: http://github.org/Juniper/contrail-packaging/commit/22ac9793701d0844dea10bbcffdd3eabe4a2f821
Submitter: Zuul
Branch: R2.20

commit 22ac9793701d0844dea10bbcffdd3eabe4a2f821
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri May 22 16:33:26 2015 -0700

Removing veth interfaces if the contrainer launch failed
Closes-Bug: 1452151
Closes-Bug: 1454976

Using inet6 argument with ip command to configure ipv6 address to container
Closes-Bug: 1455616

Configure network using statically instead of using dhclient to avoid
it writting to the compute hosts resolv.conf
Closes-Bug: 1455168

Periodically check the vrouter agent conncetion and publish the port informaton to
the vrouter agent.
Closes-Bug: 1454655

Change-Id: I3f59055da0748df2a8dddfd82cf2eb062377dfab

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova-docker (master)

Reviewed: https://review.openstack.org/177972
Committed: https://git.openstack.org/cgit/stackforge/nova-docker/commit/?id=12b11b31b6f4274efa88f3011d2e4b884ed0e30c
Submitter: Jenkins
Branch: master

commit 12b11b31b6f4274efa88f3011d2e4b884ed0e30c
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Apr 27 14:12:12 2015 -0700

    During opencontrail attach vifs the containers use dhclient to lease an ip address.
    When bulk container create request is sent, the /var/lib/dhcp/dhclient.leases file is overwritten.
    Due to the corrupted file Successive docker containers are not getting IP lease
    as the dhclient command is failling .

    Fixing it by creating separate lease files for each docker container.
    Closes-Bug: #1446929

    Removing veth interfaces if the contrainer launch failed
    Closes-Bug: #1452151

    Using inet6 argument with ip command to configure ipv6 address to container
    Closes-Bug: #1455616

    Configure network using statically instead of using dhclient to avoid
    it writting to the compute hosts resolv.conf
    Closes-Bug: #1455168

    Periodically check the vrouter agent conncetion and publish the port informaton to
    the vrouter agent.
    Closes-Bug: #1454655

    Change-Id: I345dddefa3036b29a2ee987216e1094ba9bd23e8

Changed in nova-docker:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.