Juniper Openstack

[R2.20-Build 8]:Docker: veth interfaces for errored docker containers are not getting removed after containers deletion

  1. Series trunk
  2. Bug #1452151
Bug #1452151 reported by alok kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
Fix Committed
High
Ignatious Johnson Christopher
Juniper Openstack r2.20-fcs
Trunk
Fix Committed
High
Ignatious Johnson Christopher
Juniper Openstack r3.0-fcs
nova-docker
Fix Committed
Undecided
Ignatious Johnson Christopher

Bug Description

if docker container fail to launch and later deleted, then its veth interfaces are not getting deleted.

there are many veth interfaces with status DOWN (for deleted containers too, please see the snapshot)

root@nodec62:~# nova list --all-tenants
+--------------------------------------+------------------+--------+------------+-------------+---------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------------+--------+------------+-------------+---------------------------+
| 0bd72c54-b0c1-4c0b-be5e-d26e7f7d1ec3 | docker1 | ACTIVE | - | Running | vn1=1.1.1.3 |
| eec1b0cf-1796-4efd-9dca-f8579d90de2c | docker2 | ACTIVE | - | Running | vn1=1.1.1.4 |
| 9e95ce6e-6c29-495e-bc2a-d1819a2d2d73 | docker2-87966004 | ERROR | - | NOSTATE | |
| cce2e48d-d244-4fe0-b5b0-a7648ef169e2 | docker3 | ACTIVE | - | Running | vn2=2.2.2.3 |
| 3fb640c4-f605-46d5-90bc-71da5f6f34c4 | vn1-vm1-55174744 | ACTIVE | - | Running | vn1-97972758=72.178.213.3 |
| e24bb66c-0f2e-4ff3-bfe4-8422f1a794ef | vn1-vm2-46926800 | ACTIVE | - | Running | vn1-97972758=72.178.213.4 |
+--------------------------------------+------------------+--------+------------+-------------+---------------------------+

root@nodec62:~# ifconfig
docker0 Link encap:Ethernet HWaddr 56:84:7a:fe:97:99
          inet addr:172.17.42.1 Bcast:0.0.0.0 Mask:255.255.0.0
          UP BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

eth0 Link encap:Ethernet HWaddr 00:25:90:c5:59:8a
          inet6 addr: fe80::225:90ff:fec5:598a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:3948488 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3380203 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1371797400 (1.3 GB) TX bytes:689987324 (689.9 MB)
          Memory:df980000-dfa00000

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:8063 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8063 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1383708 (1.3 MB) TX bytes:1383708 (1.3 MB)

pkt0 Link encap:Ethernet HWaddr 8a:53:87:c9:8c:4c
          inet6 addr: fe80::8853:87ff:fec9:8c4c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:3155786 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3186874 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:687665086 (687.6 MB) TX bytes:494928085 (494.9 MB)

veth1361875b Link encap:Ethernet HWaddr fe:ee:99:b9:29:6d
          inet6 addr: fe80::fcee:99ff:feb9:296d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:18 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2257 (2.2 KB) TX bytes:2022 (2.0 KB)

veth16121cf5 Link encap:Ethernet HWaddr 0a:e0:b5:d3:bb:de
          inet6 addr: fe80::8e0:b5ff:fed3:bbde/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:20 errors:0 dropped:0 overruns:0 frame:0
          TX packets:22 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2641 (2.6 KB) TX bytes:2106 (2.1 KB)

veth5eb047ef Link encap:Ethernet HWaddr 06:c1:0d:3c:ea:af
          inet6 addr: fe80::4c1:dff:fe3c:eaaf/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:15 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1710 (1.7 KB) TX bytes:1730 (1.7 KB)

veth61e11527 Link encap:Ethernet HWaddr 46:51:ec:d2:44:45
          inet6 addr: fe80::4451:ecff:fed2:4445/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:19 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2299 (2.2 KB) TX bytes:2022 (2.0 KB)

veth6abbe8a8 Link encap:Ethernet HWaddr c2:aa:97:d6:c4:a9
          inet6 addr: fe80::c0aa:97ff:fed6:c4a9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:20 errors:0 dropped:0 overruns:0 frame:0
          TX packets:28 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2641 (2.6 KB) TX bytes:2350 (2.3 KB)

veth79d950b3 Link encap:Ethernet HWaddr b2:15:41:ec:b9:49
          inet6 addr: fe80::b015:41ff:feec:b949/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:76 errors:0 dropped:0 overruns:0 frame:0
          TX packets:63 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:12099 (12.0 KB) TX bytes:7088 (7.0 KB)

veth7be3eb9f Link encap:Ethernet HWaddr 3a:6e:de:0e:36:ce
          inet6 addr: fe80::386e:deff:fe0e:36ce/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:93 errors:0 dropped:0 overruns:0 frame:0
          TX packets:97 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:21737 (21.7 KB) TX bytes:11408 (11.4 KB)

veth7d8ec904 Link encap:Ethernet HWaddr 36:81:0a:0f:f7:ce
          inet6 addr: fe80::3481:aff:fe0f:f7ce/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1817 (1.8 KB) TX bytes:1868 (1.8 KB)

veth93699ca3 Link encap:Ethernet HWaddr de:54:6d:69:84:b7
          inet6 addr: fe80::dc54:6dff:fe69:84b7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:77 errors:0 dropped:0 overruns:0 frame:0
          TX packets:56 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:12141 (12.1 KB) TX bytes:6410 (6.4 KB)

veth9878e6bd Link encap:Ethernet HWaddr e2:5a:d6:2a:2e:40
          inet6 addr: fe80::e05a:d6ff:fe2a:2e40/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:92 errors:0 dropped:0 overruns:0 frame:0
          TX packets:73 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:13657 (13.6 KB) TX bytes:7608 (7.6 KB)

vethb5b020a4 Link encap:Ethernet HWaddr 4a:a2:b3:86:34:a7
          inet6 addr: fe80::48a2:b3ff:fe86:34a7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:19 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2299 (2.2 KB) TX bytes:2014 (2.0 KB)

vethe16bab3b Link encap:Ethernet HWaddr f6:71:bc:af:77:5b
          inet6 addr: fe80::f471:bcff:feaf:775b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:52 errors:0 dropped:0 overruns:0 frame:0
          TX packets:59 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:8038 (8.0 KB) TX bytes:6043 (6.0 KB)

vethe6def323 Link encap:Ethernet HWaddr aa:8c:71:46:4e:86
          inet6 addr: fe80::a88c:71ff:fe46:4e86/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:182 errors:0 dropped:0 overruns:0 frame:0
          TX packets:190 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:24605 (24.6 KB) TX bytes:18471 (18.4 KB)

vethec4622f4 Link encap:Ethernet HWaddr de:a0:47:5c:ec:59
          inet6 addr: fe80::dca0:47ff:fe5c:ec59/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1817 (1.8 KB) TX bytes:1850 (1.8 KB)

vethf0571e88 Link encap:Ethernet HWaddr 12:ad:db:04:ae:97
          inet6 addr: fe80::10ad:dbff:fe04:ae97/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1817 (1.8 KB) TX bytes:1850 (1.8 KB)

vhost0 Link encap:Ethernet HWaddr 00:25:90:c5:59:8a
          inet addr:10.204.217.102 Bcast:10.204.217.255 Mask:255.255.255.0
          inet6 addr: fe80::225:90ff:fec5:598a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:700378 errors:0 dropped:52 overruns:0 frame:0
          TX packets:210470 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:231214997 (231.2 MB) TX bytes:89781260 (89.7 MB)

virbr0 Link encap:Ethernet HWaddr 4e:46:7e:47:35:fd
          inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
          UP BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

setup info:

env.roledefs = {
    'all': [host1,host2,host3,host4,host5],
    'cfgm': [host1,host2,host5],
    'openstack':[host2],
    'control': [host2,host1],
    'compute': [host3,host4],
    'collector': [host2,host1],
    'webui': [host1],
    'database': [host1,host2,host5],
    'build': [host_build],
}

env.hostnames = {
    'all': ['nodec64', 'nodeg18', 'nodec62', 'nodec11', 'nodec12']
}

Tags: docker vrouter
Revision history for this message
alok kumar (kalok) wrote :
Ashish Ranjan (aranjan-n)
information type: Proprietary → Public
Revision history for this message
Ignatious Johnson Christopher (ijohnson-x) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/177972/3

Changed in nova-docker:
status: New → In Progress
assignee: nobody → Ignatious Johnson Christopher (ijohnson-x)
Revision history for this message
alok kumar (kalok) wrote :

I see veth interfaces not getting deleted for containers which came up fine and then later deleted.

it's on R2.20 build 16.

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : master

Review in progress for https://review.opencontrail.org/10730
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : R2.20

Review in progress for https://review.opencontrail.org/10731
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/10731
Committed: http://github.org/Juniper/contrail-packaging/commit/22ac9793701d0844dea10bbcffdd3eabe4a2f821
Submitter: Zuul
Branch: R2.20

commit 22ac9793701d0844dea10bbcffdd3eabe4a2f821
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri May 22 16:33:26 2015 -0700

Removing veth interfaces if the contrainer launch failed
Closes-Bug: 1452151
Closes-Bug: 1454976

Using inet6 argument with ip command to configure ipv6 address to container
Closes-Bug: 1455616

Configure network using statically instead of using dhclient to avoid
it writting to the compute hosts resolv.conf
Closes-Bug: 1455168

Periodically check the vrouter agent conncetion and publish the port informaton to
the vrouter agent.
Closes-Bug: 1454655

Change-Id: I3f59055da0748df2a8dddfd82cf2eb062377dfab

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/10730
Committed: http://github.org/Juniper/contrail-packaging/commit/70bbb08f57773280a40c6f1565f853918c0bb120
Submitter: Zuul
Branch: master

commit 70bbb08f57773280a40c6f1565f853918c0bb120
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri May 22 16:09:24 2015 -0700

Removing veth interfaces if the contrainer launch failed
Closes-Bug: 1452151
Closes-Bug: 1454976

Using inet6 argument with ip command to configure ipv6 address to container
Closes-Bug: 1455616

Configure network using statically instead of using dhclient to avoid
it writting to the compute hosts resolv.conf
Closes-Bug: 1455168

Periodically check the vrouter agent conncetion and publish the port informaton to
the vrouter agent.
Closes-Bug: 1454655

Change-Id: I79360edd27ca59743f5ee6afee928f77c6a31435

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova-docker (master)

Reviewed: https://review.openstack.org/177972
Committed: https://git.openstack.org/cgit/stackforge/nova-docker/commit/?id=12b11b31b6f4274efa88f3011d2e4b884ed0e30c
Submitter: Jenkins
Branch: master

commit 12b11b31b6f4274efa88f3011d2e4b884ed0e30c
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Apr 27 14:12:12 2015 -0700

    During opencontrail attach vifs the containers use dhclient to lease an ip address.
    When bulk container create request is sent, the /var/lib/dhcp/dhclient.leases file is overwritten.
    Due to the corrupted file Successive docker containers are not getting IP lease
    as the dhclient command is failling .

    Fixing it by creating separate lease files for each docker container.
    Closes-Bug: #1446929

    Removing veth interfaces if the contrainer launch failed
    Closes-Bug: #1452151

    Using inet6 argument with ip command to configure ipv6 address to container
    Closes-Bug: #1455616

    Configure network using statically instead of using dhclient to avoid
    it writting to the compute hosts resolv.conf
    Closes-Bug: #1455168

    Periodically check the vrouter agent conncetion and publish the port informaton to
    the vrouter agent.
    Closes-Bug: #1454655

    Change-Id: I345dddefa3036b29a2ee987216e1094ba9bd23e8

Changed in nova-docker:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.