agent crash upon lb vip-delete

Bug #1504243 reported by Senthilnathan Murugappan
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.0
New
Medium
Divakar Dharanalakota
R2.20
New
Medium
Divakar Dharanalakota
Trunk
New
Medium
Divakar Dharanalakota

Bug Description

Looks like lb vip-delete triggered both create and delete netns events.
The core file and logs will be at /cs-shared/bugs/<bug-id>

supervisord logs:
2015-10-08 06:55:01,131 INFO exited: contrail-vrouter-agent (terminated by SIGABRT (core dumped); not expected)

From test logs:
2015-10-08 06:55:00,887 - DEBUG - vip b7e0b20a-e5f2-46c6-9678-b17fd260d541 deleted successfully

vrouter.log:
2015-10-08 Thu 06:55:00:831.483 PDT harmonic-01 [Thread 140713299773184, Pid 17102]: loadbalancer property change vip_.address - +20.1.1.4 vip_.admin_state -0 +1 vip_.protocol - +HTTP vip_.protocol_port -0 +80 vip_.connection_limit -0 +-1 Members: Monitors:
2015-10-08 Thu 06:55:00:876.052 PDT harmonic-01 [Thread 140713726703360, Pid 17102]: Unkown type:
2015-10-08 Thu 06:55:00:876.099 PDT harmonic-01 [Thread 140713726703360, Pid 17102]: NetNS event notification for uuid: 311b1e84-76f4-450b-b344-e42c3daf3477 usable
2015-10-08 Thu 06:55:00:876.141 PDT harmonic-01 [Thread 140713726703360, Pid 17102]: NetNS run command: /usr/bin/opencontrail-vrouter-netns create loadbalancer e51e6671-58c6-4420-95e8-155af7224efd a5d037e3-f249-40cc-a26b-5118b62c82d9 2cbb0d6f-e07c-4b16-baa8-51dde2fdbd19 --vmi-left-ip 10.1.1.6/24 --vmi-right-ip 20.1.1.4/24 --vmi-left-mac 02:58:f8:f9:3c:00 --vmi-right-mac 02:16:b8:e1:d4:9e --gw-ip 20.1.1.1 --cfg-file /var/lib/contrail/loadbalancer/d1a3b926-664e-4097-9ad9-560b94c338bd/etc/haproxy/haproxy.cfg --pool-id d1a3b926-664e-4097-9ad9-560b94c338bd --update
2015-10-08 Thu 06:55:00:876.643 PDT harmonic-01 [Thread 140713726703360, Pid 17102]: Service run command queued: /usr/bin/opencontrail-vrouter-netns create loadbalancer e51e6671-58c6-4420-95e8-155af7224efd a5d037e3-f249-40cc-a26b-5118b62c82d9 2cbb0d6f-e07c-4b16-baa8-51dde2fdbd19 --vmi-left-ip 10.1.1.6/24 --vmi-right-ip 20.1.1.4/24 --vmi-left-mac 02:58:f8:f9:3c:00 --vmi-right-mac 02:16:b8:e1:d4:9e --gw-ip 20.1.1.1 --cfg-file /var/lib/contrail/loadbalancer/d1a3b926-664e-4097-9ad9-560b94c338bd/etc/haproxy/haproxy.cfg --pool-id d1a3b926-664e-4097-9ad9-560b94c338bd --update
2015-10-08 Thu 06:55:00:924.747 PDT harmonic-01 [Thread 140713672124160, Pid 17102]: Unkown type:
2015-10-08 Thu 06:55:00:924.815 PDT harmonic-01 [Thread 140713672124160, Pid 17102]: service-instance properties change pool_id: -d1a3b926-664e-4097-9ad9-560b94c338bd +00000000-0000-0000-0000-000000000000
2015-10-08 Thu 06:55:00:924.843 PDT harmonic-01 [Thread 140713672124160, Pid 17102]: Unkown type:
2015-10-08 Thu 06:55:00:924.872 PDT harmonic-01 [Thread 140713672124160, Pid 17102]: NetNS event notification for uuid: 311b1e84-76f4-450b-b344-e42c3daf3477 not usable
2015-10-08 Thu 06:55:00:924.890 PDT harmonic-01 [Thread 140713672124160, Pid 17102]: Stopping service instance!
2015-10-08 Thu 06:55:00:924.930 PDT harmonic-01 [Thread 140713672124160, Pid 17102]: Service stop command queued: /usr/bin/opencontrail-vrouter-netns destroy loadbalancer e51e6671-58c6-4420-95e8-155af7224efd a5d037e3-f249-40cc-a26b-5118b62c82d9 2cbb0d6f-e07c-4b16-baa8-51dde2fdbd19 --cfg-file /var/lib/contrail/loadbalancer/d1a3b926-664e-4097-9ad9-560b94c338bd/etc/haproxy/haproxy.cfg --pool-id d1a3b926-664e-4097-9ad9-560b94c338bd
2015-10-08 Thu 06:55:01:032.043 PDT harmonic-01 [Thread 140713709909760, Pid 17102]: ID-PERM not set for object <default-domain:TestLbaas-13454226:default-domain__TestLbaas-13454226__d1a3b926-664e-4097-9ad9-560b94c338bd__2__left__2> Table <__ifmap__.virtual_machine_interface.0>. Converting to DELETE
2015-10-08 Thu 06:55:01:032.115 PDT harmonic-01 [Thread 140713709909760, Pid 17102]: VN change sync for Port default-domain:TestLbaas-13454226:default-domain__TestLbaas-13454226__d1a3b926-664e-4097-9ad9-560b94c338bd__2__left__2
2015-10-08 Thu 06:55:01:032.363 PDT harmonic-01 [Thread 140713709909760, Pid 17102]: Unkown type:
2015-10-08 Thu 06:55:01:033.670 PDT harmonic-01 [Thread 140713884899264, Pid 17102]: NetNS run errors:
Traceback (most recent call last):
  File "/usr/bin/opencontrail-vrouter-netns", line 9, in <module>
    load_entry_point('opencontrail-vrouter-netns==0.1', 'console_scripts', 'opencontrail-vrouter-netns')()
  File "/usr/lib/python2.7/dist-packages/opencontrail_vrouter_netns/vrouter_netns.py", line 466, in main
    vrouter_netns.args.func()
  File "/usr/lib/python2.7/dist-packages/opencontrail_vrouter_netns/vrouter_netns.py", line 423, in create
    netns_mgr.set_lbaas()
  File "/usr/lib/python2.7/dist-packages/opencontrail_vrouter_netns/vrouter_netns.py", line 161, in set_lbaas
    self.ip_ns.netns.execute([self.LBAAS_PROCESS, '-f', self.cfg_file, '-D', '-p', pid_file, '-sf', pid])
  File "/usr/lib/python2.7/dist-packages/opencontrail_vrouter_netns/linux/ip_lib.py", line 496, in execute
    check_exit_code=check_exit_code)
  File "/usr/lib/python2.7/dist-packages/opencontrail_vrouter_netns/linux/utils.py", line 86, in execute
    raise RuntimeError(m)
RuntimeError:
Command: ['sudo', 'ip', 'netns', 'exec', 'vrouter-e51e6671-58c6-4420-95e8-155af7224efd:d1a3b926-664e-4097-9ad9-560b94c338bd', 'haproxy', '-f', '/var/lib/contrail/loadbalancer/d1a3b926-664e-4097-9ad9-560b94c338bd/etc/haproxy/haproxy.cfg', '-D', '-p', '/var/lib/contrail/loadbalancer/d1a3b926-664e-4097-9ad9-560b94c338bd/etc/haproxy/haproxy.cfg.pid', '-sf', '29762']
Exit code: 1
Stdout: ''
Stderr: '[ALERT] 280/065501 (30979) : Could not open configuration file /var/lib/contrail/loadbalancer/d1a3b926-664e-4097-9ad9-560b94c338bd/etc/haproxy/haproxy.cfg : No such file or directory\n'

2015-10-08 Thu 06:55:01:059.867 PDT harmonic-01 [Thread 140713709909760, Pid 17102]: !!!! ERROR !!!! Task caught fatal exception: vector::_M_range_check TaskImpl: 0x7ffa81162640

Tags: lbaas vrouter
Revision history for this message
Senthilnathan Murugappan (msenthil) wrote :

Issue was observed on a setup with R2.20-64 agent and R2.20.1-4 svc-monitor

Changed in juniperopenstack:
assignee: nobody → Divakar Dharanalakota (ddivakar)
Revision history for this message
Hari Prasad Killi (haripk) wrote :

A similar bug was fixed in R2.21 : https://bugs.launchpad.net/juniperopenstack/+bug/1474272, this fix is not present in R2.20 build 64.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.