VM boot fails rarely due to auth error in contrail-api

R4.0.1.0 Build 36 Ubuntu 14.04 Mitaka

Build : 4.0.1.0-36
CoreLocation : /cs-shared/test_runs/nodec54/jenkins-SMLite_ubuntu-14-04_mitaka_Multi_Interface_Container_Sanity-128
cores : {'10.204.217.122': [], '10.204.217.206': [], '10.204.217.93': [], '10.204.217.95': [], '10.204.217.94': [], '10.204.217.96': []}
LogsLocation : http://10.204.216.50/Docs/logs/4.0.1.0-36_jenkins-SMLite_ubuntu-14-04_mitaka_Multi_Interface_Container_Sanity-128_1502956157.24/logs/
Report : http://10.204.216.50/Docs/logs/4.0.1.0-36_jenkins-SMLite_ubuntu-14-04_mitaka_Multi_Interface_Container_Sanity-128_1502956157.24/junit-noframes.html
Topology :
DISTRO : "Ubuntu 14.04.5 LTS"
SKU : mitaka
Config Nodes : [u'nodec54']
Control Nodes : [u'nodec54']
Compute Nodes : [u'nodel1', u'nodei10']
Openstack Node : [u'nodec53']
WebUI Node : [u'nodec54']
Analytics Nodes : [u'nodec55', u'nodec56']
Database Nodes : [u'nodec54', u'nodec55', u'nodec56']
Physical Devices : [u'yuvaraj', u"'yuvaraj'"]

Testcase test_multi_vn_repeated_policy_update_with_ping had failed because a VM got into ERROR state
Test case logs:

2017-08-17 10:47:00,739 - DEBUG - VM is in ERROR state now
2017-08-17 10:47:00,739 - WARNING - VM in error state. Asserting...
2017-08-17 10:47:01,709 - ERROR - Fatal Nova Exception while getting VM detail
2017-08-17 10:47:01,710 - DEBUG - None
2017-08-17 10:47:01,716 - INFO - Deleting VM vmc1
2017-08-17 10:47:01,716 - ERROR - VM vmc1 has failed to come up
2017-08-17 10:47:01,717 - ERROR - Fault seen in nova show is: {u'message': u"An unknown exception occurred.\nNeutron server returns request_ids: ['req-f13fbebc-f8b4-4c57-9a62-ab8da4af9c78']", u'code': 500, u'details': u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1929, in _do_build_and_run_instance\n filter_properties)\n File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2119, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'created': u'2017-08-17T05:16:29Z'}
2017-08-17 10:47:01,717 - ERROR - Nova failed to get host of the VM

----

nova-compute.log :
2017-08-17 10:46:25.428 2672 ERROR nova.network.neutronv2.api [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7] Traceback (most recent call last):
2017-08-17 10:46:25.428 2672 ERROR nova.network.neutronv2.api [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 254, in _create_port
2017-08-17 10:46:25.428 2672 ERROR nova.network.neutronv2.api [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7] port = port_client.create_port(port_req_body)
2017-08-17 10:46:25.428 2672 ERROR nova.network.neutronv2.api [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 97, in with_params
2017-08-17 10:46:25.428 2672 ERROR nova.network.neutronv2.api [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7] ret = self.function(instance, *args, **kwargs)
2017-08-17 10:46:25.428 2672 ERROR nova.network.neutronv2.api [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 662, in create_port
2017-08-17 10:46:25.428 2672 ERROR nova.network.neutronv2.api [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7] return self.post(self.ports_path, body=body)
2017-08-17 10:46:25.428 2672 ERROR nova.network.neutronv2.api [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 363, in post
2017-08-17 10:46:25.428 2672 ERROR nova.network.neutronv2.api [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7] headers=headers, params=params)
2017-08-17 10:46:25.428 2672 ERROR nova.network.neutronv2.api [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 298, in do_request
2017-08-17 10:46:25.428 2672 ERROR nova.network.neutronv2.api [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7] self._handle_fault_response(status_code, replybody, resp)
2017-08-17 10:46:25.428 2672 ERROR nova.network.neutronv2.api [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 273, in _handle_fault_response
2017-08-17 10:46:25.428 2672 ERROR nova.network.neutronv2.api [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7] exception_handler_v20(status_code, error_body)
2017-08-17 10:46:25.428 2672 ERROR nova.network.neutronv2.api [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 84, in exception_handler_v20
2017-08-17 10:46:25.428 2672 ERROR nova.network.neutronv2.api [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7] request_ids=request_ids)
2017-08-17 10:46:25.428 2672 ERROR nova.network.neutronv2.api [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7] InternalServerError: An unknown exception occurred.
2017-08-17 10:46:25.428 2672 ERROR nova.network.neutronv2.api [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7] Neutron server returns request_ids: ['req-f13fbebc-f8b4-4c57-9a62-ab8da4af9c78']
2017-08-17 10:46:25.428 2672 ERROR nova.network.neutronv2.api [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7]
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager [req-ca9dcd72-ef36-4ca7-98bf-0527492adcd7 836edc92171443af930efedcc5dcc89d da9f8df1a2b54dd7b54afa138abdc032 - - -] Instance failed network setup after 1 attempt(s)
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager Traceback (most recent call last):
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1573, in _allocate_network_async
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager bind_host_id=bind_host_id)
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 666, in allocate_for_instance
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager self._delete_ports(neutron, instance, created_port_ids)
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager self.force_reraise()
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager six.reraise(self.type_, self.value, self.tb)
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 655, in allocate_for_instance
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager security_group_ids, available_macs, dhcp_opts)
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 295, in _create_port
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager network_id, instance=instance)
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager self.force_reraise()
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager six.reraise(self.type_, self.value, self.tb)
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 655, in allocate_for_instance
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager security_group_ids, available_macs, dhcp_opts)
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 295, in _create_port
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager network_id, instance=instance)
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager self.force_reraise()
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager six.reraise(self.type_, self.value, self.tb)
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 254, in _create_port
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager port = port_client.create_port(port_req_body)
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 97, in with_params
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager ret = self.function(instance, *args, **kwargs)
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 662, in create_port
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager return self.post(self.ports_path, body=body)
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 363, in post
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager headers=headers, params=params)
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 298, in do_request
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager self._handle_fault_response(status_code, replybody, resp)
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 273, in _handle_fault_response
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager exception_handler_v20(status_code, error_body)
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 84, in exception_handler_v20
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager request_ids=request_ids)
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager InternalServerError: An unknown exception occurred.
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager Neutron server returns request_ids: ['req-f13fbebc-f8b4-4c57-9a62-ab8da4af9c78']
2017-08-17 10:46:25.430 2672 ERROR nova.compute.manager
2017-08-17 10:46:25.457 2672 ERROR nova.compute.manager [req-ca9dcd72-ef36-4ca7-98bf-0527492adcd7 836edc92171443af930efedcc5dcc89d da9f8df1a2b54dd7b54afa138abdc032 - - -] [instance: a575ed1e-66f7-480a-97ab-71a1959f4ee7] Instance failed to spawn

neutron-server.log:

2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource [req-f13fbebc-f8b4-4c57-9a62-ab8da4af9c78 6184245f56cc46e6b8167354874cfca5 b620542e3e4c4aae96c132e9272484f0 - - -] create failed
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource Traceback (most recent call last):
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 84, in resource
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource result = method(request=request, **args)
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 410, in create
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource return self._create(request, body, **kwargs)
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource self.force_reraise()
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 521, in _create
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource obj = do_create(body)
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 503, in do_create
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource request.context, reservation.reservation_id)
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource self.force_reraise()
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 496, in do_create
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource return obj_creator(request.context, **kwargs)
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin_base.py", line 460, in create_port
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource port = self._create_resource('port', context, port)
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 275, in _create_resource
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource obj_name=res_type)
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 257, in _transform_response
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource plugin_base._raise_contrail_error(info, obj_name)
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin_base.py", line 102, in _raise_contrail_error
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource raise NeutronException(**info)
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource NeutronException: An unknown exception occurred.
2017-08-17 10:46:25.443 3574 ERROR neutron.api.v2.resource
2017-08-17 10:46:25.445 3574 INFO neutron.wsgi [req-f13fbebc-f8b4-4c57-9a62-ab8da4af9c78 6184245f56cc46e6b8167354874cfca5 b620542e3e4c4aae96c132e9272484f0 - - -] 22.22.22.10 - - [17/Aug/2017 10:46:25] "POST /v2.0/ports.json HTTP/1.1" 500 338 2.189677

contrail-api log:
22.22.22.55 - - [2017-08-17 10:46:25] "GET /obj-perms HTTP/1.1" 200 7278 0.466019
WARNING:keystonemiddleware.auth_token:Authorization failed for token
WARNING:keystonemiddleware.auth_token:Identity response: {"error": {"message": "Could not find token: 798fc1d4841b4d429b2318cb5a8c29d6", "code": 404, "title": "Not Found"}}
WARNING:keystonemiddleware.auth_token:Authorization failed for token
WARNING:keystonemiddleware.auth_token:Authorization failed for token
WARNING:keystonemiddleware.auth_token:Identity response: {"error": {"message": "Could not find token: 798fc1d4841b4d429b2318cb5a8c29d6", "code": 404, "title": "Not Found"}}
WARNING:keystonemiddleware.auth_token:Authorization failed for token
127.0.0.1 - - [2017-08-17 10:46:25] "POST /virtual-machine-interfaces HTTP/1.1" 401 213 0.023053
WARNING:keystonemiddleware.auth_token:Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admin_tenant_name configuration options was deprecated in the Mitaka release in favor of an auth_plugin and its related options. This class may be removed in a future release.
WARNING:keystonemiddleware.auth_token:Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admin_tenant_name configuration options was deprecated in the Mitaka release in favor of an auth_plugin and its related options. This class may be removed in a future release.
WARNING:keystonemiddleware.auth_token:Authorization failed for token
WARNING:keystonemiddleware.auth_token:Identity response: {"error": {"message": "Could not find token: 798fc1d4841b4d429b2318cb5a8c29d6", "code": 404, "title": "Not Found"}}
WARNING:keystonemiddleware.auth_token:Authorization failed for token
127.0.0.1 - - [2017-08-17 10:46:25] "POST /virtual-machine-interfaces HTTP/1.1" 401 213 0.023003
ERROR:contrail-api:<class 'cfgm_common.exceptions.HttpError'>
Python 2.7.6: /usr/bin/python
Thu Aug 17 10:46:25 2017
....
....
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/bottle.py", line 856, in _handle
    return route.call(**args)
  File "/usr/lib/python2.7/dist-packages/bottle.py", line 1721, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/context.py", line 37, in wrapper
    return fn(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/__init__.py", line 1073, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_interface.py", line 410, in plugin_http_post_port
    return self.plugin_create_port(context, port)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_interface.py", line 355, in plugin_create_port
    net_info = cfgdb.port_create(context, port['resource'])
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 2771, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 4063, in port_create
    port_id = self._resource_create('virtual_machine_interface', port_obj)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 419, in _resource_create
    obj_uuid = create_method(obj)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 42, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 479, in _object_create
    rest.OP_POST, obj_cls.create_uri, data=json_body)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 855, in _request_server
    retry_after_authn=retry_after_authn, retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/context.py", line 37, in wrapper
    return fn(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 87, in _request
    op, url, data, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 928, in _request
    op, url, data=data, retry_after_authn=True)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/context.py", line 37, in wrapper
    return fn(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 87, in _request
    op, url, data, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 956, in _request
    raise HttpError(status, content)
HttpError: HTTP Status: 401 Content: Authentication required
22.22.22.53 - - [2017-08-17 10:46:25] "POST /neutron/port HTTP/1.1" 500 156 1.679831

keystone log:
vjoshi@nodeb6:/tmp/vjoshi/nodec53$ grep "req-616cc1f4-1a0e-41a9-b54e-e9e7c6aad3a0" keystone/keystone-all.log
2017-08-17 10:46:26.444 6134 DEBUG keystone.middleware.auth [req-616cc1f4-1a0e-41a9-b54e-e9e7c6aad3a0 836edc92171443af930efedcc5dcc89d 8ccca1335bb34a51967abace2fc2f94d - default default] RBAC: auth_context: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'836edc92171443af930efedcc5dcc89d', 'roles': [u'admin'], 'user_domain_id': 'default', 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=CardxurIRWG7ANYuQ9995g, audit_chain_id=CardxurIRWG7ANYuQ9995g) at 0x7f5b87f60178>, 'project_id': u'8ccca1335bb34a51967abace2fc2f94d', 'trust_id': None, 'project_domain_id': 'default'} process_request /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:221
2017-08-17 10:46:26.445 6134 INFO keystone.common.wsgi [req-616cc1f4-1a0e-41a9-b54e-e9e7c6aad3a0 836edc92171443af930efedcc5dcc89d 8ccca1335bb34a51967abace2fc2f94d - default default] GET http://22.22.22.53:35357/v3/auth/tokens
2017-08-17 10:46:26.445 6134 DEBUG keystone.common.controller [req-616cc1f4-1a0e-41a9-b54e-e9e7c6aad3a0 836edc92171443af930efedcc5dcc89d 8ccca1335bb34a51967abace2fc2f94d - default default] RBAC: Authorizing identity:validate_token() _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:80
2017-08-17 10:46:26.445 6134 DEBUG keystone.common.controller [req-616cc1f4-1a0e-41a9-b54e-e9e7c6aad3a0 836edc92171443af930efedcc5dcc89d 8ccca1335bb34a51967abace2fc2f94d - default default] RBAC: using auth context from the request environment _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:85
2017-08-17 10:46:26.450 6134 WARNING keystone.common.wsgi [req-616cc1f4-1a0e-41a9-b54e-e9e7c6aad3a0 836edc92171443af930efedcc5dcc89d 8ccca1335bb34a51967abace2fc2f94d - default default] Could not find token: 798fc1d4841b4d429b2318cb5a8c29d6
2017-08-17 10:46:26.451 6134 INFO eventlet.wsgi.server [req-616cc1f4-1a0e-41a9-b54e-e9e7c6aad3a0 836edc92171443af930efedcc5dcc89d 8ccca1335bb34a51967abace2fc2f94d - default default] 22.22.22.54 - - [17/Aug/2017 10:46:26] "GET /v3/auth/tokens HTTP/1.1" 404 365 0.022875
2017-08-17 10:46:26.558 6134 INFO eventlet.wsgi.server [req-616cc1f4-1a0e-41a9-b54e-e9e7c6aad3a0 836edc92171443af930efedcc5dcc89d 8ccca1335bb34a51967abace2fc2f94d - default default] 22.22.22.54 - - [17/Aug/2017 10:46:26] "POST /v2.0/tokens HTTP/1.1" 200 4205 0.186484
vjoshi@nodeb6:/tmp/vjoshi/nodec53$