Stopping ES during initial indexing leaves orphaned indexes

Need to determine if this is desired or not, but if you do not have any indexes created and start a bulk indexing job e.g.:

searchlight-manage index sync --force

Then in the middle of indexing stop elasticsearch, this will result in an error and leave orphaned index and aliases (-listener and -search).

Please note, that doing any recovery here is questionable (since ES is offline). I do wonder if there is anything we can do about this like tying into the orphaned-es-indexs [0] blueprint and publish information about starting an index job (e.g. into .searchlight index) at the beginning of a job and then doing something smart the next time it runs.

]0] https://blueprints.launchpad.net/searchlight/+spec/orphaned-es-aliases

Here is the error:

2016-05-16 10:55:08.763 CRITICAL searchlight [^[[00;36m-] ^[[01;35mConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x10a736110>: Failed to establish a new connection: [Errno 61] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x10a736110>: Failed to establish a new connection: [Errno 61] Connection refused)
^[[00m
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00mTraceback (most recent call last):
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m File "/Users/ttripp/dev/openstack/searchlight/.tox/venv/bin/searchlight-manage", line 10, in <module>
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m sys.exit(main())
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m File "/Users/ttripp/dev/openstack/searchlight/searchlight/cmd/manage.py", line 380, in main
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m return CONF.command.action_fn(*func_args, **func_kwargs)
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m File "/Users/ttripp/dev/openstack/searchlight/searchlight/cmd/manage.py", line 239, in sync
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m plugin_obj.initial_indexing(index_name=index_names[gname])
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m File "/Users/ttripp/dev/openstack/searchlight/searchlight/elasticsearch/plugins/base.py", line 142, in initial_indexing
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m self.setup_data(index_name)
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m File "/Users/ttripp/dev/openstack/searchlight/searchlight/elasticsearch/plugins/base.py", line 176, in setup_data
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m index=index)
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m File "/Users/ttripp/dev/openstack/searchlight/searchlight/elasticsearch/plugins/helper.py", line 174, in save_documents
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m actions=self._prepare_actions(documents, versions))
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m File "/Users/ttripp/dev/openstack/searchlight/.tox/venv/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 188, in bulk
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m for ok, item in streaming_bulk(client, actions, **kwargs):
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m File "/Users/ttripp/dev/openstack/searchlight/.tox/venv/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 160, in streaming_bulk
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m for result in _process_bulk_chunk(client, bulk_actions, raise_on_exception, raise_on_error, **kwargs):
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m File "/Users/ttripp/dev/openstack/searchlight/.tox/venv/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 89, in _process_bulk_chunk
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m raise e
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00mConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x10a736110>: Failed to establish a new connection: [Errno 61] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x10a736110>: Failed to establish a new connection: [Errno 61] Connection refused)
2016-05-16 10:55:08.763 TRACE searchlight ^[[01;35m^[[00m