kolla-ansible

kibana-6 migration will not run

  1. Series ussuri
  2. Bug #1901614
Bug #1901614 reported by Alexander Diana
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Triaged
Medium
Unassigned
Train
New
Medium
Unassigned
Ussuri
In Progress
Medium
Mark Goddard

Bug Description

**Bug Report**

What happened:

upgrading to kibana-6 index crashes with syntax errors regarding status codes, and also may get stuck on pre-existing index errors.

What you expected to happen:

kibana-6 upgrade does not crash

How to reproduce it (minimal and precise):

try and migrate from kibana to kibana-6 index, will receive syntax errors and problems which were reported and closed without changes in other tickets:
https://bugs.launchpad.net/kolla-ansible/+bug/1875812
https://bugs.launchpad.net/kolla-ansible/+bug/1881862

these issues are still present.

I have attached a fixed playbook which worked for our deployments.

**Environment**:
* Kolla-Ansible version: stable/ussuri (same in master)

See original description
Revision history for this message
Alexander Diana (rouk) wrote :
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to kolla-ansible (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/759833

Revision history for this message
Mark Goddard (mgoddard) wrote :

^ Added a CI test for the migration. Let's see what it does.

Revision history for this message
Mark Goddard (mgoddard) wrote :

The CI test passed in Train. I notice you are using Ussuri though. Is that upgrading from Train to Ussuri? Which OS are you using?

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a53/759833/2/check/kolla-ansible-centos-source-prometheus-efk/a5364d1/primary/logs/ansible/migrate-elk

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a53/759833/2/check/kolla-ansible-centos-source-prometheus-efk/a5364d1/primary/logs/ansible/test-prometheus-efk

Revision history for this message
Mark Goddard (mgoddard) wrote :

I had wondered if it was an issue with idempotence, but it seems not. I ran the upgrade twice (on Train) and it worked.

Revision history for this message
Alexander Diana (rouk) wrote :

I can still reproduce it, running debian 10 kolla host (py3.7, ansible 2.9.14), latest ussuri containers, running stable/ussuri kolla.

Revision history for this message
Mark Goddard (mgoddard) wrote :

It's still not clear to me why you are running upgrade rather than deploy. Are you upgrading from Train to Ussuri?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to kolla-ansible (stable/ussuri)

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/760084

Revision history for this message
Mark Goddard (mgoddard) wrote :

Trying again on Ussuri. One difference is the Ansible version.

Mark Goddard (mgoddard)
Changed in kolla-ansible:
importance: Undecided → Medium
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/760182

Revision history for this message
Mark Goddard (mgoddard) wrote :

Failed on Ussuri. I changed status_code to a comma-separated list and it worked.

Revision history for this message
Alexander Diana (rouk) wrote :

the other issue is that elastic will return errors due to ansible eating the escaped quotes in the reindex .kibana step.

In my playbook, i had to convert from escaped double quotes, to unescaped single quotes, which is valid elasticsearch "painless" syntax and removes the possibility of issues.

Revision history for this message
Mark Goddard (mgoddard) wrote :

How do those errors show? The deployment succeeded without that for me. Are they runtime errors seen when using ES after deployment?

Revision history for this message
Alexander Diana (rouk) wrote :

They are reported by ES as rejected requests.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 10.2.0

This issue was fixed in the openstack/kolla-ansible 10.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla-ansible (stable/train)

Change abandoned by "Mark Goddard <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/759833

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by "Radosław Piliszek <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/759833
Reason: I was just testing something gerrit

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.