Scale-in: cluster-relation-departed and db-router departed hook error

Bug #1874479 reported by Andrew McLeod
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL InnoDB Cluster Charm
Fix Released
High
David Ames
MySQL Router Charm
Fix Released
High
David Ames
charm-interface-mysql-innodb-cluster
Fix Released
High
David Ames
charm-interface-mysql-router
Fix Released
High
David Ames

Bug Description

unit-mysql-innodb-cluster-2: 15:57:02 INFO unit.mysql-innodb-cluster/2.juju-log db-router:54: Invoking reactive handler: hooks/relations/mysql-router/provides.py:66:departed:db-router
unit-mysql-innodb-cluster-2: 15:57:03 DEBUG worker.uniter.jujuc running hook tool "juju-log"
unit-mysql-innodb-cluster-2: 15:57:03 ERROR unit.mysql-innodb-cluster/2.juju-log db-router:54: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
TypeError: departed() missing 1 required positional argument: 'self'

unit-mysql-innodb-cluster-2: 15:57:03 DEBUG unit.mysql-innodb-cluster/2.db-router-relation-departed Traceback (most recent call last):
unit-mysql-innodb-cluster-2: 15:57:03 DEBUG unit.mysql-innodb-cluster/2.db-router-relation-departed File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/charm/hooks/db-router-relation-departed", line 22, in <module>
unit-mysql-innodb-cluster-2: 15:57:03 DEBUG unit.mysql-innodb-cluster/2.db-router-relation-departed main()
unit-mysql-innodb-cluster-2: 15:57:03 DEBUG unit.mysql-innodb-cluster/2.db-router-relation-departed File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
unit-mysql-innodb-cluster-2: 15:57:03 DEBUG unit.mysql-innodb-cluster/2.db-router-relation-departed bus.dispatch(restricted=restricted_mode)
unit-mysql-innodb-cluster-2: 15:57:03 DEBUG unit.mysql-innodb-cluster/2.db-router-relation-departed File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
unit-mysql-innodb-cluster-2: 15:57:03 DEBUG unit.mysql-innodb-cluster/2.db-router-relation-departed _invoke(other_handlers)
unit-mysql-innodb-cluster-2: 15:57:15 DEBUG juju.worker.uniter.remotestate update status timer triggered
unit-mysql-innodb-cluster-2: 15:57:15 INFO juju.worker.uniter awaiting error resolution for "relation-departed" hook

Tags: scaleback
Revision history for this message
Andrew McLeod (admcleod) wrote :
Revision history for this message
Marco Filipe Moutinho da Silva (mfmsilva) wrote :

Same issue with cluster-relation-departed hook.
Part of the logs:

unit-mysql-innodb-cluster-2: 14:25:01 INFO unit.mysql-innodb-cluster/2.juju-log cluster:0: Invoking reactive handler: hooks/relations/mysql-innodb-cluster/peers.py:95:departed:cluster
unit-mysql-innodb-cluster-1: 14:25:01 ERROR unit.mysql-innodb-cluster/1.juju-log cluster:0: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
TypeError: departed() missing 1 required positional argument: 'self'

Andrew McLeod (admcleod)
tags: added: scaleback
summary: - db-router departed hook error
+ cluster-relation-departed and db-router departed hook error
Revision history for this message
David Ames (thedac) wrote : Re: cluster-relation-departed and db-router departed hook error

TRIAGE:

The use of when_any for the peer interface handler is the root cause:

class MySQLInnoDBClusterPeer(reactive.Endpoint):

...

    @reactive.when_any('endpoint.{endpoint_name}.broken',
                       'endpoint.{endpoint_name}.departed')
    def departed(self):

Changed in charm-interface-mysql-innodb-cluster:
status: New → Triaged
Changed in charm-mysql-innodb-cluster:
status: New → Triaged
Changed in charm-interface-mysql-innodb-cluster:
importance: Undecided → High
Changed in charm-mysql-innodb-cluster:
importance: Undecided → High
Changed in charm-interface-mysql-innodb-cluster:
assignee: nobody → David Ames (thedac)
Changed in charm-mysql-innodb-cluster:
assignee: nobody → David Ames (thedac)
Revision history for this message
David Ames (thedac) wrote :

This is a scale in problem when removing a node from the cluster.

Changed in charm-interface-mysql-router:
status: New → Triaged
importance: Undecided → High
assignee: nobody → David Ames (thedac)
Changed in charm-mysql-router:
status: New → Triaged
importance: Undecided → High
assignee: nobody → David Ames (thedac)
summary: - cluster-relation-departed and db-router departed hook error
+ Scale-in: cluster-relation-departed and db-router departed hook error
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-interface-mysql-router (master)

Fix proposed to branch: master
Review: https://review.opendev.org/728580

Changed in charm-interface-mysql-router:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-mysql-router (master)

Fix proposed to branch: master
Review: https://review.opendev.org/728582

Changed in charm-mysql-router:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-interface-mysql-innodb-cluster (master)

Fix proposed to branch: master
Review: https://review.opendev.org/728585

Changed in charm-interface-mysql-innodb-cluster:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-mysql-innodb-cluster (master)

Fix proposed to branch: master
Review: https://review.opendev.org/728587

Changed in charm-mysql-innodb-cluster:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-interface-mysql-router (master)

Reviewed: https://review.opendev.org/728580
Committed: https://git.openstack.org/cgit/openstack/charm-interface-mysql-router/commit/?id=ffd0fc0dc48ed9d85911894a8038397ddce8ef69
Submitter: Zuul
Branch: master

commit ffd0fc0dc48ed9d85911894a8038397ddce8ef69
Author: David Ames <email address hidden>
Date: Fri May 15 14:32:56 2020 -0700

    Scale-in fixes

    Properly handle departed and broken hooks. Keep correct state during
    both scale in and scale out.

    Change-Id: Ibc87371f441318e2082ff4028d9f57ae6b796035
    Closes-Bug: #1874479

Changed in charm-interface-mysql-router:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-interface-mysql-innodb-cluster (master)

Reviewed: https://review.opendev.org/728585
Committed: https://git.openstack.org/cgit/openstack/charm-interface-mysql-innodb-cluster/commit/?id=5f87bb1ddec3868864d2a8d1bf9b66ae5dca4e48
Submitter: Zuul
Branch: master

commit 5f87bb1ddec3868864d2a8d1bf9b66ae5dca4e48
Author: David Ames <email address hidden>
Date: Fri May 15 15:17:59 2020 -0700

    Scale-in and Scale-out of mysql-innodb-cluster

    Handle adding and removing units to the mysql-innodb-cluster.

    Change-Id: Ie658c42b095bcff822cdfb0b771d41704ddc85ea
    Closes-Bug: #1877546
    Closes-Bug: #1874479

Changed in charm-interface-mysql-innodb-cluster:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-mysql-router (master)

Reviewed: https://review.opendev.org/728582
Committed: https://git.openstack.org/cgit/openstack/charm-mysql-router/commit/?id=05525dad0270c9a9a386e81202879ef6ad5d5688
Submitter: Zuul
Branch: master

commit 05525dad0270c9a9a386e81202879ef6ad5d5688
Author: David Ames <email address hidden>
Date: Fri May 15 14:43:15 2020 -0700

    Scale-in of mysql innodb cluster nodes

    Handle departed and broken db-router relations elegantly.
    Remove Eoan as a test target.

    Closes-Bug: #1874479
    Depends-On: Ibc87371f441318e2082ff4028d9f57ae6b796035
    Change-Id: Ida5e8fef458abc38cc50ad8749f315eadcb499e0

Changed in charm-mysql-router:
status: In Progress → Fix Committed
Changed in charm-mysql-innodb-cluster:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-mysql-innodb-cluster (master)

Reviewed: https://review.opendev.org/728587
Committed: https://git.openstack.org/cgit/openstack/charm-mysql-innodb-cluster/commit/?id=7cbf9a88b001d73d290bb123f5bba010f51319b3
Submitter: Zuul
Branch: master

commit 7cbf9a88b001d73d290bb123f5bba010f51319b3
Author: David Ames <email address hidden>
Date: Wed May 13 11:37:31 2020 -0700

    Scale in and scale out

    Handle adding and removing units of the mysql-innodb-cluster gracefully.

    Review and land:
    https://github.com/openstack-charmers/zaza/pull/353
    https://github.com/openstack-charmers/zaza-openstack-tests/pull/275

    Closes-Bug: #1874479
    Closes-Bug: #1877546
    Closes-Bug: #1877616
    Depends-On: Ie658c42b095bcff822cdfb0b771d41704ddc85ea
    Change-Id: I39210c760e7204f84365d5dcb4eeedbc0889041f

David Ames (thedac)
Changed in charm-mysql-innodb-cluster:
milestone: none → 20.05
Changed in charm-mysql-router:
milestone: none → 20.05
Changed in charm-mysql-innodb-cluster:
status: Fix Committed → Fix Released
Changed in charm-mysql-router:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.